Efficient form tests

Model-based form testing refers to an approach in which data and UI models of a low-code platform automatically generate test data sets and UI localizers. An SMT solver calculates valid test data sets from validation and calculation rules. The code generated from this fills forms with hundreds of fields and thousands of rules fully automatically, instead of manually or with high-maintenance individual tests.

Key Takeaways

• Over 255 form variables across 16 states, each with more than 100 fields per UI and individual forms with up to 100,000 fields, make manual testing virtually impossible.
• An SMT solver-based test data generator recognizes contradictory validation rules during generation and thus prevents faulty models from being deployed in the first place.
• Selectors and test code can be automatically derived from the UI model of the low-code platform so that an entire form can be filled or tested with a single line of code.
• What would take several hundred hours of manual testing can be automated in twelve to fourteen hours per federal state.

Why official forms can hardly be tested manually anymore

Official tax forms reach a level of combinatoriality that makes manual testing impossible. The example project comprises over 255 form variants, spread across 16 federal states. Each federal state has its own variant, and each form has over 100 fields on the interface.

This dimension continues to grow for individual tax types. A letter tax form has around 2400 fields and around 2200 validation and calculation rules. One particular tax even has up to 100,000 fields. Such quantities exceed any human capacity to simply go through a form from top to bottom.

Added to this is the constant change. The technicality changes per year and per federal state. Even a valid test data set once found loses its validity again as soon as a rule shifts. This means that the workload increases in every direction.

Even if I find a valid test data set that validates everything, it is constantly changing. The effort immediately goes through the roof, in every direction.

Lilia Gargouri

The maintenance trap also affects automated tests

Simply starting to automate the user interface does not solve the problem, but rather postpones it. Even the automation of a single happy path with this variety of forms ends up directly in the maintenance trap. In addition to the implementation of a single test case, there are ongoing maintenance costs as soon as fields or rules change.

The second bottleneck lies before that: before a tester can even test, they need a valid test data set. This step blocks both manual and automated testing equally. Without valid data, no test can get off the ground.

The consequences for the project were clear. Everything has to be tested in four-week sprints with a release at the end, and achieving uniform test coverage across the entire project becomes more difficult with each additional form variant.

Using models as a source for testing

Lilia Gargouri and Simon Bergler’s approach starts with the models that already exist. The platform is developed with a low-code platform in which each technicality is modeled first. There are two models behind each form: a data model and a UI model, each as a JSON file.

The data model contains the fields as well as the validation and calculation rules. The clerks and business analysts pour the technicality into this instead of implementing it. For example, they define a string field for the name or a date field for a date and formulate rules in the platform’s rule language.

A simple example: If a name is specified but no date of birth, a rule fires with an error message to enter the date of birth. It is precisely this set of rules that makes the model the ideal source for testing. If you wanted to program 2200 rules in the classic way instead, you would fail due to implementation and maintenance.

How valid test data is created from the data model

The team automatically generates valid test data sets with high coverage from the data model. The JSON file is converted into an arithmetic model for this purpose. An SMT solver from the open source area solves this system of equations and finds a test data set with as many occupied fields as possible.

The team added project-specific validations to the open source solver, for example for the assignment of the tax number. The result ends up in an Excel file: Each column is a complete, valid test data set, immediately usable for manual or automated testing.

This step has a side effect that uncovers errors early on. If rules in the model inadvertently exclude each other, things go wrong as soon as the test data is generated. The solver cannot find a solution and the modeling error becomes visible before the JSON files are deployed.

If the technicality changes, the rules change and the team simply regenerates the test data. The existing tests then run again immediately.

The UI model provides the code for the surface tests

The second generic approach extracts the automation of the user interface directly from the UI model. In this model, the modelers determine which labels a field has, how fields are grouped, whether a string field appears as a text field or text area and whether a selection is displayed as a checkbox or radio button.

The team draws all the information for the UI test automation from this model. Neither the validations nor the rules are needed to fill in the fields, as it does not matter whether the data entered is valid. The selectors or locators for the tool used are created from the field definitions.

Each element is identified by the combination of field type and heading, such as a text field with the label “Name”. A library provides the appropriate method for each field type, which can fill or check the element. For a text field, it retrieves exactly the method that serves a text field.

Two lines of code for an entire form

For the tester, the effort is reduced to just a few lines. One command reads “fill the form”. The tool reads the Excel file with the valid test data records and processes the form blindly from top to bottom.

A second line checks the entire form and ensures that it actually contains what is expected. Two lines of code as preparation thus cover a complete form, regardless of the number of fields.

This flexibility allows different test depths with the same code. The tester can go through the entire process, i.e. create a form, fill it out, complete it and archive it. However, he can also only go as far as a specific point, use this run as preparation and start from there manually or with a targeted feature test.

What the model-based approach saves in terms of time

The test duration shows why manual testing is not an option here. If you run all automated tests for a single federal state, you end up with a total test duration of 12 to 14 hours. Extrapolated to 16 federal states, this is the real testing effort.

Translated into manual work, the effort would be many times higher, estimated at several hundred hours, at least around 500. The automated tool, on the other hand, tests quickly and evenly.

The team has built up 300 to 400 concrete tests in a short space of time, plus around 20 dedicated feature tests. Much of this is recycled because the modules can be widely reused.

The benefits extend beyond the test role. If someone finds an error, the developer first has to navigate to the affected area to find the cause. Instead of manually retracing this path for two hours, the prepared navigation takes eight to ten minutes.

The next step: check PDF content automatically

After submitting a form, a PDF is created, and this is the next check field. The end user is simulated via the Happy Path, the long form is filled out and sent. This is followed by a confirmation and finally a PDF for download, which contains sensitive data and legal guidelines that must not be violated.

The process is well positioned for this test because the test data set used is known. The team knows exactly which fields it has filled in and can compare the PDF content accordingly in minute detail. However, testing PDF content is no longer web UI testing, but a separate task that no one wants to implement or maintain themselves.

The team is therefore working on a generic library for this, together with the subsidiary that is responsible for the test tool used. Such a solution is particularly relevant for the insurance industry and for e-government.

Accessibility becomes its own test field

Accessibility is becoming a major issue in both the public and insurance sectors. Tax forms, which citizens use to submit their income tax electronically, must be flawless in terms of accessibility.

The problem for testers: without accessibility expertise, you don’t know which rules to check and how to react to an error. Familiarizing yourself with the standard library for accessibility takes time, which is lacking in the sprint.

The answer follows the same pattern as with form testing. The team builds an accessibility library based on the standard library with its checks and rules. A team of accessibility experts provides the additional requirements, and user-friendly error codes are defined together. The tester calls up “check this interface” and is shown in readable run logs marked in red at which point which error code applies.