The Software Testing Survey is a long-term study on the state of software testing in German-speaking countries that has been conducted every few years since 2011. It records roles, methods, project models and tools. The results show, among other things, how the proportion of agile projects has shifted and how widespread specific test techniques such as equivalence partitioning or statement coverage actually are.
Key Takeaways
- The software testing survey has been published every four years since 2011 and will provide comparative data over a period of almost 15 years for the first time in 2024.
- Agile practices with a strong focus on code are highly valued, while collective responsibility concepts such as collective code ownership and pair programming were rated as important by less than 50 percent of respondents.
- Traditional test techniques such as equivalence class method and statement coverage are on the decline; 18 percent of respondents did not know statement coverage as a test method at all in 2020.
- For the first time, the current survey specifically asks about the use of AI at various levels, from requirements generation to test case creation, and compares this use with the use of external service providers.
- All results, from the management summary to the technical report with raw data, are publicly accessible and free of charge because the participating universities contribute publicly funded working time.
What the software testing survey measures
The Software Testing Survey is a recurring survey on the state of testing in practice that has been conducted every few years since 2011. It records how testers, test managers and companies in German-speaking countries work, which procedures they use and how projects change over time.
So far, there have been runs in 2011, 2015/16 and 2020. With the current edition, data is available for almost 15 years, which allows comparisons to be made over a long period of time. It is precisely this long-term perspective that is at the heart of the project.
Today, the survey is supported by the German Testing Board together with participating universities. One company originally involved was deliberately removed following feedback in order to make the data recognizable as independent and to avoid reservations about a corporate context.
Why the results are openly accessible
All results of the software testing survey are publicly available. This is not a minor matter, but a conscious fundamental decision by those involved.
The reason lies in the funding. The work of the universities flows into the project with public funds. A clear principle is derived from this: what is publicly funded should also be publicly accessible.
As long as universities are involved, there is no other way to do it. What is publicly funded must also be publicly accessible.
Karin Vosseberg
The publication is made in several stages for different audiences. First a management summary with the most striking points, then a more compact brochure with the most important results, and finally a detailed technical report with several hundred pages and all the raw data. For most readers, the summary and the short brochure are the most useful.
What the survey asks in terms of content
The survey begins with the question of which role the participants assign themselves to. This is particularly interesting in light of the fact that project landscapes have shifted from clear distinctions between roles to agile structures in which these distinctions are becoming blurred.
One finding from 2020: although the proportion of agile projects was very high, many participants continued to assign themselves to traditional roles. It may make more sense to ask about skills instead of roles, i.e. where someone sees their main skills. Then the self-assignment becomes more understandable.
The process models have completely reversed the ratio. While the proportion of agile projects was initially around 25 percent compared to around 75 percent of traditional, phase-oriented models, it is now exactly the opposite. In practice, however, the models often mix, which is why the current edition breaks down hybrid methods as a separate category.
Saying agile does not mean working agile
The survey not only checks whether someone describes themselves as agile, but also which agile practices they actually find important for quality assurance. This makes it possible to recognize whether an agile mindset has really arrived or just the label.
The results from 2020 show a clear pattern. Program code-related practices were highly valued. Organizational practices, which make up the mindset, were well behind.
- Collective code ownership, i.e. the shared responsibility of everyone for the code, was only considered relevant by around 50 percent.
- Pair Programming appears to be used by less than 50 percent, although it is an important element of quality assurance.
The strong fixation on the one goal - the code must run - has increased. Other aspects are being pushed into the background. If everyone looks at quality together, this is a safety net that cannot be replaced by pure runnability.
Classic test techniques are losing visibility
Traditional test methods have declined noticeably in practice. Equivalence partitions, boundary value analysis, statement and decision coverage play a much smaller role than they used to.
One detail stands out: statement testing, i.e. the test for statement coverage, was not known at all by 18% of participants in 2020. This is a remarkably high figure for an established method.
One possible explanation is automation. In agile projects with short cycles and test automation, the dashboard automatically provides key figures such as statement coverage. The method behind it disappears into the tool and is no longer perceived as an independent process.
This raises an open question for quality assurance. Is coverage now only used to check whether the existing test cases cover everything? Or is it still used as a procedure to systematically generate new test cases? This shift is an indication of where skills are lacking and where training and further education should start.
How the current issue covers AI
The current survey focuses specifically on AI. A general question on AI was already included previously, so that a development over time can be seen. New additions include specific questions on its use at various levels, from requirements generation to test case creation.
Three dimensions are surveyed: whether AI is already being used, what potential the participants see in it and whether external people are used for the respective tasks. The combination of these questions allows for an evaluation that is interesting in its own right.
The open question is whether organizations that use AI employ fewer external staff. Whether this correlation exists can only be determined from a sufficiently large number of responses. This is precisely why the survey needs a high level of participation.
All questions are reviewed when each issue is revised. Topics that no longer provide any new insights are removed. Questions on outsourcing, for example, were removed because the last runs showed that this is no longer a relevant topic.
Industry alone explains little
No major industry-specific differences can be identified across several evaluations. This is a finding that contradicts an obvious expectation.
One would assume that regulated sectors such as automotive or medical technology test differently than the rest. The data does not show this difference as clearly as expected. For anyone who wants to align their own work with industry standards, this is a relevant classification.
How to use the data for yourself
The results function as a benchmark for determining your own position. You can use the values and check where you or your company stand in the overall picture.
For companies, the newer questions in particular provide starting points. If many participants see high potential in AI, this is an incentive to take a look at it themselves. Interest in the training sector also shows where skills are in demand.
For the self-employed and freelancers, the survey provides orientation on the market. It helps them to decide where their own further training makes sense and in which direction they want to specialize. Industry-specific evaluations are also possible on request, for example for the financial or automotive sector, as far as the data allows.
Why participation determines the informative value
The quality of the results depends on the number of responses. Without a sufficient number of participants, it is not possible to make reliable statements, and then the survey is of no use to either the participants or the evaluators.
There are three questionnaires for different target groups:
| Questionnaire | Target group | Scope |
|---|---|---|
| Management | Executives | Shorter, focus on organizational issues and framework conditions |
| Operational staff | Testers, test managers | More detailed, about half an hour |
In each of the past three surveys, around 1,000 people started and around 700 to 800 answered the questions by the end. This is not representative in the strict sense, as one percent of all those working in the field would have to take part. Nevertheless, the basis is sufficient for good, reliable statements.
The current survey runs from September 1 to 30. An initial management summary will follow soon after, and the first results with a focus on AI will be presented at the QS Day in early November.
