Results of the 2024 software testing survey

The Software Testing Survey 2024 is an industry-wide survey on the status quo of software testing in German-speaking companies that has been conducted every four to five years since 2011. Around 800 participants from development, testing and management responded. Key findings: a third already use AI for coding and quality assurance, while basic test techniques such as equivalence partitioning and security tests are used far too rarely.

Key Takeaways

• A third of the testers surveyed already use AI for coding and quality assurance, compared to almost 50 percent of those working in programming alone.
• Test techniques such as equivalence partitioning and boundary value analysis are only used by around 40 percent of respondents, although ISTQB certifications have been teaching these basics for over 20 years.
• 12 percent of testers still do not automate regression testing, and a quarter do not even use automation for load and performance tests.
• Test-driven development has been steadily losing ground for years: from 67 percent in 2016 to 48 percent in 2024, although agile process models are increasing at the same time.
• Two thirds of operational employees see an increased need for further training in IT security, but at the same time consider themselves to be well prepared, which shows a clear self-perception gap.

The software testing survey has been delivering reliable trends instead of gut feeling since 2011

The German Testing Board’s software testing survey surveys the status quo of testing in German-speaking countries: what teams are actually doing, where the challenges lie and what changes have emerged over the years. It has been running at regular intervals since 2011, with editions in 2011, 2016, 2020 and 2024, meaning it now covers almost 15 years.

The 2024 survey is based on over 800 participants. This is a statistically viable figure, even if the in-depth analysis reaches its limits in some places: When many individual questions are correlated with other categories, the number of answers per intersection decreases and trends can be read rather than hard statements.

The evaluation is scientifically supported. Instead of a quick LinkedIn survey, it is based on null hypotheses, correlations and a technical report several hundred pages long. There are three sets of results for different reading needs: a management summary, a brochure and the full technical report.

Why three questionnaires show more than one

The survey separates roles, and it is precisely this separation that makes many results meaningful. There is a questionnaire for quality assurance managers, one for operational staff and another. In this way, perspectives can be compared directly.

The difference is clear when it comes to delegation. Managers tend to want to outsource as much as possible. Operational managers differentiate more strongly: they would rather hand over technical tasks such as test automation, while they are critical of delegating softer disciplines such as architecture or security management to external parties.

The participants are spread across a wide range of sectors and company sizes. The focus is on the financial sector, the public sector and the automotive industry, which is comparable to the Bitkom figures. Both corporations with over a thousand employees and small companies with 10 to 20 employees are represented.

AI has arrived in coding, still reluctant in testing

aI was surveyed for the first time in 2024, and the result is clear: one third of participants are already using AI for coding and quality assurance. This is a remarkable penetration, considering that the widespread availability of generative tools was only around two and a half years ago.

The figures are highest in programming itself. Almost 50 percent of those working in operations are already using AI there or are planning to do so in the near future. The figures are significantly lower for testing tasks.

There is a rule of thumb: the more technical a task, the more trust is placed in AI. For design decisions, project and test management, i.e. the more human factors, the values decrease significantly.

This has consequences for career orientation. Pure development is likely to become less important as a sole profile. By contrast, those who take on the role of architect, who bring components together and negotiate interfaces between people, are investing in the right direction.

If you work with AI and don’t know what a whitebox is, don’t know what an equivalence partition is: how am I supposed to assess that what the AI generates for me is reasonably useful? Frank Simon

Test automation has stagnated since 2020

There has been little movement in test automation for years, and that is the sore point of the current survey. Automation increased from 2011 to 2020, but has stagnated since 2020. It remains anchored in the lower test levels.

The figures for regression testing are sobering. 12 percent of participants state that they do not automate regression testing at all, even though this test in particular lends itself to automation.

The load and performance test is even more drastic. A quarter of respondents do not automate them. It is difficult to imagine load testing being carried out manually, which is probably due to a perception problem: In agile projects with CI/CD pipelines, load and performance often does not appear in the daily process, but is dealt with in higher-level groups and not even seen by the individual tester.

End-to-end test automation is widely discussed at conferences and in companies. This is not reflected in the figures. Remarkably little happens there.

Self-image and reality diverge when it comes to security testing

The biggest discrepancy can be seen in security, load and performance testing. Testers consider their work there to be effective, but at the same time they know that customers may very well end up with security problems. This gap is untenable in the long term.

Overall, the security test is one of the most poorly reported tests. This is due to a lack of basic knowledge: 12 percent of respondents said they did not know what a whitebox test was. A real security test is technically demanding, and those who are not familiar with the whitebox artifact probably have little to do with it.

Nevertheless, many recognize their own vulnerability. 66 percent of operational employees state an increased need for further training in IT security, a third also in load and performance testing. The self-assessment is correct: Before you can evaluate what tools and AI produce, you need substantive knowledge.

Test techniques are increasingly being forgotten

The classic test techniques are being used less frequently instead of more often. Boundary value analysis and equivalence partitioning are at just under 40 percent, although equivalence analysis in particular is one of the most intuitive procedures of all. There has been no progress.

This is surprising against the background of over 20 years of certification by the ISTQB and German Testing Board. With so many Certified Tester courses, procedural knowledge should be more widely anchored. Instead, the discussion revolves almost exclusively around automation, and hardly at all around how to narrow down the test cases to be automated in a meaningful way.

More tools do not automatically solve the problem. Modern tools have long provided whitebox information and coverage levels. If users still claim to be unfamiliar with the concept, there is little to suggest that AI can help here.

A similar picture emerges for test management. Only just under 50 percent of respondents use a test management tool at all. The rest probably work with Excel. A tool is no substitute for understanding, but for a discipline such as quality, this rate is low.

Agile or hybrid? Many teams can’t decide

Hybrid is as common as agile in 2024, and this reveals an inconsistency. Around the same number of participants describe their approach as agile as hybrid. Behind hybrid is often an inability to commit: People work in a supposedly agile way, but central decisions are still made by the boardroom.

The distribution of roles also speaks the same language. Most assign themselves to the classic roles of developer or tester. Genuinely agile roles such as Scrum Master, Product Owner or member of the agile team are each below 5 percent. Some of those who describe themselves as agile team members nevertheless find themselves in a classic, plan-oriented process model.

Test-driven development is particularly revealing, as there is a clear negative trend here:

For someone who appreciates agile techniques, the trend is pointing in the wrong direction. Pair programming, collective code ownership and stand-up meetings as a quality assurance measure are all below 50 percent. The development-related practices are widespread, while the practices that change mindset and organization are lagging behind.

What testers and test managers should take away from the figures

Certificates remain relevant to careers. In 2024, a third of managers still state that certification is mandatory and has no alternative. A certificate promotes a career, it does not hinder it.

When it comes to security testing, active promotion within your own teams pays off. A safety error rarely forgives a second attempt. It is precisely this discipline that is dismissed in the survey as less exciting, which remains difficult to understand in view of the real risks.

The certification courses themselves could be rethought. A four-day course teaches the vocabulary, but not how to implement it in your own context. It would make sense to provide support afterwards, for example through mentoring or user groups in which you can bring a specific problem and discuss it. This way, procedures are not just learned as concepts, but internalized.

Despite faster machines and better automation, the statement about the quality of testing remains the real achievement. Those who have mastered the procedures can assess what tools and AI actually deliver.

Where you can find the results

All results are available free of charge and without registration at softwaretest-umfrage.de. The management summary and the individual statistical evaluations of all questions, including graphics, are already available there, as are the results of previous surveys.

The detailed brochure with the best-of analyses comprises around 50 to 60 pages and is due to be published at the end of May. The full technical report is expected at the end of the summer, much earlier than in previous years. Individual presentations and shorter reports will accompany the publication, including at the German Testing Day and in SQ magazine.