Current testing trends are moving along two parallel tracks: AI as a tool for testers, and the testing of AI-based systems. Survey data shows that AI support in testing already ranks second, right behind programming. Agile models, above all Scrum, now dominate across industries, including safety-critical fields such as aviation and medical technology.
Key Takeaways
- AI-based test generation currently works only as an assistant: experts have to review and rework its output, and blindly trusting generated tests is not defensible.
- Among the testing practitioners surveyed, programming and testing already use AI the most, while requirements engineering gets far less AI support, despite obvious use cases such as detecting duplicate requirements.
- According to the survey data, Scrum is clearly the leading process model, and even safety-critical industries such as aviation and medical technology now officially permit agile development.
- In practice, scenario-based testing for autonomous systems struggles less with the basic idea than with two concrete hurdles: choosing representative scenarios and the modeling effort behind each test case.
Testing Trends: AI in Testing Has Two Sides
The biggest testing trends right now concern AI, and AI in testing covers two different tasks that are evolving in parallel and at different speeds. One is testing AI-based systems. The other is testing with AI, meaning AI as a tool for the person doing the testing.
If you lump the two together, you mix up very different levels of maturity. One field is still being prepared. The other has already arrived in day-to-day practice.
Tilo Linz assesses where both fields stand based on what he sees at customers, at conferences and through the Trends in Testing event series, which imbus has been running for more than ten years.
Where Testing AI-Based Systems Stands Today
Almost every company that develops software seriously is currently looking at which features in its products AI could improve or extend. That goes for pure software products and also for hardware-based systems with software inside: machines that are supposed to become smarter through AI.
In these companies, prototypes and experiments are under way to find out where AI actually makes their products better.
For software testing, this means the job of testing such systems is being prepared but not yet done for real. People are getting up to speed, going to conferences and reading up. But the systems aren’t ready for production yet, so the testing hasn’t gone live.
Tilo Linz expects that moment to arrive quickly and with force over the next few years. Testers will then face the question of whether they have really found everything the system is supposed to do, and all the bugs hiding in it. His assessment is clear: these systems will contain more bugs than what testers deal with today.
Testing with AI: Programming Leads, Testing Follows Close Behind
A Trends in Testing survey shows a clear ranking of where AI tools are already used in software development. Programming comes first, testing second and requirements engineering third. Project management trails far behind.
That programming leads was to be expected. Today’s common IDEs come with built-in AI assistants that let you modify code or ask questions about it directly.
Third place for requirements engineering is the surprise. Requirements are written documents made of language, which makes them obvious material for analysis by language models. One explanation: the respondents come from software testing and may have less insight into what their colleagues in requirements engineering actually do.
The low score for project management is puzzling, too. Whether agile or hybrid, project management is a frequent source of problems when steering doesn’t run smoothly. Tilo Linz sees untapped potential there.
AI in Testing Means Assistance, Not Autopilot
In testing, AI is used to generate test data and test cases. A test case is more than a data set: it also includes the procedure for running the test and the expected result. Companies are trying out both, at different levels of maturity.
One concrete example is AI-based generation of security tests built on the OWASP criteria catalog. You describe what the application does, hand over the current catalog and have the AI generate test procedures for a specific criterion.
The result is not a finished test. The security testing specialist in charge has to rework it before it actually runs.
That is the key point: these systems work as assistants. You can’t rely on the AI with an attitude of “if it’s green, the test has run.” The AI helps you get to a starting point faster.
“An interesting dialog develops, and you have to pick out what helps you and throw away the parts where you think: that’s odd.”
(Tilo Linz)
Where Test Automation with AI Is Heading
The next logical step combines AI-based generation of test sequences with keyword-driven testing. If you already have a library of building blocks in which individual keywords are reliably automated, an AI tool can keep rearranging those building blocks into new procedures.
Combine that with generated test data, and fully automated generation of test sequences comes within reach. It is the natural continuation of today’s building blocks, not a replacement for them.
Agile Isn’t Dead, It Has Become Routine
The claim that agile is dead doesn’t hold up in practice. Trends in Testing surveys from 2023 and 2024 show the opposite: agile approaches dominate, with Scrum clearly in the lead. The German Testing Board’s survey shows the same trend.
Kanban comes after Scrum. V-model projects still exist, but they are mostly legacy projects that aren’t worth converting anymore. Anything new that starts today is usually agile.
This has reached critical industries as well. In aviation and medical technology, the responsible authorities now officially permit an agile approach, provided the necessary safety nets are in place. They no longer insist on the V-model, as they used to.
Anyone who concludes from failed agile projects that agile is over is missing two things. First, the fair question is whether more projects fail now than under waterfall, or significantly fewer. If it’s fewer, that speaks for the agile approach. Second, projects often fail because agile was only introduced on the surface.
Teams that do Scrum superficially without really applying the techniques and practices muddle through just as they did in phase-based projects. If the project fails, the model isn’t to blame. Pinning it on the phase model or on agile gets you nowhere.
Testing in Agile Projects Still Needs the Classic Test Levels
The test levels known from the V-model remain relevant in agile work. Unit testing, integration testing and system testing are still needed as levels of abstraction, just in a different context and with different weight.
Tilo Linz takes this view in his book “Testen in agilen Projekten” (Testing in Agile Projects). It looks at agile primarily from the tester’s perspective, is structured along these test levels and explains the techniques that belong to each.
The first edition came out about ten years ago. The current edition had to take in a good deal of new material, up to DevOps and the tool-driven developments of recent years. So you don’t throw out what worked before. You put it in a new context.
Requirements Engineering Has the Biggest Untapped AI Potential in the Short Term
If the low usage in requirements engineering isn’t just a blind spot of the testers surveyed, this is where the biggest short-term potential lies. Once requirements are captured, they can be classified, analyzed and compared along many dimensions.
A simple and effective use case is detecting duplicates. Two people write the same requirement in slightly different words, a well-known source of errors in development.
The tool shouldn’t decide on its own which duplicate stays. It works like a code analysis tool: it warns that two similar requirements exist and asks you to check them again or merge them. AI as a checker and assistant, not as the decision-maker.
Scenario-Based Testing and Its Two Variants
Scenario-based testing comes in two forms that are easy to confuse. The first is the umbrella term used by ISTQB and the GTB: you write down usage scenarios much like a user story, map out flow variants A, B and C and check how the test object responds. That is a generalization of familiar test procedures and classifications.
The second variant is the specific form used for testing autonomous vehicles. Here the scenario is a traffic scenario, for example a modeled intersection with pedestrians, traffic lights and cars coming from different directions. The scenario is described formally and fed into a simulator, and the vehicle’s control system has to get through the intersection without a crash.
This second form is the genuinely new kind of scenario-based testing, and it is much harder to handle, because many autonomous agents move independently of each other.
The Two Challenges of Scenario-Based Testing for Autonomous Vehicles
The first challenge is selection. Which scenario is worth modeling? If a vehicle handles one intersection, that doesn’t mean it handles every intersection. Deciding whether a scenario is relevant and representative amounts to an abstract form of equivalence partitioning. Several formalisms and standardization initiatives are working on it.
The second challenge is the modeling effort. A scenario has to be detailed enough to serve the test purpose and expose the vehicle to exactly the aspects under test. At the same time, the effort mustn’t get out of hand, so that modeling, reviews and new versions stay manageable.
Together, these two make scenario-based testing demanding, which is why it isn’t used as widely as people originally hoped. Work is under way on recipes, languages and simulators that make the approach possible with less effort.
The approach matters beyond cars. It applies to every autonomous agent that moves on its own, from small robots to delivery drones.
Frequently Asked Questions
Why are there two distinct tasks when it comes to AI in testing?
Testing AI systems and testing with AI have different levels of maturity. Companies are currently preparing to conduct testing on AI-based systems: prototypes are being developed, people are familiarizing themselves with the technology, but testing in this area is rarely fully operational yet. AI as a tool for testers, on the other hand, has already become part of everyday work. Those who confuse the two draw incorrect conclusions about the current state of affairs.
In which disciplines of software development is AI already used most extensively?
A survey by Trends in Testing revealed the following order: programming in first place, testing in second, requirements engineering in third, and project management far behind. Programming’s lead can be explained by the AI assistants in common IDEs. Since the respondents came from the software testing field, the low score for requirements engineering may also be a perception bias.
Can AI-generated test cases be adopted without review?
No. In testing, AI acts as an assistant, not an autopilot. When generating security tests based on the OWASP criteria catalog, the result is not a runnable test, but rather a foundation that the responsible specialist must revise. Through dialogue with the model, one extracts what is useful and discards what is questionable. The attitude of “if it’s green, the test has run” does not hold up.
Is agility on the decline?
Practical experience does not support this. Surveys by Trends in Testing from 2023 and 2024 show agile approaches as dominant, with Scrum clearly in the lead and Kanban following behind. The German Testing Board’s survey points in the same direction. V-model projects mostly continue as legacy projects for which a transition is no longer worthwhile. New projects generally start out agile.
Is agile development permitted in safety-critical industries such as aviation and medical technology?
Yes. The relevant authorities now officially permit an agile approach in these sectors, provided the necessary safeguards are in place. The V-model is no longer required, as was previously the case. This does not mean that quality assurance is no longer required; it simply no longer has to be delivered through a phase-oriented model.
Do unit tests, integration testing, and system testing still need to be performed in agile projects?
Yes. The test levels familiar from the V-model remain necessary as levels of abstraction, just in a different context and with different weightings. Each level has its own techniques. What worked well in the past isn’t thrown overboard, but rather reclassified and supplemented with topics such as DevOps and tool-driven development.
Which AI use case in requirements engineering delivers immediate benefits?
Detecting duplicates. Two people may phrase the same requirement slightly differently, a well-known source of errors in development. The tool should not decide which version to keep, but rather, like a code analysis tool, warn that two similar requirements exist and prompt the user to review or merge them. Recorded requirements can also be classified and compared across many dimensions.
Why has scenario-based testing of autonomous vehicles been used less than expected so far?
Two hurdles are holding it back: the selection of scenarios and the effort required for modeling. Determining whether a traffic scenario is relevant and representative amounts to an abstract form of equivalence partitioning, since successfully navigating one intersection says nothing about all intersections. At the same time, a scenario must be detailed enough without the modeling, review, and development of new versions becoming too time-consuming. Work is underway on languages and simulators.
