Using language models in software testing means applying AI to specific, bounded tasks where language is already the raw material: generating test documentation, reviewing keyword implementations against defined rules, translating technical defect messages into plain language, and identifying alternative text for accessibility checks. Smaller, well-defined tasks yield reliable results. Reasoning models outperform standard chat models on rule-based review tasks.
Key Takeaways
- Reasoning models outperform standard chat models for code review tasks: switching from GPT-4o to O1 was what made automated keyword review produce usable results.
- Small, tightly scoped AI functions deliver faster adoption than large end-to-end automation, because testers can verify the output and stay in control of the process.
- Self-healing test automation that uses embeddings to repair broken locators risks masking real defects, because the system cannot distinguish an accidental change from an intentional one.
- As AI agents gain autonomy and operate without a human between each step, the quality requirements for their output rise sharply, which closes the gap between using AI and testing AI.
Start with small functions, not a general testing model
The most reliable way to bring language models into testing is to solve narrow, well-defined problems rather than build one all-purpose testing model. Klaudia Dussa-Zieger and her team followed this path deliberately: instead of a fine-tuned general testing LLM, they built small functionalities that each help a user with a single, concrete task.
That framing matters more than tool choice. When you know exactly which problem you want to solve, you can pick the right model, feed the right context, and judge the result. When you start from “we have to use AI”, you get vague pilots that never land.
The team’s own history shows how early some of this began. Their first AI experiments date to 2019, using a visual classification algorithm to recognize moving objects on the GUI. That was before generative models entered their work. The language-model applications came later, driven by a simple observation: testing runs on text, on specifications, test scripts, and reports.
Test specification works best as a draft, not a finished product
An LLM should give a tester a starting draft, not a final artifact. Klaudia’s first larger prototype derived test specifications from requirements using a retrieval-augmented generation (RAG) approach, feeding project-specific information into a generic model. The first models used were Llama 3.1, later Claude 3.5.
The initial version ran a complete cycle with no human intervention. It worked, but not well. The point was never full automation. The point was to remove the blank page, so a tester starts with a draft instead of an empty document.
The testers in that company had a stronger reaction than expected. They felt left out of a process that touched their craft. The fix served two goals at once: it improved the final quality and it brought the testers on board by giving them one place to feed in their own experience and information. That input carries real value, and a system that ignores it loses both quality and trust.
Documenting keywords: one button, more uniform output
Automatic documentation of keywords is a small function that pays off fast. In the team’s test management tool, testers specify tests using keywords, which are effectively code, an automated script. Like most programmers, they dislike writing documentation.
The solution was a single button. It reads the implementation of a keyword and produces a header, a description of the parameters, and a longer description of what the keyword does. The tester presses the button, reads the result, and is done.
The gain went beyond speed. Because everyone now uses the same function, the descriptions became more uniform across the team. Consistency, not just convenience, turned out to be the lasting benefit.
Why reviews only worked with reasoning models
Automated review of keyword implementations failed with standard chat models and succeeded only with a reasoning model. This is the sharpest lesson in the team’s work: model type, not just model access, decides whether a use case works.
The setup fed the team’s rules for implementing keywords and a glossary of allowed terms into the model as additional context. Early attempts with ChatGPT-4o did not produce good enough results. Moving to a reasoning model, O1, made the difference.
The working version catches most of the obvious issues within a second or two. Testers typically run two or three cycles until the keyword looks clean formally, after which the automation engineers review the logic. The reviewed developers welcomed it for a practical reason: it removed the tedious, boilerplate part of reviewing so they could focus on the interesting problems.
This split is worth keeping in mind when you plan your own functions.
| Function | Model type used |
|---|---|
| Keyword documentation | Standard model |
| Keyword review against rules and glossary | Reasoning model (O1) |
There will not be one generic solution that covers everything. Different tasks need different capabilities, and matching the two is part of the design work.
Translating technical defects into language a domain tester understands
An LLM can convert a technical defect message into a description a domain user can read. Anticipated errors usually carry a friendly message, but a genuine technical failure throws something dense and hard to parse. Domain testers often cannot tell what it means.
The team’s function feeds the defect message together with the test case that produced it into an LLM, which returns a plain-language explanation. This is another translation problem, and language is exactly what these models handle well. It worked cleanly.
Everyday coding support is already routine
Using AI for coding and environment setup has quietly become normal practice inside the team. Colleagues use Copilot for coding without treating it as a special event. One developer also uses it to write Vagrant scripts that set up the test environment automatically, which fits a DevOps approach where the environment is built by code.
The pattern he described is worth copying. The generated script gets him to about 95 percent, a boilerplate result he gets in seconds. The last five percent he finishes himself, because he knows what the script should look like and can check it quickly. What would take fifteen minutes to half an hour becomes a fast draft plus a review.
Human in the loop makes accessibility testing faster, not autonomous
You do not need a perfect result when a qualified reviewer stays in the loop. The team tested this on digital accessibility, an area long assumed to cap out around 40 percent automation because a human has to judge whether text is acceptable.
They tried it anyway, using an agent setup with MCP to generate alternative text for images, the case that requires interpreting a picture. It worked, though not perfectly. Sometimes the system proposes alt text the specialist would reject as a duplication that adds no meaning.
The value is speed, not autonomy. The system runs through all the images quickly, and the human reviews each result and decides whether it holds up. That division of labor is the realistic model for many testing tasks today.
Where LLMs still fail: reverse engineering embedded code
Some testing work is beyond current models, and reverse engineering old embedded code is a clear example. Two projects write unit tests for existing C code for regulatory reasons, work that resembles archaeology: you dig into poorly documented code with no requirements and reconstruct meaning from the source alone.
Asked directly whether an LLM could help, the team’s AI expert said no. The teams do not want line-coverage tests. They want test cases that make sense, and that level of understanding is not there. Writing a keyword header or checking against rules is one thing. Concentrating on undocumented legacy code to write meaningful tests is another.
His view was blunt: reaching that level would demand heavy investment in training or adapting the models, and by then a small team can do the work itself.
Higher-level test automation: partial help, mixed results
For test automation, LLM support ranges from solid to unresolved depending on the layer. Automation is programming, so Copilot-style coding help works as well here as it does for normal development.
On the system level, through the graphical user interface, AI can help recognize objects that move or get renamed slightly. Word embeddings from LLMs have been used to find alternative locators when the first one breaks, which points toward self-healing test cases.
Self-healing carries a real risk. If embeddings quietly repair broken locators, they can cover up genuine defects, and it becomes hard to tell an accidental change from a real mistake that deserves attention. One presentation on this idea at RoboCon 2025 was no longer being pursued the following year. The direction is open, not proven.
Free up capacity for the characteristics that get dropped first
The strongest wish for AI in testing is speed on routine work, so testers can spend attention where quality usually erodes. Functional tests come first under time pressure. Performance and security follow. Usability is the first characteristic dropped when time runs short.
If AI clears the routine functional load fast enough, teams can protect the qualities that normally get neglected. Test reporting is another natural fit, since summarizing information into text is exactly what a language model does well, and writing that summary is effort many testers would rather hand off.
Using AI and testing AI are converging again
The gap between using AI for testing and testing AI itself is closing as systems gain autonomy. For a long time these looked like two separate concerns, with a human always sitting between the model and the next step.
Autonomy changes the equation. MCP, the model context protocol, was released in November 2024, and with it the number of agents running on their own is rising. The more autonomy an agent gets, the better its output has to be, because there is no human in between to catch a weak result before the next processing step.
I think with the advent of more autonomous systems the gap between using AI and testing AI is closing again, because the more autonomy you give to an AI agent, the better it has to be. — Klaudia Dussa-Zieger
That shift turns autonomy into a testing question. Deciding how much liberty to give a system is a risk-based decision, the same logic that has always driven testing. An autonomous agent that books appointments or moves money needs access to your banking and health data, and the level of access such agents require is reason enough to test them hard. As agent farms and orchestrated teams spread, the quality of the individual agents becomes the thing that matters most.
Frequently Asked Questions
Should a team fine-tune one general-purpose model for testing, or build small AI functions?
Small, tightly scoped functions are the safer bet. Klaudia Dussa-Zieger’s team deliberately skipped a fine-tuned general testing LLM and built single-task helpers instead. With a clearly defined problem, you can choose a suitable model, supply the right context and judge the output. Projects that start from “we have to use AI” tend to produce vague pilots that never land.
Can a language model produce a finished test specification from requirements?
No. A prototype built with retrieval-augmented generation, first on Llama 3.1 and later Claude 3.5, ran a full cycle without human intervention and worked, but not well. The realistic goal is a draft. The tester starts from a filled page instead of an empty document and adds the project experience the model has no access to.
Does the type of model matter when reviewing test code against internal rules?
Yes, decisively. Automated review of keyword implementations against a team’s rules and a glossary of allowed terms produced unusable results with the standard chat model GPT-4o and only became viable with the reasoning model O1. The working version flags most obvious issues in a second or two, and testers usually run two or three cycles before an engineer checks the logic.
How much of accessibility testing can realistically be automated with AI?
Accessibility work was long assumed to cap out near 40 percent automation, because a human has to judge whether a text is acceptable. An agent setup that generates alternative text for images pushed past that, though not perfectly: some suggestions merely duplicate information and a specialist rejects them. The benefit is speed, while the reviewer still decides.
Can language models write meaningful unit tests for undocumented legacy embedded code?
No, and that is a clear limit. Adding unit tests to existing C code for regulatory reasons resembles archaeology: no requirements, poor documentation, meaning reconstructed from the source alone. Such teams do not want line-coverage tests, they want test cases that make sense, and that level of understanding is missing. Closing the gap would demand heavy investment in training or adapting models.
What is the risk of self-healing test automation?
Automatically repairing broken locators can hide real defects. Word embeddings can find an alternative locator when the first one fails, but the system cannot distinguish an accidental change from an intentional one, so a genuine mistake passes unnoticed. An idea presented at RoboCon 2025 was not pursued afterwards. The direction is open, not proven.
Why does agent autonomy raise the bar for testing AI systems?
With no human between the steps, nothing catches a weak result before it feeds the next processing stage, so the required output quality rises sharply. Since the model context protocol appeared in November 2024, more agents run on their own. How much liberty to grant is a risk-based decision: an agent that books appointments or moves money needs banking and health data.
