Using language models in software testing means applying AI to specific, bounded tasks where language is already the raw material: generating test documentation, reviewing keyword implementations against defined rules, translating technical defect messages into plain language, and identifying alternative text for accessibility checks. Smaller, well-defined tasks yield reliable results. Reasoning models outperform standard chat models on rule-based review tasks.
Key Takeaways
- Reasoning models outperform standard chat models for code review tasks: switching from GPT-4o to O1 was what made automated keyword review produce usable results.
- Small, tightly scoped AI functions deliver faster adoption than large end-to-end automation, because testers can verify the output and stay in control of the process.
- Self-healing test automation that uses embeddings to repair broken locators risks masking real defects, because the system cannot distinguish an accidental change from an intentional one.
- As AI agents gain autonomy and operate without a human between each step, the quality requirements for their output rise sharply, which closes the gap between using AI and testing AI.
Start with small functions, not a general testing model
The most reliable way to bring language models into testing is to solve narrow, well-defined problems rather than build one all-purpose testing model. Klaudia Dussa-Zieger and her team followed this path deliberately: instead of a fine-tuned general testing LLM, they built small functionalities that each help a user with a single, concrete task.
That framing matters more than tool choice. When you know exactly which problem you want to solve, you can pick the right model, feed the right context, and judge the result. When you start from “we have to use AI”, you get vague pilots that never land.
The team’s own history shows how early some of this began. Their first AI experiments date to 2019, using a visual classification algorithm to recognize moving objects on the GUI. That was before generative models entered their work. The language-model applications came later, driven by a simple observation: testing runs on text, on specifications, test scripts, and reports.
Test specification works best as a draft, not a finished product
An LLM should give a tester a starting draft, not a final artifact. Klaudia’s first larger prototype derived test specifications from requirements using a retrieval-augmented generation (RAG) approach, feeding project-specific information into a generic model. The first models used were Llama 3.1, later Claude 3.5.
The initial version ran a complete cycle with no human intervention. It worked, but not well. The point was never full automation. The point was to remove the blank page, so a tester starts with a draft instead of an empty document.
The testers in that company had a stronger reaction than expected. They felt left out of a process that touched their craft. The fix served two goals at once: it improved the final quality and it brought the testers on board by giving them one place to feed in their own experience and information. That input carries real value, and a system that ignores it loses both quality and trust.
Documenting keywords: one button, more uniform output
Automatic documentation of keywords is a small function that pays off fast. In the team’s test management tool, testers specify tests using keywords, which are effectively code, an automated script. Like most programmers, they dislike writing documentation.
The solution was a single button. It reads the implementation of a keyword and produces a header, a description of the parameters, and a longer description of what the keyword does. The tester presses the button, reads the result, and is done.
The gain went beyond speed. Because everyone now uses the same function, the descriptions became more uniform across the team. Consistency, not just convenience, turned out to be the lasting benefit.
Why reviews only worked with reasoning models
Automated review of keyword implementations failed with standard chat models and succeeded only with a reasoning model. This is the sharpest lesson in the team’s work: model type, not just model access, decides whether a use case works.
The setup fed the team’s rules for implementing keywords and a glossary of allowed terms into the model as additional context. Early attempts with ChatGPT-4o did not produce good enough results. Moving to a reasoning model, O1, made the difference.
The working version catches most of the obvious issues within a second or two. Testers typically run two or three cycles until the keyword looks clean formally, after which the automation engineers review the logic. The reviewed developers welcomed it for a practical reason: it removed the tedious, boilerplate part of reviewing so they could focus on the interesting problems.
This split is worth keeping in mind when you plan your own functions.
| Function | Model type used |
|---|---|
| Keyword documentation | Standard model |
| Keyword review against rules and glossary | Reasoning model (O1) |
There will not be one generic solution that covers everything. Different tasks need different capabilities, and matching the two is part of the design work.
Translating technical defects into language a domain tester understands
An LLM can convert a technical defect message into a description a domain user can read. Anticipated errors usually carry a friendly message, but a genuine technical failure throws something dense and hard to parse. Domain testers often cannot tell what it means.
The team’s function feeds the defect message together with the test case that produced it into an LLM, which returns a plain-language explanation. This is another translation problem, and language is exactly what these models handle well. It worked cleanly.
Everyday coding support is already routine
Using AI for coding and environment setup has quietly become normal practice inside the team. Colleagues use Copilot for coding without treating it as a special event. One developer also uses it to write Vagrant scripts that set up the test environment automatically, which fits a DevOps approach where the environment is built by code.
The pattern he described is worth copying. The generated script gets him to about 95 percent, a boilerplate result he gets in seconds. The last five percent he finishes himself, because he knows what the script should look like and can check it quickly. What would take fifteen minutes to half an hour becomes a fast draft plus a review.
Human in the loop makes accessibility testing faster, not autonomous
You do not need a perfect result when a qualified reviewer stays in the loop. The team tested this on digital accessibility, an area long assumed to cap out around 40 percent automation because a human has to judge whether text is acceptable.
They tried it anyway, using an agent setup with MCP to generate alternative text for images, the case that requires interpreting a picture. It worked, though not perfectly. Sometimes the system proposes alt text the specialist would reject as a duplication that adds no meaning.
The value is speed, not autonomy. The system runs through all the images quickly, and the human reviews each result and decides whether it holds up. That division of labor is the realistic model for many testing tasks today.
Where LLMs still fail: reverse engineering embedded code
Some testing work is beyond current models, and reverse engineering old embedded code is a clear example. Two projects write unit tests for existing C code for regulatory reasons, work that resembles archaeology: you dig into poorly documented code with no requirements and reconstruct meaning from the source alone.
Asked directly whether an LLM could help, the team’s AI expert said no. The teams do not want line-coverage tests. They want test cases that make sense, and that level of understanding is not there. Writing a keyword header or checking against rules is one thing. Concentrating on undocumented legacy code to write meaningful tests is another.
His view was blunt: reaching that level would demand heavy investment in training or adapting the models, and by then a small team can do the work itself.
Higher-level test automation: partial help, mixed results
For test automation, LLM support ranges from solid to unresolved depending on the layer. Automation is programming, so Copilot-style coding help works as well here as it does for normal development.
On the system level, through the graphical user interface, AI can help recognize objects that move or get renamed slightly. Word embeddings from LLMs have been used to find alternative locators when the first one breaks, which points toward self-healing test cases.
Self-healing carries a real risk. If embeddings quietly repair broken locators, they can cover up genuine defects, and it becomes hard to tell an accidental change from a real mistake that deserves attention. One presentation on this idea at RoboCon 2025 was no longer being pursued the following year. The direction is open, not proven.
Free up capacity for the characteristics that get dropped first
The strongest wish for AI in testing is speed on routine work, so testers can spend attention where quality usually erodes. Functional tests come first under time pressure. Performance and security follow. Usability is the first characteristic dropped when time runs short.
If AI clears the routine functional load fast enough, teams can protect the qualities that normally get neglected. Test reporting is another natural fit, since summarizing information into text is exactly what a language model does well, and writing that summary is effort many testers would rather hand off.
Using AI and testing AI are converging again
The gap between using AI for testing and testing AI itself is closing as systems gain autonomy. For a long time these looked like two separate concerns, with a human always sitting between the model and the next step.
Autonomy changes the equation. MCP, the model context protocol, was released in November 2024, and with it the number of agents running on their own is rising. The more autonomy an agent gets, the better its output has to be, because there is no human in between to catch a weak result before the next processing step.
I think with the advent of more autonomous systems the gap between using AI and testing AI is closing again, because the more autonomy you give to an AI agent, the better it has to be. — Klaudia Dussa-Zieger
That shift turns autonomy into a testing question. Deciding how much liberty to give a system is a risk-based decision, the same logic that has always driven testing. An autonomous agent that books appointments or moves money needs access to your banking and health data, and the level of access such agents require is reason enough to test them hard. As agent farms and orchestrated teams spread, the quality of the individual agents becomes the thing that matters most.
