AI-powered documentation testing refers to the use of language models as simulated user personas that not only evaluate documentation but also actively work with it. A persona—such as a junior developer—attempts to complete a task using the documentation. If the persona fails to do so, this reveals specific gaps in the document.
Key Takeaways
- AI agents with simulated personas provide a practical way to test documentation: An agent configured as a junior developer used 1.8 million tokens when it failed, while the senior developer agent solved the same task with 300,000 tokens.
- Persona simulations using LLMs revert to generic model behavior as soon as the role description is not formulated strictly enough, which distinguishes them from the prototype stage.
- An LLM’s temperature parameter controls how much the output deviates from the most likely path: low for reproducible, deterministic results; high for unexpected test results.
- No LLM can reconstruct domain knowledge and architectural decisions that are not documented, because they are simply not embedded in the code.
- Semantic anchors such as MECE measurably increase the likelihood of structured, non-overlapping AI outputs, but do not guarantee them.
AI Personas as a Tool for Testing Documentation
AI-powered documentation testing works by assigning a specific personality to a language model and having it attempt to actually carry out the tasks described in the documentation. Instead of merely evaluating a set of instructions, the agent is supposed to use them. Ingo Eichhorst refers to such an agent, for example, as a junior developer: new on the job, not in the mood to read everything, who skims the documentation and experiments.
The testing concept behind this is simple. If someone with little background knowledge wants to get the software up and running quickly, the documentation must make that possible. If the simulated junior developer can’t do it, there’s a real problem with the instructions—not with the user.
Ingo first developed this approach two and a half years ago with a payment provider based in Saudi Arabia and Dubai. At that time, language models were used to simulate people in order to test how well they could find the documentation and whether they could use it to implement the software. This idea led to a prototype that applies the same logic to an existing pipeline.
How the Testing Agent Fits into the Pipeline
Persona testing is a backward-looking process: something already exists, and the AI evaluates how good it is. Ingo distinguishes between two ways AI helps with code and documentation. The forward approach generates something; the backward approach performs appraisal of what already exists.
Many teams already run CI/CD pipelines for their documentation. The documentation passes through quality gates, undergoes testing, and must meet certain conditions. The Persona Agent is an additional component in this very chain. At the end, it generates a kind of feedback form that shows where improvements are needed before the documentation is accepted.
While the review is already AI-supported, creating the documentation remains largely a manual process with this approach. The prototype must first achieve maturity before it’s ready for production.
Different personas find different weaknesses because they approach the task differently. In addition to the junior developer, there’s a senior developer. Ingo even simulated his mother as a persona.
The appeal lies in representing groups that are missing from standard tests. A usability test at a university primarily provides students as test subjects. They know what AI is and know where to click to get a result. Such results are skewed. Simulated personas can also capture outliers and unfamiliar ways of thinking.
This is especially true when deciding whether to launch a product in another country. What are the prevailing mindsets there, and how does the culture differ? These scenarios can be simulated in advance, rather than discovered only after launch.
There is a technical hurdle with simulation. The model easily reverts to its training data. Ingo’s simulated mother was eventually able to write PHP after all because the model reverted to its original mode. A persona must therefore be defined very rigidly and precisely.
Why a Simulated Junior Developer Failed Because of a URL
The junior developer failed because of a detail that a human would hardly have noticed. The task was to use the Ionos Model Hub documentation to find the cheapest AI model, ask it for the answer to a question from The Hitchhiker’s Guide to the Galaxy, and check whether “Adams” appeared in the answer. A “pass” meant the documentation was good enough.
The junior agent loaded a page and received a 404 error. URLs at OpenAI follow a specific scheme, whereas the Ionos URL is structured differently. Clearer instructions were lacking at this point. The agent kept trying, kept getting it wrong, and consumed a massive amount of resources in the process.
The difference is evident in token consumption:
| Persona | Token Consumption | Behavior |
|---|---|---|
| Senior Developer | approx. 300,000 tokens | implemented correctly, done |
| Junior Developer | approx. 1.8 million tokens | trial and error, RTFM disabled |
Such errors are revealing because modern models can operate a browser. A process runs in the background where the model “talks” to itself. This thought process can be analyzed to identify where a line of reasoning takes a completely wrong turn.
An agent once complained that it couldn’t read the JavaScript because it had downloaded the wrong page. This page is only relevant to machines. A human would never have had this problem. But as soon as an AI boots up a VM instead of a human, too much JavaScript in the documentation suddenly becomes a real problem.
There’s No 100-Percent Guarantee When Generating Documentation
Anyone who uses language models to generate documentation must let go of the idea of foolproof output. A calculator always gives 2 for 1 plus 1; otherwise, it’s broken. An LLM responds differently. With GPT-3.5, there’s about a 55.7 percent chance that 1 plus 1 will result in 2.
This variation is intentional. The temperature parameter controls how much a model deviates from the most likely token. When set high, the model sounds more human and tends to produce unexpected outputs, which is ideal for a quirky tester. When set low, it sounds more robotic and always returns the same result for the same query.
Even at a low temperature, a small change in the input can lead to a completely different result. That’s why the forward path needs external boundaries. In coding, this takes the form of a rules file—similar to developer documentation—that specifies how the model should be developed. It doesn’t guarantee anything; it merely increases the likelihood that the model will adhere to these rules.
Semantic anchors increase the likelihood of good results
Semantic anchors are terms with strong influence in the neural network, around which the output is aligned. Ingo frequently uses MECE: Mutually Exclusive, Collectively Exhaustive. The individual parts do not overlap and, together, are complete.
If the AI is to list test cases for a system, the instruction “create a list that is MECE” yields a list that is delineated by quality attributes or clear functions, with no overlap and no gaps. Such anchors shift the result from being usable in perhaps 20 percent of cases to being usable in 80 percent of cases.
Domain knowledge isn’t in the code, and AI won’t take it off your hands
The actual knowledge of why a piece of software exists cannot be derived from the code. You can abstract code into pseudocode and deduce its function from it. However, the domain knowledge and business requirements aren’t contained there.
In well-written code, comments occasionally explain why a particular solution was chosen. This overlap of abstraction levels is helpful, but an LLM doesn’t have it—not even in the generated code.
If you don’t maintain Architectural Decision Records and don’t document why you made a decision, you’ll end up guessing later. You can ask the LLM what the most likely reason was, but that’s just crystal ball gazing—only a better crystal ball.
Ingo Eichhorst
If you don’t document decisions, you’ll have forgotten them in two years. No AI can take this task off your hands.
An intermediate layer bridges the gap between engineering and management
An AI-generated report can create a common basis for discussion between developers and management. In a startup, an agent summarizes what all developers—both human and AI-assisted—did the previous day. The report is based on the Git diff and the associated tickets, condensed into a one-minute report.
The personality behind the agent is designed as a slightly overqualified software engineer, with a bit of an air about them. It is precisely this disconnect that creates an impact. The report is written in business language, and technical terms are explained. A CEO with even a basic understanding of technology can use it to understand what happened.
This middle ground forces both sides to meet each other halfway. Developers know what they’ve built and can relate the agent’s assessment to their work. Management understands the text without needing a translation. Before, there was no common language.
This approach also has a critical side. When the agent rated a decision as suboptimal and the CEO asked for clarification, the developers had to admit that the decision was indeed a bad one. Such moments are useful because we learn the most from our mistakes.
