AI-powered documentation testing uses language models as simulated user personas that don’t just rate documentation but actually work with it. A persona such as a junior developer tries to complete a task using only the documentation. If it fails, that points to concrete gaps in the document.
Key Takeaways
- AI agents with simulated personas test documentation in practice: an agent set up as a junior developer burned through 1.8 million tokens and still failed, while the senior developer agent solved the same task with 300,000.
- Persona simulations built on LLMs slide back into generic model behavior as soon as the role description isn’t strict enough, and that is what still keeps them at the prototype stage.
- An LLM’s temperature parameter controls how far the output strays from the most likely path: low for reproducible, deterministic results, high for unexpected test results.
- No LLM can reconstruct domain knowledge and architecture decisions that were never documented, because they simply aren’t in the code.
- Semantic anchors like MECE measurably raise the odds of structured, non-overlapping AI output, but they don’t guarantee it.
AI Personas as a Tool for Documentation Testing
Documentation testing with AI works like this: a language model gets a specific personality and then tries to actually carry out the tasks the documentation describes. Rather than just rating a set of instructions, the agent has to use them. Ingo Eichhorst describes one such agent as a junior developer: new on the job, not in the mood to read everything, skimming the docs and trying things out.
The testing idea behind it is simple. If someone with little background knowledge wants to get the software running quickly, the documentation has to make that possible. If the simulated junior developer can’t manage it, the problem is in the instructions, not with the user.
Ingo built the first version of this approach two and a half years ago with a payment provider from Saudi Arabia and Dubai. Back then, language models simulated people to test how well they could find the documentation and whether they could implement against it. That idea grew into a prototype that plugs the same logic into an existing pipeline.
How the Persona Agent Fits into the Pipeline
Persona testing is a backward-looking process: something already exists, and the AI assesses how good it is. Ingo distinguishes two ways AI helps with code and documentation. The forward path generates something. The backward path reviews what’s already there.
Many teams already run CI/CD pipelines for their documentation. The docs pass through quality gates, get tested and have to meet certain conditions. The persona agent is one more building block in exactly that chain. At the end, it produces a kind of feedback form showing where the documentation needs work before it’s accepted.
While the review is already AI-supported, writing the documentation stays largely manual in this approach. The prototype still has to mature before it’s ready for production.
Personas Find Gaps That Real Test Groups Miss
Different personas find different weaknesses because they approach the task differently. Besides the junior developer, there’s a senior developer. Ingo even simulated his own mother as a persona.
The appeal is being able to represent groups that are missing from normal tests. A usability test at a university mostly gets students as participants. They know what AI is and where to click to get a result. Results like that are skewed. Simulated personas can also cover outliers and unfamiliar ways of thinking.
That matters especially when you’re deciding whether to take a product to another country. What mindsets prevail there, and how is the culture different? You can simulate these scenarios in advance instead of discovering them after launch.
Simulation has a technical catch. The model easily falls back on its training data. At some point, Ingo’s simulated mother could write PHP after all, because the model slipped back into its default mode. A persona therefore has to be defined very strictly and precisely.
Why a Simulated Junior Developer Failed Because of a URL
The junior developer failed over a detail a human would hardly have noticed. The task: use the documentation of the Ionos Model Hub to find the cheapest AI model, ask it for the answer to a question from The Hitchhiker’s Guide to the Galaxy, and check whether “Adams” appears in the reply. A pass meant the documentation was good enough.
The junior agent opened a page and got a 404 error. URLs at OpenAI follow a particular scheme, and the Ionos URL is built differently. Clearer instructions were missing at exactly that point. The agent kept trying, kept getting it wrong, and burned through a huge amount of resources along the way.
The token count makes the difference visible:
| Persona | Token Consumption | Behavior |
|---|---|---|
| Senior developer | about 300,000 tokens | implemented cleanly, done |
| Junior developer | about 1.8 million tokens | trial and error, skipped the docs |
Failures like this are revealing because modern models can operate a browser. In the background, a process runs in which the model talks to itself. You can analyze that reasoning to see where a line of thought takes a completely wrong turn.
One agent once complained that it couldn’t read the JavaScript, because it had downloaded the wrong page, one that only matters to machines. A human would never have had that problem. But as soon as an AI spins up a VM instead of a person opening the page, too much JavaScript in the documentation suddenly becomes a real problem.
No 100 Percent Guarantee When Generating Documentation
If you use language models to generate documentation, you have to let go of the idea of guaranteed output. A calculator always returns 2 for 1 plus 1. Otherwise it’s broken. An LLM behaves differently. With GPT-3.5, the chance that 1 plus 1 comes out as 2 is about 55.7 percent.
That variation is intentional. The temperature parameter controls how far a model strays from the most likely token. Set high, the model sounds more human and tends to produce unexpected output, which suits a quirky tester. Set low, it sounds more robotic and returns the same result for the same prompt every time.
Even at low temperature, a small change in the input can lead to a completely different result. That’s why the forward path needs boundaries from the outside. In coding, that’s a rules file, similar to developer documentation, that describes how development should be done. It guarantees nothing. It only makes it more likely that the model sticks to those rules.
Semantic Anchors Raise the Odds of Good Results
Semantic anchors are terms with a strong pull in the neural network, and the output aligns around them. Ingo often uses MECE: Mutually Exclusive, Collectively Exhaustive. The individual parts don’t overlap, and together they’re complete.
If the AI is supposed to list test cases for a system, the instruction “make a list that is MECE” produces a list split by quality attributes or clear functions, with no overlaps and no gaps. Anchors like that shift the result from usable in maybe 20 percent of cases to usable in 80 percent.
Domain Knowledge Isn’t in the Code, and AI Won’t Do That Work for You
The real knowledge of why a piece of software exists can’t be recovered from the code. You can abstract code into pseudocode and work out what it does. But the domain knowledge and the business requirements aren’t in there.
Good code sometimes has comments explaining why something was solved a certain way. That overlap of abstraction levels helps, but an LLM doesn’t have it, not even in the code it generates.
“If you don’t keep Architecture Decision Records and don’t write down why you made a decision, you’ll be guessing later. You can ask the LLM what the most likely reason was. But that’s reading a crystal ball, just a better crystal ball.”
(Ingo Eichhorst)
If you don’t document decisions, you’ll have lost them two years from now. No AI will take that job off your hands.
A Middle Layer That Translates Between Engineering and Management
An AI-generated report can give developers and management a shared basis for discussion. At one startup, an agent summarizes what all developers, human and AI-assisted, did the day before. It works from the Git diff and the related tickets and condenses them into a one-minute report.
The agent’s personality is set up as a slightly overqualified software engineer with his nose a little in the air. That very contrast is what makes it work. The report is written in business language, and technical terms are explained. Even a CEO who isn’t very technical can follow what happened.
The middle layer pushes both sides to meet halfway. Developers know what they built and can relate the agent’s assessment to their own work. Management understands the text without a translation. Before, there was no shared language.
There’s a critical side to it as well. When the agent rated a decision as suboptimal and the CEO asked about it, the developers had to admit the decision really had been a bad one. Moments like that are useful, because mistakes are where we learn the most.
Frequently Asked Questions
How can you determine whether technical documentation is truly sufficient for beginners?
A language model is assigned a specific role, such as that of a junior developer, and then attempts to complete a specific task based solely on the documentation. It’s not the text that’s evaluated, but the result: If the persona fails, the gap lies in the instructions, not with the user. The junior developer deliberately skims the text and experiments, just as they would in a real-world work environment.
At what point in the development process does it make sense to include automated documentation testing?
As an additional component in an existing CI/CD pipeline for documentation, where the documentation goes through quality gates anyway. At the end, the agent provides a kind of feedback form listing the areas that need to be revised before approval. This is the backward approach: something already exists, and the AI evaluates how good it is.
What do simulated personas offer that traditional usability tests don’t cover?
They represent groups that are missing from real test groups. A usability test at a university primarily involves students: they know what AI is and where to click. Such results are skewed. Simulated personas also account for outliers and unfamiliar ways of thinking, such as when a product is to be adapted for another country with a different culture.
How costly is incomplete documentation for an AI agent?
The difference is immediately apparent in token consumption. In the test described, the senior developer agent completed the task cleanly using around 300,000 tokens. The junior agent needed around 1.8 million tokens for the same task and still failed because, after encountering a 404 error, it kept trying incorrect URL variations over and over again.
Why do AI agents struggle with documentation that humans can handle with ease?
Because they access it differently. One agent once downloaded the machine-readable version of a page and complained that it couldn’t read the JavaScript. A human would never have had this problem. But as soon as a VM opens the page instead of a human, too much JavaScript in the documentation becomes a real obstacle.
Why doesn’t a language model always provide the same answer to the same question?
Because the output is probability-based: With GPT-3.5, there’s about a 55.7 percent probability that 1 plus 1 equals 2. The temperature parameter controls how much the model deviates from the most probable token. Set low, it acts more robotic and predictable; set high, it acts more human and unpredictable, which is perfect for a crazy tester.
How do you get structured results from a language model without gaps or duplicates?
Through semantic anchors, that is, terms with strong influence in the neural network. A commonly used anchor is MECE: Mutually Exclusive, Collectively Exhaustive. The instruction to create a MECE list leads to distinctions based on quality attributes or clear functions. Such anchors increase the rate of useful results from about 20 to about 80 percent of cases, but they do not guarantee them.
Can existing code be used to retroactively reconstruct why a piece of software was built the way it was?
No. Code can be abstracted into pseudocode, and its function can be derived from that, but domain knowledge and business requirements aren’t captured there. Without Architectural Decision Records, all that remains later is speculation. Asking an LLM for the most likely reason is like reading a crystal ball, only a better one. If you don’t document your decisions, you’ll have lost them in two years.
