Automating LLM testing means making non-deterministic AI systems systematically testable. Because language models do not produce consistent outputs, acceptance ranges and evaluation rubrics replace traditional yes/no assertions. In the “judge” approach, a second LLM evaluates the outputs of the system under test, calibrated against human expert evaluations.
Key Takeaways
- Non-determinism is not a bug in LLMs, but a design feature: Anyone who treats it like flakiness and excludes it is testing the system in the wrong way.
- In RAG systems, most errors occur in retrieval, not in generation: A properly fine-tuned LLM reliably delivers useful answers.
- According to studies, the “LLM-as-Judge” approach achieves 80 to 85 percent agreement with human judgments, which corresponds to the level of agreement between two people.
- Quality gates for LLM testing require acceptance thresholds rather than binary judgments: A boundary value of 80 to 90 percent replaces the traditional red-green principle.
Testing AI systems means accepting non-determinism as a feature
Anyone who has learned traditional test automation relies on determinism and consistency. A test runs, the result is reproducible, and “green” stays green. With language models, this foundation falls away. An LLM does not necessarily answer the same question the same way twice.
The usual instinct is to rule out such behavior: as a race condition, as flakiness, or as a bug in the test setup. With generative systems, that’s wrong. Non-determinism is part of the system’s design, not one of its flaws.
Anupam Krishnamurthy describes this necessary shift as follows: The expectations for deterministic, conventional software are completely different from those for a chatbot that feels as though it has a life of its own. It is precisely this change in perspective that forms the starting point for any meaningful test strategy for AI components.
Why AI Is Increasingly Becoming the Test Object, Not Just the Testing Tool
There are many articles, opinions, tools, and solutions regarding AI in testing. Far less often does anyone address the reverse perspective: AI as the test object.
Applications are incorporating more and more AI components. Jira and other widely used consumer tools now include AI components. This shifts the question from “How do I use AI for testing?” to “How do I test software that contains AI itself?”
Manual verification by a domain expert isn’t enough for this. You can’t rely indefinitely on a human to evaluate every response individually. As soon as AI components become standard in products, automated processes will be needed to test them.
Trust Is Built Through Guardrails, Not Through Complete Understanding
Traditionally, you build trust in a system through understanding: you know what the code does. With neural networks, this understanding isn’t complete, and it can’t be tested away.
The solution lies in guardrails, tests, and evaluations. Even if you don’t understand your system 100 percent, a defined framework gives you certainty: as long as the behavior stays within established limits, you can use it.
This way of thinking is new only to software developers and testers. Machine learning experts have been working this way for a long time. The task for the testing discipline is to develop its own methodologies for this approach.
How to Break Down an AI System Based on the Complexity of the Evaluation
The first concrete step is to break the system down into test tasks of varying complexity. At its core, this is the same approach as in traditional testing with different levels of abstraction, just with different goals.
The appropriate test approach depends heavily on the use case. Image generation requires different tests than text generation. AI is a tool, and your approach must be tailored to your specific solution.
Using text-based conversational models as an example, the evaluation can be organized into levels:
| Level | What is tested | How automatable |
|---|---|---|
| Deterministic | Fact-based response in a fixed format, e.g., JSON, sorted alphabetically | Classically with pytest or a familiar framework, via assertions |
| Fact-based, free text | Correctness of free-text responses with an objective core | More labor-intensive, but verifiable |
| Subjective | Tone, helpfulness, precision of the response | Most difficult, requires calibrated evaluation |
| Adversarial | Security against prompt manipulation | Dedicated test class, not optional |
Deterministic tests are quick to create and deliver immediate value. If your chatbot is supposed to always return facts in the same JSON format with a fixed order, you can verify this with a standard automated test. You should have plenty of these types of tests.
Why Adversarial Testing Is Becoming Mandatory for Language Models
With generative systems, there is no clear boundary between a natural-language instruction and the system’s behavior. A user can trigger behavior via free text that was not actually intended.
This creates security risks that didn’t exist before. Adversarial testing specifically checks whether the model can be pushed outside its intended scope by manipulative inputs.
You must not overlook this test if you’re taking a systematic approach. It belongs alongside the fact-based and subjective tests, not after them.
How One LLM Evaluates Another: The LLM-as-Judge Approach
For free-text and subjective criteria, an approach is gaining traction that uses another LLM as an evaluator. A second model determines whether the response from the system under test falls within an acceptable range.
That sounds easier than it is. The valid objection: Don’t two language models share the same blind spots? How do you ensure that the LLM judge evaluates what a human would also evaluate?
In practice, you can reduce the risk by using different models for the judge and the system under test, since they’ll then share fewer blind spots. You can also choose a more powerful model for the judge than for the system under test. This scales better and results in lower operating costs for thousands of users compared to manual evaluation.
More important than the choice of model is alignment with domain expertise. You need structured rubrics: clearly defined criteria with precise conditions and examples. Examples noticeably improve an LLM’s judgment quality—a pattern we’re familiar with from the everyday use of chatbots.
RAG as a Test Object: Why the Responsibility Remains with You
If you use a language model directly from a provider, you can outsource part of the quality assurance to that provider. With a RAG system, that’s no longer possible.
Retrieval Augmented Generation uses the language model only as a component. You build everything else yourself, which means the responsibility for quality remains with you. This automatically increases the need to perform thorough testing.
RAG is also widely used because it’s a low-hanging fruit: you expand an LLM’s dataset without having to retrain it. That’s exactly why it serves as a good example of a realistic test object.
How to Isolate the Source of an Error Between Retrieval and Generation
RAG can be clearly divided into two parts: retrieval and generation. This separation is the key to figuring out where an error lies.
Retrieval fetches relevant data from a vector database. This part typically doesn’t involve an LLM but rather classic machine learning methods such as cosine similarity, similarity search, and semantic similarity. Generation takes this data and uses it to generate the response.
In my experience, most problems lie in retrieval. If you feed an LLM the right data, it will produce a reasonable result. Anupam Krishnamurthy
That’s why it’s worth building targeted tests for retrieval and, separately, for generation. This allows you to isolate the cause rather than evaluating the entire system as a black box. The principle is identical to traditional testing: test in small increments, separating components and their interactions.
From a Green Pipeline to an Acceptable Framework
Deterministic testing knows only red and green: a “no-go” or a “go.” With AI systems, you need a more flexible basis.
Instead of passing or failing binary tests, criteria fall within a range of values—for example, between 50 and 100 percent acceptance. As a quality gate, you set a threshold: if the rating is above 80 or 85 percent, the result is considered acceptable. With subjective criteria, there is no strictly correct or incorrect answer anyway.
You calibrate these thresholds with human experts. Studies on the agreement between humans and LLM judges show agreement rates of around 80 to 85 percent. Agreement between two humans is often no higher than that. An LLM judge, therefore, does not need to be more perfect than a second human opinion; it simply needs to meet the human standard.
Model benchmarks give you a false sense of security
Public model benchmarks convey an artificial sense of security. They claim that a model is as good as a PhD, thereby providing a benchmark that says little about your specific context.
The problem lies in the method. Models are trained on standard data sets, much like a student who is given the exam questions in advance and memorizes them. The resulting test then measures memorization, not suitability for your specific use case.
In production, this will come back to haunt you if your specific context has different requirements than the benchmark. One thing remains true: No vendor can anticipate your context because it’s too specific to your use case. That’s exactly why you need to set your own benchmarks.
Your Existing Skills Aren’t Wasted
The shift to AI testing doesn’t devalue testers’ experience. The analytical approaches—breaking down the system, isolating components, and testing in a context-specific manner—still apply.
The difference lies in dealing with non-determinism. This requires creative and critical thinking, but much of what you’ve learned from years of experience as a quality professional has high reusability.
The real appeal lies elsewhere. AI testing forces you to question what you take for granted: How do we think, how do we arrive at answers, and what does “correctness” even mean? You learn these fundamentals intuitively and normally never question them. When testing generative systems, you have to do exactly that.
