Trusting AI agents in software systems means ensuring that automated processes produce correct, intended outputs through checks, guardrails, and validation layers built around the AI model itself. This requires solving two core problems: communicating requirements clearly enough for the system to act on them, and verifying that the output actually matches what was wanted, which testers call the Oracle problem.
Key Takeaways
- AI systems amplify both mistakes and successes at scale, so the checks, guardrails, and validation processes built around the model matter more than the model itself.
- Testing AI requires a shift from deterministic pass/fail checks to monitoring trends and mean time between failures, because non-deterministic outputs cannot be verified with a single green test.
- The communication problem with AI agents is structurally identical to the bug-report problem with humans: vague input produces generic, context-free output that misses the actual need.
- As AI-generated code becomes a black box, test specifications and acceptance criteria become the primary source of truth, making the tester’s skill set central rather than peripheral.
- AI democratizes software creation by removing the need for programming knowledge, which surfaces long-ignored organizational problems such as document version control and missing single sources of truth.
Trust does not live in the model, it lives in the system around it
The unit of trust in AI is not the model, it is the system you build around it. A language model predicts, it does not analyze and it does not think. Trust has to come from the process, the checks, and the guardrails wrapped around that prediction.
This is the same standard you already apply to people. You trust a colleague because of a track record, shared understanding, and the ability to verify what they hand back. With an AI agent, the same scaffolding has to exist: a process, balances, and validation steps that catch the system when it drifts.
When that scaffolding works, the same logic applies as with software you subcontract. You write the spec, you accept the delivery, and you check whether you got what you asked for. Henri Terho frames the whole question of AI trust around two old problems that testers already know well.
The two problems that decide whether you can trust an AI agent
Two classic testing problems return with AI, and both have to be solved before trust is possible: the communication problem and the Oracle problem.
The communication problem is about conveying what you actually want. When someone tells an AI “fix my code now”, the result is poor for the same reason a bug report saying “production is broken” is useless. There are no logs, no context, no specifics. People get angry at the output without noticing they gave the system nothing to work with.
Humans paper over these gaps with experience. A colleague fills in the missing context from years of shared work. A model has no such experience, so the prompt, the requirements, and everything you feed it have to be specific enough to stand on their own.
The Oracle problem is about knowing whether the output is right. Even when the AI produces something, you still have to decide whether it is correct or merely plausible. That question gets philosophical fast inside a company. “Right” might mean the boss asked for it, or that the feature makes money, or that it matches a real specification. Without a clear answer, you cannot judge the output.
Answer both problems and the trust you are looking for follows. Skip them and you are guessing.
Why generic AI output fails you
A generic prompt produces a generic answer, and that is the failure mode most people walk into. Ask a general-purpose chatbot for a business strategy as a small software business owner, and you get a clean, well-structured strategy that fits no one in particular.
The output looks complete. It has all the expected points and reads well. But it carries none of your context. It does not know you sell from Finland into another country, or that German rules apply, unless you said so.
The fix is unglamorous: more context, more specifics, better requirements. The reason this annoys people is simple. Writing a precise prompt is more work than typing “fix this”, and most people would rather have the fast answer than the correct one.
Testing AI means watching trends, not a single green check
AI testing shifts from a single pass-fail result to statistical behavior over time. Conventional software is deterministic. You give it an input, you expect one answer, and if you do not get it, you have a bug. That makes testing straightforward.
AI models are non-deterministic, so a single green test proves little. You watch trends, mean time between failures, and behavior across many runs. This is closer to how machine shops and aviation think about reliability than to a one-point check.
Plan for rare failures. Black swan events appear in these models because you cannot predict every output. A test suite that only confirms “green once” gives you false confidence.
There is a moving-target problem on top of this. The models change underneath you when providers update them. When the components of your system keep shifting, stability becomes something you have to engineer deliberately, not something you inherit.
Code becomes a black box, so the spec becomes the truth
As AI generates more of the application, the specification and the test become the only reliable source of truth. If the software passes what you specified, it is fine, and you do not need to read the generated code.
This pushes practices like behavior-driven and test-driven development from “nice tooling” to the core of the work. The real question becomes how you write down what you want, in a form precise enough to act as acceptance criteria.
There is a trap waiting here. People will overcorrect and write enormous specs, and you end up with millions of lines of specification the way teams now have millions of lines of code. Those specs will conflict. Systems will behave strangely, and you will spend your time debugging the spec rather than the code.
The abstraction level rises sharply. Writing a full open specification for a system as large as a CRM platform is genuinely hard, because no single person can lay out everything such a system should do.
AI pressure comes from outside IT, not from within it
The push for AI arrives from the business side, not from the engineering core. People from marketing now walk up and ask whether AI can solve a concrete problem they have. That kind of inbound request rarely happened with traditional software development.
This changes who drives the work. Demand comes from people closer to where the business creates value, who want a tool that tracks clients or surfaces what is happening, rather than from someone whose identity is built on being the database expert.
AI is an approachable enabler because it does not require fluency in a programming language. You can prompt it in plain language, and that democratizes computing. The flip side is that quality becomes questionable, because almost anyone can now produce software without anyone checking whether it holds up.
AI is not gonna take your job. They’re not gonna automate you, it’s going to augment you and the way that you work. And there’s a lot of work to be done around this area. — Henri Terho
Old organizational problems resurface, and now you have to solve them
AI forces unsolved human problems back to the surface, because the system needs a single source of truth that people never bothered to define. When AI becomes the main interface to a company’s knowledge, you have to write down what the organization actually does.
Consider a document store with twenty versions of the same file. Which one is correct? Answering that needs real context: whether a version went to a client, which one, and whether it was revised afterward. The problem sounds trivial and is not.
Software decisions have always carried hidden business decisions. A developer who locked a company into one cloud platform made a far-reaching call that shaped everything afterward, often without anyone noticing at the time. AI makes these buried decisions visible and demands they be addressed.
To handle the verification side, build guardrails into the platform. One option is to have multiple AI instances examine and discuss whether a given output is actually good, so the system checks itself rather than trusting a single pass.
Why a tester’s skill set fits the AI era
The testing mindset maps directly onto what AI work demands. The job already revolves around defining criteria, writing them into specs and test cases, and verifying outputs. That baseline puts testers ahead of many other roles in IT.
What you add on top is statistical thinking. Move away from caring about one test going green and toward watching how results trend across runs. Then widen your reach beyond the tester’s box.
Testing and validation will absorb parts of neighboring roles, including pieces of DevOps and programming. Expanding into those areas raises your value, and the broader your context, the better your judgment, exactly as more context improves an AI’s output.
The dominant reaction to all this change is fear, and it is a natural response to a shift this large. The fear of being automated away is the one worth setting aside first. The work ahead augments testers rather than replacing them, and there is a great deal of it.
Frequently Asked Questions
What has to be in place before you can rely on an AI agent’s output?
Trust comes from the system built around the model, not from the model itself. A language model predicts, it does not analyze and it does not think. The scaffolding that makes it trustworthy is a process, balances, and validation steps that catch drift. It works like subcontracted software: you write the spec, accept the delivery, and check whether you got what you asked for.
How do you decide whether an AI’s answer is actually correct?
You need an answer to the Oracle problem: a definition of what “right” means before you judge the output. Inside a company that gets philosophical fast. Right might mean the boss asked for it, that the feature makes money, or that it matches a real specification. Without that decision, you cannot tell a correct result from a merely plausible one.
Why do short prompts like “fix my code” produce disappointing results?
Because the input carries no context. The failure is structurally identical to a bug report that says “production is broken”: no logs, no specifics, nothing to act on. People fill such gaps for each other from years of shared work, but a model has no shared history. A generic prompt returns a clean, well-structured answer that fits no one in particular.
Can AI systems be verified with conventional pass/fail tests?
No. Conventional software is deterministic, so one input, one expected answer, and a red test means a bug. AI models are non-deterministic, so a single green test proves little. You watch trends, mean time between failures, and behavior across many runs, closer to how machine shops and aviation think about reliability. Plan for rare, unpredictable failures.
Does anyone still need to read AI-generated code?
Not necessarily. Once AI writes most of the application, the specification and the test become the source of truth: if the software passes what you specified, it is fine. That moves behavior-driven and test-driven development into the core of the work. The trap is overcorrection, producing millions of lines of conflicting specification and debugging the spec instead of the code.
Who is actually pushing for AI inside companies?
The demand arrives from the business side rather than the engineering core. Marketing people walk up with a concrete problem and ask whether AI can solve it, a kind of inbound request that rarely happened with traditional software projects. AI is approachable because it needs no fluency in a programming language. The downside is that quality becomes questionable when almost anyone can produce software.
Why does introducing AI expose old organizational problems?
Because the system needs a single source of truth that people never bothered to define. A document store with twenty versions of the same file is the standard example: deciding which one is correct requires knowing whether a version went to a client, which one, and whether it was revised afterwards. Software decisions have always hidden business decisions, and AI makes them visible.
What should testers add to their skill set for AI work?
Statistical thinking first: stop caring whether one test goes green and start watching how results trend across runs. Then widen the scope, since testing and validation will absorb parts of neighbouring roles, including pieces of DevOps and programming. The broader your context, the better your judgment, in the same way more context improves an AI’s output.
