Testing AI models means evaluating a magic box: a system whose behavior is neither controlled nor understood nor otherwise known to be safe. No human specifies how a machine learning model is built, so simple output checks cannot judge it. Testing it is an investigation through experiencing, exploring and experimenting, with inputs, outputs, training data and guardrails kept observable.
Key Takeaways
- Machine learning models are not developed by a conscious, intentional human; Michael Bolton compares them to strains from petri dishes. Nobody knows how they work, nobody has their source code.
- Testing AI models with output checks is hopeless: that approach works for deterministic software developers understand and can instrument, and a large language model is neither.
- Nothing that matters very much should depend on magic; a bot writing a funny poem for a party carries no risk, a bot giving medical advice carries a big one.
- Anyone asking how to test LLMs without source code can still monitor inputs and outputs and know the training data, guardrails, context window and outside calls, which shrinks the magic.
- Large language models fool us like a mentalist on stage, because patterns of text suggest insight while the insight comes from the reader making sense of the output.
What Does Testing AI Models Have to Do With Magic?
Testing AI models starts with an admission: nobody fully knows what the thing in the box is doing. Michael Bolton calls such a system a magic box, and he means the term precisely. A box is magical to the degree that its behavior is neither controlled, nor understood, nor otherwise known to be safe. Some boxes are weakly magical, some strongly. A language model sits at the strong end. The question of how to test LLMs then turns into a question of how much magic you are willing to depend on, and what you do about the rest.
The term comes out of a book. Michael and his co-author have been writing Taking Testing Seriously: The Rapid Software Testing Approach for about thirty years, the last five of them in earnest. Near the end they needed a chapter on AI, and they knew it would be out of date before the ink was dry. So they wrote a chapter on magic instead. Magic does not go out of date. The technology in the box changes, the problem of depending on something you do not understand stays.
Weak magic is everywhere, and it is mostly harmless. The Python interpreter is a magic box to most of the people who use it. They know conceptually how it works, just not in detail. What makes it safe to rely on is social proof. A worldwide community uses it, a smaller group understands and controls its internals, and when the rabbit pulled out of the hat turns out to be a rat, somebody fixes it. That is what “otherwise known to be safe” means. A language model offers far less of that.
Machine Learning Models Grow in Petri Dishes
Regular software is constructed by a person who specifies what it should do. A machine learning model is not developed by a conscious, intentional human being. Michael’s picture for it is a row of petri dishes. You grow many cultures, one of them kills off the nasty bacterium, and you pull that one out because it looks like the best. Nobody designed it. It was selected.
That is how antibiotics were found before anyone could read the genes behind them. Today proteomics and genomics explain the mechanism, and CRISPR can edit it. For language models that stage has not arrived. Nobody has the source code, and even with the source code a good part of the behavior would remain stochastic. The consequence for testing is plain. You cannot instrument the model the way you instrument your own code, so the magic has to be reduced from the outside.
Should You Rely on Magic When Something Matters?
Michael’s answer is a short set of principles, and the first one is the sharpest. If something matters very much, you probably should not depend on magic to get it. If it matters very much and there is no other way, you may try magic, but then it is your job to minimize how much you depend on it.
The third principle is where testers come in. If something matters and you are using magic to get it, responsibility demands that you test it. People in the software business have a duty of care to their clients and to their clients’ customers, and a magic box does not suspend that duty. The fourth principle follows from the third. If you are going to test it, maximize testability. The fifth covers the rest. If you cannot test something, or cannot test it thoroughly enough, you have to disclaim it. Even then, if you put it in front of other people, the responsibility stays with you.
Risk decides how hard these principles bite. A model that writes a funny poem about a colleague for the Christmas party carries no risk; if the poem is bad, you throw it away. A bot that gives medical advice carries a lot. In that case it may be wise to be careful about what the bot is allowed to say, or to keep magical technology out of that place entirely.
How Can You Maximize the Testability of an LLM System?
Much of what testers call testability is unavailable here, because the model itself cannot be instrumented. What remains is the box around the model. Michael’s list of things a team can observe and control:
- inputs and outputs, monitored closely
- the training data that went into the model
- the guardrails on both sides, before the prompt and after the response
- the context window, its size, what is in it and what was retrieved into it
- the calls the system makes to databases and outside services
Every item on that list that a team is aware of, can control and can observe takes some magic out of the box. There is a GPT in the middle, and you cannot open it. But you can put small windows into the bigger box, or probes, or little cameras. Testability of an AI-based system is mostly built at that level.
Something else follows from the list. Everything in an AI-based system that is good old-fashioned software can be tested the way software has always been tested. The deterministic parts, the retrieval, the plumbing, all of it is checkable. The model in the middle is the part that needs a different approach.
Why Simple Output Checks Fail Against Language Models
Reducing testing to checks of output from functions has always been a narrow idea of the job. Michael’s verdict is that it is useful for developers working with deterministic software they understand and can instrument, and hopeless against an LLM. A check assumes that the same input produces the same output and that one expected value settles the matter. A language model breaks both assumptions.
What works is the definition of testing that Michael and his colleagues have refined for years: evaluating a product by learning about it through experiencing, exploring and experimenting. That definition makes anything testable, including a magic box. You can always gain experience with something. You can always evaluate it by learning about it. You can always run experiments on it. The process is exploratory by nature, because there is no predetermined way of learning about something that nobody understands yet.
Teams feel this shift in practice. Those who have been moving away from scripted test cases now face the model and ask the questions that were always the right ones. What are the risks? What quality do we want? What threatens the value of the product for the people who matter to us? Testers have been asking those questions for decades. AI makes more people listen.
Testing an LLM Is Like Peeling an Onion
Michael’s own experiments with language models are iterative and spiral inward. He compares the work to peeling an onion. The more you peel, the more there seems to be, another layer and another one below it. It is the scene from Peer Gynt where the hero peels an onion, finds layer after layer and concludes that nature is witty.
What comes out of the layers are patterns of inconsistency and unreliability at the lowest level, when the model is queried directly. In Michael’s experiments Haiku 4.5 goes off the rails more quickly and more bizarrely than Haiku 3.5 did. The more interesting finding is that all of the models do it. He suspects this is what sits underneath the complaint that Claude Code used to be good and lately seems to have gotten worse. The behavior at the bottom does not explain why that happens, but you can watch it play out.
Two things make this work practical. First, an experimental framework that runs the same experiments over and over, so patterns and weird outliers become visible and software helps pick them up. Second, tools for volume. Testing a language model needs so much statistical analysis that anything beyond a first hurdle test requires generated requests and responses in large numbers. One good use of an LLM, ironically, is to have it generate thousands of variations on the same question and then watch what changes in the answers. A single added or changed word can dramatically affect the output. The tester wants to know how dramatically, how often, and to what degree the meaning gets inverted.
The least little thing can cause no change at all or a completely bizarre new behavior. That unpredictability is why the work stays exploratory. An observation prompts a hypothesis, the hypothesis a new procedure, the procedure a new discovery.
How Do Language Models Fool Us Like a Mentalist?
Mechanically, a language model strings one token after another. Stephen Wolfram’s essay “What Is ChatGPT Doing and Why Does It Work?” covers that half of the question well. The psychological half is in an essay by Baldur Bjarnason called “The LLMentalist Effect”, which describes how language models replicate the mechanisms of a psychic’s con. Michael recommends both to testers who work with these systems.
The con works like this. A cold reader says something quickly and looks for a hit. Last week was a little busy for you. There was a technical complication, Wednesday or Thursday, some piece of software that cost you half an hour. Whether the reader is right does not matter. What matters is what happens in your head, because you immediately start trying to attach the statement to an event. Patterns of text fool us into believing there is insight where there is none. Whatever insight appears comes from the reader’s own effort to interpret the output and make sense of it.
Michael’s favorite magicians are Penn and Teller, because they often show exactly how a trick is done. In their cups and balls routine they use clear plastic cups, so the audience sees every move. You see how clever magicians are, and you see how easy you are to fool. Knowing the mechanism defends you the next time someone performs a different trick. The same holds for technology. Understand how a language model works mechanically, understand what it does to the human mind, and you are harder to fool. Then the credit goes where it belongs, to the humans doing the thinking, and the conversation with the client can turn to the practical question. Is this thing giving us consistent, reliable results, and does reliability even matter in this case?
Testers Are Paid to Learn on Behalf of Others
The testing that treats a product as something to investigate, with a special focus on looking for problems, was ready for AI before AI arrived. In Michael’s view this is testing’s moment. The narrow version of the job, checking outputs against expected values, has no chance against a magic box. The broad version has everything it needs, plus tools to scale it.
“We’re being paid to learn stuff on behalf of other people. What’s a better job than that?”
(Michael Bolton)
Testing has always been fun when it is done right. Michael’s advice to anyone whose team complains that AI is not testable: understand the technology, engage with it, use tools, and keep the risk question in front of everything else. Where the stakes are high, prefer less magic. Where magic is unavoidable, test it, disclaim what you cannot test, and accept the responsibility that remains.
