AI in Testing 2026: Tool, Colleague, Test Object
First things first: I see GenAI as a tool. And for tools, one rule applies: “A fool with a tool is still a fool.” I like using AI as a sparring partner, getting into a dialogue, working my way through the results. But tool or sparring partner, it’s just like real life: if I don’t know the subject, the other side can tell me anything. I have no way to judge it. That hasn’t changed. What has changed is how good the tools have become, and how many people are using them. EuroSTAR 2026 received around 550 submissions, its program chair recently reported, and about 300 of them were about AI in testing. The share with real hands-on experience: much smaller. Big on opinion, short on practice.
In my GenAI testing workshops I take the opposite approach: we don’t ask what’s theoretically possible, we look at what already works today. And quite a bit has shifted there. Generating test cases from requirements? Everyone does that now. Copilot for scripting? Standard equipment, not worth mentioning anymore. The question has flipped: from “What can AI do for you?” to “What do you need to check when AI is part of the team?” Five fields I’m currently seeing in projects and trainings.
Specification instead of test-case prompts
The reflex is tempting: dump the code into the prompt, “write me some tests”, done. Research now has a name for what happens next: the misguidance effect. If there’s a bug in the code, the model dutifully writes tests that lock in the faulty behavior as the expected one. The bug gets a green checkmark. A related study with eight models and over 22,000 program variants fits the picture: as soon as the code changed semantically, the pass rate of the generated tests dropped to 66 percent, and over 99 percent of the failing tests had still passed on the old version. The tests had memorized the old behavior, not understood the requirement.
The reverse works: specification-based prompting. Give the model the requirement, the expected behavior, the constraints, and let it derive the tests. No 80-page requirements document needed. A decent user story with acceptance criteria will often do, an API contract, a state table. Tooling is moving fast here, GitHub released version 1.0 of its Spec Kit in August. But honestly: this is simply a proper test basis. We’ve been preaching that in testing for decades.
What to watch out for: Prompt from the specification, not from the code. And if there is no specification, that’s your finding right there.
Exploratory testing: the agent does its own clicking
In 2025, exploratory testing with AI meant getting test ideas from a chat. That still works, but it’s no longer the interesting part. Playwright now ships three agents out of the box: the planner explores the application on its own and writes a test plan in Markdown, the generator builds tests from it, the healer repairs broken ones. Via MCP, the agent actually clicks through the application instead of fantasizing about it.
It’s impressive the first time you watch it run. And still: the agent finds what is reachable. Not what is risky. It doesn’t know your history, the trouble with that one interface, the module that breaks with every release. Exploratory testing lives on hypotheses, experience, curiosity. The loop delivers breadth. You deliver the questions.
What to watch out for: Treat the agent’s results as raw material, not as a test concept. Prioritizing by risk stays with you.
Test data: the domain logic is solved, provisioning is not
For me, this is the strongest tension in the whole topic. The model is excellent at domain variants: edge cases, broken address formats, exotic contract scenarios, all in seconds. But the prettiest generated table is useless if it never reaches the test system. A survey by tool vendor Perforce among 518 enterprise leaders shows the scale: 99 percent wait longer than one business day for a fresh production copy, 42 percent wait weeks or months. And that’s no outlier, the World Quality Report also counts secure, scalable test data among the most common stumbling blocks. Masking, referential integrity, provisioning: as much of a slog as ever. The LLM solves the domain logic, not the logistics.
What to watch out for: The bottleneck sits in the pipeline, not in the prompt. Before you generate data, figure out how it gets into the system.
Checking what the AI builds
Code generation has become everyday business. The syntax is almost always right. The security is not: Veracode tests over 150 models with a consistent methodology, and around 45 percent of generations contain a known vulnerability. The rate has been hovering between 45 and 55 percent for two years, and model size changes almost nothing about it. The models are getting more eloquent, not more secure.
And the obvious idea of letting AI check its own work? A study at MSR ‘26 examined 3,109 pull requests: when they were reviewed only by code review agents, the merge rate was 45 percent. With human-only review, 68. AI checking AI is apparently not enough. This is where genuinely new testing work is emerging in 2026: review gates for generated code, security checks, and the uncomfortable question of who still understands what’s being merged.
What to watch out for: Treat generated code like code from a new colleague. Fast, diligent, and with no feel for the baggage in your codebase. The review isn’t a formality, it is the test.
Testing AI features: evals instead of assertions
The fifth field turns the tables: no longer AI in testing, but testing of AI. If your product gets a chatbot, an assistant or a RAG search, you’re testing a system that answers the same input differently. assertEquals falls flat. Instead: evals, golden datasets, LLM-as-judge, guardrails. Different tools, different thinking, but at the core the old question: What is good enough, and how do I recognize it?
An Applause survey of almost 1,100 respondents shows how real this is: 44 percent have switched off live AI features again because the operating costs outweighed the value. More than four in ten. For me, this is currently the most common new testing task out there, and few teams are prepared for it.
What to watch out for: Define what “good enough” means before release, and keep measuring it in production. An AI feature without an eval suite is an accident waiting to happen.
The work doesn’t disappear, it moves
One pattern runs through all five fields: the work doesn’t disappear. It moves. From writing to checking, from executing to judging. Elmar Jürgens put it nicely at the Software Quality Days: “Don’t stop doing as much quality assurance as you do now because it feels like we’re just moving faster …”
And checking takes judgment. Judgment doesn’t come from watching, it comes from doing: from debugging, from building test data, from arguing about requirements. If you want a more systematic route: even the ISTQB now has its own GenAI syllabus. When I lack the knowledge, things get dangerous. That has always been true. But in 2026 it’s truer than ever, because by now all the results look right. Whether they actually are is something only people with judgment will catch.
