What an amazing time we are living in. I think it’s super exciting and I’m thrilled about the possibilities that are opening up for us, especially in software development. I’ve just seen Gartner’s hype cycle, where GenAI is just at its peak - i.e. before it falls into the valley of tears. But what comes next will be exciting. Where will AI be used in a sustainable and productive way - apart from SEO optimization and text generators?
We would do well not to fall into blind actionism - but on the other hand not to bury our heads in the sand when it comes to AI. It’s a balancing act! I have also noticed that my personal opinion on AI is constantly changing. Nevertheless - or perhaps because of this - I would like to share my current state of ignorance here 😉
Software testing with AI
…is a big topic right now, as it is in all industries. There is of course a lot of potential here: test case creation, test data generation, execution, error analysis - a playing field for AI tools. It feels like every tool now has the addition AI in its name - even if not much has changed. So take a close look at what’s really behind it.
And we should always ask ourselves: what problem do we actually want to solve with AI? It’s tempting to be dazzled by the latest technology, but the key to success lies in using it in a targeted way to overcome specific challenges in our projects. And when I look at my customers, it’s less about even more efficient test execution and more about things like test data management and the classic: how do I overcome the media discontinuity between the requirements of the requester and the technology?
But how do you get started with AI in testing? Quite simply: start. Experiment. Try it out. This is pioneering work, much is not yet finished, not yet developed and not yet conceived. The first websites were not created with a CMS - but with notepad.exe.
Testing AI
When we talk about the testing of AI itself, we are thrown back to the primal question that we tend to avoid in software development: What does quality actually mean to us?
When I come into new projects and companies, this is always my first question: What does quality mean to you? Oh. Ambiguity. Um. Reference to ISO 25010, specifically? Silence.
Traditional quality criteria such as functionality must be rethought for AI systems. Other criteria such as accuracy, learning ability, adaptability, data quality and statistical criteria are becoming much more important here.
Rethinking necessary
As testers, test managers and quality engineers, we need to rethink both fields. Both in the use of AI and in the testing of AI. We have to let go of what we love - and learn new things. This is not always comfortable, but it is necessary.
And there is help available: The popular protagonists are a good place to start using AI: ChatGPT, Midjourney, Gemini, Copilot, etc. Just give them a try 🙂 Or you can use AI as part of a training course, for example the A4Q Practical Tester And for testers? The ISTQB Certified Tester AI Testing training and certification is a good option here. Incidentally, this certification has been around since 2021 and its predecessors even longer. After all, AI hasn’t just been around since ChatGPT. And somehow it’s just software 😉
Frequently Asked Questions
Should you wait it out or jump right in amid the AI hype?
Neither one nor the other: Blindly jumping on the bandwagon and burying your head in the sand are both bad answers. GenAI was at the peak of the Gartner Hype Cycle—meaning it was just before expectations began to decline. What’s interesting is what remains afterward—namely, the question of where AI is being used productively and sustainably, beyond SEO copy and text generators.
Does the “AI” in a testing tool’s name always mean real AI?
No. It feels like every tool has gotten the “AI” label in its name by now, even if the product itself hasn’t changed much. That’s why it’s worth taking a closer look at what’s actually behind it. The more useful starting question is, in any case: What problem in your own project is the AI supposed to solve?
In practice, where are the biggest pain points where AI could help with testing?
Not in test execution. Test case creation, test data generation, execution, and defect analysis all have potential. In client projects, however, it becomes clear that the focus is less on even more efficient execution and more on test data management and the classic issue: the disconnect between the client’s requirements and the technical implementation.
Do you need a fully developed tool strategy before using AI in testing?
No. This is about pioneering work: Much of it isn’t finished yet, hasn’t been developed yet, and hasn’t even been conceived yet. So getting started, experimenting, and trying things out is the more realistic approach. To draw a comparison: The first websites weren’t created with a content management system either, but with notepad.exe.
Why is it so difficult to define quality in AI systems?
Because the question of quality often remains unanswered even with traditional software. In new projects and companies, the question of what quality means to the team is usually met with confusion and a reference to ISO 25010. When asked for specifics, there is silence. When testing AI, we come back to precisely this fundamental question.
Which quality criteria become more important than traditional ones when testing AI?
Accuracy, learning ability, adaptability, data quality, and statistical criteria are coming to the forefront. Traditional criteria such as functionality aren’t disappearing, but they need to be rethought when it comes to AI systems. For testers, test managers, and quality engineers, this shift in thinking applies in two areas: the use of AI and the testing of AI.
Are there training and continuing education programs for AI testing?
Yes. For testers, the ISTQB Certified Tester AI Testing training and certification program—which has been available since 2021, with even earlier precursors—is a good option. AI is therefore not a topic that began with ChatGPT. To get started with AI on its own, the popular tools—tried out in your own daily work—are sufficient for now.
