What a crazy time we’re living in. I find it incredibly exciting, and I’m thrilled by the possibilities opening up for us, especially in software development, and that includes AI in software testing. I just looked at Gartner’s hype cycle (as of spring 2024): GenAI is sitting right at the peak, just before the plunge into the trough of disillusionment. But what comes after that is where it gets interesting. Where will AI find lasting, productive use, beyond SEO tweaks and text generators?
AI is changing two things in software testing right now. Testing with AI gives test case design, test data and defect analysis new tools. And anyone testing AI systems has to work out all over again what quality actually means.
We’d do well not to fall into knee-jerk action, but we also shouldn’t bury our heads in the sand when it comes to AI. It’s a balancing act! I also notice that my own opinion on AI keeps shifting. Still (or maybe that’s exactly why), I’d like to share my current state of ignorance here 😉
AI in Software Testing: Testing with AI
Like in every other industry, testing with AI is a big topic right now. And there’s a lot of potential: test case design, test data generation, test execution, defect analysis. A playground for AI tools. It feels like every tool has added “AI” to its name by now, even when not much has actually changed. So take a close look at what’s really behind it.
And we should keep asking ourselves: what problem do we actually want to solve with AI? It’s tempting to be dazzled by the latest technology, but what pays off is using it in a targeted way on the specific problems in our projects. When I look at my clients, it’s less about even more efficient test execution and more about things like test data management and the classic: how do I bridge the gap between what the requester has in mind and the technology?
So how do you get started with AI in testing? Simple: start. Experiment. Try things out. This is pioneering work. A lot of it isn’t finished yet, hasn’t been built yet or hasn’t even been thought of. The first websites weren’t built with a CMS either. They were built with notepad.exe.
Testing AI
When we talk about testing AI itself, we get thrown back to the fundamental question we like to dodge in software development: what does quality actually mean to us?
Whenever I come into a new project or company, that’s always my first question: what does quality mean to you? Oh. Confusion. Um. A reference to ISO 25010. And specifically? Silence.
Traditional quality criteria such as functionality need to be rethought for AI systems. Other criteria become much more important here: accuracy, the ability to learn, adaptability, data quality and statistical criteria.
Why Testers Need to Rethink
As testers, test managers and quality engineers, we need to rethink in both areas: in using AI and in testing AI. We’ll have to let go of some things we’ve grown fond of, and we get to learn new ones. That isn’t always comfortable, but it’s necessary.
And there’s help. To start using AI, the popular big names are an obvious choice: ChatGPT, Midjourney, Gemini, Copilot, etc. Just give them a try 🙂 Or you can use AI as part of a training course, for example the A4Q Practical Tester.
And for testers? Here, the ISTQB Certified Tester AI Testing training and certification is a good fit. By the way, this certification has been around since 2021, and its predecessors even longer. AI didn’t start with ChatGPT, after all. And in a way, it’s just software 😉
Frequently Asked Questions
Should you wait it out or jump right in amid the AI hype?
Neither one nor the other: Blindly jumping on the bandwagon and burying your head in the sand are both bad answers. GenAI was at the peak of the Gartner Hype Cycle, meaning it was just before expectations began to decline. What’s interesting is what remains afterward, namely the question of where AI is being used productively and sustainably, beyond SEO copy and text generators.
Does the “AI” in a testing tool’s name always mean real AI?
No. It feels like every tool has gotten the “AI” label in its name by now, even if the product itself hasn’t changed much. That’s why it’s worth taking a closer look at what’s actually behind it. The more useful starting question is, in any case: What problem in your own project is the AI supposed to solve?
In practice, where are the biggest pain points where AI could help with testing?
Not in test execution. Test case creation, test data generation, execution, and defect analysis all have potential. In client projects, however, it becomes clear that the focus is less on even more efficient execution and more on test data management and the classic issue: the disconnect between the client’s requirements and the technical implementation.
Do you need a fully developed tool strategy before using AI in testing?
No. This is about pioneering work: Much of it isn’t finished yet, hasn’t been developed yet, and hasn’t even been conceived yet. So getting started, experimenting, and trying things out is the more realistic approach. To draw a comparison: The first websites weren’t created with a content management system either, but with notepad.exe.
Why is it so difficult to define quality in AI systems?
Because the question of quality often remains unanswered even with traditional software. In new projects and companies, the question of what quality means to the team is usually met with confusion and a reference to ISO 25010. When asked for specifics, there is silence. When testing AI, we come back to precisely this fundamental question.
Which quality criteria become more important than traditional ones when testing AI?
Accuracy, learning ability, adaptability, data quality, and statistical criteria are coming to the forefront. Traditional criteria such as functionality aren’t disappearing, but they need to be rethought when it comes to AI systems. For testers, test managers, and quality engineers, this shift in thinking applies in two areas: the use of AI and the testing of AI.
Are there training and continuing education programs for AI testing?
Yes. For testers, the ISTQB Certified Tester AI Testing training and certification program (available since 2021, with even earlier precursors) is a good option. AI is therefore not a topic that began with ChatGPT. To get started with AI on its own, the popular tools, tried out in your own daily work, are sufficient for now.
