Scenario-based testing is a new test method that comes out of autonomous driving: the test object stays the same, and what varies is the dynamic environment it has to cope with. For the future of software testing, that means testers need to be able to model those environments.
Tilo Linz is CEO and co-founder of imbus AG, a leading solution provider for software quality and software testing, and has worked in software quality for more than 25 years. As founder and chairman of the German Testing Board e.V. and a founding member of ASQF e.V. and ISTQB, he has played a major part in shaping and advancing education and training in the field, both nationally and internationally. Tilo Linz is the author of “Testen in Scrum-Projekten” and co-author of “Basiswissen Softwaretest” (dpunkt.verlag).
The Challenges Ahead for Software Testing
What challenges will software testing, and software testers, face in the future? What’s coming our way?
Before I talk about software testing, let me say something about “looking into the future,” which we worked on at imbus a few years ago. When people look into the future, they are wired to keep the current situation in mind and extrapolate it in a straight line. We picture what is already happening, just in a more futuristic form, and we stay shaped by the present and the past. That’s why looking ahead is especially hard in IT, where everything changes so fast. Many quantum leaps simply aren’t on our radar, and that’s the problem.
So the question is: how do we break out of linear thinking, so that we can also sense disruptions and genuinely new things? Futurologists use concrete, professional methods for this. One of the most important forecasting techniques is the so-called scenario technique. Starting from what you observe today, you work out the most important influencing factors and megatrends. You develop stories, spin them further, and arrive at things that could suddenly realistically happen. One of those worlds then becomes reality.
Together with a futurologist, imbus has published “Trends in Testing” studies for the years 2025, 2035 and 2050 over the past years. We have updated them several times, a few years apart. If you leaf through them in hindsight, some of it is nonsense, but some of it really came true. The shocking part was that a lot of it happened much faster than expected. You can download the study free of charge from our download area.
Of course, I can’t predict exactly which challenges software testers in particular will face in the coming years. But I’m happy to share what the challenges could be in the different worlds. I reread the study earlier; it is structured around 2025, 2035 and 2050. At the time of our conversation in 2022, 2025 was just two and a half years away. When we first published the study, that was the distant future. We see “Structure Ware” as an important trend in it, and I’m convinced that this will be the concrete area of application.
“Structure Ware” is about the fact that 3D printing is now used in the real world, wherever hardware and similar things are produced. A few years ago that was brand new; today it’s standard in industry. At home, too, it has become a normal hobby. You send whatever you want to produce to a device as code, and then it materializes. The traditional view is that you’re sending a CAD drawing. In reality, you’re sending software.
If you take that further and have powerful devices, they won’t be called 3D printers anymore, but bioprinters and so on. That production line can sit anywhere in the world. Or, the other way around, you order from Amazon and instead of a parcel you get “Structure Ware” in your inbox. It goes to the sophisticated 3D printer in your home, and the finished product comes out. What comes out in the end depends entirely on the “Structure Ware” that was sent.
What This Means for Testing
How does this affect testing itself? What are the approaches? Have you been thinking in that direction?
Here it’s actually the other way around. People who earn money by creating plans for 3D printing don’t think of themselves as software coders. They are engineers doing traditional part drawings. At some point they’ll hit a certain complexity threshold where something doesn’t work or where they’ve left something out. Then they’ll need a lot of the standard testing techniques, or they’ll be surprised to discover that there is a whole guild of software testers who can help.
Scenario-Based Testing: New Demands on Testers
What do I have to do differently if I’m a tester or test analyst? Do I have to do anything differently at all? Are the requirements different?
The test manager will have to master new things. Some new test methods are emerging right now, and this isn’t the distant future. A lot has happened around autonomous driving, and software and system testing is part of that. There’s an enormous amount going on in the field, with new publications every day.
Scenario-based testing is being developed right now, a new test method that traditional software testing doesn’t yet know in this formalized form. It will help bring autonomous vehicles to market, and I’m convinced the whole topic of robotics hangs on it too. Autonomous driving is already very demanding. Once a two-legged robot is walking around, it gets even more complex. The software tester has to be able to model these different environments, and that’s what scenario-based testing is about.
We’re no longer dealing with a test object lying on a lab bench and dozens of versions of it. We have a dynamic environment with many variations. In scenarios A, B and C, the parameters change the scenario, but the test object stays the same. It flips the relationship between test environment and test object.
Where Artificial Intelligence Comes In
There’s a lot of talk about artificial intelligence. From a testing perspective, there are two angles. On the one hand, there’s AI software somewhere that needs to be tested. On the other, there’s using AI for testing. Have you thought about the future in that direction?
Yes, exactly those two perspectives matter, and we deal with both. On the one hand, we have a research project running on testing AI-based agricultural machinery. Surprisingly, that’s one of the first real application areas for autonomous machines. Driving straight across a field isn’t critical. The whole integration of the systems is an issue, and so is the question of how to test the AI parts. The systems are supposed to learn how much fertilizer to apply, where the optimum lies, and so on.
AI for testing shows up for us mainly in autonomous driving. People are trying out various new methods to get the explosion of test cases and the complexity under control. That includes AI-based algorithms that help select from a potentially infinite number of scenarios.
How Development and Test Processes Are Changing
Beyond technology, there’s also the process side: how development and test processes are changing. Do you have a sense of where that’s heading?
There will be significantly more automation, and it may well get simpler, too. You’ve just read the book on keyword-driven testing. By 2050, the idea that non-IT people can click together process steps or test sequences will surely be commonplace.
In autonomous driving, testing and simulation will blur together. Until now, automotive has used simulation from a development perspective, to see how the prototype behaves in various directions. That will now merge much more with testing, validation and safety assurance, because you can’t afford to drive across the same intersection in a million different attempts.
DevOps and Agile
Everyone is doing DevOps and agile right now. Seen through a QA lens, where does it go from here?
As you say, it works wonderfully and it will stay a fixed part of the picture. Recently, while reading our own website, I came across the topic of the V-model, which in my view hardly matters anymore. Where teams still have to work according to the V-model, they always try to use a hybrid, so they can live the DevOps idea in some places: get to deployment as fast as possible and automate it end to end. In my opinion, there’s no other way anymore.
Recommendations for Software Testers and Test Managers
A lot of challenges and changes lie ahead for testers, test managers and everyone working in testing. What would you recommend to someone who is firmly in the saddle in software testing today, to prepare and stay fit for the future?
You have to be open to what’s happening around you, starting now. The key is to keep up with technological developments. Whether something affects you directly yet isn’t that important. Experience shows that you think you’re miles away from something, and next week the customer calls. My tip for everyone in software testing: be open, be curious and dig into what’s happening, even if your current project doesn’t require it. That goes for developers and testers alike. The differences are shrinking anyway.
You should at least try to roughly understand how newly emerging technology works. With machine learning, for example, you should know roughly how the algorithms work and what they do, even if you couldn’t implement one yourself. The same goes for scenario-based testing. You should know the term and know what a scenario is.
That goes hand in hand with continuous learning. The Foundation Level, for example, is still needed, and it’s the entry point. You can also see that all the training courses are still in demand. As a coach and trainer, you keep getting asked to support people.
One outcome of the pandemic dilemma that I find interesting and good: traditional classroom training still exists, and it’s picking up again. At the same time, online training has become a solid alternative. At the start of the pandemic, people saw it as an emergency measure. Now it’s an additional, perfectly normal delivery format. We now offer a third format as well, blended learning. Instead of running a course for three days straight, we stretch it over four weeks, for example, at a rhythm and pace the learners set themselves. That only became possible through the mix of online and on-site. New forms of learning play an important role and make it easier to stay on the ball, because you’re no longer forced to follow a schedule someone else dictates.
I don’t see the combination of the pandemic and technological change as purely negative. This is where the circle closes and we’re back at the beginning, with the look into the future. A pandemic, for example, is also a kind of impact that throws everything off course. In hindsight, that’s often exactly the point where you switch from one path to another and completely different things get developed.
Frequently Asked Questions
Why are predictions about the future of IT so often wrong?
Because people project the present linearly and imagine the future as a more futuristic version of today. This approach leaves out leaps and disruptions. That’s why futurology uses scenario analysis: You identify the most important influencing factors and megatrends, develop stories based on them, and spin those stories further. In retrospect, some of the predictions from such studies turned out to be nonsense, others came true, and many much faster than expected.
What is meant by the trend term “Structure Ware”?
Structure Ware refers to the code that is used to create a physical product at the recipient’s end. In 3D printing, it appears that a CAD plan is being sent, but in reality, it is software. Taking this a step further, instead of a small package, a structure arrives in your mailbox that a powerful device at home materializes. What comes out depends solely on this transmitted structure, that is, on software quality.
How does scenario-based testing differ from traditional software testing?
It reverses the relationship between the test object and the test environment. In traditional testing, there is a test object of which there are countless versions. In scenario-based testing, the test object remains the same, while the dynamic environment changes: Parameters define Scenario A, B, or C. The tester must be able to model these environments. The method originated in the context of autonomous driving, and robotics is closely tied to it.
What role does artificial intelligence play in the testing environment?
Two distinct approaches should be distinguished. AI systems themselves become the test object: In 2022, imbus conducted a research project to test AI-based agricultural machinery (one of the first real-world applications of autonomous machines), in which system integration and machine learning components were to be evaluated. Conversely, AI-based algorithms help with testing by making meaningful selections from the potentially infinite number of driving scenarios.
Why do testing and simulation blur the lines in autonomous driving?
Because it’s not feasible to actually drive through the same intersection a million times. In the automotive sector, simulation has long been used in development to observe the behavior of a prototype. It is increasingly overlapping with testing, validation, and verification. At the same time, automation is advancing, even to the point where non-IT professionals can set up test procedures with a few clicks.
Does the V-model still hold significance in the age of DevOps and agility?
In 2022, Tilo Linz no longer saw it as particularly significant. Where work still had to follow the V-model, hybrid approaches emerged to implement DevOps principles in specific areas: getting to deployment quickly and automating end-to-end. He described agile work and DevOps as an integral part of the process; in his view, there’s hardly any other way to do things anymore.
How do you stay on top of your game as a tester?
By exploring new things, even if they aren’t required for your current project. Experience shows: You think you’re miles away from it, and then the next week the client calls. It’s enough to have a rough understanding of how things work in principle (for example, what machine learning algorithms do) without being able to implement them. The difference between developers and testers is getting smaller anyway.
