Systematic software testing means specifically selecting the relevant inputs from a practically infinite number of possible inputs: those with high risk and high everyday significance. Testing does not break software, but reveals where it already is. Sufficient test coverage is achieved by methods such as equivalence partitions and boundary value analysis, not by blind trial and error.
Key Takeaways
- Testing is not a destructive act: Not a single test case breaks software, but makes visible where it is already broken.
- Complete testing is mathematically impossible: three parameters at 16 bits take around 89 years with 100,000 tests per second, at 32 bits it is 10 to the power of 15 years.
- The Foundation Level qualification is a driver’s license, not proof of driving ability: it allows you to play along in traffic, real testing can only be learned through years of practice.
- Agility has not torn down the wall between development and testing, it has only made it a little more permeable; in the minds of many of those involved, it still stands.
- Developers who fulfill clear coverage criteria with equivalence partitions and boundary values know when enough testing has been done; those who run 300 automated test cases without criteria do not.
Testing doesn’t break software, it makes defects visible
Testing does not destroy software. It shows where it is already defective. This distinction sounds like nonsense, but it shapes the self-image of an entire discipline.
No test case damages a program. If a part of the code is faulty, it already was. The test only brings it to light. Anyone who describes testing as a destructive activity is reversing the logic.
Testing is constructive because it requires thinking. Testers consider which test data and which test environment can be used to demonstrate a problem that development has not thought of. They have to think ahead: Has the negative case been taken into account? Did they think about February 29? Such gaps remain decades after they were first described.
Testing doesn’t break anything, it shows where the software is broken. Not a single test case breaks any software.
Andreas Spillner
Why complete testing is impossible
No program in the world is ever presented with all possible sequences for execution. This is not a question of discipline, but of mathematics.
A math example makes this tangible. An old system with 16 bits, three numbers to be entered, a test automation that manages 100,000 tests per second: it would take around 89 years to try out all combinations. With 32 bits, you end up with an order of magnitude of 10 to the power of 15 years.
This does not mean that testing is pointless. It means that you have to choose. Which processes run every day? Where is there a high risk of something failing? This is exactly where systematic testing comes in, instead of blindly reaching in somewhere.
If you have internalized this one number, you have gained a lot in testing: No program ever does everything it could. So the question is not whether you select, but how.
Sufficient instead of tested
The term “testing” is misleading. Anyone who claims that something has been tested must have made all possibilities work once. As the bit example shows, this is never possible.
A different goal makes sense: sufficient test cases for the specific problem. Not extensive, not detailed, but appropriate to the risk. A highly critical system requires many methods and great depth of testing. Vacation planning for employees does not need this.
Test methods help with this selection. With the equivalence class method, many input parameters can be condensed to a few representatives. This gives you a resilient test case set without drowning in combinatorics.
The image of a spider’s web hits the nail on the head. A good web is evenly shaped so that something sticks. It is not narrow-meshed in one place and large-meshed in another. It is precisely this even coverage that test methods provide.
Clear exit criteria provide security
Exit criteria decide whether you can go home satisfied. If you know that equivalence partitions are covered, boundary values have been checked and decision tables have been given a green tick, you have a reliable signal of sufficient coverage.
The counter-example is common. An automation runs through with 300 test cases, and nobody knows exactly what these 300 test cases are actually testing. Then the criterion that says: You have tested enough is missing.
The structured method provides developers with exactly what they would otherwise be missing: an indication of when they can stop. Without this criterion, the feeling remains that testing is a bottomless task that is hardly worth starting.
Agility has torn down the wall between testing and development
The biggest game changer in testing over the last two decades has been agility. The Agile Manifesto dates back to the year 2000, long before the topic became more prevalent in curricula and standard works.
There used to be a wall between development and testing. Developers threw the product over the wall, testers threw back the error message and there was no talking. Agility has broken down this separation because it puts teams at the center instead of fixed handovers.
The requirement itself is older than agility. It has long been propagated that developers and testers should work together and not against each other. What is new is that a method makes this cooperation a principle.
One remaining wall remains: the wall in the head. It can be seen where collaboration would be possible, but old reflexes continue to have an effect.
Communication is productivity, even if management doesn’t see it
Talking to each other is the core of agile work, and it is precisely this that is often not recognized as work in companies. There is often no time set aside for exchange and cooperation because nothing seems to come of it.
In fact, the opposite is true. Communication often produces the most results, but the results are not immediately visible on paper. This gap between impact and visibility is the real sales problem.
One practical way is to camouflage communication spaces. A workshop or a short coaching session creates a formal framework in which exchanges can take place. As soon as the first effects become visible, tolerance grows.
Testers and developers benefit from each other. The developer learns testing know-how, the tester learns development knowledge. You need both in an agile team today, for example for test automation.
There is no favorite method, only suitable methods
There is no universal test method. Which method or which combination is right always depends on the system and its environment.
There is no silver bullet, and certainly not in testing. Instead, the more critical the system, the more intensive and varied the methods. The lower the risk, the less effort is justified.
This attitude takes the pressure off. Those who do not look for the one right method, but for the right one for the problem in question, make better decisions. The choice of method is a question of context, not fashion.
Error culture determines whether defects lead to learning
A good error culture means owning up to mistakes and learning from them. In Germany, this culture is often weak, and not just in software development.
The contrast is made clear by an image. When a rocket explodes on take-off and everyone cheers because a lot has been learned from the failure, this is an attitude that is difficult to imagine in Germany. It is more common to duck, hide and sweep under the carpet.
Celebrating is not the goal. Learning is. Some companies have a culture in which mistakes are discussed openly so that they are not repeated. Others duck away. The difference determines whether a defect becomes a learning opportunity or a taboo.
Personal openness is part of this. Anyone who deals with their own mistakes publicly instead of hiding them helps themselves above all, even if no one else can learn from them directly.
The Foundation Level is a driver’s license, not a master’s certificate
A test certificate at Foundation level is comparable to a driver’s license. You can play along from now on, you know the rules and the signs, but you can’t drive yet.
The knowledge behind it is not new. Many of the methods were already described in the 1970s and are still valid today. The merit of a good basic work lies not in innovation, but in clear terminology and solid preparation.
The actual competence grows afterwards. Only after years of dealing with real systems do you learn to test, just as it takes years to really drive a car. The certificate is the ticket, not the goal.
A syllabus as a guide also makes writing about the subject much easier. If the content is predefined, the most difficult work is eliminated: the constant question of what to include and what not to include. Embedded aspects or real-time topics, for example, deliberately do not belong in the basic level.
Why developers find it hard to get excited about systematic testing
Convincing developers of systematic testing remains difficult. The idea of testing is deeply ingrained: testing seems like an infinitely large task that is hardly worth starting.
This is precisely where the leverage lies. Structured methods provide developers with exit criteria. They transform a seemingly limitless task into one that has a clear end.
The difference can be seen in two evenings. With equivalence partitions covered, boundary values checked and decision tables complete, you go home satisfied. With 300 test cases and no knowledge of their purpose, you are left with the uneasy feeling of not having a criterion for sufficient testing.
In the end, it takes passion for the cause. Anyone who approaches a topic with conviction will produce good results, almost regardless of the field.
Predictions about the future of testing are rarely reliable
Predictions about technology often miss the mark, and this also applies to testing. Expectations such as the strong modularization of software, in which systems are only assembled from intensively tested building blocks with verified interfaces, have not materialized in this form.
Waves of hype reinforce this pattern. A few years ago, blockchain was seen as a technology that would make contracts superfluous. This has not materialized to the extent promised. Such technologies often end up on a plateau of real use, far removed from the original promise.
Artificial intelligence is currently under similar scrutiny. It is impossible to say for certain whether it is a prolonged hype or a permanent change. Only one thing is certain: the expectation that AI will make testing superfluous because it will simply be told what to do is repeating an old pattern. It used to be said that formal specification and code generation would make testing unnecessary. That was not the case.
Our own misjudgement also teaches us humility. Today, almost everyone uses technology that was dismissed as superfluous, such as television pictures on cell phones. Looking into the future remains difficult.
