What is Regression Testing?
Regression testing checks, after a change to a piece of software, whether previously working behaviour still works correctly. The ISTQB syllabus puts it soberly: regression testing confirms that a change, including an already tested defect fix, has no adverse consequences. Those consequences can hit the changed component itself, other components of the same system or even connected systems. The environment counts as well: an update of the operating system or the database can trigger the same effects as a code change.
The name already tells you what this is about. Regression means falling back: something that worked suddenly does not work any more. Such defects rarely appear where the work was just done. They appear in the side effects next to it. Which is exactly why testing only the change itself is not enough.
One classification point, because it goes wrong so often: regression testing is not a test level of its own. It does not line up behind component testing, integration testing, system testing and acceptance testing; it runs across all of these levels. Wherever something has been changed, it belongs.
When Regression Testing is Due
Five triggers cover most cases:
- After a defect fix. The classic. The fix has been tested, but did it break something elsewhere?
- Before a release. The final evidence that the accumulated changes of a cycle have left existing behaviour untouched.
- After a refactoring. By definition, the behaviour is supposed to stay the same. Whether it actually does, you only know after the test.
- After the update of a dependency. New framework version, new library, new database driver: foreign code, same effect.
- After changes to configuration or environment. Even without a single changed line of code, system behaviour can shift.
An impact analysis clarifies how much of this is needed: which parts of the software could the change touch? The ISTQB syllabus explicitly recommends one before the scope of the regression test is fixed. Without this analysis, only two bad options remain: testing too much or testing too little.
Why Regression Testing Gets Expensive
The effort for regression testing has an unpleasant property: it grows faster than the changes that trigger it. After all, what gets checked is precisely the functionality that is not supposed to be affected by the change. The share of tests to be executed therefore quickly exceeds the share of software that was changed. A small fix, a large test scope.
Silke Reimer described to me in the podcast what this feels like in real life, using IVU as the example: around 25,000 test cases, a complete run of 6.5 hours. Add to that up to ten branches maintained in parallel, because several releases sit with customers. A single defect fix travels through four or five branches if in doubt and triggers the full suite every time. Even ten large machines in continuous operation were no longer enough.
The seemingly frugal alternative, running everything only once per night, carries its own price. When a test fails in the morning, the search only begins: which of yesterday’s commits was it? This analysis time shows up in no effort estimate and still falls due every week.
Regression Testing and Test Levels
Regression testing and test levels sit on two different axes: test levels describe when and by whom testing happens, regression testing describes why testing happens again. That is why it exists on every level. The unit suite that runs after every commit is a regression test at component level. The nightly end-to-end run is one at system level. And the automated tests that protect accepted increments are one at acceptance level.
The ISTQB syllabus says it directly: confirmation testing and regression testing are required at all test levels where defects were fixed or changes were made. Whoever plans regression testing as a separate phase at the end of the project has misunderstood the concept and given away its greatest advantage: fast feedback close to the change.
Regression Testing vs. Retesting
Regression testing and retesting answer two different questions and still get mixed up constantly. The ISTQB syllabus calls it confirmation testing, in everyday project language it is usually retesting; both mean the same thing. Confirmation testing confirms that a specific, previously found defect has actually been fixed. Regression testing confirms that the fix has not broken anything elsewhere. The two belong together, in exactly this order: first check that the defect is gone, then check whether something new got broken along the way.
Two details from the syllabus are practically relevant. Preferably, the same person who originally found the defect performs the confirmation test; they know the way there best. And when time or money is short, the confirmation test may be limited to the test steps that triggered the failure. Regression testing offers no such shortcut, because nobody knows in advance where the side effect sits.
Regression Testing and Test Automation
Regression testing and test automation are often mentioned in the same breath, but they are not the same thing. A regression test does not have to be automated. It is merely the most natural candidate for it: regression suites run many times, the number of test cases grows with every release, and exactly this repetition is what makes automation pay off. All the more remarkable is one number from the Softwaretest-Umfrage 2024, a survey of the German-speaking testing community: 12 percent of the participants do not automate their regression tests at all.
Automation is a means here, not an end. For the telematics app of the insurer HUK-Coburg, the first test automation delivered green runs that nobody trusted: while the automation passed completely, manual regression testing found 20 to 25 critical defects. The team shut the automation down entirely after four years and started over. Today around a third of the roughly 150 regression test cases are automated, with a clear goal and, this time, with trust in the results. A green regression run that nobody trusts protects nothing.
How test automation is set up in general, how the test automation pyramid helps and where automation pays off is a topic of its own with a page of its own. Here the question comes one step earlier: which tests need to run again after a change at all?
Test Case Selection: Full, Selective and Prioritized Regression Testing
For this question there are three strategies:
Full regression testing executes the complete suite on every change. It is the safest variant and perfectly fine for small suites. Beyond a certain size it becomes unaffordable, see the 6.5 hours above.
Selective regression testing executes only the test cases actually affected by the change. The prerequisite is knowing which test case uses which parts of the software. That is exactly what test case selection is.
Prioritized regression testing orders the test cases by risk, defect history or usage frequency, so that likely failures become visible early. It complements selection when even the reduced suite still runs for a long time.
The IVU case from the podcast shows what selection can deliver. A nightly run logs which files, classes and functions each test case opens. On every check-in, only the test cases affected by the changed files run, fully automatically and across the language boundary between C++ and Java. The selection saves 50 to 60 percent of all test cases:
| Before | After (on average) | |
|---|---|---|
| Java side | 2.5 hours | 1 hour, often 10 to 15 minutes |
| C++ side (selection at function level) | 4 to 4.5 hours | 10 to 15 minutes |
| Test cases saved | 50 to 60 percent |
The most important step, though, was not the technology but the validation. For four to six weeks the complete suite kept running in parallel, while the system logged what the selection would have picked. The result: not a single genuine failure missed. Only this evidence earned the method the team’s trust.
For prioritization, process mining supplies a data basis that replaces gut feeling. Production data shows which workflows actually run, and how often. In one case from the podcast, the most elaborate workflow had over 600 distinct variants, but the top 10 covered around 80 percent of all production runs. Whoever has test cases for these ten covers what really happens and can justify why the remaining 590 go untested.
Visual Regression Testing
A visual regression test compares screenshots of the user interface with previously approved reference images and raises a flag when the appearance changes unintentionally. It finds defects that functional tests walk right past: a shifted layout, a covered button, unreadable text on a new background. The functional test stays green, the page is broken anyway.
On the tooling side, a close look pays off. Playwright ships visual regression testing in its TypeScript version directly in the test runner, diff images included. The Python version needs external libraries for this, some of which are no longer actively maintained. Differences like these help decide which variant fits your own team.
Tools
The tool spectrum is the same as for automated testing in general, because regression tests run at all levels: Playwright, Selenium and Cypress for the user interface, REST-assured or Postman at API level, plus the unit test frameworks of the respective language. With keyword-driven approaches such as Robot Framework, non-technical testers write their regression test cases themselves.
The deciding factor is less the individual tool than the embedding. Regression tests only develop their value when they run in the CI pipeline on every change, not as a special operation before the release. The closer the test to the change, the cheaper the defect it finds.
From Practice
On regression testing, I am above all the one who listens. For years, practitioners have been telling me in my podcasts how they keep their test suites under control, and two of these conversations have shaped my view of the topic.
Silke Reimer described in the episode “Automatic test case selection for regression testing” how IVU brought its 6.5-hour suite down to minutes. What stuck with me was less the technology than the way the team’s scepticism was handled: weeks of parallel operation, full suite against selection, until the numbers spoke for themselves. Felix Doppel told the counter-story from HUK-Coburg in “Test automation of mobile apps”: an automation glowing green while the manual regression testers found the critical defects. His team had the courage to shut down four years of work and start again.
Both stories tell the same thing at their core. A regression test is worth as much as the trust a team has in it. And this trust does not come from a tool’s promise but from evidence: a parallel run, traceable numbers, defects found. Whoever builds a regression test always builds both: the test cases and the trust in them.