How Stiftung Warentest tests

Vacuum cleaners, toothbrushes, laptops - Stiftung Warentest has been putting products through their paces for decades. But how is such a test actually created, who buys the devices and how are soft criteria such as handling made comparable? The answer: with cross-functional teams, undercover purchasers, strict verification processes and a clear mission: independence and transparency. There are surprising parallels for software testers - from user glasses to dealing with uncomfortable test results.

Podcast Episode: How Stiftung Warentest tests

In this episode, I talk to Johannes Stiller from Stiftung Warentest about how one of Germany's best-known consumer organizations tests - and where the parallels to our software testing lie. Johannes gives an insight into how decisions are made about which products to test a year and a half before publication, how undercover buyers procure toothbrushes and laptops anonymously and why subjective criteria such as "handiness" can still be measured. Particularly exciting: the role of cross-functional teams, the verification process shortly before go-live and dealing with manufacturers who sometimes don't exactly react enthusiastically to test results. If you want to know what objectivity, transparency and independence look like in practice, you should listen in.

"It happens quite often that we are the bearers of bad news." - Johannes Stiller

Johannes Stiller completed his doctorate in nuclear and particle physics in Heidelberg in 2016. During this time, he worked intensively on site at CERN on the ALICE experiment. Due to his expertise in tracking algorithms, he then switched to software development in the automotive industry. A major milestone was his significant involvement in the safety braking systems for the Parkshuttle Rivium autonomous parking shuttle system. After moving to Berlin, he dedicated himself to the series development of driver assistance systems such as Emergency Assist and Travel Assist at the VW Group, including ASPICE-compliant development, testing and approval. Since August 2025, Johannes Stiller has held a management position at Stiftung Warentest, where he is particularly motivated by the opportunity to work on a wide range of topics on a scientifically sound basis.

Highlights der Episode

• Objective tests need blindness: masking product names, changing order, not informing testers.
• Stiftung Warentest plans tests a year and a half in advance - spontaneity kills comparability.
• Subjective criteria become measurable through trained, stable teams over years.
• Manufacturers only find out test results shortly before publication - no chance of manipulation.
• Concealed purchasing on the real market instead of manufacturer deliveries ensures that the test is realistic.

How Stiftung Warentest tests - and what software testing can learn from it

The claim: Independent, objective, user-oriented

When we think of software testing, we often think of source code, bugs and automated test suites. However, the conversation between Richie and Johannes Stiller from Stiftung Warentest made it clear that there are also many structural parallels and aha moments when testing toothbrushes, laptops and treadmills.

Stiftung Warentest has been an institution for decades when it comes to independent consumer tests. Its primary goal: product tests that support consumers so that they can make good decisions on their own. The testers there see themselves as neutral representatives of the user's perspective - much like software testers who try to identify weaknesses or strengths for end users.

The choice of topic: What is being tested at all?

For many, a magazine full of freshly tested devices, from mattresses to laptops, seems like a matter of course. But the road to the test is a long one. Ideas for topics are sometimes collected and planned a year and a half in advance 05:07. The basis for the selection is what consumers are interested in. The team evaluates inquiries on its own website and observes current market trends and seasonal trends. The wish: tests must be useful for users, up-to-date and practically measurable.

But coordination is also important. An advisory committee critically examines the proposals, often based on the question: Can a product even be compared fairly? In the end, a clear topic plan is created, which various team members design together.

What does a test procedure look like?

Implementation begins with the test program. This is based on existing standards - or, if these do not exist, they are created to the best of our knowledge and with a view to practical application 08:02. The organization behind it: A project manager devotes himself intensively to the product, develops the test program, obtains feedback from external testing institutes and colleagues. The test criteria range from objectively measurable things such as power consumption to more subjective impressions, such as the handling of a toothbrush.

Especially for criteria that are difficult to measure, test groups are carefully trained and products are covered, for example, so that they cannot recognize the brands. This minimizes bias and ensures comparability - similar to a blind test. The results of different testers are combined so that a meaningful overall picture emerges.

Teamwork: diversity of perspectives as a principle

Particularly exciting for the software world: Stiftung Warentest deliberately relies on interdisciplinary teams. Market analysts determine which products or innovations are currently relevant. Scientists, technicians, editors and consumer representatives discuss the selection and test execution together. Johannes Stiller emphasizes that this mixture of market orientation, specialist knowledge and clear processes ensures solid test results.

Quality teams in software development also benefit when technical understanding, user perspective and market intuition work together, just as the cross-functional teams in IT do. What counts in the end is that not only the technical analysis is right, but that the test results are understandable and comprehensible for the customer.

Communication: transparency, verifiability, openness to criticism

Another point: transparency and communication. Before results are published, there is a critical verification process. The team checks all measurement data together, asks about anomalies and checks the consistency again. Manufacturers are informed, but have no influence on the content or timing of publication. Problematic products are even reported to the companies in advance in order to protect consumers.

Of course, Stiftung Warentest is not always popular - some manufacturers react nervously or even aggressively if their product performs poorly 21:58. But the long-term influence on product quality can be seen in the fact that many companies base their development on the findings of such tests.

Users also get in touch: they give feedback, criticize and make suggestions. Johannes Stiller mentions an alert reader service that picks up on tips from the community. This keeps testing a lively, open process.

Whether testing treadmills, laptops or software modules: In the end, it's always about honest feedback, reliable criteria and a clear focus on the user perspective. Openness, teamwork, transparency and a good dose of perseverance characterize both Stiftung Warentest and successful software testing teams.

Anyone who needs inspiration for their next software test will find plenty of valuable suggestions when looking at other testing worlds - and perhaps also a little confirmation for their own work.