Test Procedure for ZDF

Automation as a panacea? In a large media library project with over 1000 different end devices, this strategy quickly reaches its limits. The combination of automated and manual testing covers different types of errors: While scripts check thousands of broadcast pages, the human eye finds visual bugs and usability problems that no framework recognizes. The trick is not in the either-or decision, but in the intelligent division - and the courage to take the more pragmatic route in software testing, even against management guidelines.

Podcast Episode: Test Procedure for ZDF

How do you automate tests when applications are used by a wide range of devices - from new to outdated? The desire to automate everything cannot be fulfilled here, as the differences between the various operating systems are too complex. A combination of automation and manual testing manages the balancing act here. Anika and Benedikt take care of quality assurance for ZDF’s mobile applications and share their strategies for finding errors and how they decide which tests are automated and which are carried out manually.

“You have to find a good balance: What do I automate and on how many devices and what do I actually want to map? And how can I close this gap?” - Benedikt Broich, Anika Strake

Benedikt has been part of Appmatics since 2016 and supports the development of QA processes. As team lead, he works at the interface between test management and partners such as Klett, Kicker & Eurowings.

Anika Strake, Team Lead Test AutomationAnika has been working at Appmatics since 2019. She initially joined as a QA automation engineer. Today she is Team Lead Automation and supervises test techniques for partners such as ZDF, Migros & BVG.

Highlights der Episode

• Automation does not replace manual testing - only the combination really finds all critical errors.
• Manual testers are faster at analyzing errors and device variants than maintenance-intensive automated tests.
• 90% of failed cases are real bugs, only 10% are automation errors - setup is worthwhile.
• Exploratory testing uncovers critical gaps that are only then transferred to automation.
• Automation needs constant maintenance - without blocked resources, the suite collapses.

The balancing act of test automation and manual tests at ZDF

In this episode, I talk to Anika Strake and Benedikt Broich about the challenges of a test automation project for ZDF. They emphasize the importance of striking a balance between manual and automated testing to ensure both efficiency and flexibility.

Background of the project

Benedikt begins by explaining how Appmatics was awarded the contract by ZDF. The original tender called for complete automation of the tests. However, it quickly became clear that this was not the most efficient solution. Many apps did not have tags and IDs set correctly, which made automation more difficult. They also had to ensure that older devices were also covered. The solution: a mixture of automated and manual tests.

The role of manual tests

Anika explains in detail the importance of manual tests in the project. Especially with old devices or specific OS versions, they encounter unexpected problems. Manual testing allows them to isolate and fix bugs faster. While automated tests can provide a broad overview, manual tests allow a deeper investigation of specific problems. This two-pronged approach significantly increases test coverage.

Automated tests: opportunities and limitations

Automated tests offer many advantages, especially for repeatable tasks such as checking links or performing standardized actions in apps. However, Benedikt also points out the limitations of this method. Changes to the front end or updates to operating systems can easily interfere with automated tests. This once again demonstrates the value of manual tests in order to be able to react to such changes at short notice.

Evaluation of test cases

An important aspect of the project is the regular evaluation of which test cases should be automated and which should be carried out manually. Anika mentions that they pay attention to several factors: Does automation increase test coverage? Are the functions critical? And how complex is the manual testing effort? These considerations help to optimize the use of resources.

Results and findings

After a year of intensive work, the team was able to make significant progress. The combination of manual and automated tests proved to be extremely effective. They found errors that they would never have discovered through pure automation, especially on edge devices or in specific usage scenarios. This experience clearly shows that a balanced approach to test automation is crucial for success.