Properties Based Testing

What do you do when your distributed system has a billion possible error combinations and a human can think through maybe twenty of them? Property-based testing turns the classic approach on its head: instead of writing test cases, you describe the properties of your system - a generator then automatically creates thousands of scenarios that also uncover hidden bugs, such as those that only occur after 17 consecutive steps. The method does not replace software testing, but makes it drastically more efficient.

Podcast Episode: Properties Based Testing

Properties based testing is a method that checks constant output properties and complements traditional test strategies. This technique is particularly suitable for complex and microservices-based systems as it allows the appraisal of a wide range of inputs. Nikhil explains with several real life examples how PBT works, what the advantages are and where the method reaches its limits.

“The fundamental proposition of properties based testing is when the system gets too complicated, it is not possible for human to write every test cases.” - Nikhil Barthwal

Nikhil Barthwal is passionate about developing decentralized systems. He has several years of professional experience in large corporations as well as in smaller start-ups and acts as a mentor for various start-ups. He is also a guest speaker at various international conferences and gives talks on topics related to decentralized systems and software quality.

Highlights der Episode

• Property-based testing automatically generates millions of test cases from system properties instead of manual cases.
• Shrinker reduces complex error sequences to the minimum sequence for human debugging.
• For distributed systems with thousands of services, n³ combinations exceed any manual test coverage.
• Cultural change is harder than tool introduction: start small, show success, let it grow organically.
• Destructive testing with millions of cloud calls can drive costs out of control - consider resources.

How properties-based testing increases quality assurance

In this episode, I spoke with Nikhil Barthwal about the applications of Properties Based Testing. As software systems become increasingly complex, traditional testing methods are reaching their limits, making way for more advanced techniques such as properties-based testing.

The challenge of testing large-scale systems

Testing thousands of microservices is a huge challenge. Nikhil explains how properties-based testing addresses this by automatically generating test cases based on system behavior, reducing the need for manual test case creation.

Core components of properties-based testing

There are three main components: Modeling Language, Generator, and Shrinker. Nikhil uses an example of an arithmetic service to illustrate how these components interact within property-based testing frameworks to efficiently identify potential errors.

Practical applications and frameworks

Nikhil talks about various technologies that support property-based testing. He also highlights QuickCheck in Haskell as the origin of this concept, which has since been translated into several languages.

Achieve reliability beyond human imagination

Properties Based Testing aims to outperform human capabilities by handling an immense number of permutations in system behavior. This ensures reliability and uncovers errors that might only have shown up under certain conditions or over time.

Implementation of properties-based testing in organizations

Nikhil suggests starting small with properties-based testing and gradually expanding its use within an organization. He emphasizes the importance of cultural change and convincing teams to implement new methods for improved results.

Limitations and precautions with properties-based testing

Properties-based testing has limitations. For example, it is not suitable where tests incur significant costs or are destructive. It is crucial to weigh up these factors before implementation.

A new ally for testers

Properties Based Testing does not replace the tester, but acts as an assistant that increases productivity by narrowing down potential problems for human analysis.