Holistic Testing

Holistic testing is an approach where quality activities are woven into every stage of the software development lifecycle, not confined to a single phase. It combines a whole-team responsibility for quality with testing that runs continuously through discovery, planning, building, deploying, and observing how customers actually use the product in production.

Key Takeaways

• Lack of shared understanding is the primary driver of rework: teams build and test a feature, only for the product owner to reject it because the requirement was never fully explored.
• Testing activities belong at every stage of the development cycle, from evaluating business ideas in discovery through observing customer behavior in production, not only in a dedicated test phase.
• Risk assessment during planning prevents defects before a single line of code is written, yet many teams skip it even when they practice test-driven development and continuous integration.
• Testers who build relationships with operations engineers, site reliability engineers, and designers gain visibility into production risks that never surface inside the delivery team alone.
• A visual, collaboratively built test strategy on a mind map outperforms a written document because teammates engage with it, update it, and generate ideas they would not reach through reading alone.

What holistic testing actually means

Holistic testing combines two ideas: the whole team owns quality, and testing happens all the way around the software development lifecycle. It is not a phase at the end and not the mini-waterfall stuffed into the last day of a sprint.

Lisa Crispin and Janet Gregory built the model on the infinite DevOps loop, with testing activities at every stage. The inspiration came from Dan Ashby’s continuous testing model in 2016. They wanted to enlarge on it, because the standard DevOps loop you find online shows a single “test stage,” which most people read as automated tests in continuous integration.

That narrow reading is the problem. There is work to do across the entire cycle to build quality into a product. Reducing it to one box on a diagram tells teams the wrong story.

Lisa and Janet deliberately call them stages, not phases. Stages are not gated, and there are no handoffs between them. The vocabulary matters, because “phase” invites the old habit of throwing work over a wall.

The cycle starts long before the first line of code

Quality work begins at discovery, where the business decides what to change next. Even here, someone with a testing mindset can ask hard questions. Is this idea congruent with what the business has done before? What problem does it solve for the business, and what problem does it solve for the customer?

The planning stage is where you get the most value for the least effort. The team defines what matters most about a feature: to the customer, to the team maintaining the software, and to the business. From there you choose the quality attributes that matter most.

Quality attributes often go unspoken because business people assume the delivery team will simply know what to do. Security and performance fall into this gap. Lisa prefers the term “quality attributes” over “non-functional requirements,” since calling them non-functional sends the wrong message about how important they are.

Risk thinking belongs in planning, not after the fact. Lisa has walked into teams doing test-driven development, pair programming, and continuous integration, yet taking the requirements handed to them by product without question. Preventing bugs means analyzing risk early, slicing the work, and prioritizing, because a human brain cannot hold more than about six priorities at once.

Story-level planning is the next layer. Dig into each story and build shared understanding of what you are about to build. A large share of defects can be prevented before anyone writes code.

Small batches lower your risk

Frequent deployment in small batches is the safer path, not the riskier one. If you deploy a tiny change and it breaks, you know exactly what broke.

During building, the team decides how to instrument the code for monitoring and observability. Exploratory testing can already happen at the story level. Once the code reaches a test environment closer to production, the quality attributes get their turn, alongside automated checks.

Some testing only humans can do. Exploratory testing, accessibility testing, and other human-centric work cannot be automated away. They sit beside the automated suite, not in competition with it.

Testing in production is an option where the domain allows it. Where it does not, you can still observe what is happening to customers or release a feature to a small group first. A testing mindset helps spot patterns, anomalies, and risks as changes go live, so the team can respond quickly.

The right side of the loop is where testers hold back

Many teams stop paying attention the moment a feature is released. For them the work is done, and they move on to the next thing. That is the side of the loop where most testers have never had a chance to build skills.

The reason is rarely laziness. People are overloaded and have no time to think about what happens after release. Work gets thrown to operations so the team can start the next item.

Testers often hesitate here because the tools feel foreign. They worry they cannot read log files or use monitoring dashboards. Many of those tools are genuinely easy to use now, and the skills testers already have transfer directly.

If you are not on a team that lives by “we build it, we run it,” build relationships with the people who do run it. Ask the site reliability engineers and operations specialists what they see in production and which risks are hurting the team. Feed what you learn back into the testing strategy so it covers those risks.

The whole-team approach reaches past the delivery team. You need relationships with designers, with marketing people who hold the analytics, and with platform engineers if none are embedded in your team. Observe how customers actually use a feature. Sometimes they ignore it, sometimes they use it in a way no one anticipated, and either way that learning feeds the next business decision.

Quality is a people business more than a tech business

Tools and tech stacks do not make teams successful. Lisa points to thirty years of research showing that the right people, enabled to do their best work, are what matters. Sometimes enabling them just means getting out of their way.

Testers tend to ask questions nobody else thinks of. That single habit can shift a team. If you are killing yourself testing at the end of a sprint, find one person who feels the same, a developer who sees the same problem, and brainstorm how to pull more people in.

Make the problem visible and make your contribution to quality visible too. When developers understand what is happening, they do not want testers stuck with soul-crushing work at the end of every iteration. People want good quality, and once the cost becomes visible, they tend to solve the problem together.

Lisa frames the tester as a translator between two languages.

A lot of times as a tester, I felt like I can bring the business people and the tech people together and get them having a conversation where they actually understand each other. Lisa Crispin

That role does not require coding. Testers need technical awareness, an understanding of the product architecture, and the ability to use IDEs and monitoring tools. The team already has strong programmers. What it often lacks is someone who can speak to the business in its own language and to the engineers in theirs.

Internal quality is the team’s call, not the product owner’s

Internal quality is what matters to the people building the software, separate from the external quality that matters to customers. Kent Beck drew this line years ago, and the Agile testing quadrants carry it forward.

A product owner should not dictate how software gets implemented. The team was hired for that expertise. How to host it, how to containerize it, whether to practice test-driven development, all of that is the development team’s decision.

Internal quality pays for itself because teams read code far more than they write it. Code that is hard to understand creates constant toil. Make the code clear, document it well internally, and a new team member knows where things are and how the process works.

Installability and portability belong in this conversation too. If you ship desktop software, build installability in for yourself and your customers from the start. If the product may move to other platforms later, think about portability now rather than retrofitting it.

Why rework is a symptom of missed conversations

Most rework traces back to a lack of shared understanding. The team thinks it knows what a story means, builds it, tests it, hands it to the product owner, and hears that it was not what they wanted.

The fix is cheap and the failure is expensive. A team can invest a little more time in planning and save a great deal of time downstream. When a defect reaches production, customers start calling support, and the cost climbs.

“That wasn’t really a bug, it was a missing requirement” is a phrase Lisa hears often, and it does not hold up. A missing requirement is a missed conversation. Flow diagrams, state diagrams, and context diagrams during planning surface those gaps before they turn into rework.

Make the strategy visible so people actually use it

A test strategy buried in a document nobody reads has little effect. Lisa wrote elaborate test plans in waterfall projects in the 1990s and was proud of them, yet no one else ever looked at them, not even her fellow testers.

A mind map on a shared virtual whiteboard works better because it invites collaboration. Everyone can be in the board at the same time, adding ideas and brainstorming together. Testing activities live on the map, get checked off, and gain screenshots or screencasts as evidence when something breaks.

Visuals trigger thinking that prose does not. Something on a whiteboard sparks an idea in someone else, the kind of idea that saves the team real time. They help people think laterally and work around their unconscious biases.

Involving the team in creating the map is the point, not just showing it to them. When people add to the strategy themselves, they stay connected to it and are more likely to return to it a week later.

One team Lisa worked with resisted mind mapping on a story everyone called easy. They tried it anyway, since an easy story would map quickly. An hour later they had a large map that clearly covered more than a single story. Trying it once is often what reveals the value.

Where to start when the team is under pressure

Start at the planning stage. In a context full of deadlines and stressed people, planning is where a small change returns the most, because missing shared understanding is what causes so many later problems.

The model gives example testing activities, but they are prompts, not commandments. Lisa is explicit that holistic testing is a thinking tool, not a rule. Teams have added their own concerns to it, such as heavy UX design work when user experience drives the product.

Tailor the model to your context and add whatever you need. The activities on the right side of the loop deserve attention too, since that is where most testers have had the least chance to learn. Plenty of resources exist to close that gap.