Quality hunting is an alternative to bug hunting in which small teams gather information about a system’s quality for its stakeholders. Teams of about four people explore the system for no more than half a day, guided by a goal set beforehand, such as usability. Their report turns raw data into knowledge that supports the decision to go live.
Key Takeaways
- Rik Marselis developed quality hunting out of frustration with bug hunting, because for him testing means getting information about the quality level of a system.
- At the first quality hunt in a EuroSTAR test lab, one team found about eight problems and could still judge how well the system works for disabled users.
- A quality hunting team has about four people, half the size of an average agile team, and a session lasts three hours or at most half a day.
- Teams should sketch the structure of their report at the start, so the session fills that structure with the information their stakeholders need.
- Shortly before a release that bundles several sprints is a good moment for a quality hunt; teams that ship every user story right away cannot fit it in.
What Is Quality Hunting?
Quality hunting is a short, time-boxed exploratory session in which small teams gather information about a system so that stakeholders can judge its quality. Rik Marselis developed it as an alternative to bug hunting. The format keeps the playful competition of a bug hunt, but teams collect evidence about quality characteristics instead of a count of defects.
Among test design techniques, quality hunting sits closest to a session-based exploratory test: a fixed time frame, a clear mission, learning while testing. What sets it apart is the result. A quality hunt ends with a statement that answers one question for the people who decide: is the system good enough? Rik describes the method in the book Amplified Quality Engineering, and the new ISTQB syllabus Quality in DevOps names it as one of its approaches.
A List of Bugs Says Little About Quality
Bug hunting starts from the premise that testers should find bugs. Rik’s frustration with that premise started the whole idea. In his view, a bug is a signal. It shows that quality is lower than expected, and then someone can act on it. The actual job is to know the quality level.
“Testing is about getting information about the quality level of your system.”
(Rik Marselis)
The method took shape in a test lab at the EuroSTAR conference, where Rik and his colleagues ran the first quality hunt as a hands-on exercise. Some teams still worked in bug-hunting mode and came back with a big pile of problems. That pile said nothing about whether the system was good enough.
Another team reported only about eight problems, but it had looked at how the system would be used. Its verdict: good enough for a general audience, not good enough for people with disabilities, because some choices were not available to them. That team delivered an evaluation of quality, and the experience led Rik to describe the method more formally.
Every Hunt Starts With the Stakeholder’s Question
Bug hunting has an obvious goal. Quality hunting needs some preparation, because the goal has to be defined first: what information do we need, and for whom? The team can settle this itself, or whoever writes the assignment does it for them.
For a first hunt, Rik’s advice is simple. Ask your stakeholders what information would help them sleep well at night. Whoever carries the decision, be it the product owner, the users or management, should get what they need to stop worrying about the system. The hunt then gathers the inputs for exactly that information.
Quality characteristics give the assignment its shape. Whether the system does what it is supposed to do still matters, but the more revealing question is often how it does it. Some non-functional characteristics suit the format better than others. A performance test is a poor match for a three-hour team session. Usability is hard to specify up front and easy to explore in a short time frame, which makes it a good candidate for a quality hunt.
How Does a Quality Hunting Session Work?
A quality hunt is organized like a game. Several teams compete, the stakeholder picks a winner, and there is a prize. Rik recommends keeping the prize small, since the point is to get good information. The competition still pays off, because teams push each other to look further.
The typical setup:
- Team size of about four people, roughly half of an average agile team of seven
- No more than half a day, around three hours
- Four to six teams in parallel, which yields a lot of information in very little time
Within the time box, each team decides how to approach the assignment and then works the way an exploratory tester does. Think of a test, run it, learn something, check whether it brings the team closer to the information it needs. Little is prepared in advance; what the team sees triggers the next step. Every so often, the team steps back and recalls the end goal: a specific kind of information for the stakeholders.
How a team gets there is up to the team. The only fixed point is the goal, the best information possible within the time frame.
Teams With the Same Assignment Look in Different Places
Give several teams the same assignment and they will take totally different directions. Rik’s usability example shows how. Some teams check whether the system looks good and follows the company’s guidelines or style rules. Others think about accessibility and ask how people with disabilities can use it.
For the stakeholder, this makes picking a winner hard, because there is value in every direction. Rik therefore suggests handing out a few smaller prizes. In the end, the teams often win together, because their combined findings give the stakeholder a lot of information in a short time.
How Should Teams Report What They Found?
A bug hunt produces a list of bugs. The format of a quality hunt report depends on the team, but Rik’s advice holds for testing in general: think about the structure of your report at the start. What message do you want to convey, and to whom? Once the structure stands, the session fills it in. During the hunt, the team checks which information it already has and what is still missing.
Rik ties this to the hierarchy of data, information, knowledge, and wisdom. Testing first produces data, which the team assembles into information. What stakeholders should receive is knowledge, so that they actually know the quality level. The wisdom to decide whether to go live stays with them. A report full of raw data about what the team did skips the step that matters.
The Best Moment Comes Right Before a Release
How often to run a quality hunt depends on the situation. One workable rule is to do some quality hunting before every release to production. Teams that push every single user story to production right away, in true DevOps style, will not find a slot for it. Most teams work in sprints and release the result of several sprints together. Rik sees the time just before such a release as the ideal moment.
Because anyone can join a team, a hunt at that point also works as an addition to acceptance testing. End users, managers, and other people involved get first-hand experience of the system’s quality.
The approach also keeps people in the loop where automation dominates. In DevOps environments where the motto is to automate everything, quality hunting preserves human curiosity and a critical mind, which is why the ISTQB syllabus includes it. The same holds for AI in testing, where people still have to ask what the goal was and whether they actually get what they need.
