Quality assurance in large agile companies means more than just testing: quality is considered throughout the entire development cycle, from requirements to operation. Teams work independently, quality specialists act as coaches and sparring partners. Interfaces between teams are automatically secured via consumer-driven contracts. Quality awareness is spread to all teams through guilds, quality buddies and training courses.
Key Takeaways
- Quality is not the task of a dedicated test person, but must be considered and supported by all team members throughout the entire development lifecycle.
- The Quality Buddy role enables teams without a dedicated Quality Specialist to still have someone who actively triggers quality issues and acts as a multiplier between the team and the guild.
- Consumer Driven Contracts have become a standard task at Otto as soon as a new interface between two teams is created, thus largely automating interface testing.
- The declared aim of a Quality Specialist is to make themselves superfluous: Only when the team has the same know-how and mindset is the task complete.
From a separate test team to quality responsibility within the team
Otto has dissolved the classic test team and shifted responsibility for quality to the development teams. The company used to work in a waterfall-like manner: a separate testing team waited until development delivered the software, then carried out manual technical tests and reported back its feedback.
The change began when Otto developed and operated the store and later the app completely in-house. With this in-house development, the mindset shifted and the separate test team was absorbed into the development teams. The testers initially continued to carry out their usual testing activities, but sat directly in the team, with shorter feedback loops.
This first step was not yet the goal. Dominique Petrich, Quality Manager at Otto, describes the target image differently: away from the classic ISTQB roles such as tester and test manager, towards sparring partners, communicators, moderators and coaches in the team.
Quality is created across the entire lifecycle, not just in testing
Quality must be considered throughout the entire lifecycle of a requirement, from conception to going live and into operation. Testing itself is only one part of it. The actual goal is broader: processes, the team and interaction are just as much a part of it.
This results in an unusual self-definition of the quality role. A quality manager works towards making themselves superfluous in the team.
I’m done when my colleagues in the team have the same know-how as me, the same mindset, the same awareness of quality.
- Dominique Petrich
The role name reflects this change. The departmental Quality Specialist later became the Technical Quality Manager, primarily to create a uniform designation within the company.
Why team ownership has driven the transformation
The agile transformation at Otto did not come as an instruction from above, but was co-designed by everyone involved. This joint design increases acceptance because no one has a process imposed on them.
With its in-house development, Otto built up the teams cross-functionally. The aim behind this: Everyone in the team should be able to think and contribute to everything. Optimization never stops. Just as software is developed iteratively, a team also develops iteratively.
Personal responsibility comes with obligations. However, the teams want to live this responsibility and improve themselves. It is precisely this willingness to optimize that ensures that quality issues are tackled from within the team instead of being imposed from outside.
How the know-how for testing is distributed within the team
The first lever was the distribution of testing expertise across all shoulders. Instead of one QA person alone conceiving, writing and executing tests, everyone in the team learns how to do it. A single test resource would otherwise be a bottleneck.
The concrete topics are initially technical: how do I write tests, with which frameworks, how do I execute them. From there, the team gradually works its way through the lifecycle.
Otto does not prescribe a fixed tech stack for the teams. The teams are largely autonomous and decide on programming languages, frameworks and tools themselves. Only certain things are fixed and organized centrally.
This is what the work process looks like in the teams
Otto works with a mix of Scrum and Kanban, known internally as Scrumban. The teams have chosen the practices that suit them, instead of reproducing a framework one-to-one.
In Dominique’s team, work is carried out via a board with clear phases: Analysis, development, an acceptance, going live and a subsequent review process. Stories are added to the prioritized backlog at any time and are processed. There are no dedicated sprint schedules with fixed pre-commitments.
Estimates are still made, but for a different purpose. The estimation only serves to uncover the need for discussion. If the estimates differ widely, the team members have a different understanding of the story and need to talk again. The estimates themselves are not included in the stories and are not used as a measure of success.
The two-week rhythm is maintained in order to keep the teams synchronized and to break down overarching plans, from the company to the division to the individual team.
A quality guild against lost synergies
With around 60 teams in the e-commerce sector, a risk arises: synergies are lost and quality is treated reactively instead of preventively. Once a process has been set up, it is not touched again, and only after a major error does someone look at the causes.
This is exactly what preventive quality thinking is supposed to prevent. In response, four colleagues founded the Quality Guild, a regular exchange format for everyone involved in quality. It focuses on current challenges, mutual support, interfaces and learning from each other.
The figures show why an overarching format is necessary: there are around 15 quality specialists and around 60 teams. This means that not every team has a dedicated QA person, and this is deliberately accepted.
Quality Buddy and Security Champion: roles that anyone can take on
For teams without a dedicated QA person, Otto has created the role of Quality Buddy. This person holds up the flag for quality in the team, triggers the topic in the event of anomalies and acts as a multiplier between the team and the guild.
The role is not tied to a specific profession. A quality buddy can be a developer, a technical expert or someone from the UX area, regardless of the actual role.
The same pattern is repeated for security. Each team has a security champion who has their own exchange format and anchors security topics more firmly in the team.
This is supplemented by specific offerings: two workshops in which teams work specifically on their quality topics, a training program for new colleagues and occasional presentations, some with external speakers.
Why the basics need to be repeated
Even experienced developers benefit from regularly refreshing the basics of testing. Onboarding often assumes that new colleagues have a certain mindset and knowledge. Nevertheless, it is worth revisiting the basics.
These include equivalence partitions, decision trees and the clean creation of test cases. Most people have heard these concepts before, but they tend to fade into the background in everyday life. The feedback is clear: it was good to have heard this again.
How Otto tests between teams: Consumer Driven Contracts
Otto uses Consumer Driven Contracts, or CDC for short, for testing between two teams. These automated interface tests run in the CI/CD pipelines of the respective teams.
This is now a matter of course. Anyone who sets up a new interface to another team has the automated interface testing as a standard task in the backlog. In this way, Otto achieves a very high degree of automation across individual or a few teams.
It becomes more difficult beyond two teams. As soon as end-to-end testing becomes necessary across several departmental boundaries, the proportion of manual work and communication increases significantly. There is not yet a ready-made solution for this, and this is precisely where Otto sees one of its biggest unresolved issues.
Securing non-functional requirements via specialized teams
Otto covers security, load, performance and availability via dedicated teams, combined with roles in the product teams. There is a separate team for security with mandatory training for everyone.
For technical security, Otto relies on tooling for the platforms used. As all repositories are located in GitHub, GitHub services are mandatory for all teams. The infrastructure in the Amazon Cloud provides automated scans that alert teams to anomalies.
For load and performance, a dedicated team runs tests against the environment on a regular basis. If something is noticed, the team contacts the affected product teams. They can also work with the load testing experts to design their own tests for their services.
The store must not fail, otherwise it will be expensive. This is why alerting and monitoring are the responsibility of each team.
Disaster recovery as a recurring exercise
A Disaster Recovery Day takes place once a year. This involves simulating an emergency, such as a compromised AWS account that forces a complete rebuild. The aim is to restore all interfaces as quickly as possible in a highly automated process and in collaboration with other teams.
These days reveal where there are still gaps. In the meantime, individual teams also regularly carry out such exercises themselves in order to further automate the process and become faster in the event of an emergency.
Usability via UX roles, A/B testing and real customer feedback
Otto ensures usability via UX specialists directly in the front-end teams. UX designers and UX managers bring usability and accessibility requirements to the team, whether for the app or webshop.
In addition, there is systematic customer feedback. Otto works with regular surveys and A/B testing, in which features are initially only rolled out for some of the customers. Both are measured: how the features perform and how the feedback turns out.
Otto operates its own UX lab on the campus in Hamburg. Real customers are invited there, look at features on the screen, are tracked and give direct feedback on whether they were able to use something intuitively.
Static analysis is the responsibility of the individual teams
There is no uniform standard for the use of static analysis at Otto. Some teams use it heavily, others hardly at all. This is also an expression of team autonomy.
In concrete terms, this means that some teams use linting and have test coverage checked locally, for example during pairing or by the QA person. Other teams build quality gates into their CI/CD pipelines, define a minimum test coverage and run automated analyses on them.
This bandwidth is intentional. Instead of imposing static analysis from the outside, the decision remains with the team.
The biggest open construction sites for the next few years
End-to-end testing across multiple areas remains the biggest technical challenge. Large projects that affect the store, backend and several areas should be tested with as little manual effort as possible and integrated into the agile process. This topic will be with Otto for some time to come.
AI is also becoming part of everyday working life. Otto is already working heavily with algorithms and is trying out how AI can support programming work, for example when writing tests, and is collecting best practices.
New channels are also generating a need for testing. In future, shopping will not only take place via web stores and apps, but also via smart TVs or cars, for example. Each new channel must be integrated, tracked and tested.
The biggest structural trend concerns the QA role itself. The number of QA specialists will not grow in line with the number of teams in the future. The goal is for teams to be fully responsible for quality.
This means that the role of the quality manager will shift upwards. Instead of moving from team to team, they will move into overarching activities: creating offers, enabling teams and driving quality awareness across teams and departments.
