Test automation works best when a company introduces it step by step, starting with a test automation pilot project. A gradual approach gives you time to adapt techniques and tools to the company’s needs, to adjust the processes and, above all, to bring the company’s expectations in line with what test automation can actually do. In practice, three basic phases have proven themselves for us: pilot project, application and roll-out.
Test Automation Pilot Project: Proof of Concept
The pilot project phase consists of several activities: preparing the pilot, running it and evaluating the results. Each of them ends with something concrete that the next step builds on.
Preparing the Pilot Project
Preparation starts with an analysis of the current state. We capture the process descriptions, the tools in use and the existing dependencies by going through the relevant documentation. In addition, we usually interview everyone involved to find out how the processes are actually lived and how that compares with the documentation. The gap between the two is sometimes considerable.
The focus of this analysis is on finding improvement potential that comes from reducing work that has so far been done by hand. We also look at what it would take to realize that potential, for example tool support, the need for modeling, test coverage or traceability. The result is a catalog of criteria that a test automation solution should meet as well as possible.
The next step is a market analysis of the available concepts, the tools that go with them and their dependencies, based on that catalog of criteria. All available solutions are analyzed and rated against these criteria and then compared with each other. Depending on the time and money available, the pilot phase is then run with the best few solutions from this comparison.
Running the pilot with every solution makes no sense, because some offerings will very likely be an obvious mismatch for the company’s needs. I also advise against running the pilot with just one tool, the top-ranked one, because that narrows your view of the pros and cons far too much. My recommendation: run the pilot with the three top-ranked tools.
Running the Pilot
The pilot now tackles a small task that takes only a few person-days. Subject matter experts can be involved in the work, but the test automation experts should always be in charge. For each tool, we record the path to the solution, the advantages and disadvantages we ran into along the way, and the solution itself.
Evaluating the Test Automation Solution
After the pilot, the experience gained is evaluated and presented to all participants in a workshop. This is where the differences between the tools are reassessed. Hands-on experience in particular can overturn the ratings made earlier on paper. The workshop has to settle two fundamental questions:
- After this pilot, should we recommend using test automation (locally limited) in real projects?
- Which tool should that be done with?
If the answer to the first question is yes and the second question has an answer as well, the last milestone of this phase is reached, and the application phase begins.
Applying Test Automation in One Department
The application phase comes down to two steps: choosing the department or group in which test automation will be used commercially, and the application itself.
When choosing the department, one of the key points is to reuse the experience from the pilot project and make the adjustments needed for a transition from pilot to application that runs as smoothly as possible. I recommend that every decision-maker pays attention to choosing people with the right expertise here too, because the introduction is still at a sensitive stage. As part of this adjustment, the tool also has to be selected and the processes adapted. To judge success later, it is worth choosing the relevant metrics carefully.
After this preparation comes the application itself. Especially the first time around, users should be in constant contact with experts so they can discuss and reflect on their experiences, obstacles and progress. At first this happens through training and coaching alongside the actual work, with the goal that the department can work independently.
Using the metrics defined at the start, the chosen test automation technology can then be given a final assessment of its commercial suitability. This experience should provide enough information to decide whether to roll the technology out across the company.
Rolling Out Test Automation
In the roll-out phase, the use of test automation is formalized. The development process is adapted to test automation, and a matching training program is set up. Each department appoints people who monitor and steer the process improvement. Especially during first use, coaching should accompany the work. The long-term goal of the roll-out is that employees internalize the process without outside support and, whenever needed, adapt it to a business environment that keeps changing.
Introducing Test Automation in Agile Projects
In agile projects, these three phases still happen, but in compressed form. Usually a test automation tool is tried out on one part of the application during the running project. The result is evaluated, for example in a retrospective, and if it works, the tool is gradually rolled out to the rest of the project. Tool use is much more dynamic here: tools are switched now and then, and instead of a single test tool there is often a whole set of tools in use.
Frequently Asked Questions
Why is it better to introduce test automation gradually rather than company-wide right away?
A gradual introduction allows time to adapt techniques and tools to the company’s needs. Equally important is adjusting processes and aligning expectations: Many stakeholders overestimate what test automation can achieve. A three-phase approach has proven effective: a pilot project as a proof of concept, commercial implementation in one department, and subsequent rollout.
How do you determine which requirements a test automation solution must meet?
The first step is to analyze the current state: process descriptions, tools in use, and existing dependencies are documented using available documentation, supplemented by interviews with stakeholders. The focus is primarily on the potential to reduce manual tasks, as well as on topics such as modeling, test coverage, and traceability. This results in a set of criteria for evaluating the tool.
Is the existing process documentation sufficient as a basis for the analysis?
No. Documented processes and actual practices sometimes differ significantly. That’s why interviews with all stakeholders are part of the preparation process, to capture the actual workflow and compare it with the documentation. Only this picture reveals where manual tasks actually occur and what potential for improvement test automation can unlock.
Is it sufficient to conduct the pilot project using the highest-rated tool?
This is not recommended, as it severely limits the view of the pros and cons. It is recommended to conduct the pilot using the three top-ranked tools from the market analysis. Testing all available solutions is also impractical: some offerings are clearly not suited to the company’s needs and are eliminated during the evaluation phase.
How large should the task in the pilot project be, and who should carry it out?
The task should remain small and take only a few person-days to complete. Subject matter experts can be involved, but test automation experts should always take the lead. For each tool used, the approach taken, the advantages and disadvantages experienced, and the solution itself are documented so that the evaluation is based on comparable experiences.
How do you determine whether a test automation solution is suitable for company-wide deployment?
By key performance indicators (KPIs) defined prior to the implementation phase. These should be selected carefully, as they form the basis for the final evaluation of the technology’s commercial viability after it has been implemented in a single department. Only this experience provides the foundation for deciding on a company-wide rollout.
What does the rollout involve besides distributing the tool?
The rollout formalizes the implementation: The development process is adapted to test automation, and a training program is set up. Within the departments, individuals are designated to oversee and manage the process improvement. During the initial implementation, coaching should be provided alongside the application. The long-term goal is for employees to drive the process without external support and to adapt it as needed.
Does this phased approach also apply to agile projects?
In a condensed form. Typically, a tool is tested on a specific part of the application during an ongoing project; the results are evaluated (for example, during a retrospective) and, if successful, the tool is gradually rolled out across the project. The approach is more dynamic: tools are occasionally switched out, and often a tool landscape emerges rather than a single test tool.
