At best, the introduction of test automation in a company takes place gradually. This serves to adapt the techniques and tools to the needs of the company. But also to adapt the processes and, in particular, to align the company’s expectations with the possibilities of test automation. In practice, an introduction in 3 basic phases, which represent this step-by-step introduction, has proved successful for us: Pilot project, application and roll-out.
Pilot project test automation - proof of concept
The pilot project phase comprises several activities: the preparation, implementation and evaluation of the pilot.
Preparation of the pilot project
During the preparation phase, the current status is analyzed. To this end, the process descriptions, tools used and existing dependencies are recorded by analyzing the associated documentation. Furthermore, interviews are typically also conducted with all those involved in order to record the processes in place and the comparison with the documentation. The difference is sometimes significant. During this analysis, priority is given to identifying potential for improvement resulting from the reduction of previously manual activities. Corresponding activities to leverage this potential (e.g. use of tools, need for modeling, test coverage or traceability) are also analyzed. This results in a catalog of criteria that a test automation solution should fulfill in the best possible way.
The next step is the market analysis of the available concepts, the associated tools and the corresponding dependencies based on the previously identified list of criteria. All available solutions are analyzed and evaluated and then compared. Depending on the time and financial possibilities, the pilot phase is then carried out with the best x solutions. Since there are most likely some offers that obviously do not fit the company’s needs, it makes no sense to implement all solutions. I also advise against carrying out the pilot with only one (the best-ranked) tool, as this narrows down the view of the advantages and disadvantages considerably. I recommend carrying out the pilot with the 3 top-ranked tools.
Implementation of the test automation pilot project
The pilot is now carried out with the aim of solving a smaller task that only takes a few person-days. The implementation itself can be carried out with the involvement of the technical experts. In any case, the experts for test automation should be in charge. The path to the solution, the advantages and disadvantages experienced in the process and the solution itself are recorded for each tool used.
Evaluation of the test automation solution
Following implementation, the experience gained is evaluated and presented to the entire group of participants in a workshop. Here, the differences between the tools used are reassessed. In particular, the practical experience of using the tools can revise the assessments made previously. Two fundamental questions will be clarified during the workshop:
- Should the (locally limited) use of test automation in real projects be recommended after completion of this pilot?
- Which tool should be used for this deployment?
If the first question has been answered positively and an answer has also been found for the second question, then the last milestone of this phase has been mastered and the application phase begins.
Application
The application phase essentially comprises two steps: The selection of the department / group in which the commercial application of test automation is to take place, and the application itself.
When selecting the department, one of the key points is to reuse the experience from the pilot project and make the necessary adjustments to ensure the smoothest possible transition from the pilot to the application. I recommend that every decision-maker pay attention to the selection of suitable people, as the introduction is still in a sensitive phase. Furthermore, the appropriate tool must be selected and the processes adapted as part of the adaptation. To evaluate success, it is also advisable to carefully select the relevant key figures.
The preparation phase is followed by the application phase. Especially during the initial application, it is recommended that users are in constant contact with experts in order to discuss and reflect on their experiences, obstacles and progress. Initially, this is done with training and coaching sessions accompanying the application with the aim of independent work in the department.
With the help of the key figures defined at the beginning, the selected test automation technology can be finally evaluated after the application phase with regard to commercial suitability. This experience should provide enough information to decide whether the technology should be rolled out company-wide.
Roll-out of test automation
In the roll-out phase, the use of test automation is formalized. The development process is adapted to the test automation and a corresponding training program is set up. Responsible persons are appointed in the departments to monitor and control the process improvement. Application coaching should be provided, especially during the initial application. The long-term goal of the roll-out is for employees to internalize the process without external support and adapt it to the changing business environment as required.
Test automation in agile projects
In agile projects, these phases are shortened. In most cases, a test automation tool is tested on a section of the application during the ongoing project. The result is evaluated in a retrospective, for example, and rolled out successively to the project if successful. Here, the use is much more dynamic, tools are changed occasionally and there is often a tool landscape instead of a single test tool.
Frequently Asked Questions
Why is it better to introduce test automation gradually rather than company-wide right away?
A gradual introduction allows time to adapt techniques and tools to the company’s needs. Equally important is adjusting processes and aligning expectations: Many stakeholders overestimate what test automation can achieve. A three-phase approach has proven effective: a pilot project as a proof of concept, commercial implementation in one department, and subsequent rollout.
How do you determine which requirements a test automation solution must meet?
The first step is to analyze the current state: process descriptions, tools in use, and existing dependencies are documented using available documentation, supplemented by interviews with stakeholders. The focus is primarily on the potential to reduce manual tasks, as well as on topics such as modeling, test coverage, and traceability. This results in a set of criteria for evaluating the tool.
Is the existing process documentation sufficient as a basis for the analysis?
No. Documented processes and actual practices sometimes differ significantly. That’s why interviews with all stakeholders are part of the preparation process—to capture the actual workflow and compare it with the documentation. Only this picture reveals where manual tasks actually occur and what potential for improvement test automation can unlock.
Is it sufficient to conduct the pilot project using the highest-rated tool?
This is not recommended, as it severely limits the view of the pros and cons. It is recommended to conduct the pilot using the three top-ranked tools from the market analysis. Testing all available solutions is also impractical: some offerings are clearly not suited to the company’s needs and are eliminated during the evaluation phase.
How large should the task in the pilot project be, and who should carry it out?
The task should remain small and take only a few person-days to complete. Subject matter experts can be involved, but test automation experts should always take the lead. For each tool used, the approach taken, the advantages and disadvantages experienced, and the solution itself are documented so that the evaluation is based on comparable experiences.
How do you determine whether a test automation solution is suitable for company-wide deployment?
By key performance indicators (KPIs) defined prior to the implementation phase. These should be selected carefully, as they form the basis for the final evaluation of the technology’s commercial viability after it has been implemented in a single department. Only this experience provides the foundation for deciding on a company-wide rollout.
What does the rollout involve besides distributing the tool?
The rollout formalizes the implementation: The development process is adapted to test automation, and a training program is set up. Within the departments, individuals are designated to oversee and manage the process improvement. During the initial implementation, coaching should be provided alongside the application. The long-term goal is for employees to drive the process without external support and to adapt it as needed.
Does this phased approach also apply to agile projects?
In a condensed form. Typically, a tool is tested on a specific part of the application during an ongoing project; the results are evaluated—for example, during a retrospective—and, if successful, the tool is gradually rolled out across the project. The approach is more dynamic: tools are occasionally switched out, and often a tool landscape emerges rather than a single test tool.
