Test automation metrics are figures derived from the raw data of test tools that show what test automation brings to a project. A self-built Metric Collector gathers data from Jira, GitLab and Jenkins every day, keeps the history and feeds a Power BI dashboard that test coordinators and management read in separate views.
Key Takeaways
- Test automation metrics gain their value from history: only the trend over weeks shows whether open tests and bug counts really drop before a release.
- Ivaylo Brüssow calls counting automated test cases bean counting; useful test metrics answer what the project gains from the automation.
- A self-written Metric Collector in Python runs on Jenkins every evening and pulls data from Jira, Xray, two GitLab instances, SonarQube and Dependency-Track.
- When Git holds more test scripts than Jenkins runs, or tests are missing in Jira, the test automation dashboard flags a gap and the team contacts the application owners.
- Eight weighted metrics produce a score from 0 to 5 per application, modeled on the Nutri-Score, which upper management receives by email once a month.
Test Automation Metrics: Counting Test Cases Isn’t Enough
Around 100 applications take part in the initiative Ivaylo Brüssow and Andrej Nod work on, and for every one of them somebody wants to know where test automation stands. In the pilot projects, the numbers came from manual work: count the test cases, count the keywords in Robot Framework, type it all into a spreadsheet. With 100 applications that no longer holds, and the raw number of automated test cases says little about what the automation delivers anyway. So their team built its own Metric Collector: it merges data from Jira, GitLab and Jenkins into a Power BI dashboard, and the test automation metrics it produces carry a history and a documented formula.
The pressure came from two directions. During a merger program, the team offered test automation as a service and had to show upper management what had come out of the funds it received. And internally the team wanted to know why the logs in some areas showed a flat line: test cases sitting there, nothing running anymore. Does the area need support, or has its focus shifted?
“It’s irrelevant how many test cases I’ve automated as a raw number. The question is, what do I gain from that?”
(Ivaylo Brüssow)
The team first clarified which data it holds itself and which metrics follow from it. Every metric is documented: whoever sees a value on the dashboard also sees the formula behind it. If a coefficient sits well below 1, the application owner should be able to work out for themselves how to get up to 1.
Which Metrics Show Effectiveness Better Than the Number of Test Cases?
The first metric anyone asked for was code coverage. Ivaylo had to decline: without additional tools in the individual projects, it can’t be measured centrally. What the team offered instead were two metrics that come out of the existing systems and have since become the main metric.
| Metric | What the team measures with it | Measurable from the existing systems |
|---|---|---|
| Code coverage | The first metric that was requested | No, only with additional tools per project |
| Test coverage | Whether every change and every Jira story is traceably covered by at least one test case | Yes, from Jira with Xray |
| Degree of automation | Share of automated tests in the total number of tests | Yes, from Jira, GitLab and Jenkins |
Only side by side do the two metrics say something. One application has plenty of test scripts in its Git repository but hardly any test cases in Jira, so its degree of automation looks poor even though automation happened. Andrej takes such a finding to the application team and asks for the data to be entered, because the count comes from the sources everybody uses. “You’re counting wrong” came up more than once; on closer inspection the data was missing or in the wrong format. As a side effect, the metric checks daily whether the process is being followed, without anyone compiling an audit every few months.
How Does Data From Jira, GitLab and Jenkins End Up on One Dashboard?
Data was never scarce, just scattered. Jira delivers its daily snapshot, Jenkins and GitLab theirs, and each tool only knows its own corner. The team wanted the numbers from all systems on a single tile of one test automation dashboard, overlaid and timestamped. So it wrote the Metric Collector, name included: a piece of Python software that starts on Jenkins late every evening, once the day’s Jira tickets are closed and all pipelines have run.
Connected are Jira with Xray, two GitLab instances, the Jenkins instances and OpenShift pipelines, plus SonarQube and Dependency-Track. Each metric has its own worker that prepares its data and stores it with history. Nobody rebuilds Jira’s reports; the Collector pulls only the numbers it needs and brings them to a common denominator.
Most of the work sits in the mapping. The same application has one name in Git, another in Jira and a third in Jenkins. Out of that grew a large mapping table, supplemented in places by automated matching: Jira lists every application taking part in a release along with the release contents; on top of that the Collector lays the test coverage, the executions on Jenkins and the number of test scripts in Git. Only in this overlay do the gaps show. Scripts that sit in the repository but run on no Jenkins. A curve that once went up and then went quiet.
What Do Historized Metrics Show Before a Release?
Jira shows the current state too. What it doesn’t show is the trajectory: how did things look at the start of the test phase, how do they look shortly before the release? The expectation is clear. The curve of tests still to be worked through goes down, and so does the number of defects. If it doesn’t, the dashboard shows it weeks ahead. Andrej calls that a “car crash in slow motion”: the release is heading into a wall, and the trend shows it long before the date arrives. A positive trend, in turn, lets the team release with a clear conscience because coverage and testing time line up. That beats a PowerPoint traffic light somebody sets back to green before the steering committee.
For release planning there is the “release bracket”: a filter that hides everything that doesn’t count for the specific release. The history has a second effect: the Collector gathers more data than the dashboard evaluates today, so if someone turns up in a year with a new metric, the data has been sitting there for months and only needs defining and visualizing.
One Score From Eight Metrics for Management, the Details for Test Coordinators
Who sees what depends on the role, and the team settled the target groups in a workshop. A manager wants a rough overview. For that, the dashboard aggregates several metrics into a score from 0 to 5, like a Nutri-Score with a green and a red zone. Eight metrics feed into it with different weights. Anyone who wants to improve their score has to change more than one number; that is deliberate. Once a month the score goes out automatically by email to upper management: where do we stand, which projects lead, where has the best score been reached.
Test coordinators get their own view with predefined filters and the same data in more depth. Whoever wants to go further opens the tools themselves: SonarQube, Jira, Git. Same data on every level, filtered and combined differently. Power BI does the visualization because the group already used it; a colleague from the team built the dashboard with its filters, views and partly role-based perspectives.
Workshops With the Business Units Decide on Acceptance
The test metrics emerged in three rounds. Before the project there were workshops on how well or badly the automation was running and which handful of metrics would say something about it. After the proof of concept of the Metric Collector, the team agreed internally on the metrics the Collector should feed. And once the first prototype ran, representatives from the business units joined, from developers to test leads to heads of department, and discussed for several hours. The team’s message: we have the instrument, it can deliver this and that, tell us what you need, because we’re building it for you.
Ivaylo calls the feedback mixed: the dashboard is accepted, the team wants more of that and a lot more actual use. Some requirements hit the limits of Power BI, and then it’s a question of building around them or taking a different route. After a year and a half the initiative isn’t finished. But the gut feeling is gone. “We have roughly this much, I’d guess” has turned into “we have exactly this much”, with the formula right next to it.
