Test reporting is a management tool that shows test managers where a project stands, how it got there and where it is heading. A proven structure is divided into a 3x3 matrix: three technical columns for test preparation, test execution and errors, combined with three rows for actual status, progress and data quality.
Key Takeaways
- Test reporting based on a 3x3 matrix of test preparation, test execution and defects covers the actual information requirements of the stakeholders in 95 percent of all cases, according to experience from several projects.
- Forecasting is a core function of test reporting: if you use historical execution data to project when the test objective will be achieved, you create the basis for concrete decisions on deadlines, resources or target corrections.
- Poor data quality falsifies any actual status: An automated internal control system that monitors defined mandatory attributes such as sub-project affiliation makes incorrect data maintenance visible before it corrupts reporting to the outside world.
- Consistency in format pays off directly: Stakeholders who are familiar with the same reporting layout from previous projects are immediately oriented without having to re-explain every key figure.
Test reporting is a management tool, not a mandatory diagram
Test reporting is not decided in a calm project, but in a stormy one. Matthias Gross describes this with the image of a boat trip: any ship can be steered at 28 degrees and with a slight swell. Only when the waves are five meters high on the left and right and the horizon disappears does it become clear whether the guidance and steering work.
It is precisely in this situation that reporting provides the value for which it exists. It not only shows a status, it is the basis for decisions: testing longer, investing more or going into production with a broader target corridor.
Those who see reporting as a compulsory exercise produce diagrams. Those who see it as a management tool derive measures from it. The difference is not in the tool, but in the question of whether the figures lead to action.
Why every reporting starts with a goal
Without a defined goal, you are heading nowhere. If you set sail without knowing the harbor, you are adrift. In testing, the goal is roughly clear: as error-free as possible in production, with high test coverage. However, hardly any project reaches the ideal point of 100 percent coverage and zero errors.
This is why Matthias works with a target corridor instead of a single point. If the ship lands twenty meters to the left or thirty meters to the right, this is often acceptable. A go-live with thirty-five blocker bugs and seventy percent coverage, on the other hand, is clearly outside the corridor.
The goal must be agreed with the team, project management and client because it defines the quality produced. Nevertheless, the first step is to ensure that the goal exists and is named. Only then can a position be determined.
The 3x3 matrix: reporting that is sufficient in 95 percent of cases
Complete test reporting fits into a matrix consisting of three columns and three rows. The three columns stand for the technical artifacts, the three rows for the dimensions in which each artifact is considered.
The columns represent the path of the software:
- Test preparation: everything on the way to the finished test case.
- Test execution: how the tests are doing in execution.
- Defects: Number, criticality and processing of errors found.
The rows provide a different view for each column:
- Actual status: the current position, often as a pie chart. Example test preparation: 80 percent completed, 20 percent open.
- Progress: how the status came about over the last few days and weeks.
- Measurement quality (ICS): an internal control system that measures the reliability of the data.
If all nine fields are filled with a graph or key figure, the reporting is sufficient in the vast majority of cases. Most stakeholders only want to know three things anyway: How is the preparation going, how is the execution going, how are the errors looking.
Filling in the nine fields is more work than it initially looks. Matthias reports that this is exactly where teams using the approach for the first time start to sweat. The effort pays off as soon as the fields are in place and serve as a control instrument.
How the progression becomes a reliable forecast
The history line is not there for decoration, it feeds the forecast. Based on the historical progress and the know-how from the project, it is possible to project how long the team will still need to reach the target with the current performance.
This forecast regularly clashes with a fixed date. If the go-live is in calendar week 25, but the projection is calendar week 28, there is a three-week difference on the table.
With this information, measures can be initiated before things get tight. There are three options: postpone the release, invest more in execution or live with a wider target corridor. Without a forecast, you won’t notice the difference until the hypercare phase hits.
Measure data quality before reporting lies
A report is only as good as the data behind it. The third line, the internal control system, expresses data quality as a percentage: 100 percent means clean data, a drop to 80 percent is an alarm signal.
The attributes that are actually required for reporting, such as status, date or sub-project, are measured. The status is usually well maintained. Fields such as the sub-project are more problematic if the tool does not enforce a clean selection.
If the ICS value plummets from 100 and is suddenly at 80, this is an indicator for me that I need to take countermeasures. I already know that I am communicating the wrong actual status to the outside world, but I can intervene in the background.
- Matthias Gross
There are nine sub-projects in Matthias’ current project. The field is not mandatory and allows multiple values. A test case with two sub-projects or with an unknown value falsifies the evaluation because it either counts twice or is completely lost.
The rule behind this is clear: a test case belongs to exactly one valid sub-project. An automated system checks each test case against the stored list and calculates the ICS value. If a new sub-project is added, the list is expanded.
Matthias keeps this value internally. It would have to be explained to external parties, which is why it is used for internal control and does not flow into the external dashboard. He is the control instance that prevents the reporting from reporting a position that is three hundred meters away from the real one.
How reporting can be automated
The first four steps are manual set-up work, the fifth automates them. Reporting is used regularly, so the steps defined once must be reproducible without manual work.
Matthias’ team downloads the data via the Jira API with the Expand Change Log add-on. This puts the complete ticket history into the Python script. Clean snapshots are created there, condensed to the weekend, suitable for weekly reporting in the waterfall.
The summarization follows a clear logic. A ticket with three changes in a week only counts with the last status at the weekend. A ticket that has not been touched for weeks still receives its snapshot.
The processed raw data ends up in Excel, where it becomes a dashboard via pivot tables. This separation is deliberate: The data preparation remains robust in Python, the flexible assembly of new views is possible in Excel and can be done without programming knowledge.
| Layer | Tool | Task |
|---|---|---|
| data source | Jira API (Expand Change Log) | read complete ticket history |
| Preparation | Python | Create snapshots, summarize to weekend |
| Evaluation | Excel with pivot tables | Dashboard, flexible views |
| distribution | PDF from the dashboard | weekly same view for management |
Management receives a PDF with the same key figures every week. Stakeholders with technical expertise have direct access to the Excel and can create their own views, such as which department is currently holding which test cases.
Consistency beats a flood of metrics
The greatest benefit comes from always having the same presentation. Matthias has been using the same 3x3 matrix in every project for years. Anyone familiar with the procedure from previous projects will immediately find their way around: preparation here, execution there, errors there.
This consistency is the antithesis of the overloaded Jira dashboard with thousands of pie charts that nobody knows what to do with. Only the level of detail is varied, not the structure.
In the case of a larger migration with several sub-projects, the matrix initially shows the overall project. By zooming in, the identical nine-field view can be created for an individual sub-project such as Finance. This gives sub-project management and overall project management depth of detail without changing the basic structure.
Further evaluations are often not new key figures, but different sections of the same data. A department view instead of a sub-project view, for example, can be created quickly as soon as the affiliation is available as an attribute.
Chat with the data without touching the linear reporting
An AI experiment has supplemented reporting, not replaced it. Automated, weekly reporting remains untouched because its strength lies precisely in its consistency.
The team has also created a way to chat with the data. The data prepared in Python is transferred to a PostgreSQL database, connected via n8n, Ollama and WebUI. Questions can be asked and new views or ideas retrieved via a chat interface.
The main effort was put into the system prompt until the solution worked reliably. The result has a clear place: it provides spontaneous, explorative answers. It does not replace the inviolable basis with the same key figures.
