Public sector software testing has one question at its center: how does the client of an e-government system know that the delivered software really meets every requirement? Citizens only accept these systems if they work without errors, are easy to use, secure and fast. This article shows how systematic acceptance testing answers that question, using the e-government platform of the Free State of Saxony as the example.
In 2005, as part of an e-government initiative, Saxony started building a central infrastructure platform for e-government applications. It provides the components that e-government processes need. New applications plug into these base components, which makes them quick to build and easy for users to reach. The base components include the citizen services portal Amt24, the form service, the Sachsenatlas geoportal, the integration framework, a central content management system and components for electronic payments, electronic signatures and encryption.
For the people responsible for the platform, one rule is fixed: no component goes live until the acceptance tests are complete and the acceptance criteria are met. Systematic testing has proven its value again and again, for custom software as well as for off-the-shelf products. Since 2005, thousands of defects have been logged and fixed in the defect management system, and tens of thousands of test cases have been executed. The defects range from minor layout problems to severe failures that would have been disastrous in production. Testing also exposed wrong or incomplete requirements, which could then be fixed before go-live, for example through a change request.
The sections below follow the test process step by step and show what matters when you test an e-government platform in a structured way.
How Public Sector Software Testing Works: The Test Process
1. Test Planning
To avoid surprises later, test planning should start together with the planning of the development project, with its own budget, staffing and schedule for the test project. The test objectives drive the estimate of the test effort. For systems with high public visibility, performance and usability matter as much as correct functionality. Systems that perform critical functions or process personal data have to be tested for security vulnerabilities. When old and new business systems are integrated, interfaces and data transfer need thorough testing.
Standards help with effective test planning: the test process according to ISTQB, software quality characteristics according to ISO 9126-1:2001 (since replaced by ISO/IEC 25010) and test documentation according to IEEE 829:2008 (today ISO/IEC/IEEE 29119-3).
2. Test Preparation
Test preparation covers the test specification, test tools, the test environment and the procurement or creation of test data. A dedicated test center has proven especially valuable here. Many e-government projects bring together test specialists from IT and testers from different business departments and agencies. The test center gives them shared workstations and a place to discuss methodical and domain questions, so they can work together across the line organization. Many questions about the expected behavior of the application get settled quickly, often while the test specification is still being written.
3. Test Execution
Under the fixed-price contracts common in the public sector, acceptance often has to happen within a tight time window. Saxony has therefore made a habit of defining milestones with interim versions while development is still running. A release test of each interim version shows early how good the delivery is and leaves time to fix defects before the software is handed over for acceptance. The acceptance test itself is then only a regression test of meaningful test cases from the release tests.
Testers from the business departments focus on whether the most important use cases can be carried out, and they bring their experience with typical weak spots. Test specialists check that interfaces conform to standards and make sure every defined requirement is covered by tests.
4. Test Control
The project manager needs regular updates on test progress, defect numbers and emerging problems. If communication between the agencies’ IT systems fails, for example, measures such as opening network connections or adding server capacity can be started well before the go-live date. Because every defect is recorded in the defect management system, current issues are visible at any time, and project management can settle disputes about individual defect reports quickly (bug or feature? change request?).
5. Test Completion
The core of the test report that the test team hands to the project manager is a description of the residual risks and an evaluation of the delivered software against the acceptance criteria. These criteria are, of course, part of the contract with the supplier, and they can take different forms. For the e-government platform, a required test coverage is usually combined with defect-based metrics.
Example:
- 90% of all test cases with high or medium priority must be executed.
- No blocking defects and at most 10% of the major defects found may still be open.
Based on the acceptance recommendation in the test report, the project manager decides on final acceptance and any further steps.
Government Software Testing in Practice: The Saxony Test Center
When Saxony put the e-government platform and its first components out to tender in 2004, it was clear that this would set the standard for later projects. A test center was set up in which test specialists from ANECON specify and execute functional test cases together with testers from the business departments of the Free State. Security and usability tests follow common standards such as BITV, BSI, W3C or OWASP. Load and performance tests are a core part of the test projects as well, because they also determine the response times for the service level agreements with the operator.
Since testing started in Saxony, every project on the platform has had to go through these types of tests, plus a check against the requirements of the operating environment. Two things have proven to be key success factors: the collaboration of test specialists and business testers in the test center, and the change, release and defect management processes that are now routine. The effort pays off. Demands such as the EU Services Directive and users’ growing expectations of e-government call for more and more functionality and ever tighter integration of the IT systems involved. Their success depends heavily on user acceptance, and users expect a system that is error-free, secure and fast.
Frequently Asked Questions
Why isn’t it enough to perform testing just before software goes live?
Testing late in the process leaves little time to fix any defects that are found. It makes more sense to agree on milestones with interim versions during development and to verify them through release tests. This provides an early indication of the quality of the deliverables. The actual acceptance testing then consists only of regression testing of the most relevant test cases from the release tests.
When should test planning begin in a project?
At the same time as the development project is planned. The budget, staffing, and schedule are allocated specifically for the test project, not diverted from the development budget as an afterthought. The test objectives form the basis for estimating the effort required: they determine whether the focus is on performance efficiency, usability, security, or interfaces in addition to functionality.
What are the primary test objectives for publicly accessible government applications?
In addition to correct functionality, the primary focus is on performance efficiency and usability, because these systems are accessible to the public. If a system processes personal data or performs critical functions, testing for security vulnerabilities is also required. When old and new business systems are integrated, interfaces and data transmission must be tested comprehensively.
Does systematic testing offer any benefits beyond identifying program errors?
Yes. On Saxony’s e-government platform, testing since 2005 has also uncovered erroneous and incomplete requirements that could be corrected via change requests before the system went live. The range of errors found extended from minor layout issues to serious errors that would have been catastrophic in production. Both custom software and off-the-shelf products were affected.
What are the advantages of bringing testers together in one location?
A dedicated test center brings together test specialists from IT and subject-matter testers from various departments and government agencies to work side by side. They can exchange ideas on methodological and technical issues and collaborate across organizational silos. In Saxony, this approach has already allowed many questions regarding the application’s expected behavior to be resolved quickly during the test specification phase.
How do domain testers and test specialists divide up the work?
Testers from the business units focus on the feasibility of the most important use cases and contribute their experience with typical weak spots. Test specialists verify that interfaces comply with standards and ensure that all defined requirements are covered by test cases. Both perspectives complement each other; neither replaces the other.
How can acceptance criteria be formulated in a measurable way?
A combination of required test coverage and defect-based metrics has proven effective. For the Saxon e-government platform, for example, the following applied: 90 percent of all test cases prioritized as high or medium must be executed; no blocking defects may remain unresolved; and no more than 10 percent of the identified major defects may remain open. Such criteria should be included in the contract for work and services with the supplier.
What purpose does the test report serve at the end of the project?
It provides the project manager with the basis for deciding on final acceptance. At the core of the report are a description of the residual risks and an evaluation of the delivered software against the agreed-upon acceptance criteria. From this, the test team derives an acceptance recommendation. The project management, not the test team, then decides on any further actions.
