ISTQB in practice at Bucher and Suter

Test setup in telecommunications software means developing functional test cases, load testing and your own automation tools step by step and often without ready-made market solutions. External requirements from partners such as Cisco provide concrete exit criteria. Internally, ISTQB certifications and regular team reviews ensure the long-term transfer of knowledge and quality.

Key Takeaways

• Lack of market solutions for specialized load testing forced Bucher & Sutter to develop their own test automation software from scratch because no available tool covered the combination of voice gateway control and CRM simulation.
• Cisco requirements, such as the 99.99 percent target for correctly handled calls, taught the team that the same numbers can mean different things depending on how you phrase the failure case.
• ISTQB certifications up to Advanced level serve as a common vocabulary within the team at Bucher & Sutter and ensure that even experienced testers regularly review their knowledge.
• The move to the cloud increases the importance of test automation in CI/CD pipelines and at the same time brings new requirements for IT security and data privacy that dedicated on-premise environments did not have in this form.

When the test basis is missing, testing begins with research

Third-party systems without documentation force testers to do exploratory testing. Testers testing interfaces between unknown platforms are not given a ready-made list of requirements to work from. Instead, they have to work out the expected behavior themselves, step by step.

In the case of telecommunications solutions that mediate between the Cisco world and large CRM providers, this was precisely the initial situation. Some of the systems to be connected were unknown and their proprietary features were barely documented. The expected results, i.e. how the system should react to a certain call or process, could not be derived from specifications.

Alexander Meister describes the development of such a test catalog as research work. Use cases were developed in repeated consultation with the experts from the platform involved, often two to three times a month. Over time, this process resulted in a catalog of 400 to 500 test cases, which were then adapted to the different platforms.

This type of test case development requires more mental work than simply ticking off predefined criteria. You don’t just look for errors, you first define what would be correct in the first place.

The change from developer to tester is also a change of role in your head

Switching from development to testing means turning your perspective around. A developer is proud of what their software can do. A tester is interested in the opposite.

What you can do is all well and good, but I’m interested in everything that doesn’t work well. I’m interested in what you can’t do.

Alexander Meister

This change of perspective is difficult at first. The point is not to look for faults in people, but in the software. Whether this is a question of character or a development that you go through remains to be seen. In any case, it’s an attitude that you have to acquire, not one that comes automatically with the job.

Why there was no off-the-shelf tool for load testing

Machine-oriented systems cannot be tested with standard load tools. As soon as tests start very deep in the layers, for example directly at voice gateways, there is no interface that a generic tool could serve. The test has to start at API level.

This is exactly what made setting up the load testing environment so complex. On the one hand, load generators simulated the voice load including the complete voice stream, i.e. real data traffic over the line instead of pure pseudo signaling. On the other hand, something had to be created that abstracted the agents of a call center and the underlying CRM world.

This counterpart was not available to buy. It had to be developed from scratch as a separate automation software. If you want to run load testing in such a special environment, you often build the test robots yourself.

One condition is non-negotiable: the test infrastructure must be more stable than the system it is supposed to test.

You can’t test anything if the software you build around it breaks down first.

Alexander Meister

A load environment needs to be sealed off, otherwise you’re testing phantoms

Shared infrastructure distorts long-term load testing. If you start a 48-hour run on an environment that is open for other purposes on the side, you risk abnormal ends for completely irrelevant reasons.

An illustrative example: A run runs for 46 hours, then someone downloads an update via an open network share and the test is over. You start all over again without there ever having been a real defect.

The consequence is a dedicated load system. Your own core switch, your own servers, the rack exclusively for this purpose, not accessible from the outside, access only via a console. Only this isolation ensures that a failed test points to a real fault and not to a malfunction from outside.

Specified exit criteria force precision

For OEM-labeled products, the client specifies the test criteria. The test exit criteria were fixed because the product was running under a third-party label. This meant that the steps that had to be achieved were not a matter of negotiation, but a condition of delivery.

Such specifications generate weekly acceptance tests that show what works and what does not. They also force linguistic precision. For example: 99.99 percent of calls must be handled correctly. Is that the same as saying that 0.01 percent may fail?

With ten million calls, the difference quickly adds up. What initially sounds like splitting hairs becomes a tangible difference with large volumes. It is precisely such discussions that shape the way a test team reads and defines criteria.

Quality remains a team matter, even if there are test specialists

Testing does not scale as a one-man show. What started with one person in 2006 grew early on: the first employee joined in 2007, and today nine people take care of testing and quality assurance.

The importance of quality assurance was recognized right from the start, along with the effort it requires. Quality assurance does not happen on its own, it is not a sure-fire success. This early insight has supported the development of the team.

It is important to note that although there are test specialists, responsibility for quality lies with the whole team. Specialization does not replace shared responsibility, it complements it.

How knowledge transfer is organized in a test team

Knowledge flows best via fixed exchange formats, not randomly. Two rhythms run in parallel in the test team.

• Once a month, the whole team: Here, test-specific topics are examined, such as problems from the current month or new software that could be of interest. Everything to do with testing is put on the table.
• Individual meetings every two weeks: The QA manager sits down with each person in turn to clarify project-specific topics. Anything that comes up there can be included as an item in the next team meeting.

This ensures an exchange of information across the project teams. Responsibilities and the appropriate competencies are specifically assigned to individual people instead of floating around diffusely.

Certification creates a common language

ISTQB certifications serve above all to create a common vocabulary. If everyone talks about the same thing, testers don’t talk at cross purposes. The Foundation Level forms the basis for this, which new team members bring with them or catch up on.

The benefits work in both directions. New people join with a common conceptual framework. Those who have been with us for a while stay on the ball thanks to the recurring contact with newly certified people.

The other levels build on this, depending on the focus. Those who go in the direction of test management take a different path than someone with a focus on technical testing or test analysis. The aim is to certify people up to Advanced Level so that they can take this basis with them as a backpack.

In addition to building up expertise, contact with the testing world outside your own company is also important. Such acquaintances create an exchange that would not be possible internally alone.

The cloud shifts the focus to automation and data privacy

The move to the cloud takes test automation to a new level. With CI/CD, setting up clean pipelines becomes central to ensuring that tests run reliably and get rolling. This makes automation more important because it becomes a prerequisite for delivery.

At the same time, IT security and data privacy come to the fore. This issue is different in the cloud than with a system that runs on a dedicated server. Anyone relocating data and services must rethink security and protection.

Both topics, automation and security, are the current big issues. They show a pattern that runs through the entire setup: Each new platform brings its own challenges, and testing grows with them.