Fuzzy Testing

Fuzzing refers to the systematic testing of software with random or deliberately manipulated input data in order to uncover unexpected behavior such as crashes or security vulnerabilities. Classic fuzzing works without prior knowledge of the system. Customized fuzzing incorporates knowledge of input formats and protocols so that syntactically valid but boundary-value test data can be automatically generated in the millions.

Key Takeaways

• Fuzzing without format knowledge fails with validations: A randomly generated IBAN number hits the checksum in only about one in 8,000 cases, which keeps the fuzzer busy for days without ever penetrating the actual program logic.
• Customized fuzzing combines known syntax rules with controlled random variations so that only individual fields such as the transfer amount remain uncertain, while the rest remains valid and processable.
• Any program that processes external data must be able to cope with invalid, negative and mathematically boundary value inputs such as NaN or negative amounts, because such values can propagate virally through downstream systems.
• LLMs are not suitable as primary test data generators because they learn from existing patterns and thus miss precisely those unknown borderline cases that reveal errors in the first place.
• The official test suite for e-bill (X-bill) only includes a few dozen cases with 150 data fields, which makes automated test generation a requirement, not an option.

What is fuzzing?

Fuzzing is the systematic application of invalid, unexpected or purely random inputs to a program to see how it reacts. Does it crash? Does it hang up? Does it get out of sync?

The term was coined by American professor Bart Miller at the end of the 1980s. Miller worked from home via a simple telephone line and a modem on his mainframe computer. During the frequent thunderstorms in Wisconsin, electrical discharges disrupted the connection, and the bits and bytes arrived at the mainframe with changes.

He noticed that many programs could not cope at all with such garbled input. He turned this into a programming task for his students: generate random, meaninglessly jumbled inputs and use them to test common programs of the time. The result was sobering. The majority of UNIX programs at the time could be thrown out of sync in this way.

The scientific publication was initially rejected by the reviewers. Their argument: Why would anyone ever send such meaningless input to a program? At the end of the 1980s, Miller knew every user of his mainframe computer personally. It was only with the Internet and anonymous users that it became clear how real the danger was.

Why every program must be secured against malicious input

Any program that processes external data must not rely on this data being well-formed. This is the central consequence of the idea of fuzzing.

An attacker can poke random data into a system from a distance. If the system no longer responds at some point, there may be a gap that can be deliberately widened until the system can be manipulated. For this reason, fuzzing is now a must for any software that processes third-party data.

The topic is therefore at the interface between security and robustness. It is not just about security vulnerabilities, but also about ensuring that a system remains stable under unexpected inputs.

How a single input can blow up a system

Even small manipulations of valid data can have a considerable impact. The example of a bank transfer makes this tangible.

A normal transfer reads: from Andreas to Richie, one euro. In 99.9 percent of cases, this is exactly what happens. It gets interesting when targeted changes are made:

• Negative values: A transfer of minus one euro. If a script somewhere has forgotten that amounts could be negative, a hole is created.
• Infinity: An amount of an infinite number of euros. If this fails, the bank is also in trouble.
• Not a Number (NaN): A floating point special value. Everything that is offset against NaN becomes NaN itself. A single accepted NaN transfer could infect the day’s total, the balance sheet and the share price.
• Buffer overflow: An email address with 100,000 characters instead of the usual 256. If the destination buffer overflows, appended characters can be used to overwrite critical data.
• Command injection: Instead of a name, the field contains a smuggled-in command that deletes the database at the other end.

Banks are secured against such attackers. The question is whether this also applies to every FinTech and every intern code. Fuzzing starts with random data and pokes here and there until the system shows a weakness.

Any program that processes data from outside cannot rely on the data from outside being well-formed. It must be able to cope with erroneous, random and malicious data.

Andreas Zeller

Fuzzing is automation over hours, not minutes

Fuzzing is not manual, but automated over long periods of time. A fuzzer typically runs for 24 hours, often days, sometimes weeks, generating hundreds of thousands to millions of trials.

The advantage of pure randomness is that the fuzzer needs no prior knowledge and has no bias. It tries one input after another and also finds errors beyond the known attack catalogs. This is precisely why the chance of a hit is good, even if a run takes a long time.

A classic beginner’s mistake is a lack of logging. If you send random data for a week and when it crashes you no longer know which input triggered it, you are left empty-handed. Recording is mandatory.

A protected environment is just as important. Injected commands can trigger any behavior, including deleting your own file system or the very database in which the test data is stored. Never fuzz on a production system.

Why time bombs are particularly difficult to find

Not every error is immediately apparent. A malicious datum can be introduced at an early stage and only manifest itself many interactions later. This decoupling of cause and effect is called a time bomb.

Such errors are particularly difficult to troubleshoot because there is a large gap between the trigger and the visible effect. For an attacker, this is a perfidious tool: he infiltrates the date and waits until it causes damage weeks later.

This is why good fuzzing not only records the last interaction, but the entire sequence. Once you have a reproducible error sequence, there are automatic processes that reduce it. Out of a thousand interactions, they filter out the two or three that are actually relevant to the error.

Why pure chance fails with structured formats

Pure random data does not get very far with complex input formats. A bank immediately rejects random bytes because a bank transfer has a strict structure: a well-formed document, checksums, valid fields.

The IBAN shows the problem in numbers. The two check digits after the country code only match random data in about one in 100 cases. The two-letter country code matches about one in 676 cases. Together, you end up with around one hit per several thousand attempts for a valid IBAN alone.

And a bank transfer needs two valid account numbers, yours and mine. That multiplies to around 64 million combinations. A fuzzer spends a whole day just to generate a few valid IBANs. It never gets to the actual logic, such as checking the amount field.

The core: data packets are deliberately built to be robust against simple errors. The very mechanisms that protect a bank transfer over a thunderstorm-prone line also block the naive fuzzer.

Customized fuzzing marries chance with knowledge of the system

Customized fuzzing combines the randomness of fuzzing with conventional test methods that require knowledge of the system. Instead of generating everything at random, the fuzzer is taught the rules of the input format.

Tell the generator how an IBAN is structured: two letters, two check digits, then the account number. From now on, it will only generate data that follows this pattern. The probability of syntactically valid entries increases dramatically.

The decisive step is to generate syntactically valid data that nevertheless contains nonsense content. Comparable to a grammatically correct German sentence that makes no sense in terms of content. The data packet must first be accepted so that a manipulated value such as minus one euro arrives in the deeper processing at all.

This is how targeted control is created. You set up the transfer according to the rules, with sender, recipient, account numbers and amount, and then remove all checks in one or two places. For the amount, for example, you chase through arbitrary values. The result: a million transfers in one minute, almost all of which are correct, and a much more efficient test than with pure chance.

Why security and software testing fuzz differently

Security and classic software testing pursue different goals with fuzzing, and this shapes the method.

In security, you often don’t want to assume any knowledge about the system under attack. If you penetrate with insider knowledge, your performance counts for less. That’s why security sticks to the classic method: randomly inject everything, with as little prior knowledge as possible.

When testing your own software, the situation is reversed. You know the system and want to make it robust. So you actively use your knowledge of the format and possible vulnerabilities. If the IBAN check works reliably, you ignore it and point the fuzzer at a fresh, recently changed digit.

This is exactly where customized fuzzing helps. Existing test cases are calibrated to known problems and rarely find new ones. A generator that specifically targets a new feature quickly puts its finger on the wound.

Complex data formats are no longer testable without generation

The complexity of modern data formats exceeds what can be tested by hand. The electronic invoice in X-Rechnung format has been mandatory since this year and is a good example.

The X invoice recognizes around 150 different data fields, including constructions such as sales outside the EU. The official test suite, on the other hand, only includes a few dozen test cases. With so few handwritten invoices, it is impossible to cover 150 fields with their extreme values and negative values.

Nobody types in such quantities by hand any more. Without automated generation of test data, such extensive testing is not feasible.

Then there is the insecure channel. Electronic invoices arrive by email and anyone can claim to be someone else. Where once a human read the paper and only limited variants were possible, now fresh, barely matured implementations meet an open door.

The open challenges: Coverage, oracles and LLMs

Fuzzing does not solve everything. Three problems remain stubborn.

Coverage: Many special cases are not in the data format, but in the processing. The fact that a recipient is on a blacklist cannot be deduced from the input data, there is no field for this. A good fuzzer recognizes which branches have not yet been taken in the program run and concentrates on them. However, even good fuzzers only manage to take some of the branches.

The oracle problem: Has the system done the right thing? Common fuzzers only check for generic errors such as crashes or hangs. Many ignore an error message because security people are looking for the uncontrolled crash. As a tester, on the other hand, you want to check whether an error message is appropriate and whether the right thing is happening. Building good tests for this remains difficult.

Limitations of LLMs: Machine learning learns from existing data and generates similar results. This works for applications, code and images that others have already made. For test data, you often want exactly the opposite. If you ask an LLM to generate 100 more transfers from 100 existing ones, the new ones look like the old ones. Then you could just use the training data for testing.

LLMs are certainly useful for individual data fields, such as generating valid IBANs. However, for the creative task of generating something that has not yet been tested in this way, they are extremely unsuitable. This is where the creative tester is needed to find errors before an external attacker comes across them.