“There’s a budget again. It just needs to say AI on it.” - Richard Seidl
Test data management is currently experiencing a renaissance. Driven, of course, by AI and the possibilities it offers: better, more accurate test data, simple generation, effortless management across system boundaries. Ideas are bubbling over as to what will be possible … or would be … or well, maybe … hm.
There are a few classic challenges in software development. But while we have halfway solved those around releases (pipelines) and test environments (cloud), for example, test data is another matter. Especially in application landscapes with many different systems and data repositories, test data initiatives can quickly get out of hand. It’s not easy either, because the challenges are manifold:
- Data, data, data - there is simply a lot of it. Countless tables and fields and thousands, sometimes even millions, of data records. All of which are cross-linked and contain import/export data in directories.
- Compatibility - Each system has its own schema, its own data organization, which does not necessarily fit together. A clean data architecture across system boundaries is not easy. Different responsibilities then add further complexity.
- Synthetic vs. real data - Synthetic test data helps me immensely with my structured test cases (limit values, ECs, etc.) - but it is also not a reality.
- And as soon as you get to real data with all its peculiarities, errors, etc., data protection is just around the corner: anonymization and pseudonymization require a lot of energy and time.
- Do we want to go one better? Then our test data must also map historicization and time travel. Yeah - jackpot.
The AI will fix it, right?
But no problem at all. Just throw all the rules, requirements etc. into an AI and then we generate our test data across all systems and almost in real time - a dream. But I’m pretty sure it won’t work that easily. There are already some very nice approaches to generating and managing test data. However, my observation is that this is often a case of treating symptoms. I would rather ask two other questions.
What data do I really need? (And of these: which do I really need?) Just because we can store everything doesn’t mean we have to. It’s so easy to add a field to a table - but the effects can be dramatic. So: Just leave it out and delete all structures that are not needed. A (test) data radical cure!
Do I have a suitable data architecture? With cross-system architectures, I see many interfaces and dependencies, but hardly an overall picture of the data content, data flows and where which data is stored in a meaningful way. So that they are not stored redundantly and circularly. It’s slowly becoming a shoe.
And if the data is halfway decent, then I’ll think about something with AI 😉
Frequently Asked Questions
Why is test data management still considered an unsolved problem, even though releases and test environments have long been automated?
There are pipelines for releases and the cloud for test environments: these staples of software development have been reasonably well addressed. Test data, however, is not. It is tied to legacy application landscapes with many systems and separate data repositories, and that’s exactly where test data initiatives quickly spiral out of control. The problem lies less in a technical tool than in the structure of the data itself.
What makes test data in large system landscapes so complex?
The sheer volume and its interdependencies. There are countless tables and fields, plus thousands to millions of data records that are linked in every direction, supplemented by import and export data scattered across various directories. Add to that compatibility: Each system comes with its own schema and data organization, and differing areas of responsibility further increase the complexity.
When do synthetic test data reach their limits?
Synthetic test data are extremely helpful for structured test cases, such as boundary values or equivalence partitions. However, it does not reflect reality. It does not capture the nuances and errors found in production data. Anyone who wants to test these effects cannot avoid using real data—and that entails a whole different level of effort.
Why isn’t using real data in testing a simple solution?
As soon as real data comes into play, data privacy becomes an issue. Anonymization and pseudonymization require a great deal of time and effort, and this effort is not a one-time cost. The gain in realism therefore comes at the cost of ongoing additional effort, which should be factored into the planning of a test data initiative.
Why don’t test data generation tools often solve the problem?
Because they treat the symptoms. There are certainly promising approaches to generating and managing test data, but they address the wrong issue as long as two upstream questions remain unanswered: What data do you really need, and does a suitable data architecture even exist? Only after that is it worth looking into generation, including AI.
What are the benefits of consistently removing data structures?
Just because everything can be stored doesn’t mean it should be stored. A field in a table can be added quickly, but the consequences can be dramatic: more dependencies, more test data to populate, and more maintenance effort. A radical overhaul—in which all unnecessary structures are eliminated—tackles the test data problem at its root.
How can you tell if a robust data architecture is missing?
By the presence of many interfaces and dependencies combined with a lack of an overall picture. When no one can say which data stores exist, how data flows, or where specific data should logically be stored, redundant and circular data stores emerge. It is precisely this structure that makes cross-system test data so tedious later on.
