Modern IT Systems Are Never Finished, and Always Ready
Digitalization calls for software that is developed, tested and delivered without pause, which is why DevOps continuous testing has become a necessity rather than an option. Development, testing and delivery no longer run as separate activities, they run together, and the users take part in the development process.
DevOps is one answer to the demand for continuous development. In principle, a modern IT system is never finished. It has only ever reached a stage at which part of the user requirements are met. The requirements keep evolving, and so do the conditions around them. The challenge for DevOps is to set up systems so that they can be developed further all the time and still be ready for use at a defined level of quality. Users have to accept that the system keeps changing, yet they must also be able to rely on what has already been achieved while development goes on.
Digitalization and DevOps
Digitalization is a very abstract term for automating every area of human life. One formal definition reads: “The term digitalization generally refers to the changes in processes, objects and events that come with the increasing use of digital devices. In the original, narrower sense, it means creating digital representations of physical objects, events or analog media.”
The prerequisite is that all data about these areas of life is captured in digital form, as bits and bytes. Analog data processing interprets data as changing flows or changes of state. Digitalization records those states as bit patterns, which can be stored and processed again.
That gives us the chance to automate every business application and reshape it at will. The catch: automating an application is not a one-off project. Systems change regardless of how automated they are, and once they are automated, they only change faster. Once you have started automating an application, you have to keep at it. There is no way back. Along the way, the know-how needed to run the system by hand fades as well.
As promising as it sounds, digitalization is a pact with the devil. Automation also means dependency. Goethe’s Faust sends his regards.
Three Goals of Digitalizing an Organization
Digitalizing an organization comes down to three main goals:
- storing as much data as possible electronically,
- automating as many processes as possible,
- networking as many system actors as possible.
Data is all the information a system needs to do its job: customer data, employee data, order data, production data, inventory data. It can sit on electronic storage media, on paper or in employees’ heads. Digitalization aims to have all of it on electronic media, because that is the only way to make it available to everyone and everything.
Processes are the operational workflows: order processing, production planning and control, warehousing, shipping, invoicing. They are carried out by actors, and those actors can be people or machines. In a semi-automated process, some actors are people and some are machines. In a fully automated process, only machines are at work. Many processes can be fully automated, others only partly. One goal of digitalization is to automate as much as possible.
Within an organization, the actors communicate with each other and with actors outside their own system. They are networked. They exchange data through paper, in conversation or by electronic message. Devices, as actors, can also talk to each other directly, without a person in between, which is why we speak of the Internet of Things (IoT). A company is a network of communicating nodes, and one goal of digitalization is to connect as many of those nodes as possible, so that everyone and everything can exchange data with everyone and everything else.
In principle, digitalization lets a company react faster to changes in its environment. Both the data and the operations on it are recorded as bit patterns, and bit patterns are easy to overwrite. New ones appear within seconds and change what the data means and how it is processed. That gives us the chance to change our work processes constantly.
It all sounds so simple. All we have to do is capture the data electronically, program the processes and digitalize the communication. But everything that gets digitalized also has to be tested. Errors creep in, information is wrong or, worse still, data ends up in dangerous combinations. In that respect nothing has changed. Testing holds up the fast implementation of requirements, especially when testing and maintenance are separated from implementation. When a new law was passed, it used to take months until it was implemented in the software. That can no longer be justified. Testing and handover have to speed up.
The Road to Digitalization
First we need to understand exactly what is new about digitalization. Electronic data processing has been around for a long time, so have automated processes, and broadband data transmission goes without saying. And the fact that wireless communication plays such a big role in our everyday lives is not exactly new either. What may be new is the combination of all of this. In the end, a company is digitalized when all of its data is stored electronically, all of its work processes run fully automatically and all of its employees are connected to each other and to their devices.
Is it really sensible to go that far, or should we stop halfway? How far do we want to take digitalization? Here, too, testing sets the limits. We can only store, automate and transmit electronically as much as we can vouch for. The data we store must not be corrupted. The questions we ask the data management system must be answered correctly. The processes we trigger must run according to the applicable regulations and business rules. And the messages we send must not be altered or intercepted on the way. Everything we digitalize has to be tested and it has to be secured.
Digitalization brings an enormous testing effort with it, and that effort lands on us. It starts with testing the data and continues with testing the processes and testing the data transfer. The effort is only manageable if we work systematically and automate the testing itself as far as possible.
How to Measure the Degree of Digitalization
The degree of digitalization can be understood as the sum of three ratios:
- electronically stored data relative to all business-relevant data,
- automated processes relative to all business processes,
- networked system nodes relative to all system nodes.
This measure may not mean much from a business point of view, but it is a good starting target for IT. True digitalization, if there is such a thing, means drastic changes to existing products and business models, in other words new systems. Whether users are ready for that, or have any choice, is an open question.
Electronic Storage of All Business Data
Since cloud computing arrived, nobody doubts that we can physically store all our data. Cloud providers offer practically unlimited storage. The question is whether we store useful data or just junk. The data also has to be correct; invalid values must not creep in. That gives two criteria for the data:
- it is useful,
- it is correct.
To find out whether data is useful, test it with every business application, including the query and reporting systems. That shows which data is used and which is not. The goal of this test is an inventory of all the data that is actually in use. Whether the applications themselves are useful is for the users to decide.
To find out whether data is correct, check it against the rules that define correctness. There are two ways to do that. The first is to compare data with other data: new data against old data, or data in one application against the same data in another. Wherever the data is used, its value ranges should be identical.
The second is to compare actual values with expected values. The simplest form of this check is a visual inspection of the data by a subject matter expert. That kind of check, however, is slow and prone to errors. The most elegant form is an automated check of values against rules, known as assertions. An assertion states which values a data element should have. It can be defined as a value range, a set of alternative values, a calculated value or a relation to other values, among other things. The purpose is to find invalid values and sort them out.
This automated approach does require rules for every data element. Neither approach guarantees that the data is 100 percent correct, but both catch most of the invalid data.
The release process may speed up, but the quality of the product must not be sacrificed for it. Quality assurance for electronically stored data and automated processes is a precondition for digitalizing a business, and it only works if the users are involved. They have to accompany the release process and accept each step right away. That means representatives of the end users are on board from the very beginning. Without them, quality cannot be secured.
Automating Business Processes
Many processes in organizations are already automated, some of them by very old software, the so-called legacy systems. Where processes are highly standardized, the software solutions come packaged as standard systems. That applies to a large share of business applications, and more and more production, logistics and traffic control systems are covered by standard products as well.
Off-the-shelf packages are the preferred choice for process automation. They cost less, they are more reliable and easier to maintain. Above all, the vendor vouches for their quality and keeps developing them. Standard systems have been tested many times over and reach a level of reliability that custom solutions rarely achieve.
Even so, many users still feel compelled to knit their own custom solutions. They can start on a greenfield and build everything from scratch, or they can start from ready-made components and assemble them into a whole, developing any missing pieces themselves. What matters is that new application systems consist of as many prefabricated, pretested components or services as possible.
A service-oriented architecture with microservices is a promising way to get there. Despite all progress in software technology, new development still comes with enormous costs and risks, and testing automated processes is enormously expensive. To be considered trustworthy, every possible path through a process has to be tested with every possible state. Without automation, it can take years to test large, complex processes adequately.
Test automation speeds up testing, but automated testing has its limits too. A person still has to assign the test data, and only a subject matter expert can specify the expected results. Once the test of a new application has proven itself, it can be mutated and repeated automatically, but it has to get that far first.
Why Testing Is the Main Obstacle to Process Automation
That is why testing remains the main obstacle to automating processes. The software required is quickly written or generated, but it can take a long time and a lot of money before it is confirmed in every respect as a complete and reliable solution. As long as people develop software, it will contain errors, and faulty software makes it risky to automate business processes. Testing reduces the risks of automated processes, just as it reduces the risks of electronically stored data.
Static analysis can also tell us something about the quality of new software, but only dynamic testing shows whether the automated process really behaves the way it should. Anything that delivers this proof faster and more cheaply is welcome. Still, the fastest and cheapest solution remains software that has already proven itself, either as a whole or as individual building blocks.
Networking All Nodes
The third pillar of digitalization, next to electronically stored data and automated processes, is data exchange everywhere. Information has to be transmitted everywhere and at all times: from the sensors on devices, from the monitors that watch the automated processes, from the data stores and from the people who capture data. Every element of a system, people and machines alike, has to be networked. There is human-to-human, human-to-machine and machine-to-machine communication (see also Internet of Things).
That communication, however, has to work without fault. Software can monitor itself, but only if it can send and receive messages. The exchange of digital messages is the glue that holds a digital environment together, and it has to be free of interference and secure.
Secure communication means that everything sent is also received. Nothing sent may be intercepted, copied or listened in on along the way. Messages have to stay sealed, and only the intended recipient may open them. That applies to people and devices alike.
More and more, devices and software components send messages to each other. The sender, whether a person, a hardware device or a software service, has to be sure that the data arrives at its destination unchanged. The recipient has to be sure that the data was not misused, manipulated or intercepted on the way. Only a secure communication network can guarantee that. This is why data transfer has to be tested as well.
For that we need tools that generate and send messages, tools that receive and validate them, and tools that trace messages on their way through the network. These tools have to detect when and where messages are intercepted. Communication testing makes sure that data transfer works properly.
Monitoring data transfer plays a special role when testing a digitalized company, because it is not only about whether messages are delivered correctly, but also about whether they are delivered securely. Attacks on the communication network have to be simulated to see how the network reacts. Interception, redirection and unauthorized use of messages must be ruled out. The network can only be considered secure if every one of these simulated attacks is detected and blocked. That includes attempts by unauthorized people to break into the network, and of course every possible way in has to be tested. Secure message delivery is a precondition for digitalization.
All of this shows how much effort goes into testing communication. Anyone who is serious about digitalization has to be prepared to pay a high price for quality. Otherwise they will always be vulnerable.
Digitalization Requires DevOps Continuous Testing
To meet the challenge of digitalization, we need to look at how testing can keep up.
With so much to test in so little time, test automation has to be pushed forward. That alone is not enough, though. The users have to be part of testing. They cannot wait on the sidelines until testing is finally declared complete. In fact, the users are responsible for testing, because only the end users can decide whether testing is sufficient for their purposes. They are the ones who carry the risk if something goes wrong. Experts advise them, of course, but only the end user can decide whether a product is ready for use.
The user is also responsible for checking the data. They need to know what state the data is in and whether that state is good enough for operations. This means the data has to be monitored continuously by automated tools, and even the smallest deviation from the expected state has to be reported. The data is monitored for as long as it is in use. The same goes for the automated processes.
Because the processes keep evolving, they have to be tested again and again. Testing never ends. After every change, however small, the software has to be retested. Like the state of the data, the process flows have to be monitored at all times and every deviation from the expected behavior recorded. Finally, every communication has to be traced and every message checked. Messages that deviate from the norm are set aside for closer inspection, and the content of every message is checked against its specification, with any deviation reported.
Digitalization Requires Products, Not Projects
Anyone who enters the digitalization arena has to let go of project thinking. Projects are one-off activities, limited in time and budget, with a beginning and an end. In a digitalized world there should be no time-limited projects anymore, only products: systems that are developed, tested and delivered on an ongoing basis.
Testing becomes part of the operational handover. Until now, development, testing and acceptance were always kept apart. There were walls between these activities. In DevOps, these activities flow into one another.
Exploratory testing and everyday use can hardly be told apart. The end user is tester and user at the same time. When errors occur, they try to work around them. The errors are reported in any case, but they must not stop the system from going live, as long as they do not prevent further operation.
DevOps Promises to Be the Solution
For this problem, DevOps promises a solution. More and more software systems chase a moving target, so they must never be seen as finished products. They stay in development for their entire lifetime. They only ever cover a subset of the requirements, and that subset keeps shrinking over time. At first, a system might cover more than 90 percent of the digitalization requirements, but over time that share drops to 80 percent and lower.
Software systems have always lost value over time. Belady and Lehman recognized this back in the 1970s and captured it in their laws of software evolution. A software system represents a particular application, travel booking, for example. It becomes obsolete to the extent that it no longer keeps up with that application: the travel provider wants to offer more and more services that the software does not cover. The software falls further and further behind and is worth less and less to its users.
When Belady and Lehman published their laws of software evolution, conventional change management could still handle the rate of change. New requirements stayed below 5 percent of all requirements. Users filed change requests for requirements that were missing or poorly covered. These were prioritized and planned into upcoming releases, and sooner or later they were implemented. Later, the annual rate of change for more and more systems climbed above 10 percent. Conventional change management soon reached its limits. The functionality of the software fell further and further behind, and the pressure on IT kept growing.
Now, with digitalization, requirements change at rates of more than 20 percent. Conventional change management is completely overwhelmed. For more and more applications, a lengthy test procedure before release is no longer an option. They have to be released earlier, even if they have not been tested enough, and the remaining errors have to be fixed in production. The reasoning is simple: the damage caused by those errors is smaller than the damage caused by a delayed delivery.
What matters is that the delivered systems stay under the control of product management. All errors and other defects in the system are logged and tracked. Sooner or later they are fixed, but they must never be the reason not to ship a release. Continuous delivery comes first.
An Example of Continuous Delivery
One application that practices continuous delivery is the payroll system of the Austrian states of Upper Austria and Burgenland. The clerks in the HR department are also responsible for testing and delivery there. The system, with a VisualAge front end and a PL/I back end, is being rebuilt in Java piece by piece. The old environment doubles as the test environment.
For the past year and a half, the front end has been reimplemented in Java one component at a time. Each component is taken out of the old system and rewritten in Java. It then goes to the responsible clerks, who compare the behavior of the new component with that of the old one.
The clerks test the new Java components alongside their daily work. If they find deviations, they report them to the developers, who fix them right away. Until then, the end users keep working with the old component. The current system is a mix of new and old components.
By the beginning of 2017, a third of the use cases had been replaced this way. All front-end components are due to be in Java by the end of 2018. Until then, VisualAge and Java components run side by side in the same environment. What counts is that the service continues without interruption.
Summary
Digitalization means that a company’s application systems are in constant change. They have to be developed further all the time to keep pace with the real world. Development, testing and delivery happen side by side, and that calls for different forms of organization. Not every application is suited to a DevOps organization, but for more and more front-office applications, time to market takes priority over quality. For those applications, DevOps is the right solution.
Frequently Asked Questions
How can we measure the extent to which a company has been digitized?
The degree of digitization consists of three components: electronically stored data as a proportion of all business-relevant data, automated processes as a proportion of all processes, and networked system nodes as a proportion of all nodes. From a business management perspective, this metric is of limited value, but it is useful as a target for IT. A business would only be fully digitized once all three components have been fully implemented.
Why is there no turning back when it comes to application automation?
Anyone who automates an application will, over time, lose the know-how to operate it manually. At the same time, automated systems change faster than before, so you have to stay on top of them. Automation therefore also means dependency. Automating an application is not a one-time project, but a commitment to ongoing development, testing, and deployment.
How do you determine whether stored operational data is usable?
Two criteria apply: The data must be useful and accurate. Usefulness is demonstrated by testing the data with all operational applications, including query and reporting systems; the result is a list of the data actually used. Accuracy is verified by comparing the data with other data sets or automatically against rules that define value ranges, alternative values, calculated values, or relationships. Neither approach provides a 100% guarantee.
For process automation, is off-the-shelf software or an in-house development the better option?
In 2017, off-the-shelf software packages were considered the preferred solution: more affordable, more reliable, easier to maintain, extensively tested, and backed by the vendor’s guarantee of quality and ongoing development. New in-house developments remained associated with high costs and risks. Those who choose to build their own solutions should use as many off-the-shelf and pre-tested components or services as possible. A service-oriented architecture with microservices was a promising approach for this.
Can test automation fully offset the increasing testing effort?
No. Test automation speeds up repetitions: Once a test has been validated, it can be modified and run again. In 2017, however, it was still the case that humans had to assign the test data, and only subject matter experts could specify the expected results. Static analysis reveals quality characteristics of the software, but only dynamic testing can determine whether an automated process truly behaves as required.
Should software that hasn’t been sufficiently tested be released?
In 2017, when requirement change rates were very high, this was precisely the approach advocated: The damage caused by a delayed release outweighs the damage caused by remaining defects. The condition is that all bugs and defects are documented and monitored, will be fixed later, and product management retains control over the delivered systems. They must not be allowed to hold up a release.
Is DevOps the right organizational model for every application?
No. Not all applications are suitable for it. In 2017, DevOps was primarily recommended for front-office applications where delivery speed takes precedence over quality. In these cases, development, testing, and delivery flow seamlessly together rather than being separated by silos, and the end user serves as both tester and user. A prerequisite is an organization that replaces project-based thinking with continuous product development.
