Kubernetes load testing is about whether a containerized application meets expectations for response time and scaling under load. Autoscaling and load balancing change every run, so a reliable test needs a concrete risk question and a lab setup with both scaled back. Development, operations, and QA have to pool metrics, logs, and application knowledge; no department manages that alone.
Key Takeaways
- A load test done for the sake of doing one is potentially wasted time, says Stephan Schwarz, because no decision about a risk comes out of it.
- Every system has a bottleneck, and fixing one only reveals the next, so Kubernetes load testing has to show which one is currently limiting the workload.
- Kubernetes performance testing rarely points to CPU or RAM; the real culprits have been faulty caching, an external OIDC provider, database servers, session handling, and firewalls.
- Many teams set no requests and limits for their containers, or set them poorly, because without a load test nobody knows how the software behaves under load.
- Load tests in the CI/CD pipeline are the icing on the cake. Infrastructure-heavy projects can start by recording latency metrics in the end-to-end smoke test they already run.
Kubernetes load testing starts with a question
Kubernetes load testing pays off only when the team knows beforehand which question the test is supposed to answer. Stephan Schwarz calls a load test that is run so that a load test has been run “potentially wasted time”. A decision should follow from the result. Otherwise a number sits in a report.
The questions Stephan has in mind are about risk. Do we accept it? How high is it, and what does it consist of? Can we mitigate it, and how? Once a mitigation is defined, the next load test checks whether it holds. That chain turns a test run into a basis for decisions.
What does performance actually mean?
Performance means that expectations are met. They differ from company to company, and an organization often notices only during its first load test that nobody internally means the same thing. Everyone says the system “has to be performant”, but nobody has written down what that means in seconds or error rates.
Stephan offers an example. Is a fast error message better for the customer than waiting four, five or ten seconds for a valid response? The answer depends on the business. Right behind it sits the second question, namely which risk the company accepts on purpose.
Why do so few teams run load tests?
Because a single team rarely builds the momentum on its own. Stephan Schwarz sees the same picture in most client projects. The development department wants to test, or QA says “we really should do this”, and then nothing happens. A load test that holds up needs expertise from the operations team, the platform team and sometimes a go from the C-level.
Then there is the effort. Build a test environment, sort out dependencies, provision load generators, interpret the results with several departments. That is not an afternoon’s work, and Stephan says openly that he understands why it is rarely done.
The topic typically gains traction after a postmortem. Something broke, customers noticed, and suddenly load testing sits in the top 3 action items. Starting earlier would be better.
Which test environment fits Kubernetes load testing?
The environment follows from the question. A team that wants to know how a containerized application scales under CPU load builds a lab setup: either the container alone or a dedicated Kubernetes cluster with nothing else running on it. The next question concerns infrastructure dependencies. Does the test use the database on the production cluster or its own instance in the lab cluster?
Testing against production is legitimate too. Some dependencies exist only in prod for good reasons and cannot be rebuilt in staging. A team that goes this way names the risks and accepts them knowingly.
There is a trade-off nobody escapes. The more lab-like the setup, the less it has to do with what will run later.
Kubernetes also makes repeatability hard. Stephan recalls the quote attributed to Einstein (which he probably never said) about doing the same thing twice and expecting the same result. In a Kubernetes cluster, even doing the same thing twice is difficult, because load balancers and autoscaling keep intervening. His recommendation for the start is to scale back autoscaling and load balancing as far as possible. Unless scaling itself is the question, then it stays in.
The load generator has to scale with the workload
The requirements for a load generator are modest. The tool should speak the application’s protocols, emit metrics, run the test and scale. In Kubernetes, the last point becomes mandatory, because the workload under test scales up itself. A generator that does not keep pace becomes the limiting factor and ends up measuring itself.
There are plenty of tools; Stephan names Artillery, Locust and k6. Clients usually have an observability stack that brings logs and metrics into a Grafana dashboard. A ready-to-use load generator farm is almost always missing; vanilla Kubernetes does not ship one.
Stephan’s first load test in 2018 produced miserable numbers. He wondered why his service was so bad until he realized he was testing from his own laptop, which was never built to open that many sessions in parallel. A load test always tests the whole construct, generator included.
What are typical causes of performance problems in Kubernetes?
The causes vary widely, and raw resources are rarely among them. Adding RAM or another node, the old recipe, helps in very few cases according to Stephan’s experience. In Kubernetes performance testing, the findings usually sit elsewhere.
What he has seen: caching that was not implemented properly. An external dependency nobody had on their radar, an externally hosted OIDC provider for authentication that was never designed for the load. The system itself would have coped, the login did not. Add database servers, session handling and firewalls to the list.
This is exactly why a load test needs every bit of know-how the company has. Nobody knows in advance where it will jam, and no single department can run the search alone.
There is always a bottleneck
Every system has a bottleneck. A team that has not found it yet does not have one fewer. And once it is fixed, the next one appears, like a hydra. For Stephan, that is the normal state.
What matters is knowing which one is limiting the system right now. Then the team can react when more customers arrive, plan hardware for next year or provision more in the cloud in time. Without that finding, an organization is waiting for something to happen.
Dev and ops were socialized differently
Load tests often fail at the boundary between development and operations, and Stephan Schwarz explains that with socialization. He comes from operations himself, from a local data center where the top rule was: no changes, whatever you do. Every change threatens the SLA, and outages are the worst thing for an ops person. “Never touch a running system” is a saying for a reason. For developers the opposite holds. Their job is features, and features are changes.
Silo thinking adds to it. Stephan has heard the sentence: we only do development here, and if the application does not work, that is an operations problem. He asks back whether “operations” means the ops team or the business as a whole. In the second reading, the sentence is correct.
That is why a load test is a team effort. Operations has the metrics and the logs, developers know the application, QA brings the test questions. When all three sit at one table, the part Stephan is passionate about begins. Ops shows the data from the last run, QA explains what the test did, developers find the spot in the code. The optimizations that come out of this reach further than two extra CPUs.
Nor is a load test a one-off project. One run produces a hunch about the cause, the mitigation has to prove that it works, so the team runs the test again. In the end the whole company understands its product better, management included. It may learn that 10 percent slower responses are fine for one customer because that customer caches anyway.
Do load tests belong in the CI/CD pipeline?
Stephan calls integration into the development pipeline the icing on the cake, and he does not believe every team needs it. Infrastructure-heavy projects with many dependencies are the candidates. They run an end-to-end smoke test anyway; if that test also captures the metrics and records them with the result, the team sees after every infrastructure change whether latency goes up or down.
Something simpler matters more to Stephan: that teams in Kubernetes environments define adequate requests and limits for their containers at all. In many projects they are missing or not set sensibly. He does not hold that against anyone. A development team cannot know how its software behaves under load as long as nobody has tested it. And the operations team has the metrics and the methods, but little connection to the application.
Getting started means building momentum
The first step is a list of questions. Engineers, QA engineers and operations engineers usually have a feeling for where things might jam. Those hunches become questions and named risks, along the lines of “I could imagine this is our limiting factor”.
With that list, the team goes to management and makes its case. The argument is ready: we do not know where our bottleneck is, so we are waiting for something to happen.
“If I don’t test my product, then my customer will.”
(Stephan Schwarz)
Management may decide otherwise. If SLAs and contracts allow it, a company can carry the risk. Stephan is fine with that, as long as it is a conscious decision. Firing requests at a test instance without knowing why answers no question at all. His advice: make risks transparent, make questions transparent, talk across departments and build the momentum that way. In his experience there is no other way.
