Kubernetes went from novelty to default in about a decade. As of 2024, over 60% of enterprises had adopted it as their de facto container orchestration solution, and that number is projected to exceed 90% by the end of 2027.
Kubernetes performance testing validates how containerized workloads behave under realistic load before release. Cloud-native stacks need two things that traditional tooling keeps apart: realistic traffic, and automated configuration tuning. Speedscale replays sanitized production traffic. Akamas learns the optimal configuration from each test run. Together they form a closed loop.
This article covers why traditional performance engineering breaks on Kubernetes, what each platform contributes, and how the two combine inside a CI/CD pipeline.
Why Kubernetes performance testing breaks traditional methods
Modern application architecture runs on microservices and container orchestration. The methodologies teams used to tune monoliths struggle badly when pointed at containerized applications. Going from monolith to microservices simply adds moving parts.
A single microservice usually depends on other services. Those dependency chains are difficult to map and harder to test completely. A bottleneck in one service cascades and surfaces as an issue in a service that looks unrelated. The real root cause gets buried.
Four problems account for most of the pain:
| Problem | What it does to your test results |
|---|---|
| Kubernetes configuration surface | Pod requests, limits, autoscaling policies and runtime flags interact, so a fix in one layer creates a regression in another |
| Cloud elasticity | Infrastructure changes underneath a test run, so response time and throughput are never repeatable |
| Test environment fidelity | Mocks drift from reality, and copying production data creates privacy exposure |
| Synthetic workloads | A best-guess traffic model does not reproduce what production actually does |
Kubernetes-specific performance pitfalls
Kubernetes carries its own set of performance pitfalls, and they hit reliability, user experience and cost at the same time. We covered the full list in our top 10 Kubernetes performance pitfalls. The most common ones:
- Pod CPU requests and limits set from a guess rather than from observed behaviour
- Node resources, especially memory, exhausted before the autoscaler reacts
- Cluster autoscaling that provisions the wrong capacity at the wrong time
- Application runtime settings left at their defaults
That last one deserves its own paragraph. Java applications on Kubernetes are particularly awkward, because the JVM exposes a very large configuration surface and its defaults were never designed for containers. Akamas research found that Java on Kubernetes accounts for over 50% of cloud costs according to 62% of the organisations surveyed (The State of Java on Kubernetes 2026). Node.js applications hit similar trouble for a different reason: their runtime characteristics are unusual and, in many cases, poorly documented. We go deeper on both in application runtime optimization.
Balancing performance, reliability and cost across all of that by hand is not realistic.
The cloud is not magic: elasticity destroys your baseline
Plenty of people still believe the cloud has unlimited resources. It can certainly auto-scale on demand, and for performance testing that cuts both ways. A poorly written application will consume every resource you hand it, and the bill arrives later.
Elasticity is excellent for uptime. It is terrible for baselines. Consistent, reliable test results become hard to get, because the infrastructure under a test run can change mid-test.
Multi-tenant interference makes it worse. Noisy neighbours skew accuracy, mask real issues, and produce false positives during peak testing windows. Shared, dynamically allocated resources are the opposite of the controlled, repeatable conditions that performance testing has always needed. That gap is why predicting real-world performance, or justifying an optimization to a finance team, gets so difficult.
Test environment bottlenecks in microservices testing
Environments have always been the tar pit of performance testing projects. Microservices made it worse. Building realistic, isolated, cost-effective test environments that genuinely replicate production is often the single largest bottleneck in the process, and it drags on the whole agile cycle.
Consider one service with several dependencies. Testing it in isolation means mocking those dependencies. Integration testing means running the entire chain. Traditional mocks rarely reflect the diversity of real data and have to be maintained forever. Results get distorted, and the maintenance effort is time-consuming and error-prone.
Using production data instead is a terrible idea unless it has been obfuscated. PII, financial records and health information carry serious privacy and compliance exposure. Full production datasets are often too large to copy. Periodic database dumps go stale and miss schema changes.
The outcome is predictable. Teams cannot stand up production-like environments quickly, so they compromise, and the compromises produce unreliable results and a higher risk of production failures.
Where traditional load testing tools fall short
Traditional load testing tools have to change to meet what cloud-native applications demand. Two limitations dominate.
| Limitation | Consequence |
|---|---|
| Scripted test cases | Tools like JMeter need scripting and plugins before they return anything useful, and many teams cannot fund the maintenance |
| Synthetic workload models | A workload built from anecdotal input from the test team or ops does not capture what production actually receives |
Run those tests and you still learn very little about how the application will behave in the real world.
Akamas: AI-driven tuning for Kubernetes performance testing
Akamas moves configuration decisions off trial and error and onto data. It searches the configuration space and returns the settings that deliver the best performance and efficiency, in a fraction of the time manual tuning takes.
The platform supports two scenarios:
- Akamas Offline optimizes in staging by automating performance tuning against a load test
- Akamas Insights optimizes directly in production, surfacing opportunities and issuing ready-to-apply recommendations for Kubernetes clusters and workloads
This article focuses on Akamas Offline and the Speedscale integration.
At its core, Akamas uses patented reinforcement learning. Long before large language models and the “agentic” labelling of everything, Akamas was already applying advanced AI to converge on strong configurations within hours of the first experiment.
It is worth remembering what the manual version looked like. Performance engineers gathered in a lab with a production-like environment. They ran load tests by hand, read the results, changed one setting, and repeated. Weeks or months would pass. Now consider a Java application with 500+ JVM options, plus pod resources and scaling policies, for every microservice. Manual optimization is not slow. It is arithmetically impossible.
Unlike code profiling, Akamas needs no code changes and no in-process agents. It reads KPIs from the observability tooling you already run, integrating with Dynatrace, Datadog and open source tools such as Prometheus. Reading telemetry rather than instrumenting the application keeps overhead minimal and leaves application behaviour undistorted, which matters when that behaviour is the training signal.
What “autonomous” actually means here
Autonomous means the optimization process runs against a goal you set. “Improve throughput.” “Reduce cost while holding my latency SLO.”
Akamas applies or recommends configuration changes, evaluates the result, and converges on the configuration that meets the goal. Two operating modes are available:
- Fully autonomous: changes are applied directly
- Semi-autonomous: changes are issued as recommendations that a human approves
What Akamas optimizes
Optimization here means adjusting parameters across the whole stack rather than one layer of it.
| Layer | Examples |
|---|---|
| Kubernetes | Pod CPU and memory requests and limits, replica counts, HPA settings |
| Application runtime | JVM heap and garbage collection flags, Node.js runtime options |
| Infrastructure | Cloud instance type and size |
Full-stack coverage is the point. Akamas tunes infrastructure-level and application-level settings in the same experiment, which is what makes the resulting changes reliable rather than locally optimal. We explain why single-layer tuning fails in full-stack optimization: more than just Kubernetes.
Goal-oriented optimization with SLOs and guardrails
Optimization is goal-oriented, so users define both the objective and the constraints. Objectives can be maximum application throughput, minimum cloud resource usage, or a balance across throughput, response time, resilience and cost. Tying the objective to a business outcome is what pulls performance engineering out of purely technical metrics.
Guardrails matter as much as goals. An AI optimizing without boundaries will happily cut cost and destroy latency. Akamas learns continuously from system behaviour and rejects changes that would harm performance, cost or availability. Defining SLOs constrains the search to configurations inside acceptable performance boundaries, and safety policies prevent failures reaching production.
Performance experiment orchestration
Akamas runs performance experiments in a structured loop:
- Apply a configuration
- Trigger a test
- Collect metrics
- Evaluate the result against the goal
- Choose the next configuration to try
Akamas acts as the brain of that loop, not as the load generator. Existing load testing investments still generate the workload, so teams keep best-of-breed tools rather than being locked into one methodology. This is exactly where Speedscale fits.
Speedscale: production traffic replay and service virtualization
Speedscale takes a different route to API and load testing. It isolates individual microservices so each component can be tested and tuned before release. For a complex application, tuning component by component is how you make sure the end-to-end experience rests on something solid.
Speedscale runs a three-part process:
- Observe. Capture API requests and responses to build realistic data sets
- Transform. Prepare that data for replay in another environment
- Replay. Execute load tests and mocks against new versions of the code
Capturing and replaying sanitized production traffic brings the messy reality of production into staging. You get deep insight into behaviour under realistic conditions, without risking live systems or exposing sensitive data.
Realistic environment replication and service virtualization
Traditional load testing tools cannot easily replicate the responses of a complex production environment and its dependencies. Doing that requires service virtualization, which Speedscale provides natively. It generates mock services automatically from the backend requests and responses captured during observation.
Privacy is handled in the same step. Data sanitization options include PII masking, header filtering and custom scrubbers driven by regex or JSONPath rules.
These mocks behave like production because they came from production. They refresh automatically from running systems and need no application changes and no network re-routing, which removes the maintenance burden and the result distortion that traditional mocks introduce.
What Speedscale builds is closer to an environment replication system: it simulates the active parts of the runtime environment, including a remote Kubernetes cluster, Elastic Container Service (ECS), or a virtual machine environment.
Mocking also decouples application code from downstream systems such as large language models and databases. That means testing at scale without hitting rate limits or paying per-transaction costs for real services. The full service virtualization toolkit is there, including artificial response delays so you can see how a slow external dependency ripples through the wider application. Load shaping lets you multiply captured traffic by 2x or 10x of expected production load.
Solving the environment problem removes a large slice of the setup and configuration time from any load testing project.
Performance insights and reporting
The Speedscale Traffic Viewer visualizes traffic flows in real time. You can inspect API request and response payloads, headers and metadata, and filter or search to find specific patterns and anomalies.
After a replay, reports cover the SRE golden signals:
- Errors: a dedicated dashboard
- Latency: average, 95th and 99th percentile
- Throughput: requests per second or per minute
- Saturation: CPU and memory utilization from the environment
The Traffic Viewer also includes a Graphical Service Map built from actual traffic rather than from a diagram someone drew last year. It shows dependencies, bottlenecks and data flow, which makes failure points easier to pinpoint.
Speedscale integrates with the major observability vendors in stand-alone mode too, so both platforms can read from the same telemetry stack.
How the Akamas and Speedscale integration works
Together the two products cover both halves of the problem. Speedscale supplies realistic input. Akamas decides what to change.
| Responsibility | Speedscale | Akamas |
|---|---|---|
| Generate realistic workload | Replays sanitized production traffic | Triggers the test as part of the loop |
| Provide dependencies | Auto-generated service mocks, refreshed from running systems | Not applicable |
| Measure the run | Latency, throughput, error rate, resource utilization | Ingests KPIs from observability tooling |
| Decide the next configuration | Not applicable | Reinforcement learning selects the next candidate |
| Apply the result | Test case versioning, rollback safeguards | Applies or recommends the optimal configuration |
Speedscale generates realistic test cases and high-fidelity mocks from real traffic, which removes the manual maintenance that normally makes realistic dependencies unaffordable. Akamas then orchestrates the experiments, learns from each run, and recommends optimal configurations for Kubernetes resources, JVMs, Node.js applications and other components.
The dependency runs in one direction and it matters. Akamas learns from measured performance data, so the quality of the learning is capped by the quality of the workload. Replayed production traffic raises that ceiling, because latency, throughput, error rates and resource utilization are all measured under conditions that mirror live production.
Kubernetes performance testing in CI/CD
Both products run inside a pipeline, which is where the loop actually closes:
- Every code change is validated against realistic production traffic
- Developers get immediate feedback on regressions and improvements
- Speedscale supports test case versioning and rollback safeguards, and can block a merge when a replay degrades performance
- Akamas applies its recommended configurations as a pipeline step
The result is a performance gate that runs on every commit rather than a performance test phase that runs when someone has time.
Where performance engineering goes next
Performance engineering is moving toward autonomous, intelligent systems, pushed there by AI and machine learning. There is a lot of hype in this space right now. There are also real capabilities underneath it, and the gap is closing quickly.
The direction is clear enough. Systems keep getting more complex, and human pace stops being sufficient. AI platforms spot subtle patterns and generate complex queries and candidate solutions faster than people can, which accelerates analysis and troubleshooting and spreads expertise beyond a small group of specialists. Teams get productivity back, and time for innovation with it.
Shifting left was the first step: developers testing individual features under a handful of virtual users. Embedding performance testing and tuning directly in the CI/CD pipeline is the next one, so every change is validated against realistic conditions automatically.
A continuous, AI-driven performance feedback loop removes the manual guesswork and the endless trial and error. Releases become more predictable and deployment risk goes down.
Low-code and no-code interfaces are pulling test creation away from specialists too, so more roles can design and run performance tests without heavy scripting. Specialists are not going away. Someone still has to apply context across the full spectrum of enterprise performance. But the more roles that own performance inside their own scope, the better the outcome.
Bottom line
Speedscale and Akamas solve two halves of the same problem. Speedscale feeds Akamas realistic workloads and accurate performance data. Akamas orchestrates those tests and applies the resulting configurations inside CI/CD. Neither half works well alone: realistic traffic without automated tuning is just better measurement, and automated tuning on synthetic traffic optimizes for a workload that does not exist.
Customers running Akamas optimization at scale have seen substantial results. Gaming operator Sisal cut the cost of its Kubernetes microservices by 58% (read the case study).
Performance engineering stops being a reactive, bottleneck-prone phase and becomes a continuous process that runs on every commit.
FAQs
What is Kubernetes performance testing?
Kubernetes performance testing measures how containerized workloads behave under load. It covers pod resources, autoscaling behaviour, application runtime settings and the dependencies between services. Unlike testing a monolith, it has to account for infrastructure that changes during the test.
How is traffic replay different from traditional load testing?
Traditional load testing runs a script that someone wrote from an assumed traffic model. Traffic replay captures real requests and responses from production and replays them. The workload is observed rather than guessed, so the results reflect what your users actually do.
Do I need service virtualization for microservices performance testing?
You need it whenever a service under test depends on something you cannot run in the test environment. That includes third-party APIs, LLM endpoints and production databases. Service virtualization stands in for those dependencies so the test is not blocked or billed per transaction.
Can performance testing run in a CI/CD pipeline?
Yes. Speedscale replays traffic against each build and can block a merge when performance degrades. Akamas runs its optimization experiments as a pipeline step and applies the recommended configuration. Both fit into an existing pipeline without a separate testing phase.
Does Akamas require agents or code changes?
No. Akamas reads KPIs from the observability tooling you already run, including Dynatrace, Datadog and Prometheus. There is no in-process agent and no application code change, so the measured behaviour is not distorted by the measurement.
How does this differ from APM or Kubernetes monitoring?
Monitoring and APM tools tell you what happened. They do not choose the configuration that fixes it. Akamas sits on top of that telemetry and turns it into specific configuration recommendations. It integrates with your existing monitoring stack rather than replacing it.
About the author
Scott Moore, Performance Engineering Advocate. More than 30 years in load testing and performance engineering, writing and speaking on how the discipline is changing in cloud-native environments.
Want to see this on your own workloads? Calculate your Kubernetes savings or create an account.
