If you are a cloud native company, you already know that Kubernetes is a solid choice for deploying and managing modern applications at scale. It’s flexible, powerful, and capable of handling complex workloads. However, there is also the potential for serious performance headaches.
Many people think performance optimization is just about cutting costs or squeezing efficiency out of resources. That’s part of it, but not all of it. It’s also about delivering reliable services without over-provisioning. There is a fine line between balancing performance, reliability, and costs. Nobody wants a cheap Kubernetes environment if it’s always down (or slow). That only frustrates users. Nobody wants a robust Kubernetes environment that costs so much to run that it defeats the purpose.
Kubernetes optimization is more than tuning the cluster (sorry, platform engineers). It’s a holistic effort that involves many layers, including the cluster, the workloads and application runtimes like the JVM running on it. Focusing on just one piece leaves significant optimization opportunities on the table. It’s a layered stack: the infrastructure (nodes, networking, storage), the control plane (scheduling, autoscaling), and the applications themselves. Each layer needs attention.
In this article, we’re going to look at the top performance pitfalls that we see on a regular basis from customers in their Kubernetes journey. We’ll show you how to avoid these antipatterns and share practical tips to keep your deployments smooth.
1. Settings CPU Limits Based on Theory Rather Than Experience
By far, pod resource requests and limits are the biggest challenge K8s teams face. If you’re not familiar, we described how requests and limits work and their impact on costs and reliability in a previous post.
One of the most debated aspects is CPU limits: is it best to set them for your workloads, or avoid them and just live with CPU requests? CPU limits can cause CPU throttling and significant application performance degradation, such as response-time slowdowns, even when CPU usage is low compared to the limit.
But CPU limits can also be critical for cluster stability and application performance in shared or multi-tenant clusters. So, as a developer or operator of K8s apps aiming for reliability and efficiency, what should you do?
To understand how CPU limits actually work, we ran performance tests under CPU-contention scenarios. We measured application performance when a misbehaving job, co-located on the same node, saturates the node CPUs. We tested two setups: CPU requests only (no limits), and both requests and limits set.
The result? CPU requests alone do not protect latency-sensitive workloads from a noisy neighbor. In our tests, application p95 latency rose 3x. With CPU limits in place, the CPU-hungry job could no longer saturate the node, and the impact on application performance was contained.
Why is that? This might sound counterintuitive, especially as many people expect CPU requests to prevent such situations. But there is nothing wrong here: that’s exactly how CPU resource management works on K8s (courtesy of Linux cgroups), and failing to understand it can have serious consequences for app performance and stability, as others have noted too. More details in an upcoming blog.
2. Picking the Wrong Instance types for your Nodes
Kubernetes provides built-in autoscalers at the cluster level, adding nodes when demand rises and removing them when traffic dies down. The Cluster Autoscaler (or Karpenter) watches resource requests from pending pods, spins up new nodes to accommodate them, then scales down during quiet periods to save costs.
A common issue SREs and platform engineers face is failing to keep node-group configuration matched to workload resource requirements. Teams often configure node groups with standard cloud instance types and never revisit whether the CPU-to-memory (or GPU) ratio matches the workload shape.
This can produce highly inefficient allocation at the cluster level, with significant CPU or memory (and cost) wasted. What often happens: the cluster autoscaler adds nodes for pending pods, but only one resource (say, memory) is exhausted while the other (say, CPU) sits idle and wasted.
Let’s look at an example. In this cluster, the cluster autoscaler creates nodes due to a memory shortage (right chart). In doing so it provisioned roughly 5,000 CPUs that went unused, because the workloads never requested them (left chart).
Notice that the cluster autoscaler works perfectly here. Nevertheless, a huge amount of resources and cost is wasted because of the poor instance-type configuration in the underlying node groups.
Choosing node types and scaling your cluster is like picking the right rental car for a road trip. Too small, and everyone is cramped, with no room for luggage. Too big, and you burn far too much fuel, and every gas-station stop leaves your wallet in shock. Misjudge your node instance types, or fail to scale dynamically, and you’ll either hit bottlenecks or waste a lot of resources.
3. JVM Misconfigurations for Java Applications
Java applications on Kubernetes can be particularly tricky. The Java Virtual Machine (JVM) is a highly configurable engine with 500+ options that can be tuned for performance and resource efficiency. But JVM tuning requires deep expertise and is notoriously hard. Even a common task like determining the optimal heap memory size can be surprisingly complex for Java developers.
Containers and Kubernetes add another layer of complexity: how should you configure the memory limit and heap size of your JVM? And what about CPU limits and garbage-collector type? For high-performance, reliable, efficient Java apps, JVM resource management must be aligned with Kubernetes pod resource settings. Despite the JVM’s container-aware heuristics, in practice the default configurations are wasteful, prone to reliability issues, or cause unnecessary slowdowns.
Here’s a real example. In the chart below you can see an out-of-memory kill (OOMKill) that hit application availability. Seeing the steady rise in container memory usage, the SRE team blamed a memory leak in the application – time to fix your code, dear development team! But there was no memory leak. The real cause was a JVM misconfiguration: the pod memory limit did not account for total JVM memory demand, which includes both heap and off-heap memory.
4. Poor application performance due to wrong JVM heap sizing
In the previous pitfall we covered jointly optimizing pod resources and the application runtime to avoid reliability issues. But a wrong JVM configuration can hurt application performance too.
Many developers assume the JVM performs best out of the box, especially with container-awareness features that let it size the heap automatically from pod memory limits. In reality – whether you set the heap explicitly (e.g. via -Xmx) or rely on container awareness (e.g. MaxRAMPercentage) – the resulting heap size may be well off and slow your application substantially.
To show this, we ran benchmarks with the DaCapo Java suite, which mimics real-world Java workloads. We used the G1 garbage collector on Java 21 (eclipse-temurin) and measured Spring Boot latency across different JVM max-heap sizes. Here’s what we found:
- If the heap is too small, performance suffers badly. A 20% latency impact is easy to hit, rising to 65% when the heap is too tight.
- Beyond a certain size, more heap doesn’t help. Throwing gigabytes of memory at a Java app is not useful – the extra memory is basically wasted.
As a Java developer looking to optimize your applications and microservices, don’t forget to size your JVM heap properly.
5. Resource Management for Node.js Applications
Node.js applications often hit performance and reliability issues on Kubernetes because of their unique (and, honestly, under-documented) runtime characteristics.
Node.js runs on a managed runtime called V8, which resembles the JVM in many ways, including automatic memory management via garbage collection and a multi-threaded architecture (yes – the Node.js runtime is not single-threaded).
A common problem is an out-of-memory kill (OOMKill). This mirrors pitfall #3 for the JVM: most teams misread OOMKills as a memory leak, when the real issue is how heap memory is sized relative to container memory limits.
Less known is that Node.js performance and resource usage are also very sensitive to V8 garbage collection and heap settings. In an earlier blog we benchmarked a Node.js app across different V8 heap-pool sizes. Tuning the V8 heap memory pools cut application latency by nearly 50%, with no code changes. Similar gains showed up on CPU usage – which means savings on infrastructure footprint and cost.
Check the blog for full details.
6. Misconfigured HPA Autoscaling (Autoscaling Gone Wild)
Kubernetes provides built-in application-level scaling via the Horizontal Pod Autoscaler (HPA), adding pods when demand rises and removing them when traffic dies down. HPA can be a lifesaver, but it’s not a set-and-forget solution.
HPA needs careful tuning of scaling metrics, thresholds, and periods. Scaling behaviour is highly application- and traffic-dependent. Is your app’s startup time quick or slow? Do you need to react to sudden traffic peaks, or does traffic build slowly? Depending on your performance requirements or SLOs, HPA must be configured accordingly.
Teams often don’t get HPA right, and we regularly see deployments that scale too slowly, too aggressively, or fail to scale down. A poorly tuned HPA will:
- hurt end-user performance
- cause unnecessary SLO breaches and production incidents
- waste a lot of resources
- push the cluster autoscaler to provision unnecessary nodes
Look at this example. The team rightly adopted HPA for a deployment with fluctuating traffic, but set the scaling threshold far too low. As a result, HPA scaled the deployment far too much: overall HPA scaling efficiency (CPU used / CPU requested) was just 9%, wasting most of the requested CPU.
The lesson: as with cluster-level autoscaling, simply adding HPA to your workloads may not meet the scalability and efficiency goals you’re after.
7. VPA Missteps with Requests and Limits
The Vertical Pod Autoscaler (VPA) is often the most recommended tool for setting pod resource requests and limits. VPA automatically sets container requests and limits based on observed usage.
That sounds like exactly what’s needed, but the reality is different.VPA has a number of known limitations that many teams are not aware of, and can make it inapplicable in real-world production environments.
There is one very important limitation not even mentioned in the docs. The VPA recommendation algorithm takes a purely infrastructure-only approach: it looks at resource usage and adjusts resources, with no awareness of the application running inside the pod.
Why is that a problem? Modern apps run on a runtime that manages resources such as heap memory and CPU threads. Runtimes like the JVM or Node.js V8 adjust automatically based on pod resource limits. The VPA is unaware of how these runtimes behave, so applying its recommendations without care can create performance and reliability issues.
Infrastructure-only approaches like the VPA can also slow your application down. In one customer test, the team applied VPA recommendations to a Java app and measured response time before and after. The change introduced significant slowdowns, making the app unresponsive to users.
Why did that happen? Read this blog on why it’s crucial to consider the application runtimes inside containers and why infrastructure-level metrics aren’t enough. This holds for any tool or approach that looks only at container-level resource metrics – not just the VPA.
8. Pod Sizing for Disaster Recovery (DR) and High Availability (HA)
Poorly sized pods in disaster-recovery or high-availability scenarios can jeopardize cluster resilience. If pods aren’t configured to handle failover or spikes, you risk downtime or degraded performance during critical moments.
This is a common challenge for SREs and development teams working out the best resource-allocation strategy. Teams typically size pods against currently observed load. For example, a pod requests 4 CPUs but uses just 1, so you downsize it to match the 1-CPU demand and save cost.
The problem: this ignores reliability, high availability, and business-continuity requirements. Your applications may need to withstand the failure of one or more data centers and keep response times good during infrastructure disruptions.
Properly sizing workloads for degraded scenarios means simulating the impact of 2x traffic on your workloads. And again, it’s not only about pod resources: your application runtime – like the JVM heap memory – must be considered too, to avoid unexpected bottlenecks at critical moments.
9. Configuration Chaos and Drift
Technical issues aren’t the only challenges teams face adopting Kubernetes. A more organizational or process issue we see, especially in mid-to-large organizations, is that developers push frequent changes to production, and these can introduce performance regressions or outright break things across the cluster.
Part of the problem is that developers aren’t necessarily well-versed in the Kubernetes stack – and rightly so. As we showed in this blog, configuring apps and clusters for efficiency and performance is no small feat. Developers’ goal is usually to ship features quickly, not to become infrastructure experts.
Meanwhile, SREs and platform engineers have to cope with the fallout. Poor configuration of even a single workload can impact other applications on the same node (as shown in pitfall #1 on CPU limits) or the entire cluster (for example, if a node crashes).
Kubernetes is a great platform, but every user has to be a good citizen and use it wisely so everyone else can too. At scale, proper performance isolation and reliability practices are key – and in practice this doesn’t always happen. The only way to keep the platform stable, performant, and efficient is to adopt proper processes and tools that automate the optimization work as you scale.
10. Organizational Knowledge Gaps and Prioritization Conflicts
Kubernetes thrives on collaboration, but when priorities aren’t aligned between platform engineering and application teams, it creates a deadlock that affects both performance and reliability. When application teams lack Kubernetes expertise and prioritize feature speed over stability, big problems follow.
It typically goes like this:
- Platform engineers understand Kubernetes intricacies, but application teams may lack the training to optimize their deployments. This leads to misconfigured resources or ignored best practices.
- Application teams focus on shipping features quickly, sidelining critical tasks like performance tuning, monitoring setup, or adopting Kubernetes-native practices.
- Without good collaboration, platform teams may enforce overly restrictive policies while application teams deploy suboptimal workloads. The result is friction between teams and unreliable software.
To fix this, invest in cross-team training to bridge the Kubernetes knowledge gap. Offer workshops or hands-on learning tailored to application teams. Establish clear service-level objectives (SLOs) that align platform and application priorities, balancing feature velocity with reliability goals.
Tools to Save the Day
Here are some tools and techniques to help you tune Kubernetes for better performance:
- Observability: Use tools like Prometheus, OpenTelemetry, or commercial options like Dynatrace and Datadog to understand resource usage. Don’t stop at workload-level metrics – collect application-runtime metrics too, as they are critical for success, as shown throughout this post.
- AI-powered tuning: Use tools that automate full-stack Kubernetes optimization to reduce the effort and skills required from engineering teams. Akamas autonomously optimizes the full-stack configuration of enterprise applications – from infrastructure to application layers – using reinforcement learning, live telemetry, and user-defined goals, both live in production and offline in testing.
- Load testing: Simulate traffic with tools like Locust, JMeter, or commercial options like OpenText LoadRunner and Tricentis NeoLoad. Load testing is often the best way to find scalability bottlenecks and tune workload configurations like the JVM or HPA before you deploy to production.
- Chaos engineering: Use tools like Chaos Toolkit, Chaos Mesh, or commercial options like Gremlin to test resilience against worst-case events. This helps confirm applications can sustain traffic in disaster-recovery scenarios.
- Advanced Kubernetes features: Use capabilities like taints, tolerations, affinity rules, priority classes, resource quotas, and HPAs to build reliable, efficient clusters.
Wrapping It Up
Kubernetes is a great platform to scale your cloud applications on, and we love it – but performance and efficiency won’t come for free. You can build clusters that are fast, reliable, and cost-effective by avoiding these common pitfalls. The key is to stay proactive: monitor relentlessly, tune continuously, and build a performance-engineering culture where everyone cares about performance within their own scope.
If you’re struggling with Kubernetes performance, we can help. Explore Akamas Insights to see how our AI-driven optimization platform tackles all of these challenges and cuts cost. We understand the problems companies face deploying Kubernetes at scale, and we can help you too.
FAQs
What causes poor performance in Kubernetes?
Most Kubernetes performance problems come from small misconfigurations across three layers: infrastructure (node instance types), the control plane (HPA, VPA, cluster autoscaler), and the application runtime (JVM or Node.js/V8). Optimizing one layer while ignoring the others leaves gains on the table and can even create new bottlenecks.
Do CPU limits help or hurt Kubernetes performance?
Both, depending on context. CPU limits can cause throttling and slow a workload even when usage looks low. But in shared or multi-tenant clusters they protect latency-sensitive workloads from noisy neighbors. In Akamas contention tests, removing CPU limits let a misbehaving job push p95 latency 3x higher.
Why do Java applications get OOMKilled on Kubernetes?
Usually not because of a memory leak. The common cause is a pod memory limit that doesn’t account for total JVM memory demand, which includes both heap and off-heap memory. When off-heap usage is ignored, container memory grows past the limit and Kubernetes issues an OOMKill.
Does the Vertical Pod Autoscaler (VPA) improve application performance?
Not reliably. The VPA sets requests and limits from observed infrastructure usage alone, with no awareness of the runtime inside the pod. Because runtimes like the JVM and V8 react to pod limits, applying VPA recommendations blindly can slow an application down, as seen in a customer test on a Java app.
How do you optimize Kubernetes performance across the full stack?
Tune pods, application runtimes, and nodes together instead of in isolation. Align JVM or V8 settings with pod resource limits, size node instances to workload shape, and validate autoscaling under realistic traffic. Automating this full-stack tuning – for example with Akamas – reduces manual effort and keeps configurations optimal as workloads change.
