Kubernetes gives you dense container scheduling and strong workload isolation, and I love it for that. It also forces a hard tradeoff between two things every team cares about: cloud cost and service reliability.
Kubernetes cost optimization is the practice of setting pod resource requests and limits so a cluster runs at low cost while still meeting its SLOs. The tension is structural. Set requests and limits too high and you pay for idle capacity. Set them too low and you trigger CPU throttling or OOMKills that breach latency and error SLOs. This post explains why that tradeoff exists, and shows how ML-based optimization resolved it on a real application: a 77% gain in cost efficiency while holding a 500ms latency SLO.
Kubernetes spending keeps climbing. The CNCF Kubernetes FinOps Report (June 2021) shows adoption rising fast, which partly reflects how hard Kubernetes is to run cost-efficiently. Managing performance, stability, and efficiency on Kubernetes is tough even for seasoned SREs and performance engineers. The Kubernetes Failure Stories archive is full of incidents: unexpected CPU slowdowns, sudden container terminations, and stability issues that trace straight back to resource configuration.
The rest of this article covers two things. First, why Kubernetes resource management makes cost and performance so hard to balance. Second, how our ML-based approach handles that balance automatically, applied to a real-world microservices application.
Kubernetes resource requests and limits: the fundamentals
Kubernetes cost efficiency starts with two settings on every container: resource requests and resource limits. Getting them wrong is the single most common cause of both waste and SLO breaches.
Resource requests are the CPU and memory a container is guaranteed to get. When a developer defines a pod, they declare its requests, and Kubernetes schedules the pod onto a node where that capacity is actually free.
In Figure 1, Pod A requests 2 CPUs and lands on a 4-CPU node. Pod B of the same size fits on the same node. That node is now fully requested, so Kubernetes will not place Pod C there, even if real usage is low. Requests are how Kubernetes manages true cluster capacity.

Kubernetes does not overcommit requests. You cannot request more CPUs than the cluster physically has. That is unlike virtualization, where a host can present far more virtual CPUs than it owns.
Requests are also not the same as utilization. If requests sit far above real usage, a cluster can report itself full while running at 10% CPU. Setting accurate requests is therefore central to Kubernetes cost efficiency.
Resource limits are the maximum CPU and memory a container can use. A container can burst above its request up to its limit. For example, Pod A might carry a limit of 4 CPUs and 4 GB of memory.
Kubernetes treats the two resources very differently when a workload hits its limit:
| Resource | Behavior at the limit | Impact on the application |
|---|---|---|
| CPU | The container is throttled | Latency spikes and slowdowns; the pod survives |
| Memory | The container is terminated (OOMKilled) | The pod disappears; no paging or swap as in a traditional OS |

Why CPU throttling breaks SLOs below the limit
Here is the effect most teams miss, and the one that quietly breaks SLOs. CPU throttling does not wait for usage to reach the limit. It starts well before it.
In the Akamas labs, we measured throttling that began at just 30% of the CPU limit (Figure 3). This is a consequence of how CPU limits are enforced by the Linux kernel’s CFS quota mechanism.

The service impact is severe. Latency spikes appear with no obvious cause, even at low average CPU usage, and they breach SLOs.
The obvious fix is to remove CPU limits, and it can slash latency (see the results at Buffer). It is also dangerous. One runaway container can starve your most critical services. Setting resource limits remains the best practice Google recommends.
So the requests-and-limits balance is a real tradeoff, not a solved problem. Now scale it up. A cloud-native application is dozens of microservices, each with its own CPU and memory requests and limits, autoscaling behavior, and runtime settings such as JVM options. The job becomes: set requests and limits for every microservice, account for autoscaling and runtime configuration, and still meet every performance and availability SLO, all at minimum cost.
Done by hand, across that many interacting parameters, this is effectively impossible. The good news: with the right approach, it is not.
Optimizing Kubernetes cost and reliability automatically
We took an ML-based approach at Akamas, using Reinforcement Learning to explore huge configuration spaces quickly. You set the goals and constraints. The system applies a candidate configuration, measures the result, and learns its way toward the optimum, one experiment at a time.
To show it working, we optimized the Google Online Boutique, a cloud-native demo of 11 microservices written in Go, Node.js, Java, and Python. It ships with a Locust load generator for realistic traffic. We added Prometheus for pod resource metrics and the Istio service mesh for service-level metrics.
The optimization goal: maximize efficiency, meaning higher throughput at lower cloud cost, without breaching SLOs. Figure 4 shows how the goal and constraints map to the application’s metrics.

The goal maximizes the ratio of frontend throughput to total cost. Cost is computed from the CPU and memory allocated to each microservice. We ran on AWS Fargate, a serverless Kubernetes offering priced at $29/month per requested CPU and about $3/month per GB of memory (current Fargate pricing). The SLO constraints: 90th-percentile latency at or below 500ms, and an error rate at or below 2%.
This study tuned Kubernetes only, so the parameters were the CPU and memory limits of each microservice: 22 tunable parameters in total. (More complex studies that add other parameters are covered in later posts.)
The optimizer found a better configuration in 35 iterations, about 24 hours. It improved cost efficiency by 77%, from 0.29 TPS//moatbaselineto0.52TPS//mo. Figure 5 compares the baseline and best CPU and memory limits.

The changes are instructive. Most microservices had their CPU limits cut, an obvious cost win. Two were deliberately given more CPU and memory. That mix is the point: the optimizer raised resources where reliability demanded it and cut them everywhere else, all while respecting the SLO constraints.
Figure 6 compares throughput (left) and 90th-percentile response time (right) for the best configuration (green) against the baseline (blue).

The best configuration was both cheaper and faster. Throughput rose 19%. Latency peaks fell 60%. Response time also became far more stable.
Conclusions
Kubernetes is a great platform for microservices, but it demands careful per-container configuration to stay both cost-efficient and reliable. As the numbers above show, that tuning is daunting by hand, mostly because of how Kubernetes manages resources.
Manual tuning cannot reliably hit performance, stability, and cost targets at the same time. The AI techniques we use at Akamas explore thousands of configurations, including the hundreds of settings tied to individual microservices, in hours rather than weeks.
The result: SREs, performance engineers, and developers get the configuration that best fits their own cost and reliability tradeoff, expressed as goals and SLO constraints.
If you want to see the tradeoff on your own workloads, the Kubernetes waste calculator is a quick start, and Akamas Insights applies this optimization continuously across pod resources and autoscaling. For JVM-heavy services, pair it with JVM on Kubernetes optimization.
FAQs
What is Kubernetes cost optimization?
Kubernetes cost optimization is the practice of tuning pod resource requests, limits, and scaling so a cluster runs at the lowest cost that still meets its SLOs. It targets the gap between resources requested and resources actually used.
Why do Kubernetes cost and reliability conflict?
Lower requests and limits cut cost but risk CPU throttling and OOMKills that breach SLOs. Higher values protect reliability but pay for idle capacity. The right setting is a balance, not a maximum or a minimum.
Why does CPU throttling start below the CPU limit?
The Linux kernel enforces CPU limits with CFS quotas over short scheduling periods. A bursty container can exhaust its quota inside a period and be throttled even when average usage is low. Akamas Labs observed throttling from 30% of the limit.
Should I remove Kubernetes CPU limits?
Removing CPU limits can reduce latency, but a single runaway container can then starve critical services. Google recommends keeping limits. Tuning them well is safer than removing them.
What is a Kubernetes SLO in this context?
An SLO is a measurable reliability target, such as 90th-percentile latency at or below 500ms and an error rate at or below 2%. In optimization, SLOs act as constraints the configuration must satisfy while cost is minimized.

