Notice: An abridged version of this interview has been published as Akamas contributed content by TFIR for the KubeCon+CloudNativeCon NA 2022 in Detroit, October 24-28
Stefano Doni, CTO at Kubernetes optimization company Akamas, and long-time CMG contributor and Best Paper award winner, is a recognized expert in performance engineering challenges and hot topics like microservices and container optimization.
We asked Stefano to help us understand an emerging discipline that he and his team call AI-powered Autonomous Optimization, in the context of the growing adoption of Kubernetes for microservices applications.
What is AI-powered Autonomous Optimization?
AI-powered Autonomous Optimization is a practice that uses AI to help performance engineers, developers and SREs tune applications. The goal is to maximize availability, performance, resilience and cost efficiency at the same time.
This is the term we use at Akamas. The discipline is new, so there is no standardized name for it yet. Gartner, for example, introduced “AI-Augmented Software Engineering” (AIASE) to describe the broader context of how AI improves software engineering.
Why is AI Kubernetes optimization emerging now?
Great question. The “why now” matters here.
Performance tuning and infrastructure optimization are not new. Neither is applying AI to operations: think of AIOps.
But tuning today’s applications on complex, multi-layered, hybrid infrastructure was already an impossible challenge, even for the best experts. I can say that from experience. I spent almost 15 years as a performance engineer and capacity planning expert. That is exactly why I co-founded Akamas: to use AI to help performance engineers like me.
We proved that AI is the only practical way to tune the hundreds of available parameters, such as JVM garbage collection or EC2 instance types. Among thousands of possible configurations, AI identifies the ones that deliver the best tradeoff of performance, resilience and cost. It does this while meeting SLOs, such as response time or throughput, within timeframes that fit modern release cycles.
Cloud-native applications raised the bar again. Kubernetes has to be optimized in near real time. This is not only about better performance or lower cost. On Kubernetes, there are also real performance and resilience risks. Those risks are driving adoption of AI-powered tools. Most teams still keep a human in the loop to review recommendations before AI applies them, but the direction is clear.
Why is container-only Kubernetes optimization not enough?
Two reasons. First, Kubernetes memory and CPU mechanics are not well understood. Second, the ergonomics of application runtimes, such as the JVM, are understood even less. The interplay between the two dramatically affects cost, performance and availability.
This is why we advocate an application-aware approach. We call it “AI-powered, autonomous, application-aware optimization”, which is a mouthful, so we shorten it to “triple-A optimization”.
Jokes aside, we worked with customers who optimized only at the container layer. Sizing a container by requests and CPU or memory limits alone misses large cost and performance gains. It can also cause under-provisioning and poor end-to-end performance. That is the real dilemma for anyone optimizing Kubernetes applications, until they see what AI Kubernetes optimization can do.
How does optimizing the runtime and Kubernetes together cut cost?
Let me be specific. Suppose the goal is to cut the cost of a Kubernetes environment. That cost is driven mostly by the resource demands of application pods and containers. So how do we find the minimal resource settings that do not hurt performance or reliability?
The key insight is that container resource use is largely driven by the application runtime. In a Java microservice, memory use is driven by the JVM, which actually allocates and frees memory. JVM memory use depends on JVM configuration such as heap size and garbage collector. We routinely see Java microservices with huge memory footprints simply because developers assigned 4 GB to every service as a “best practice”.
A tool that only adapts pod requests and limits to observed usage leaves real savings on the table. That is why you must optimize the runtime and the Kubernetes resource settings together. The same applies to any runtime-based language: Go, Node.js and .NET included.
What are the reliability risks of cutting CPU and memory blindly?
The risk is hurting application performance and reliability.
Kubernetes resource management is tricky. DevOps teams are often surprised that Kubernetes slows an app through CPU throttling even when CPU usage looks low. Runtimes also self-configure inside a container. We have seen the JVM change its garbage collector and memory pool sizes on its own, based on container memory and CPU requests and limits.
Why does that matter? If your cost-cutting approach is simply “reduce CPU and memory”, you are in for surprises. The application can slow down, or crash, in strange ways, when you least expect it.
What capabilities matter when choosing a Kubernetes optimization tool?
Based on my experience with customer environments, my top three required capabilities are goals, full-stack scope and safety.
- Custom goals and constraints (SLOs). Different applications need different goals, and the same application needs different goals at different times. Today the goal may be more orders processed; tomorrow it may be lower cost while holding a response-time SLO. Tools that only support fixed goals such as cost reduction, and that cannot take custom metrics and SLOs into account during the search, are not fit for the real world. Tools that report SLO impact only after applying changes are equally inadequate.
- Full-stack scope. Recommended configurations should span the layers of the application and the infrastructure. For Kubernetes microservices, both the runtime and the container layer must be considered. This is not only about capturing cost and performance gains. It is about preventing major availability and performance risks. Be cautious with tools that operate only at the infrastructure or container level.
- Safety. When you optimize in production under dynamic workloads, safety is not a nice-to-have. Safety is a major research area for us. It led us to features such as gradual optimization, smart constraints and outlier detection.
What is the future of AI-powered optimization for Kubernetes?
We have watched Kubernetes become the de facto standard for cloud-native applications. We see a parallel with the disruption VMware brought to virtualization.
As Kubernetes adoption grows, so does the need for more sophisticated automation and control. Kubernetes native scaling mechanisms, the Horizontal Pod Autoscaler (HPA) and the Vertical Pod Autoscaler (VPA), have different maturity levels and cannot be used together. Their most critical limit is that they scale on resource thresholds alone. They ignore feedback from measured application performance. They also ignore how the runtime manages its own resources. Static autoscaling settings are not always adequate under varying workloads, so service SLOs can suffer.
This resembles what happened about 20 years ago, when the first automated resource managers replaced VMware’s native Distributed Resource Scheduler (DRS). We believe AI-powered autonomous optimization can now maximize the cost efficiency of Kubernetes applications while protecting service quality at any point in time.
At Akamas we validated a “smart auto-scaling” approach that replaces VPA and makes HPA more effective. It dynamically adjusts both pod resource requests and HPA thresholds. We see this as a killer feature that will make AI Kubernetes optimization a standard part of any Kubernetes environment running critical applications. We go deeper on this in our companion piece on why utilization-based Kubernetes autoscaling can hurt reliability and cost.
Any recommended reading?
A few resources on these topics. Two of my own posts: Kubernetes resource management and CPU throttling and why lowering GC time may not improve application performance. For JVM defaults inside containers, see Microsoft’s “Containerize your Java applications”. New Relic’s 2022 State of the Java Ecosystem reports statistics from millions of Java apps, including many running in Kubernetes containers.
To see AI Kubernetes optimization in practice, explore Akamas Insights, the JVM on Kubernetes optimization solution, and Kubernetes workload optimization.
FAQs
What is AI-powered autonomous optimization?
It is a practice that uses AI to tune applications and infrastructure together. It targets availability, performance, resilience and cost in a single optimization, and can apply changes with a human in the loop or autonomously.
Why is optimizing Kubernetes applications so hard?
Kubernetes CPU and memory mechanics interact with runtime ergonomics, such as JVM heap and garbage collection. Tuning either layer alone can miss savings and trigger CPU throttling or out-of-memory crashes.
Why isn’t container rightsizing enough on its own?
Container resource use is largely driven by the runtime. Adjusting pod requests and limits to observed usage, without tuning the runtime, leaves cost savings unclaimed and can cause under-provisioning.
How is AI Kubernetes optimization different from HPA and VPA?
HPA and VPA scale on resource thresholds and cannot run together effectively. AI-driven smart autoscaling uses application performance signals, such as response time, and tunes pod requests and HPA thresholds together.
Does this apply to runtimes other than Java?
Yes. The same runtime-plus-container principle applies to any managed runtime, including Go, Node.js and .NET.
