This blog is co-authored by Kyle McMeekin, Head of Channel at Gremlin.
Chaos engineering tells you what breaks. It does not tell you which configuration stops it from breaking again.
Kubernetes chaos engineering injects controlled failure into a running cluster to expose weaknesses before users find them. The harder problem arrives immediately after. Most of what chaos experiments surface is a configuration problem, not a code problem. Gremlin’s State of Chaos Engineering study found that for more than half of customers, wrong configurations account for at least 20% to more than 80% of their sev0/sev1 incidents.
Fixing those settings by hand across hundreds of microservices is where most resilience programs stall. Autonomous optimization closes that loop. Akamas searches the configuration space with reinforcement learning. It returns settings that hold up under the exact failure conditions Gremlin creates.
What Kubernetes chaos engineering actually tests
Chaos engineering is the practice of injecting failure into a system on purpose, under controlled conditions. The same work travels under several names: Kubernetes chaos testing, fault injection, resilience experiments. Gremlin calls the individual experiments “attacks.”
An attack lets SREs verify that failover and failback mechanisms work. It surfaces dependencies nobody documented. It also reduces alerting noise, because teams learn which signals matter when something genuinely fails.
Gremlin applies attacks at several layers of the target infrastructure:
- Resource attacks: consume CPU, memory, disk or I/O on a host or pod
- Network attacks: delay, drop or blackhole traffic between services
- State attacks: shut down hosts, kill processes, change system time
Kubernetes is a natural fit for this work. Resilience and failover are among its strongest features, and cloud-native adoption keeps growing. The confidence is not always justified. Kubernetes reliability remains an open problem for a meaningful share of adopters. The CNCF’s 2020 annual survey reported reliability as a top adoption challenge for 12% of companies running containers and orchestration tools.
Reliability and performance are the same engineering problem viewed from two angles. Enterprise load testing platforms have moved in this direction. LoadRunner integrated with Gremlin so teams can drive failure scenarios while measuring performance against SLOs.
Why chaos experiments keep finding configuration, not code
Run enough attacks and a pattern emerges. The application code is usually fine. The settings around it are not.
That matches what Gremlin measured. Wrong configurations drive at least 20% to more than 80% of sev0/sev1 incidents for over half of surveyed customers. A pod memory limit set too low. A JVM heap that does not fit the container. An HPA threshold that scales the wrong way under a partial outage. These are not edge cases. They are defaults that nobody revisited.
Why manual Kubernetes tuning cannot keep up
Knowing the fix is a configuration change does not make the change easy.
A single cloud-native service exposes tunable parameters at every layer: container resources, runtime flags, application settings, middleware, instance type. Those parameters interact. Vendor defaults and published best practices often fail under a specific workload, and sometimes make performance and resilience worse.
Shorter release cycles compress the time available to test any of it.
The scale problem is measurable. In several Akamas polls, 33% of customers reported that ensuring the performance and resilience of Kubernetes applications remains an unmet challenge. One provider of critical SaaS services estimated that tuning a single microservice takes several weeks, up to 2 months of effort. A real application has hundreds of them.
Manual tuning does not scale to that. Neither does scripted automation that only replays a fixed playbook.
Autonomous optimization: turning findings into configurations
Autonomous optimization is the practice of letting AI search the configuration space and converge on settings that meet a defined goal. No human tuning loop. No prior knowledge of the application.
Akamas is built on patented reinforcement learning. It tunes parameters across multiple layers and technologies at once, and converges in hours or days rather than weeks. The approach is goal-driven: you declare constraints such as SLOs, and a goal such as maximize resilience or minimize cost.
The output is a configuration, not a dashboard. That is the difference that matters after a chaos experiment.
How the two practices form a loop
- Inject failure. Gremlin runs an attack against the target service.
- Observe the impact. Capture SLO breaches, latency, replica churn, saturation.
- Define the goal. Turn the observed weakness into an optimization constraint.
- Optimize the configuration. Akamas explores parameters across pod, runtime and infrastructure layers.
- Re-run the attack. Verify the new configuration under the same failure conditions.
Each pass narrows the gap between what the system survives today and what it needs to survive.
Chaos engineering, performance testing and optimization compared
The three practices answer different questions. Teams get the most value when they run all three, in sequence.
| Practice | Question it answers | What it produces | Where it stops |
|---|---|---|---|
| Chaos engineering | What breaks when a dependency fails? | A ranked list of real weaknesses | It does not choose the fix |
| Performance testing | Does the system hold at target load? | Pass or fail against SLOs | It validates one configuration at a time |
| Autonomous optimization | Which configuration holds under both? | Applied settings across every layer | It needs a defined goal to optimize against |
A real result: 49% lower cost, or higher resilience under load
Akamas was applied to a microservice providing a critical B2B authorization service. The goal was cost reduction. After a few dozen experiments, about 19 hours, the AI had produced several configurations better than the baseline.
| Configuration | Cloud cost | Response time | Behavior under higher load |
|---|---|---|---|
| Cost-optimal | 49% lower | Improved | Meets the cost goal |
| Resilience-oriented | 16% lower | Improved | Better response time, fewer triggered replicas |
The second configuration is the interesting one. It gave up most of the savings and bought resilience instead. That tradeoff is invisible to a team tuning by hand, because nobody has time to explore both branches.
The full autoscaling case study walks through how the replica behavior changed. The same tension between cost and reliability shows up in Kubernetes cost optimization without breaking SLOs.
What changes for SRE teams
Chaos engineering and autonomous optimization solve adjacent halves of the same job. One finds the failure scenario. The other finds the configuration that survives it.
Run together, they produce outcomes an SRE team can report on: lower MTTR and MTTD, fewer incidents, and cloud spend that falls rather than rises when reliability improves. Kubernetes resilience stops being a quarterly fire drill. It becomes a property the platform maintains between experiments.
Akamas can also hold a resilience-oriented configuration in reserve, ready to apply as a remediation action when a specific failure occurs.
Teams standardizing this across many services usually start with intent-based tuning profiles, then extend to HPA and workload scaling settings. Akamas Insights delivers the recommendations continuously from existing observability data.
Frequently asked questions
What is Kubernetes chaos engineering?
Kubernetes chaos engineering is the practice of injecting controlled failure into a running cluster to expose weaknesses before users do. Typical experiments kill pods, throttle CPU, or drop network traffic between services. The goal is verifying that failover works and that SLOs hold when a dependency degrades.
What do Kubernetes chaos experiments usually find?
Configuration problems, more often than code defects. Gremlin’s State of Chaos Engineering study found wrong configurations account for at least 20% to more than 80% of sev0/sev1 incidents for more than half of customers. Common culprits are memory limits, runtime flags and autoscaling thresholds.
How is chaos engineering different from performance testing?
Performance testing measures whether a system meets its SLOs at a target load. Chaos engineering measures whether it survives a failure. Both validate one configuration at a time. Neither one searches for a better configuration, which is what autonomous optimization adds.
Can chaos engineering fix the problems it finds?
No. Chaos engineering is a discovery practice. It produces a ranked list of weaknesses and leaves the remediation to engineers. Autonomous optimization closes that gap by searching the parameter space and returning a configuration that holds under the failure conditions the experiment created.
What tools do you need to combine chaos engineering and Kubernetes optimization?
Three pieces. Gremlin chaos engineering attacks inject the failure. An observability stack such as Prometheus, Datadog or Dynatrace measures the impact. An optimization engine such as Akamas searches the configuration space. Akamas reads telemetry from the monitoring stack you already run.
Does this reduce MTTR?
It reduces both MTTR and MTTD in practice. Chaos experiments expose detection gaps before an incident, and optimized configurations reduce the number of incidents that reach sev0 or sev1 in the first place.
See how it works
Update: Akamas turns Kubernetes telemetry into applied configuration changes across pods, runtimes and infrastructure. See what the platform optimizes, or read The State of Cloud Native Optimization 2026 for how enterprises handle Day 2 operations today.
