In the July 2026 edition: how vLLM keeps production quality at a two-week release cadence, the hidden startup cost when the JIT can’t keep up, and why open-source AI is easy to adopt but hard to run in production. The Optimal is the Akamas newsletter where DevRel Engineer Graziano Casto breaks down Kubernetes performance, reliability, and cost optimization.
Hello everyone! Graziano here.
If you’re reading this, we’re officially wrapping up July 2026, which around here means one thing: conference season is loading. Between prepping talks and getting our booth game ready for KubeCon NA, our CPU usage has been higher than usual, but at least it’s the good kind of load. Grab a coffee and let’s look at the logs for this month.
The Latest Replica
How vLLM Keeps Production Quality at a Two-Week Release Cadence
In June 2026 alone, vLLM merged 1,918 commits into main, about 64 a day, backed by a CI suite running 266 jobs across 58 hardware runner queues and 13 million job minutes. Impressive, but the real story is what happened in May: v0.20.0 shipped and needed two emergency patches within days, because nothing in CI had actually run the models end to end on the target hardware to confirm they were still fast and still correct. One change broke gpt-oss on Blackwell under tensor parallelism, another quietly tanked DeepSeek V4 throughput on GB200. Neither crashed anything; the server just got slower or wronger, exactly the kind of regression a green checkmark won’t catch. That gap is why they built a nightly performance and accuracy benchmarking layer running across 17 model-hardware combinations before anything reaches a release candidate. The lesson generalizes past vLLM: “tests pass” and “performance and correctness held” are two different questions, and if you only ask the first one, production will ask you the second one at 3am.
Read the full article here.
When the JIT Can’t Keep Up: The Hidden Cost of Container Startup
A Quarkus benchmarking deep dive pulls apart what a single throughput number hides: time to reach it. Running a REST/CRUD workload on 4 pinned cores with 100 concurrent connections, the team found the JVM’s C2 compiler threads getting starved for CPU during warmup, in one configuration spending up to 180% of a 200% time budget just waiting in the run queue instead of compiling. The app kept serving requests the whole time, liveness probes stayed green, it just ran on slower, less-optimized code for far longer than the steady-state number suggested. Virtual threads and a different OS scheduler both closed part of the gap, because fewer on-CPU threads means more scheduling room left for the compiler to actually run. The framework names in the benchmark matter less than the mechanism: if you size CPU requests off a steady-state benchmark and your pods get killed on a tight startup budget, you may be restarting the JVM before the JIT ever finishes its job, over and over, and never noticing why.
Read the full article here.
The State of Open Source AI: Adoption Is Easy, Production Is Hard
Mozilla’s new report on the open-vs-closed AI landscape has a data point that should sound familiar to anyone who’s shipped infrastructure before: 79% of developers use open models when adding AI features, ahead of closed models at 71%, but only 51% of open-model teams reach production versus 63% for closed, and that gap barely narrows with company size (53% to 57% for open, versus 54% to 73% for closed). When the report asks teams why they churn away from open models, the top answers aren’t about model quality at all: infrastructure and compute cost, security and compliance, and ongoing maintenance lead the list, in that order, in nearly every region surveyed. Model capability closed most of the way to parity years ago. What’s still open is the operational tax of actually running the thing reliably and affordably, which is a much older problem than AI and one performance engineering has been chewing on for decades.
Commits From The Lab
The articles this month all converge on the same idea: efficiency isn’t something you configure once, it’s something you keep measuring, whether the workload is a JVM warming up, an inference server under load, or a model going from adoption to production. That’s the exact gap we’ve been building tooling for in the lab.
Two things worth your attention:
- Claude Code plugins for Akamas. We built a set of plugins that let teams run our Claude Code workflows without depending on external connectivity, useful for anyone working in a locked-down or air-gapped environment who still wants AI-assisted workflows without punching new holes in the perimeter.
- The vLLM Optimization Pack. A new Akamas Optimization Pack purpose-built for tuning vLLM inference servers, the same category of engine covered in this month’s Replica. If you’re running inference at any real scale, the tuning surface (batching, memory allocation, parallelism settings) is large enough that guessing is expensive.
Curious about either? Try Akamas and see the engines at work.
The Diff
Numbers beat opinions, so here is what optimization looked like in production this month.
A single-replica Java service on Kubernetes, sized for peak guesswork instead of measured demand.
The workload’s baseline configuration topped out at around 2,000 requests/minute before hitting trouble: two container restarts causing roughly 8 minutes of total downtime during testing, an average response time of 7.5 seconds, and CPU usage that regularly exceeded what was actually allocated to the pod. We pointed continuous tuning at the JVM and container configuration together (heap sizing, GC settings, connection pool and thread pool sizing, CPU/memory requests and limits) rather than at just the container envelope. The best configuration found after only one day of automated experiments let the same single replica handle 5,000 requests/minute (288% more compared to the baseline), the full target load, with zero restarts over the full test window and average response time down to about 350 milliseconds(94,5% less). It got there while consuming only about 200 millicores and 100 MB more than the baseline was already using. The takeaway: the baseline wasn’t under-provisioned, it was mistuned. Once the JVM and the container agreed on how much room they actually had, the same box did more than twice the work at a fraction of a second, not seconds, of latency.
Catch Us
Want to meet the Akamas team in person? We’re regularly at conferences and meetups across the Kubernetes and Java ecosystems. See where we’ll be next on our Events page, and follow me on LinkedIn for the talks and sessions I’ll be at.
Keep those CPUs cool and your latencies low.
Stay optimized,
Graziano Casto, DevRel @ Akamas
