Resource Topic: AI

  • What is sovereign AI? A practical definition for teams that run their own models

    What is sovereign AI? A practical definition for teams that run their own models

    Everyone in European AI is talking about sovereignty this year. Few people mean the same thing by it. Here’s the definition I use, and the part most plans leave out.

    Sovereign AI is AI that an organization or a country runs under its own control. The models, prompts, data and infrastructure stay under its own jurisdiction. It can run on owned GPUs, in a European cloud, on a neocloud or on a mix.

    What sovereign AI keeps inside your jurisdictionA dashed boundary labelled your jurisdiction contains four blocks: models, prompts, data and GPUs. An external AI provider sits outside the boundary. Your jurisdiction ModelsOpen-weight, fine-tuned Promptsand the answers DataCustomer data, code GPUsOn-prem or in the cloud ExternalAI provider
    Sovereign AI keeps the models, the prompts, the data and the GPUs under your own jurisdiction.

    That’s the short answer. The longer one is more interesting, because sovereignty changes who pays for every token.

    Sovereign AI has two meanings

    The word gets used in two ways, and they’re mixed up all the time.

    The first meaning is national. A country wants AI built on its own infrastructure, data and talent. That’s the scale of InvestAI. The European Commission presented it in February 2025, to mobilize €200 billion for AI. That includes a €20 billion fund for AI gigafactories (European Commission, February 2025). In July 2026 the EuroHPC Joint Undertaking opened the call for up to seven of them. It expects more than €20 billion of private investment (EuroHPC, July 2026).

    The second meaning is enterprise. A company wants its own AI, on its own terms. Prompts and answers carry customer data, source code and business knowledge. The company wants them in its own data center or cloud account.

    National sovereign AI

    InvestAI aims to mobilize €200 billion for AI in the EU, with a €20 billion fund for AI gigafactories. The July 2026 call covers up to seven of them. Sources: European Commission, EuroHPC.

    Enterprise sovereign AI

    Prompts, source code and customer data stay in your own data center or cloud account. You choose the models, the GPUs and who operates them.

    For most companies, the second meaning is the one that matters. It’s the one this post is about.

    Why sovereign AI is growing so fast

    The money is already moving. Gartner forecasts worldwide sovereign cloud infrastructure-as-a-service (IaaS) spending at about $80 billion in 2026, up 35.6% from 2025. European spending grows 83% this year, and Gartner expects Europe to pass North America in 2027 (Gartner, February 2026).

    Sovereign cloud IaaS spending in EuropeGartner figures: 6.9 billion US dollars in 2025, 12.6 billion in 2026 and a forecast of 23.1 billion in 2027. Europe, sovereign cloud IaaS spending, US$ billions 0510152025$6.9B2025$12.6B2026$23.1B2027 forecast
    European spending on sovereign cloud infrastructure nearly doubles in 2026 and again in 2027. Source: Gartner, February 2026.

    Three forces sit behind those numbers.

    Control. Prompts, answers and models are business assets. Companies want them under their own rules and their own jurisdiction. In Europe, that means AI that runs under European control.

    Independence. When your AI runs at an external provider, its price, terms and availability are theirs to change. A self-hosted model keeps that decision in house.

    Regulation. Prompts and answers often contain personal data, which GDPR governs. NIS2 makes essential and important entities manage risk across their technology suppliers. In finance, DORA does the same for ICT providers. Every external AI provider is one more supplier to assess.

    Where sovereign AI runs

    Sovereign doesn’t have to mean on-premise. I see three paths, and most teams end up mixing them:

    • On-premise AI: your own GPUs, in your data center or a colocation site.
    • European cloud: GPU capacity from a provider that operates under European law.
    • Neocloud: a GPU-specialized cloud. For sovereignty, what matters is who runs it and under which jurisdiction.

    The hybrid version is the common one. Owned GPUs carry the steady load. A European cloud or a neocloud covers the peaks with whatever GPUs it has available that week.

    A hybrid sovereign setup over one dayDemand rises and falls over 24 hours. Your own GPUs serve a flat base. A European cloud or neocloud covers the peaks above it. 00:0012:0024:00 Your own GPUs European cloud or neocloud
    In a hybrid setup, your own GPUs carry the steady load and a European cloud or neocloud covers the peaks.

    The part of sovereign AI nobody budgets for: capacity

    Here’s what most sovereignty plans skip. Once you stop calling an external provider, every token runs on GPU capacity you’re responsible for. And that capacity is finite.

    Demand isn’t waiting. Goldman Sachs Research expects token consumption to grow 24 times between 2026 and 2030. Agentic AI is the main driver (Goldman Sachs, May 2026).

    Supply isn’t keeping up. New capacity takes hardware, power and permits. Memory alone costs six times what it did a year earlier (InfoQ, August 2026). Overprovisioning was a cloud habit. On prem, it’s a capital expense with a waiting list.

    Token demand 2026 to 2030
    24×
    Growth in token consumption
    Driver
    Agentic AI
    Goldman Sachs Research, May 2026
    Memory price In one year
    6×
    June 2026 vs June 2025
    Effect
    New capacity costs more
    InfoQ, August 2026

    So the sovereign question changes fast. It starts as “where do our models run?” It becomes “how many tokens can our GPUs actually serve?”

    That second question is the one we work on at Akamas. Default inference settings leave GPU capacity unused. In a hybrid setup, every GPU type needs its own configuration. Nobody can re-tune them all by hand. Akamas Inference Optimizer tunes the full inference stack under real traffic. Each GPU serves more tokens within the latency targets you set. There’s more on our sovereign AI infrastructure page.

    Final considerations

    Sovereign AI is a decision about control. It’s a good decision, and I expect many more European companies to make it.

    Just don’t stop at the architecture diagram. Control means the capacity is yours, and so is every idle GPU.

    Sovereignty is the reason to build. Efficiency is the reason it pays back.

    FAQs

    What is the difference between sovereign AI and sovereign cloud?

    Sovereign cloud is infrastructure: compute and storage that run under a given jurisdiction. Sovereign AI is broader. It covers the models, the prompts and the data, plus the GPUs they run on. Sovereign AI can run on a sovereign cloud, on owned GPUs, or on both.

    Is sovereign AI only for governments?

    No. Countries build sovereign AI at national scale. Companies build it to keep prompts, data and models under their own control. Both use the same building blocks: GPUs, open-weight models and a serving stack they operate.

    Does sovereign AI mean on-premise AI?

    Not necessarily. On-premise AI is one path. A European cloud or a neocloud under European jurisdiction also qualifies, and many teams mix all three.

    What does sovereign AI infrastructure need?

    It needs GPUs under your control, open-weight models and a serving stack such as vLLM. It also needs enough capacity for your traffic. Capacity is the hard part, because it grows only as fast as hardware, power and permits allow.

    Mario Scriminaci is Product Director at Akamas, the autonomous optimization platform for Kubernetes and AI infrastructure. He works on how teams get more out of the GPUs they run.

  • Your GPUs Are 95% Idle, and Buying More Won’t Fix It

    Your GPUs Are 95% Idle, and Buying More Won’t Fix It

    AI is reshaping every industry, and agents are at the center of that shift. Many companies are racing to put them to work building assistants that handle customer requests around the clock, pipelines that classify and route data without human input or systems that reason across tools and act on their own. The appeal is obvious. What’s easy to miss in all the excitement is that every one of those agents, no matter how capable it looks from the outside, runs on a model, and that model runs on a GPU that is the engine keeping the whole thing moving.

    Enterprises are pouring billions into GPUs to run LLMs and AI agents. For the first time in computing history, we are seeing data-center build-outs at a scale never witnessed before, where the GPU portion alone can account for as much as 70% of total investment, according to recent industry estimates.

    Then the first real utilization reports come in. The number is hard to ignore: GPU usage as low as 5%. You might be spending millions on accelerators while 95% of that capacity sits idle and yet almost every platform and engineering leader we talk to is trying to do the same thing: buy more GPUs to support the GenAI use cases moving into production this year. The bottleneck, as it turns out, is usually not capacity but configuration. On a stack as deep as the GenAI one, the gap between a default setup and a properly tuned one can easily reach 2x throughput or more on the same hardware.

    GPU utilization drives AI factory ROI: Akamas customer data showing 5% CPU usage out of 60,000 provisioned, ~$15M/year idle

    This is the central tension we explored in our recent webinar with Daniele Zonca and Roland Huß from Red Hat, who quite literally wrote the book on running generative AI on Kubernetes.

    We’ve Already Seen This Pattern with CPUs

    Here’s a number that should worry anyone treating GPUs the way they treated CPUs. Across one of our customers, with several dozen Kubernetes clusters and around 60,000 CPUs, average utilization over a full day was 5%. At cloud prices, that adds up to roughly $15 million a year spent on resources that are almost entirely idle.

    Those are CPUs. Now apply the same operational habits (over-provisioning, ignoring efficiency, assuming the answer to every capacity question is “add more”) to hardware that costs roughly 40x more per unit and is genuinely scarce. In much of Europe you simply can’t procure H100s on demand, no matter how big your budget is. If 5% utilization on cheap, abundant CPUs costs $15 million a year, the equivalent waste on GPUs is an order of magnitude worse. It can quietly block the ROI of an entire GenAI initiative.

    Why GPUs Are So Easy to Waste

    GPUs are nothing like CPUs, and that difference is exactly why they’re hard to keep busy.

    A CPU is built for latency: most of its silicon is dedicated to control logic and caches that predict what comes next, so that a handful of threads can execute general-purpose work as fast as possible otherwise a GPU is built for throughput: almost the entire chip is devoted to computation, with thousands of cores optimized for the matrix multiplications that LLM serving depends on. Think of a CPU as a fast motorbike and a GPU as a cargo ship.

    The catch is feeding that cargo ship. Every computation requires data to move from high-bandwidth memory (HBM) into the cores, and that transfer frequently becomes the real bottleneck, which is the first reason GPU utilization ends up so low.

    This gets sharper once you look at how inference actually runs, across two very different phases:

    1. Prefill processes the entire input prompt in parallel to produce the first token. It is compute-bound and drives your time to first token (TTFT), which is the waiting time your user feels before anything appears on screen.
    2. Decode generates tokens one at a time, each step reusing the KV cache built from all previous tokens. It is memory-bound and drives your time per output token (TPOT), which measures how fast the response streams out.

    Because the same hardware handles two completely different access patterns, optimizing for one does not automatically improve the other. That mismatch is a core source of complexity in the whole stack.

    The Trade-off You’re Probably Losing On

    NVIDIA’s Jensen Huang has popularized a chart worth understanding. When you plot total token throughput (your revenue proxy, since AI services are priced per token) against per-user interactivity (latency), you don’t get a single best point. You get a Pareto front: a curve of configurations where improving one dimension always costs you something on the other.

    Pareto front of token throughput per GPU versus interactivity, showing a 2x gap between default and optimal LLM inference config

    Push for maximum throughput and single-user latency suffers. Push for fast responses and you sacrifice a large share of GPU capacity.

    Your use case fixes one constraint on that curve. A chatbot needs to emit tokens faster than human reading speed (around 200ms per token) or it feels sluggish and users leave. That means you have a fixed latency limit you cannot cross.

    But along that line sit hundreds of possible configurations, and most teams are unknowingly sitting on a suboptimal one. A single misconfiguration can cut effective throughput in half, leading teams to conclude they’re maxed out and order more GPUs, when in reality they’re using only half of what they already paid for.

    With proper tuning, doubling the throughput from existing hardware is not unusual. That is the real opportunity.

    The Playbook: Getting 2x from the GPUs You Already Have

    The GenAI stack has hundreds of tuning knobs, from the model down to the kernel and Kubernetes itself. That depth is the real challenge. The skills needed to navigate it effectively are genuinely hard to find, which means most teams end up working with defaults that no one has had the time or expertise to question. Here are the levers with the highest payoff.

    AI inference as a full-stack optimization problem across seven layers, from application and routing down to GPU and hardware instance

    1. Right-size the model, then quantize it

    Defaulting to the biggest model for every request is rarely the right call. Smaller and reasoning-tuned models can match quality on the right tasks, and a well-designed agent framework can maintain accuracy while running a smaller model. Quantization can compress a model to as little as 25–30% of its original size while preserving most of its accuracy, which is a significant reduction in resource use for almost no quality cost.

    2. Route intelligently and exploit cache locality

    Pick the right model per request, but also send follow-up messages to the replica that already holds the relevant KV cache. Multi-turn chat and agentic flows send the whole growing conversation back and forth, so cache-aware routing is critical to avoid recomputing work you’ve already done.

    3. Tune the inference engine and the parallelism strategy

    The engine layer around vLLM (or any other inference server) is evolving fast. It pays to match the parallelism strategy to your model:

    • Data parallelism (replicas behind a load balancer) scales concurrency but not single-request latency.
    • Pipeline parallelism splits a too-big model across nodes by layers.
    • Tensor parallelism splits within a node using NVLink/NVSwitch to actually improve latency.

    You can combine pipeline and tensor parallelism to get the best of both, but neither is free of trade-offs. Pipeline parallelism is constrained by inter-node network latency, which can be 25–50x slower than staying on a single node. Tensor parallelism introduces its own synchronization overhead across the NVLink fabric inside the node. The right combination always needs to be validated against your specific setup rather than assumed to work out of the box.

    4. Match hardware to the bottleneck

    If your constraint is in the decode phase, the real problem is memory bandwidth. Buying the GPU with the most teraflops will not help. Disaggregated serving (separating prefill and decode pools, as projects like llm-d enable) lets you scale each phase independently and route large inputs to dedicated prefill instances. As inference scales out, expect training-grade networking such as RDMA and InfiniBand to start mattering for inference as well.

    5. Partition GPUs with care

    If a GPU is allocated but underused, you may be able to host a second model on it, but GPUs have both a compute side and a bandwidth side. Partitioning can create a noisy-neighbor effect: spare capacity on paper does not guarantee you can add load without degrading both workloads. Always validate the impact before packing more onto a single card.

    6. Measure what you optimize

    None of these changes are safe to make without proper visibility. Track LLM-specific metrics (TTFT, TPOT), lean on the OpenTelemetry semantic conventions emerging for GenAI, and use tracing tools like MLflow to follow non-deterministic, multi-hop agentic requests. This also gives you the data for internal cost allocation across teams.

    GPU-level metrics deserve special attention because they are significantly harder to interpret than application-level ones. Tools like NVIDIA’s DCGM give you utilization numbers, but those numbers are often too coarse to identify whether your actual bottleneck is compute, memory bandwidth, or something else. GPU memory bandwidth, for instance, cannot be directly measured from DCGM at all: vLLM can expose an estimation through the –enable-mfu-metrics flag, but it remains an approximation. To make things more confusing, “GPU utilization” means very different things depending on which layer of the stack you’re looking at (allocation, kernel execution, or model FLOP/s), and conflating them leads to wrong conclusions.

    7. Load test with your own traffic, not a vendor benchmark

    Traditional tools like k6, JMeter, and Locust work at the application level, but purpose-built tools like GuideLLM (part of the vLLM project) let you build realistic, record-and-replay workloads that reflect how your system is actually used. The most important point here is to always build your test dataset from real traffic, because a vendor’s benchmark on a foundation model says nothing about how the system will behave under your specific workload. The optimization space for GenAI is too workload-dependent for generic results to transfer.

    Too Many Knobs for a Human to Tune

    With hundreds of knobs spread across three fundamentally different layers (the model, the inference server, and the GPU itself), millions of possible combinations, and a target that shifts with every new use case and every new model release, this is not a one-off exercise you do at launch. It is a continuous practice.

    Each layer also requires different expertise. The people who understand model selection and quantization are rarely the same people who can tune vLLM internals or reason about GPU memory bandwidth, and finding someone who spans all three is genuinely uncommon. That makes this well beyond what any team can manage by hand.

    The emerging answer is to use AI to optimize AI systems: autonomous tuning that explores the configuration space and finds the settings that maximize throughput while staying within your latency and cost targets. This is exactly the problem Akamas was built to solve, first for traditional Kubernetes workloads and now for GenAI inference, and we have something significant coming on that front that we will be sharing very soon.

    The Bottom Line

    As enterprises move AI agents into production and pour significant capital into the GPU infrastructure to support them, the pressure to show returns on that investment is growing fast. But utilization tells a different story, and the 5% figure is not a hardware problem. It is what happens when a genuinely complex stack gets deployed with default settings and no one has the time to tune it properly.

    The good news is that the same depth that makes the GenAI stack hard to configure also creates real headroom. The difference between where most teams are parked today and what their existing hardware can actually deliver is large enough to change the ROI equation entirely.

    You do not need more GPUs to get there. You need visibility into your actual bottleneck, a realistic picture of your workload, and a systematic way to explore the configuration space. Everything else in this article follows from those three things.

    —

    Want to go deeper? Daniele and Roland’s book on running Generative AI on Kubernetes is available for free (sponsored by Red Hat), and you can watch the full webinar replay here.

    FAQs

    Why is GPU utilization so low in enterprise AI deployments?

    GPU utilization is typically low because most teams deploy GenAI stacks with default configurations and never tune them. GPUs require a continuous, balanced stream of work to stay busy, and a single misconfiguration in the inference server, parallelism strategy, or batching policy can cut effective utilization by 50% or more.

    What is the difference between prefill and decode in LLM inference?

    Prefill processes the full input prompt in parallel to generate the first token; it is compute-bound and determines time to first token (TTFT). Decode generates tokens sequentially, reusing the KV cache; it is memory-bound and determines time per output token (TPOT). Because the two phases have different bottlenecks, optimizing one does not automatically improve the other.

    How much throughput improvement is realistic without buying more GPUs?

    Doubling throughput from existing hardware is not unusual with proper tuning. The gap between a default vLLM configuration and an optimized one, covering model size, quantization, parallelism strategy, batching, and KV cache routing, routinely reaches 2x on the same hardware.

    What is disaggregated serving for LLM inference?

    Disaggregated serving separates the prefill and decode phases onto dedicated hardware pools. This lets you scale each phase independently: prefill instances handle compute-heavy prompt processing, while decode instances handle memory-bound token generation. Projects like llm-d implement this pattern on Kubernetes.

    Why can’t I just use DCGM to find my GPU bottleneck?

    DCGM provides GPU utilization metrics, but those numbers are coarse and don’t distinguish between compute saturation and memory bandwidth saturation, which are two very different problems requiring different fixes. GPU memory bandwidth in particular cannot be measured directly from DCGM; vLLM’s –enable-mfu-metrics flag provides an approximation, but it remains an estimate.

  • KubeCon 2023 Insights: Kubernetes for GenAI, Efficiency, and Sustainability

    KubeCon 2023 Insights: Kubernetes for GenAI, Efficiency, and Sustainability

    Last week I attended KubeCon + CloudNativeCon North America 2023 in Chicago. I prioritize this conference every year, because it is the fastest way to learn what the cloud-native community is building and where it is heading.

    Three themes dominated the event. Generative AI took over the keynotes, and Kubernetes turned out to be the infrastructure underneath most of it. Sustainability moved from discussion to working tooling, with projects that measure and then cut the carbon footprint of a cluster. Cost optimization filled a whole track, and nearly every session landed on the same two problems: pod sizing and autoscaler configuration.

    Below are my key takeaways on the areas I find most crucial: cost efficiency, performance, and reliability in the K8s ecosystem. I hope they are useful for others as well.

    Let’s dive in!

    KubeCon + CloudNativeCon North America 2023 opening keynote hall in Chicago
    Welcome to Chicago for KubeCon 2023. Image courtesy of CNCF.

    GenAI on Kubernetes: the next big thing in the K8s world

    I have to admit, I was surprised by how much space Generative AI took during the keynotes. It underscores once again how AI is revolutionizing every sector of the computing industry.

    But what does GenAI have to do with K8s? As Priyanka Sharma, CNCF Executive Director, put it: not many people know that Kubernetes is the infrastructure powering the GenAI rocket.

    During her presentation, Priyanka showcased a chatbot powered by a Large Language Model (LLM) straight from her laptop. There was a minor hiccup when the database pod ran into trouble. I suspect the dreaded OOM kill or CPU throttling hit the KubeCon keynote.

    The keynotes also went into the complexities of running AI workloads on Kubernetes. Representatives from NVIDIA discussed how hard it is to rightsize GPUs and manage AI workload performance. That job is markedly different from handling traditional web applications, the initial focus of K8s. A new Dynamic Resource Allocation model is in the works. It will allow better management of GPUs, including sharing one GPU among multiple pods.

    Google’s Tim Hockin gave an insightful keynote on the future of Kubernetes. As a co-founder of K8s, he emphasized the challenge that AI workloads will pose for the platform. He predicted these workloads will demand substantial resources, potentially thousands of cores. Managing those resources efficiently is crucial, and not only for cost. It matters for sustainability too.

    Efficiency and performance tuning are going to be critical for the future success of Kubernetes. We are going to see more configuration options to fine-tune resource allocation for AI workloads. That is great news. It also means K8s users get a few more “knobs” to figure out.

    Interesting times ahead if you are interested in K8s optimization!

    Kubernetes sustainability: measure the carbon footprint, then cut it

    Sustainability really took the spotlight in the keynotes and the sessions. Why the focus? Data centers gobble up about 2% of global power, and we are nudging close to that critical 1.5 °C global warming limit. It is a wake-up call for the IT world to step up.

    Before we can make a dent in this, we need the right metrics to pinpoint where the carbon goes. That is the tricky part of sustainability. A clear picture of the carbon footprint of software and infrastructure is not straightforward to get yet.

    Four projects presented at KubeCon 2023 attack that visibility gap:

    Once you have visibility, you can start optimizing. Rightsizing your K8s pods can cut CO2 emissions by 45%, according to the figures shared in the kube-green session. Or you can turn the pods off entirely. That is the goal of kube-green, a project developed by our friend Davide Bianchi of Mia-Platform. kube-green shuts down Kubernetes resources during inactive periods.

    Proper workload scheduling can also be effective, which is where the carbon-aware KEDA operator comes in.

    Overall, it was encouraging to see sustainability as a focal point and to see the progress being made. In IT, energy waste is all too common, and we all need to play a part in reducing emissions. Interest in actual carbon reduction is still growing. The good news is that cost optimization is a far more common concern, and it often leads to carbon savings as well.

    KubeCon 2023 sustainability slide showing a 45% CO2 emissions cut from rightsizing Kubernetes pods with kube-green
    You can cut CO2 emissions by 45% with kube-green

    Kubernetes cost optimization roll call

    This year, cost optimization emerged as a significant focus. Teams increasingly recognize how inefficient a Kubernetes cluster can be. With a strong optimization strategy, they can achieve substantial cost and carbon savings.

    15,000 Minecraft players vs. one K8s cluster. Justin Head demonstrated that running K8s on bare metal led to 65% cost savings. His team used the K8s Cluster API and other open-source tools to get cloud-native benefits on a bare metal setup. One clever move was tuning K8s scheduler policies for more efficient pod placement, using NodeResourcesFit.MostAllocated. See the K8s scheduler docs for more info. Slide deck here.

    Sustainable scaling of Kubernetes workloads. Vinay Kulkarni from eBay highlighted the challenge of defining pod resource requirements accurately. Efficiency is crucial, but under-resourced pods put performance and availability at risk. eBay is exploring in-place pod resizing to adjust settings without restarts, plus AI for automated pod sizing. The goal is to optimize resource use without missing Service Level Objectives (SLOs). Slide deck here.

    Kubernetes on a budget: how to get pay-per-use right. Karim Lakhani from Intuit discussed cost-optimizing a high-scale API gateway across more than 30 clusters. His team fine-tuned Horizontal Pod Autoscaler (HPA) configurations and cluster autoscaler settings. They adjusted scaling metrics and pod resource limits manually. Slide deck here.

    K8s pod autoscaling with application-aware AI. At Akamas, hosted by Dynatrace, I presented a new K8s optimization case study. We focused on configuring HPA and pod resources to minimize cost while keeping applications reliable. Our approach directly addresses the HPA tuning challenges the Intuit team described, and it makes scalable, cost-effective applications easier to reach. Slide deck here.

    Akamas CTO Stefano Doni presenting application-aware AI-powered Kubernetes optimization at the Akamas booth at KubeCon 2023
    Sharing the application-aware, AI-powered optimization gospel at Kubecon 2023!

    In summary, it is encouraging to see K8s teams increasingly aware of the vast optimization potential within the platform. K8s offers excellent capabilities, but its full efficiency and scalability benefits require proactive tuning. It is also exciting to observe the growing trend of using AI for K8s optimization. At Akamas, we have long advocated for AI-driven optimization, and we have built an enterprise platform that addresses both cost and reliability challenges in K8s.

    FAQs

    What were the main themes at KubeCon 2023?

    Three themes dominated KubeCon + CloudNativeCon North America 2023 in Chicago. Generative AI took over the keynotes. Sustainability moved from discussion to working tooling. Cost optimization filled a track, and the sessions converged on two problems: pod sizing and autoscaler configuration.

    How much CO2 can rightsizing Kubernetes pods save?

    Rightsizing Kubernetes pods can cut CO2 emissions by 45%, according to figures shared in the kube-green session at KubeCon 2023. Shutting workloads down during inactive periods saves more. The kube-green project automates that shutdown.

    Which tools for measuring Kubernetes carbon footprint were presented at KubeCon 2023?

    Four projects stood out. The Software Carbon Intensity specification calculates the carbon intensity of software. The Carbon Aware SDK identifies cleaner energy sources. Kepler monitors energy consumption through Prometheus and eBPF. The carbon-aware KEDA operator scales workloads by cloud-region emissions.

    Why are AI workloads harder to run on Kubernetes than web applications?

    AI workloads need GPUs, and rightsizing a GPU is harder than rightsizing CPU and memory. NVIDIA speakers made this point at KubeCon 2023. Kubernetes was originally built for traditional web applications. A new Dynamic Resource Allocation model was in development to manage GPUs better, including sharing one GPU across pods.

    What did teams share about Kubernetes cost optimization at KubeCon 2023?

    Justin Head reported 65% cost savings from running K8s on bare metal. eBay described in-place pod resizing and AI-driven pod sizing to protect SLOs. Intuit fine-tuned HPA and cluster autoscaler settings by hand across more than 30 clusters. Akamas presented application-aware AI for pod autoscaling.