AWS EMR clusters are almost always overprovisioned. Spark’s default settings target small clusters, not production, so teams add hardware to feel safe and the AWS bill climbs. This case study shows a different path. Akamas used AI to automatically tune Spark and EMR cluster configuration together for a real Digital Media service. The result was a 52% lower AWS bill, 11% faster job execution, and 17% lower peak memory. Akamas reached it in about three and a half days, with no application code changes.
Big data applications usually leave large gains on the table, both in performance and in cost. The underlying infrastructure, whether on-premise or in the cloud, is often inefficient and over-provisioned to protect the performance-to-cost ratio.
Vendors publish best practices, yet configuring and tuning big data applications stays hard. Performance and reliability are highly sensitive to the specific application and to the datasets it processes. Data scientists own most of these applications. They rarely have deep expertise in tuning the lower layers of the stack, and performance tests seldom run in environments that mirror production.
To show a different approach, we walk through a real case. Spark configuration and AWS Elastic MapReduce (EMR) resources were tuned automatically to cut the cloud bill, and that also shortened job execution time.
Why Spark and EMR tuning is so hard
The first challenge is the sheer number of parameters. Spark exposes about 200 parameters. The Cluster Manager adds about another 200. Together that is thousands of possible configurations to choose from before releasing to production. Spark’s defaults do not help here. As Karau and Warren note in High Performance Spark, the default settings target very small clusters and are not recommended for production.
The second challenge is the data-driven nature of big data. Performance and reliability depend heavily on both the application and its datasets:
- When input data changes over time, data may need to be re-partitioned across worker nodes to avoid out-of-memory failures.
- A new application release can change how data flows, so a previously good configuration no longer applies.
- Production often differs from test, so tuning has to happen against a production-representative environment.
Universal rules for maximizing Spark performance are therefore almost impossible to define. Karau and Warren call configuring a Spark job “as much an art as a science,” dependent on the data storage setup, the volume of data processed, and the type of job.
The third challenge is the cloud itself. Spark jobs increasingly run on managed cloud clusters rather than on-premise, because the cloud makes cluster management cheaper and easier to match to fluctuating demand. That flexibility adds options. AWS EMR alone offers about 230 combinations of instance families and sizes. Each has its own performance profile and its own cost.
Performance tuning in modern architectures reaches its highest complexity with big data applications. So is big data optimization an impossible problem? The case below says no.
The workload: a Spark batch on an ephemeral EMR cluster
The case is a critical Digital Media service serving about 1 million users per day. It runs a Business Intelligence application that analyzes content consumption from online stores, streaming services, and news sites. On average it collects, transforms, and aggregates 25 million events per day from raw logs and user transactions. Each event is a user interaction such as browsing, searching, accessing, or consuming content from the catalog.
The core component is the Analytical Engine, a Spark batch that runs several times a day on an ephemeral EMR cluster.
The customer faced long execution times and higher-than-desired cloud bills, even though the cluster was overprovisioned.
Preliminary analysis showed very low parallelism. Only a fraction of the available executors (11) did useful work. The likely cause was skewed data distribution. For most of the batch, the majority of executors sat idle, waiting for the few executors that were doing the work. A code-level fix was possible, but the customer needed to cut cloud cost quickly, without waiting for a full re-engineering.
Two optimization areas emerged, at the AWS cluster level and at the Spark level:
- Right-size the AWS cluster. Balance allocated resources against execution time, and test whether a few large instances beat many small ones.
- Find the Spark and YARN configuration that maximizes performance while guaranteeing each batch finishes within 4 hours, before the next batch starts.
Acting on either area was hard for the customer. The environment was complex, the data volume large, the time for tuning short, and the skills to select cloud instances in this context were scarce.
Letting AI find the optimal Spark and EMR configuration
We made no assumptions about which options were worth exploring. Akamas patented AI searched for an optimal configuration on its own. The goal was to reduce the cloud bill. The constraint was not to increase execution time by more than 50%.
The study progressed one experiment at a time. Each blue dot is one experiment, meaning one set of Spark and EMR parameters. The yellow bands mark “failed” configurations that either did not complete or broke the constraint. Akamas uses these failures to prune the search further.
The first experiment is the baseline, the customer’s initial configuration. Every experiment after it is a candidate that Akamas AI chose based on prior results. Most configurations scored lower cost than the baseline, which confirms the cluster was overprovisioned.
Akamas ran 50 experiments in total. Better configurations appeared within the first 24 hours. At experiment #27, after about three and a half days, Akamas converged on a much cheaper configuration: 52% lower cost.
What the cheapest configuration changed
Both the instance types and the number of worker nodes changed:
- The master node scaled down from an m5.4xlarge to an m5.xlarge, about one-fourth of the original resources. The master runs base Spark services, not core computation, so it did not need the extra capacity.
- Worker nodes went from three smaller m5.4xlarge instances to one larger m5.8xlarge, about a 33% reduction. This runs against the usual advice to prefer many small nodes. It works here because this application does not exploit massive parallelism.
- The driver received more resources, from 3 to 5 CPUs and from 17 to 32 GB.
The aggregate cluster CPU and memory usage tells the rest of the story. The baseline is the orange line, the optimal configuration is green:
- The optimal configuration reaches higher average CPU usage, so it uses the allocated CPUs better.
- The optimal configuration uses less memory, with 17% lower peak memory usage. That headroom makes the application more resilient.
There is a second benefit in the same chart. Execution time dropped by 11%, even though speed was not the goal. This is expected: a shorter cluster uptime lowers cost.
When speed is the goal: 20% faster, still 37% cheaper
Change the goal to minimize execution time and the optimal configuration shifts. It uses an even bigger instance, an m5.12xlarge, with one more executor and a much larger driver. This configuration cuts execution time by 20% while still delivering a 37% cost saving.
Conclusions
Configuring and tuning big data applications is hard, even for experienced performance engineers and data scientists. With the right approach and tool, it becomes fast and repeatable.
Akamas supports a fully automated optimization process. Workflows run experiments unattended. A single workflow can provision a cluster, submit Spark jobs, and collect the KPIs and cost metrics needed to score each configuration against the goal.
Once you define the goal (for example, a performance-to-cost ratio) and the constraints (for example, SLOs), Akamas patented AI finds an optimal set of Spark and EMR parameters in a short time, even for complex applications.
Running Spark on Kubernetes instead of EMR? The same approach applies to Spark and big-data cluster tuning on Kubernetes. For the instance-selection side of the problem, see AI-driven cloud instance optimization. To validate changes safely before production, see Akamas Offline.
FAQs
What is AWS EMR cost optimization?
AWS EMR cost optimization means tuning the EMR cluster and the Spark job together to run the same workload for less money. It covers instance type and count, driver and executor sizing, and Spark and YARN parameters.
Why are Spark’s default settings bad for production?
Spark defaults are designed so that jobs run on very small clusters. They are not meant for production scale. On production data volumes, defaults usually leave executors idle and resources overprovisioned.
Is it better to run many small nodes or fewer large nodes on EMR?
It depends on the application. The common advice favors many small nodes, but applications with low parallelism can run cheaper on fewer large nodes. In this case, one m5.8xlarge beat three m5.4xlarge instances.
How much can tuning Spark and EMR together save?
In this case study, the cost-optimized configuration cut the AWS bill by 52%. A speed-optimized configuration cut execution time by 20% while still saving 37%.
Does EMR cost optimization slow Spark jobs down?
No. Removing overprovisioned resources raised CPU utilization and cut idle time. The cheapest configuration ran 11% faster, and the fastest ran 20% faster.
