Optimizing Spark costs on AWS with AI-powered tuning

During the Neotys Jurassic PAC 2020 virtual conference, Luca Cavazzana (Software Engineer at Akamas) explained how Akamas uses AI techniques to find the optimal configuration for Spark applications. He presented a study where Akamas tuned both Apache Spark parameters and EC2 cluster size, finding an optimal trade-off between resource allocation and execution time that minimizes the overall cost.