使用 AWS Analytics 优化 Apache Spark 工作负载 [重复].pdf

1、 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.A N T 3 1 9-R 1Optimizing Apache Spark workloads with AWS Data AnalyticsSekar Srinivasan(he/him)Pr.Analytics&AI SAAWSPraveen Mohan Prasad(he/him)Sr.Analytics&AI STAM

2、AWS 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.This is a chalk talk-there will be whiteboarding and demosInteract with us as we ask questions and ask us questions This is a 300 level session we will dive deep into Spark optimization conceptsWhat to expect from this talk 2025,

3、Amazon Web Services,Inc.or its affiliates.All rights reserved.AgendaAWS Data processing for SparkPerformance Optimized Spark RuntimeKey Optimizations in actionCost and Performance*2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.Performance Challenges in Data Processing with SparkH

4、ow to lower my Spark job costs?Why does my job take long time to load data from s3?How to optimize my joins?I am running out of disk space for my shuffle dataHow to handle data skew?2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.One Engine.Many Platforms.Peak Performance.Unified

5、Data ExperienceOne consistent Spark runtime across Glue,EMR and Athena no rewrites,no silos.AWS GlueServerless data integration built on Spark for ETL and cataloging.Amazon EMRManaged Spark clusters offering scalable,governed processingAmazon AthenaServerless Spark queries delivering fast,flexible a

6、nalytics3.x.x amzn 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.4.5xOpen Source Apache Spark and Iceberg2xBetter for writes compared to opensource Spark and Iceberg5.4 xEMR Spark 4.0 compared to Spark 4.0EMR accelerates time-to-insight and optimizes performanceMethod:Trailing 1