5.李呈祥 -Apache Spark最新技术进展和3.0+展望（21页）.pdf

1、Apache Spark最新技术进展和3.0+展望李呈祥（司麟）阿里巴巴高级技术专家计算平台事业部Agenda3.0Spark on CloudData Warehouse EnhancementSpark+AIA Unified Analytics Engine for Large-scale Data ProcessingEasy-to-use APIRich Ecosystem SupportEfficient EngineData Warehouse EnhancementDelta Lake ACID Transactions Scalable Metadata Handling T

2、ime Travel(data versioning)Open Format Unified Batch and Streaming Source and Sink Schema EnforcementComing soon:Audit HistoryFull DML SupportExpectationsData Source V2 Unified API for batch and streaming Flexible API for high performance implementation Flexible API for metadata management Target 3.

3、0Runtime OptimizationDynamic optimize the execution plan at runtime based on the statistic of previous stage.Self tuning the number of reducers Adaptive join strategy Automatic skew join handlingAdaptive ExecutionEMR Runtime Filter Filter big table with runtime statistic of join key.Support both par

4、titioned table and normal table.EMR Spark Relational CacheUser may analyze data in certain access patternRegularly join 2 tables?Regularly aggregate by certain fields?Regularly filter by certain fields?Data Organization:partition,bucket,sortfile index,zorderData pre-computation:pre-filterdenormaliza

5、tionpre-aggregationMake data adaptive to compute,so spark compute faster.EMR Spark Relational CacheEMR Spark Relational CacheEasy to build and maintainTransparent to userCREATE VIEW emp_flat AS SELECT*FROM employee,address WHERE e_addrId=a_addrId;CACHE TABLE emp_flatUSING parquetPARTITIONED BY(e_ob_

6、date)EAJFP-User Query-SELECT*FROM employee,address WHERE e_addrId=a_addrId and a_cityName=ShangHaiSpark OptimizerCFPEAJP-Cached Mata-emp_flatoptimized planSpark on CloudStorage and Computing DisaggregationWhy disaggregate storage and computing:Pay as you go.Scale independently of each other.More rel