Blaze：SparkSQL Native算子优化在快手的设计与实践(1).pdf

1、Blaze：SparkSQL Native算子优化在快手的设计与实践王磊|快手大数据SQL引擎负责人2023 What is Blaze Architecture and Implementation details Current Progress and Future WorkWhat is BlazeRoadmap of Engine Spark 1.0Volcano Model Spark 2.0Whole-stage-codegen Spark 3.0Adaptive Query Execution Whats next?Vectorized ExecutionRelation wo

2、rksProjectCorporationDescriptionVeloxMetaVectorized acceleration libraryGlutenIntel&KyligencePlugin to offload SQL Engine to Native LibraryPhotonDatabricksNative vectorized engine for SparkNative CodegenAlibabaGenerate native code for SparkWhat is BlazelThe Blaze is an accelerator for Apache Sparkl

3、Blaze leverages native vectorized execution to accelerate query processing.It combines the power of the Apache Arrow-DataFusion library and the scale of the Spark distributed computing frameworkRelationship of Blaze&DataFusionSparkBlazeGlutenDataFusionVeloxPhysical PlanProtoBufSubstraitArchitecture

4、andImplementation detailsArchitecture OverflowSpark on BlazeA Simple DemoHigh-level ComponentslBlaze Session Extension：hooks the whole accelerator into Spark execution lifetimelPlan SerDe：serialization and deserialization of DataFusion plan with protobuflJNI Gateways：passing data and control through

5、 JNI boundarieslNative Operators：defines how each SparkPlan maps to native execution counterpartsExecution Flow DetailslPhysical Plan ConversionlGenerate and Submit Native PlanlNative ExecutionPhysical Plan ConversionGenerate and Submit Native PlanNative ExecutionMore Implementation DetailslCompatib

6、le with UDFlMemory ManagementlMore Efficient Operator ImplementationCompatible with UDFMemory ManagementMore efficient operator implementationlComparasion operation：arrow-rowlSort operation：sort_unstable in rustlHashMap：hashbrown(SwissTable hashmap)lColumnariz