APACHE PARQUET 中用于数据湖仓一体更新插入的快速写入复制.pdf

1、Fast Copy-On-Write with Apache ParquetXinli Shang,Mingmin Chen Uber Data Infra shangxinliapache.orgSpeaker Intro Xinli ShangSenior Manager Uber Apache Parquet PMC chair,Presto committer Mingmin ChenDirector Uber Data Infra AgendaUber data architectureUpserts challenges Apache Parquet introduction Fa

2、st Copy-On-Write within ParquetConclusion&future work Uber Data ArchitectureIngestion(Parquet,Hudi,Hoover)Online StorageEventsTelemetryFeedsKafkaData LakeCompute Fabric RealReal-Time AnalyticsTime AnalyticsData Platform&ToolsData Platform&ToolsBatch AnalyticsStream Processing(Flink,AthenaX)Complex P

3、rocessing(Spark)Data Workflow(Piper,uWorc)BI Tools(QueryBuilder,Dashbuilder,Summary)Metadata Platform(Databook,Quality,Lineage)Interactive(Presto)ETL(Hive)DashboardsReportsAnalysisMachine LearningIn-memory(Pinot)storageSecurityGlobal Data WarehouseUber Lakehouse PlatformA dataset need to be updated

4、for different use caseTrip fare is changed Change Data Capture(CDC)Change data for compliance reasonUpdating datasets is not that easyAppend only systemStructure data with compression,e.g.ParquetLocating affected data files is slow Updates in LakehouseLakehouse Data StackData-A collection of files(e

5、.g.Parquet)storing tables content Metadata-Info about a table schema,partition,file and snapshot details Index-Data structure to efficiently locate records within a tableLogic View of Table Updatetrip_uuidtrip_fair_tipsdatestr111-1111-1111-1111-111111111111111132023-06-3022222-2222-2222-2222-2222222

6、222222282023-07-01change trip_fair_tips to 5$where trip_uuid=111-1111.and datestr=2023-06-30trip_uuidtrip_fair_tipsdatestr111-1111-1111-1111-111111111111111152023-06-3022222-2222-2222-2222-2222222222222282023-07-01Update in Plain Hive Table Formatupdate trip_fair_tips to 5$where trip_uuid=345-2342.a