使用 Lakeflow 声明式管道掌握变更数据捕获.pdf

1、Mastering Change Data Capture With DLT Lakeflow Declarative PipelinesRay Zhu(Databricks)Jacob Gollub(Square)Toby Tan(Square)AgendaLakeflow Declarative Pipelines OverviewChange Data CaptureHow We Built ItCustomer Story-Square3Lakeflow Declarative Pipelines OverviewQuality ChecksGovernanceData Discove

2、ryIts difficult to build and operate reliable data pipelines5your transformation logicDependencyManagementDaily PartitionComputationCheckpointing&RetriesBackfill HandlingVersion ControlDeployment InfrastructureData LakeOrchestration Data WarehouseStreamingBIData ScienceGenerativeAIMachine LearningOR

3、CHESTRATETRANSFORMINGESTConnectDeclarative PipelinesJobsPart of LakeflowOne data engineering experience powered by data intelligenceSimple native ingestion connectorsReliable data pipelines made easyOrchestrate workflows across the entire Data Intelligence Platform3,000+Customers 7Change Data Captur

4、eChange Data FeedsOLAPOLTPMessageBusesCloud StorageidnamecityoperationsequenceNum1AliceSeattleINSERT12BobBostonINSERT21AliceSan FranciscoUPDATE3.A Simple ExampleidnamecityoperationSequenceNum1AliceSeattleINSERT12BobBostonINSERT21AliceSan FranciscoUPDATE3SCD Type 1idnamecity1AliceSan Francisco2BobBos

5、tonidnamecityoperationSequenceNum1AliceSeattleINSERT12BobBostonINSERT21AliceSan FranciscoUPDATE3SCD Type 2idnamecitystartend1AliceSeattle131AliceSan Francisco3null2BobBoston2nullAPPLY CHANGES INTOSourceKeysSequence byIgnore null updatesDelete and truncateSCD Type 1/Type 2Column listTrack history on

6、column list APPLY CHANGES FROM SNAPSHOTSourceKeysSequence byIgnore null updatesDelete and truncateSCD Type 1/Type 2Column listTrack history on column list How We Built Apply ChangesidnamecityoperationSequenceNum1AliceSeattleINSERT12BobBostonINSERT21AliceSan FranciscoUPDATE3SCD Type 1idnamecity1Alice