使用 Delta-rs 和 Delta-Kernel-rs 提供 CDC 提要.pdf

1、Using Delta-rs and Delta-Kernel-rs to Serve CDC FeedsOussama Saoudi Stephen CarmanAgendaOverview of CDF and applicationsIntroduction to delta-kernel-rs and its CDF implementationIntroduction to delta-rs,its CDF implementationProblems with that implementationHow Kernel fixes those problemsKernel Inte

2、grationDemonstration3About the AuthorsSr.Resident Solution Architect at DatabricksI work specifically with highly regulated customers in HLS and Finance customersContributor to delta-rs and occasionally delta-kernel-rs Author of constraints,table features,CDF reads,and UC vended creds in delta-rsBee

3、n writing Rust around 8 years4Steve CarmanAbout the AuthorsSoftware Engineer at DatabricksWorking on storage teamAuthor of CDF and ICT in KernelMuch Smarter than Steve5Oussama SaoudiIntroducing Change Data FeedRepresents all the changes that happened to a table between two versionsWhat is it?Opidval

4、add1100Opidvaladd2200Opidvaldelete2200add2220idval11002220v1v2v3Applications of Change Data Feed-Why maintain a materialized viewIncremental Materialized View MaintenanceIDPriceLocation8$2Ottawa97$50Ottawa5$25Montreal309$13MontrealSELECTSELECT SUMSUM(price)(price)ASAS total_pricetotal_priceFROMFROMs

5、ales;sales;Full table scan each time you check your revenue!Applications of Change Data FeedIncremental Materialized View MaintenanceOpIDPriceLocationadd8$2Ottawaadd97$50OttawaUPDATE revenue_viewSET total_revenue=total_revenue+(SELECT SUM(price)FROM sales_cdf);OpIDPriceLocationadd5$25Ottawaadd309$13

6、OttawaApplications of Change Data Feed-Incrementally process and transform new dataETLJSON blobsETLThe Delta Ecosystem Before KernelApacheFlinkTrinopolarsApachePulsarRayConnectorsProtocol changesN x McomplexityClickhouseVariant TypeChange Data FeedIdentity columnsCoordinated CommitsDelta ConnectType