通过高效的数据管道解锁 Lakehouse.pdf

1、2024 Databricks Inc.All rights reservedUNLOCKING THE LAKEHOUSE UNLOCKING THE LAKEHOUSE WITH EFFICIENT DATA PIPELINESWITH EFFICIENT DATA PIPELINESPrabodh MhalgiPrabodh MhalgiCapital OneCapital OneUsman ZubairUsman ZubairDatabricksDatabricks1June 2024June 20242024 Databricks Inc.All rights reserved2Us

2、man ZubairUsman ZubairLead TechnologistLead TechnologistDatabricksDatabricksPrabodh MhalgiPrabodh MhalgiSr.Lead Data EngineerSr.Lead Data EngineerCapital OneCapital One2024 Databricks Inc.All rights reserved Road to the Lakehouse Building Efficient Data Pipelines Key Takeaways The Road Ahead3AGENDAA

3、GENDA2024 Databricks Inc.All rights reserved4CAPITAL ONE JOURNEYCAPITAL ONE JOURNEY20142014Began to modernize Began to modernize our architecture with our architecture with RESTful APIsRESTful APIs20152015Became open source Became open source firstfirst20162016Declared all in on Declared all in on p

4、ublic cloud and public cloud and introduced Databricks introduced Databricks at Capital Oneat Capital One20172017Modernized our data Modernized our data ecosystem on cloudecosystem on cloud20202020Completed move to Completed move to public cloudpublic cloudOperationalized data Operationalized data l

5、akehouse v1lakehouse v120232023Evolved data Evolved data lakehouse to v2 with lakehouse to v2 with open data formatsopen data formats20242024Advancing to Advancing to the forefront of the forefront of leveraging AIleveraging AI2024 Databricks Inc.All rights reservedCAPITAL ONE SCALECAPITAL ONE SCALE

6、5Petabyte scale data lakeTerabytes added every day1000s of datasetsNear real-time ingestion1000s of users2024 Databricks Inc.All rights reserved6ROAD TO THE LAKEHOUSEROAD TO THE LAKEHOUSEChallenges and ObjectivesChallenges and ObjectivesPoor Query Poor Query ExperienceExperiencePerformance&Performan