向高盛传奇的Lakehouse学习数据治理.pdf

1、Legend LakehouseModernizing Data Governance with the Legend LakehouseLessons from Goldman SachsAbhishek Narang&George WuJune 11,2025Agenda2Legacy Architecture,Technical Challenges and ObjectivesDesign Principles of the Legend LakehouseLegend Lakehouse Architecture How Databricks fits into Legend Lak

2、ehouse ArchitectureData-Flow Lifecycle walkthroughReality Check Why change couldnt wait?Legacy Architecture Challenges 2 massive on-prem clusters(1000 nodes each,40+PB HDFS)Fixed on-prem hardware with no elasticity,2x data every 18 monthsDisparate architecture for batch and streaming workloadsHomegr

3、own catalog lacked support for modern,open formats Duplicate data copies across OLAP systems(on-prem and cloud)Entitlements stuck at file-system ACLs(no fine grained or dynamic controls)Scalability and Infrastructure LimitsData Management&Architecture GapsTight coupling of compute and storage limite

4、d scalability and flexibility Intricate and hard to maintain chargeback processesLimited visibility into usage patterns and data lineage,complicating audits Operational Complexity and Cost Opacity4Why the Hadoop-based stack could no longer keep up What are we trying to solve?Data Discovery&AccessDat

5、a Quality&SilosCost&ComplexityCommon pain points across data teamsHard to get access once discovereddiscoveredData is hard to find Data is hard to find Sharing is slow and Sharing is slow and opaque to opaque to data data producersproducersChallenging to standardize and experimentSilos hinder cross-

6、team collaboration Variable data qualityDeep dependency on engineers/expertsInfrastructure and data duplicationHard to use the platform as building blocksLegend Lakehouse IdeologySelf-ServiceInteroperableScalable+ServerlessFoundational Design PrinciplesSupport for Structured and Unstructured dataDat