为未来十年的增长设计数据湖屋系统.pdf

1、Engineering Data Data Lakehouse Systems for the Next Ten Years of GrowthAdi PolakData&AI Technology StrategistDatabricks2023About me Adi PolakThesis in Machine Learning and Cyber SecurityAuthor:Scaling Machine Learning bookWork experience:Fortune 500 and startupsData&AI technology StrategistOur 20 m

2、in together Overcoming CAP Theorem Boil the Ocean Build Rube Goldberg Machines Dont Budget for Complexity Dont use Managed capabilities and in house knowledge*How NOT to build data Lakehouse systems Overcoming CAP TheoremPractical aspects of CAP TheoremIn designing a database,we can choose two of th

3、e threeConsistencyAvailabilityPartition tolerancePractical aspects of CAP TheoremDistributed systems:provide two of three properties simultaneously:Consistency,Availability,&Partition tolerance.Boil the OceanLambda architecture1_DAIS_Title_SlideBuild Rube Goldberg Machines“Self-Operating Napkin”But

4、didnt budget for complexityCourtesy of RisingWave-https:/www.risingwave.dev/Forget to use Managed services and in house knowledgeMy team has specific skills and we want to build in house generative AI platformWork with lowcode to expend abilities and Turn AI problems into a data engineering problemCourtesy of Prophecy Booth#416Sum it upCAPBoil the OceanBuild Rube Goldberg MachinesDont forget to:Budget complexityExisting skills&skills planningManaged servicesThank you!L