4 Snowflake-The Evolving Landscape of Iceberg REST Catalog.pdf

1、The Evolving Landscape of Iceberg REST CatalogYufei G 2024 Snowflake Inc.All Rights Reserved Caused by:org.apache.iceberg.exceptions.NotFoundException:Failed to open input stream for file:s3:/some/path/to/table/metadata/13648-45c53fb2-5124-4541-ace3-c63ed91e1d26.metadata.jsonAgendaWhy REST Catalog?R

2、ecent Features&Future DirectionGrowing EcosystemA Server Implementation:Apache Polaris(Incubating)Storage OnlylTable still workslPoor addressing:using path for namespace and table lLack of Data Governance,e.g.,data lineagelPerformance concerns No caching,Repeated file reads for queries and metadatal

3、Compatibility concerns:old clients vs.new clientsENGINEENGINE 2024 Snowflake Inc.All Rights Reserved Traditional Iceberg CatalogAddress table nicelyProvide basic data governance featuresEngine writes metadata.json filesEngine retries for conflict resolution:lock the tablePerformance concerns:No cach

4、ingCompatibility concern:old clients vs.new clients 2024 Snowflake Inc.All Rights Reserved CATALOGCATALOGENGINEENGINEHive CatalogJDBC CatalogIceberg Optimistic ConcurrencyIceberg Table Client1Client2v1v1v2v2v3v3Happy PathIceberg Optimistic ConcurrencyIceberg Table Client1Client2v1v1v2v2v3v3v2v2Confl

5、ict ResolutionReliable and Low-latency CommitsBetter Conflict ResolutionEasy Client ImplementationData GovernanceIceberg REST Catalog Shifting Client responsibilities to the server A simple idea to change way we use the Iceberg table completely 2024 Snowflake Inc.All Rights Reserved CATALOGENGINERES

6、T CatalogShift Client Responsibilities to ServeroServer-side handles commit and retry.No table lock.Faster and more reliableoServer-side writes the metadata.json file,no compatibility issueImproved Decision Making at Commit TimeoBetter Conflicts resolutionDDL vs DML,dont failed append when it confli