范斌-加速AI推理与检索生成：在PB级数据【Alluxio-范斌】湖上实现Parquet查询1000倍性能提升.pdf

1、加速AI推理与检索生成：在PB级数据湖上实现Parquet查询1000倍性能提升Bin Fan,VP of Technology A01The HookThe Challenge:Sub-Millisecond Point Lookups on Petabyte Data Lakes?Executing point lookup queries like“SELECT ID,DATA FROM TABLE WHERE ID=123”over partitioned Iceberg data lake(Parquet)of tens or even hundreds of PBon object

2、 stores(e.g.,S3)within sub-millisecondData Lake AppsWhy This Matters-Agentic Memory:-AI Agents require instant recall of vast historical knowledge and context.-Online Feature Store:-Real-time inference demands immediate access to fresh,relevant features.-Real-time Personalization&Recommendation:-Del

3、ivering personalized experiences in milliseconds is key to user engagement and conversion.These use cases are driving the need for extreme low-latency access to large-scale data.Common Approaches&Their Limitations:OLAP EnginesHow it works:Executing point lookup queries directly against S3 Parquet vi

4、a an OLAP engine.Pros:-Mature ecosystem,well-supported.-Handles complex analytics.Cons:-Overkill:Heavyweight for simple key-value lookups.-High Latency&Concurrency:Query planning,scheduling,and full Parquet scan overheads make sub-millisecond unachievable.Data Lake Agentic AppsQuery EngineCommon App

5、roaches&Their Limitations:In-Memory KV StoresHow it works:Exporting tables or relevant data portions into an In-Memory KV Stores.Pros:-Low Latency:Fast key-value access.Cons:-Prohibitive Cost at Scale:Extremely expensive to fit Petabytes of data into memory.-Data Sync Complexity&Staleness:Requires E

6、TL pipelines,leading to data lag and consistency issues(the Dual-Store Problem).-Operational Overhead:Managing two separate data systems.In-memory KV StoreData Lake Data CopyAgentic AppsImportParquet on S3:Why Its Not Natively Sub-Millisecond?Data Lake AppsHow it works:Parquet files are self-indexed