存算分离：Apache Doris 部署新范式 - 杨勇强.pdf

1、存存算算分分离离：A Ap pa ac ch he e D Do or ri is s 3 3.0 0 部部署署新新范范式式杨杨勇勇强强 Apache Doris PMCDoris Summit Asia 2024Doris Summit Asia 2024Doris Summit Asia 2024Doris Summit Asia 2024Doris Summit Asia 2024Doris Summit Asia 2024Doris Summit Asia 2024Doris Summit Asia 2024Doris Summit Asia 2024目录存算分离技术特性01存算分离典

2、型应用02Doris Summit Asia 2024Doris Summit Asia 2024Doris Summit Asia 2024Doris Summit Asia 2024Doris Summit Asia 2024Doris Summit Asia 2024Doris Summit Asia 2024Doris Summit Asia 2024Doris Summit Asia 2024存存算算分分离离架架构构Meta ServiceCompute Group ABECacheBECacheBECacheCompute Group BBECacheBECacheBECacheS

3、3/OSS/Azure/GCP/HDFS存算一体架构存算分离当前架构存算分离目标架构DiskFEDiskFEDiskFEDiskBEDiskBEDiskBEDiskBEDiskFEDiskFEDiskFEMeta ServiceCompute Group AComputeNodeCacheCompute Group BComputeNodeCacheComputeNodeCacheComputeNodeCacheComputeNodeCacheComputeNodeCacheS3/OSS/Azure/GCP/HDFSDoris Summit Asia 2024Doris Summit Asia

4、 2024Doris Summit Asia 2024Doris Summit Asia 2024Doris Summit Asia 2024Doris Summit Asia 2024Doris Summit Asia 2024Doris Summit Asia 2024Doris Summit Asia 2024存存算算分分离离查查询询性性能能完完全全命命中中：预热之后数据都在doris page cache 或者 linux page cache部部分分命命中中：开始时三级cache都为空，顺序跑 tpcds 的查询，取第一遍的成绩完完全全未未命命中中：每个 TPCDS 的 SQL 开始

5、时清空三级缓存多多层层 C Ca ac ch he e0100200300400500600700800完全命中缓存部分命中缓存完全未命中存算分离存算一体本地磁盘 cache 压缩数据Linux Page Cache 压缩数据Doris Page Cache 解压后的数据持久化存储S S3 3A Amma az zo on nDoris Summit Asia 2024Doris Summit Asia 2024Doris Summit Asia 2024Doris Summit Asia 2024Doris Summit Asia 2024Doris Summit Asia 2024Dori

6、s Summit Asia 2024Doris Summit Asia 2024Doris Summit Asia 2024存存算算分分离离数数据据及及时时性性32 并发 flink 写入，checkpoint 周期 5s。A:引入 metaservice 的存算分离实现；B：meta 写入对象的存算分离实现。Doris Summit Asia 2024Doris Summit Asia 2024Doris Summit Asia 2024Doris Summit Asia 2024Doris Summit Asia 2024Doris Summit Asia 2024Doris Summit