郭人通-Milvus：探索云原生的向量数据库（GOTC上海会场）（19页）.pdf

1、Milvus：探索云原生的向量数据库郭人通 2021年07月10日合伙人&架构师计算机软件与理论博士兴趣领域：分布式系统、数据库、异构计算CCF 分布式计算与系统专委会委员郭人通Milvus 系统架构师Why Vector DatabaseInt,float,string,textjsonimagevideoaudiodomain specific Structured dataUnstructured dataABCDEFGABCDEFG2021.04.102021.04.10Data are Increasing Horizontally:TypesWhy Vector DatabaseD

2、ata are Increasing Vertically:SemanticsembeddingsRicher semanticsembeddingsembeddingsWhy Vector DatabasebooleanembeddingIntfloatstringTypes with high abstractionWhy Vector Database80%data growth is unstructured,over 40,000 Exabytes per year The big pictureRaw Data(Structured/Unstructured Data)Latent

3、-semantic Data(Embedding Vectors)Search&Analytics EnginesVector EngineCo-processorsQuery Parser/PlannerAbout MilvusThe Idea2018.10Milvus 0.1Release2019.041st SeedUser2019.06OpenSource2019.102020.03Milvus 1.0Release2021.03Milvus 2.0 Release2021.062019.06ProjectGraduationJoinedLF AI&DATAA brief histor

4、yStone AgeBronze AgeIron AgeRoman TimesDatabaseFrameworksFor AI ModelFrameworks For AI DataMagnetic Taps1970sCODASYL IMSRelational Model1980sSQLSIGMODVLDBERIngresSystem RDBMSs for PCDB2OracleSybaseInformix1990sMySQLNowPostgresSQLSnowflakeHBaseCloud NativeCAP2010s2016NowMATLABOpenNNTorchCaffeGPU supp

5、ortChainerTheanoTensorflowCaffe2PytorchCNTKMXNetKerasONNX2019NowFaissAnnoyhnswlibDB-pluginES-pluginPineconeMilvusProximaWe are hereKey challengesFast System Evolution01Multi-environment Deployment02Hardware Cost03Diverse Workloads04Complex,Hybrid Query05Architecture:logical log as the system backbon

6、eArchitecture:take a closer lookArchitecture:take a closer lookCoordinatorsLog BrokerLog SubscribersStorageFrom Milvus 1.0 to Milvus 2.0MilvusMilvus FunctionalitiesProxy,DDL handling,DML handling,DQL handlingMetaLog Pub-subDataetcdPulsarRocksDBS3MinIOTxnKVKVMsgStreamFrom Milvus 1