利用 CXL 技术实现内存高效的 RAG 流水线.pdf

1、Towards Memory-efficient RAG pipelines with CXL technology Siamak Tavallaei,Arun George,Kyumin Park SamsungOpen Systems for AI-ITOutline:Memory-efficient RAGRAG with CXL A brief on how RAG pipelines use memory How effective adding memory is in increasing performance How CXL Memory Expansion helps Ho

2、w CXL-enabled Memory Pooling helps dynamic pipelinesMemory usages in RAG pipeline We focus on the memory usage scenarios In particular,Vector DB use cases are interesting We observe memory demand scenarios1.Data Ingestion phase Huge transient memory demand Large memory required for creation and hand

3、ling of Vector embeddings 2.Query phase Memory pressure based on workload These scenarios qualify for memory expansion(consistent/on-demand)Semantic searchVector DBVectorizationPrompt engineeringLocal Knowledge BaseQuery generationLLMAnswerDB InsertionDB Search1.Data Ingestion2.QueryConvert knowledg

4、e base into vector embeddingsLocal documents:text,PDFs etc.User asks a questionPerform semantic search on Vector DB and get similar results(Context)Augment the question with this contextQuestion+context is used in the model to find the answerMemory demandMemory demandExplorationExploration:How can C

5、XLCXL help?Consistent memory demand:CXL-based memory expansion can help expand memory CMM-D provides memory expansionTransient memory demand:CXL-based memory pooling can handle transient memory spikes CXL 3.0 Dynamic Capacity Devices(DCD)provide the mechanismSolutions:CXL Switch+CMM-D(CXL-connected

6、DRAM)Vertical/Horizontal scaling with CMM-D CMM-D can deliver price,performance,and scalability benefits in vertical and horizontal scalingCPUServer#1RDIMMCMM-DVertical scaling with CMM-DRDIMMCPUServer#3CMM-DRDIMMCPUServer#4CMM-DRDIMMCPUServer#1CMM-DRDIMMCPUServer#5CMM-DRDIMMCPUServer#2CMM-DHorizont