简化在 EKS 上使用 AWS DLC 进行生产环境 vLLM 部署.pdf

1、 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.A I M 3 5 2Streamlining vLLM deployment for production with AWS DLCs on EKSArindam PaulSr.Product Manager,AWS GenAIVishal NaikSr.Solutions Architect,AWS GenAI 2025,A

2、mazon Web Services,Inc.or its affiliates.All rights reserved.AgendaLLM Deployment ChallengesvLLM AWS Deep Learning ContainersArchitecture Deep DiveDemoBest Practices for ProductionQ/A 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.In todays presentation“Financial Fraud Detection”

3、Agent generating a detailed report on Transaction AnalysisLeveraging vLLM on Amazon EKSDeepSeek R1 Distill Qwen 32B(https:/huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B)Sneak Peek 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.2025,Amazon Web Services,Inc.or its affilia

4、tes.All rights reserved.LLM Deployment Challenges 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.LLM Deployment ChallengesComplex Dependency ManagementCompatibility issues,across CUDA,PyTorch,NCCL,and DriversScaling Across NodesDistributed inference adds networking and coordinati

5、on overheadSuboptimal Infra UtilizationIdle memory,inefficient batching,and underused accelerators Model Storage&Loading BottlenecksLarge checkpoints cause slow startup and I/O constraints 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.2025,Amazon Web Services,Inc.or its affiliat

6、es.All rights reserved.vLLM AWS Deep Learning Containers 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.AWS Deep Learning Containers(DLCs)Pre-built Docker images for deep learning with secure and optimized runtime provided at no cost.2025,Amazon Web Services,Inc.or its affiliates