在 AWS AI 芯片上使用 vLLM 大规模部署 LLM.pdf

1、 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.A I M 3 5 5Deploying LLMs at scale with vLLM and AWS AI ChipsPinak PanigrahiYahav Biran,PhDSr.GenAI Inference Performance ArchitectAnnapurna ML,AWS Principal Archite

2、ct,Kubernetes Platforms&AWS AI ChipsAnnapurna ML,AWS 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.Warm up:Pick the odd oneLlamaOrcaGopherFalconBelugaGraniteCoyoteNot(yet)a Language Model!2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.2025,Amazon Web Services

3、,Inc.or its affiliates.All rights reserved.How are you serving LLMs in production?2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.5Performance objectives:Customers need predictable latency,throughput targets,and formal SLOs.High Availability:End-to-end reliability becomes mandator

4、y;multi-AZ,failover,zero-downtime updates Elastic Scaling:Workloads must scale up and down efficiently as demand fluctuates Capacity&Cost:GPU scarcity perf -elasticity-HA-cost/capacityPrototyping to Production Initial PoCs typically use basic GPU setups optimized for speed over scalability 2025,Amaz

5、on Web Services,Inc.or its affiliates.All rights reserved.2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.LLM Inference 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.Model Serving on AWS Traini

6、um 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.ML DevelopersData ScientistsPerformance EngineersNeuron Developer Stack 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.Neuron Developer StackML DevelopersData ScientistsPerformance Engineers 2025,Amazon Web Ser