AWS 上的 vLLM：从测试到生产以及两者之间的所有环节.pdf

1、 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.O P N 4 1 4vLLM on AWS:Testing to Production and Everything in BetweenPhi Nguyenhe/himSenior GenAI SpecialistOmri Shivhe/himSenior Open Source ML Engineer 2025,Amazo

2、n Web Services,Inc.or its affiliates.All rights reserved.AgendaLLM JourneyAI on EKSLLM Deployment PatternsConclusion 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.LLM Journey 2025,Amazon Web Services,Inc.or its a

3、ffiliates.All rights reserved.Everyone heard the storyI was talking to a Fortune 500 company last month.They built an internal Ask HR chatbot-simple Q&A about benefits,policies,that sort of thing.-5,000 employees,light usage.Their first full month?$180,000 in OpenAI API costs.The CFO called an emerg

4、ency meeting.2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.LLM JourneyFoundationDeployOptimizeAccelerateScale 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.LLM JourneyFOUNDATION 2025,Amazon W

5、eb Services,Inc.or its affiliates.All rights reserved.LLM JourneyEndpoint SprawlsTeam ATeam BTeam COpenAIClaudeBedrock 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.LLM JourneyEndpoint SprawlsNo Cost VisibilityNo Policy EnforcementNo ComplianceTeam ATeam BTeam COpenAIClaudeBedro

6、ck 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.LLM JourneyUnified GatewayEnvoy AI GatewayTeam ATeam BTeam CSelf-ManagedClaudeBedrockOpenAI 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.LLM