使用 kro 和 ACK 加快 Kubernetes 中的模型加载速度 [重复].pdf

1、 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.OPN403Kumudhan CherarajanHe/HimDelivery ConsultantAWSRamesh MathikumarHe/HimPrincipal Delivery Con

2、sultantAWSExpedite Model loading times using kro and ACK 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.GoalsProblem Statement:Existing Model Loading Latency on EKSAnalyzing approaches to enhance Model Startup timesEnable flexible,scalable and cost efficient strategiesUsing kro+A

3、CK to orchestrate and manage complete setup 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.Amazon ECRImageApplicationPythonModelContainerFew GBs10-100s of GBs10-100s of GBs=10s-100s of GBLLM Model loading on EKS takes tens of seconds to minutesTimeLarge container images inflate p

4、ull times and slow startupSizeModel artifacts traverse a long data path from Storage to GPUPathGPU resources stay idle during model load,driving up costUsageScaling slow down,impacts performance and agilityEfficiencyProblem Statement 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved

5、.Goal 1:Analyzing approaches to enhance Model Startup timeShrink container image footprint by decoupling model artifactsAccelerate Image pulls to enhance Model Startup time 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.Evaluate S3,EFS,FSx for Lustre as external Storage backendsL

6、everage SOCI to accelerate image pulls via lazy loadingGoal 2:Explore Decoupling&Pull Acceleration Approaches 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.Introduce kro to define blueprints for inference apps&infraIntroduce ACK to provision AWS services(S3,E