从系统层面提升 GPU 利用率.pdf

1、NVIDIAIMPROVEGPU UTILIZATION EROMSYSTEM LEVELClick Cheng， NVIDIA Solution ArchitectGTC China 2020#page#WHATS ABOUT THE TALKWelcomeItsFrom system level of NVIDIA perspective， proposed several ways to improve GPU utilizationDiscuss several GPU monitoring metrics which reflect real GPU utilizationiIntr

2、o each solution mechanism， usage， discuss the benefit in some test cases；Summary different solution positioning， comparison， etc；ItsNotImprove GPU utilization from scheduler levelOptiimize GPU utilization from coding level#page#OUTLINEOverviewWhats About The TalkGPU Utilization DiscussionMulti-Proce

3、ss ServiceMPS Intro，Usage，TestCasesMulti-Instance GPUMIGIntro，Usage，Test CasesTriton and VGPU BriefIntro，Test CasesQuick Summarry#page#OVERVIEW#page#BACKGROUNDWhy ls This ImportantGPU is more and more powerful， and more precious.Many applications are benefiting more from more powerful GPUWhile for s

4、ome lower-utilized application， still cant fully utilize GPUpowerful computing capability.Example， some developing scenario， inference scenario-Especially for some inference cases with critical latency limitation，which not allowed batching for inference，How to share and isolate among processes or us

5、ers on one GPU#page#GPU UTILIZATIONMetrics and ToolsGPU utilization: reflect how busy different resources on GPU are， metrics including GPUcore(CUDA core，integer， FP32， Tensor Core）， frame buffer(capacity， bandwidth）， PCle RXand TX， NVLink RX and TX，encoder and decoder， etc.Generally，when we talk ab

6、out GPU utilization， we are mostly talking about GPU utilizationof CUDA core.GPU utilization reflects an impact on delivered application performance somehow， but notnecessarily.Monitor toolsnvidia-smi or NVML，installed with GPU driverDCGM: Data Center GPU Manager， standalone package， using NVML and