SemiAnalysis InferenceMAX：人工智能前沿基准测试.pdf

1、Dylan PatelSemiAnalysis InferenceMAX:Benchmarking the AI FrontierOCP SPECIAL FOCUS:MARKET IMPACT&OCP ADOPTION STORIESSemiAnalysis:What We DoKey Trends in AI InferenceIntroduction to InferenceMAXInferenceMAX v1 ResultsFuture PlansContents2SemiAnalysis:What We DoSector Coverage Tokenomics,Applications

2、,IntelligenceAI Companies,AI Models,Cloud ServicesAccelerators,Networking,Optics,DatacentersODM and OEMsIC DesignFoundry and IDMsTesting and PackagingSemicap EquipmentSubassemblies,Materials,ConsumablesTeam MembersSemiAnalysis:What We Do42 team members globally(plus interns!)26 in North America(offi

3、ces in SF,NYC)5 in Europe11 in AsiaSemiAnalysis:What We D(186,000 subscribers on free tier)SemiAnalysis:What We D(institutional business)continued on next pageSemiAnalysis:What We D(institutional business)continued on next pageSemiAnalysis:What We D(institutional business)ToolsDataLLM Inference syst

4、ems are not a monolith:dedicated compute and system-level optimizations are the standardPrefill vs.Decode specialization:Treat prefill and decode as different workloads(compute vs memory/latency bound)to cut TTFT and ITL.Separate GPU pools for prefill and decode;transfer KV cache between pools to ra

5、ise overall utilization.KV cache as a first-class resource:Cache-aware routing(pick the worker with best cache hit&lowest load)and fast KV transfer are becoming core platform features.MoE+Expert Parallelism(DeepEP,etc.):Route tokens to a few experts across GPUs;use throughput-optimized dispatch for

6、prefill and low-latency dispatch for decodeDynamic rate matching:Continuously tune the Ctx:Gen GPU ratio per model/traffic to balance TTFT,ITL,and tokens per second per GPUAI Inference Is Becoming Increasingly SpecializedKey Trends in AI InferenceExisting Benchmarks:Lack transparency,with opaque dec