10.XSAI：以CPU的编程范式支持现代LLM核函数 .pdf

1、XSAI():Hardware Support forModern LLM Kernels in a CPU Paradigm北京开源芯片研究院(BOSC)2 2XSAI():Xiangshan-KMH+RhyMAXConsumer SoC(AIPC)Cloud SoC北京开源芯片研究院(BOSC)3 3AgendaMotivation of RISC-V+GEMMWhy GEMMWhy RISC-V CPUCurrent statusHigh-bandwidth L2-cacheAsynchronous GEMMMXFP8北京开源芯片研究院(BOSC)4 4GEMM is important

2、,even for decoding&consumer-sideFAQ:But decoding is GEMV/mem-bound?Takeaway1:Decoding in cloud is not memory-boundTakeaway2:Decoding in consumer-side can be GEMMTakeaway3:Speculative decoding is cheaper than memory channel北京开源芯片研究院(BOSC)5 5Evolution of transformer blockMHA-GQA-MLALessKVCacheoccupati

3、onHighercomputation/memratioDenseFFN-MoEFFNLessCommunicationhttps:/arxiv.org/pdf/1706.03762https:/arxiv.org/abs/2405.04434北京开源芯片研究院(BOSC)6 6Takeaway1:Decoding(MoE)in cloud is not mem-boundWith EP,DSV3 is either compute-bound or network-boundNOT memory boundHard to make profit without EPFlag-ship ope

4、n-weight models are becoming MoEDeepseek V3,Qwen3,LLAMA4https:/ 7Speculative decodingGuess multiple tokens with a draft modelVerify them in parallel(load weights once)Similar with prefill(GEMM)For QWEN3 8B+speculation tree size=64GQA:QO_heads=64x8=512,head_dim=128Up proj:M=64,K=5k,N=50kDown proj:M=6

5、4,K=25k,N=5kMain ops are GEMV GEMM during decodinghttps:/arxiv.org/abs/2211.17192Takeaway2:Decoding in consumer-side can be GEMM8 8北京开源芯片研究院(BOSC)Takeaway3:Spec-decoding is cheaper than memorySpeculative decoding VS.larger mem bandwidthIncreasing memory channels is expensiveDouble memory channelsImp

6、act on total areaDouble particlesIncrease GEMM FLOPS is cheaperDouble height of systolic arrayLimited impact on SRAM/datapathhttps:/ 9Why RISC-V CPU+GEMM?Software eco-systemScalar compilersAuto-vectorization for non-critical operationsOpenMPGeneral computingvariable tree attention mask(SpecInfer and