用于LLM推理加速器的压缩使能MRAM存储器芯片子系统.pdf

1、Compression RatioBlock sizeZeroPoint1.5X+64 byteOffline compressInline DecompressProblem#1:LLMs memory bound.HBM-based GPUs 60%utilization.Problem#2:Existing lossy compression methods degrade accuracyOpportunity:LLM inference is memory-bound 6:1 read-to-write ratioSolution:Lossless compression-enabl

2、ed MRAM memory chiplet subsystem on UCIe interface.Impact:MRAM delivers HBM-like bandwidth at 30-50%lower power,AI-specific chiplet augmentation to GPU-based inferenceImpact:Lossless compression squeezes models by 1.5X,leading to bandwidth,power and Tokens/s gainNuRAM Managed MRAM SmartMem:3X HBM BW

3、 and DRAM like GBs capacity via stacking,1000X lower standby and 2.5X denser than SRAM Decompressor IPSPECCompression granularityOnly 32B or 64BClock frequencyUp to 1.75 GHz(Samsung 4nm/TSMC N5 technology)BandwidthMatches the B/w of 1 HBM3e channel(1.2GHz)Decompression latencyPipelined decompressor;

4、Deterministic latency;Default:21/cycles+1/foreach next decompressed 64B block of the compressed package(or stream)Area and peak power*/IP instanceArea=0.04mm2 Peak Power=0.057W(Samsung 4nm,Low Power Plus(LPP)1.2GHzCompression ratio(geomean)100%lossless1.5x for Llama model data at bf161.25x-1.43x for

5、 Llama model data at OFP8(e4m3/e5m2)Compression enabled Managed MRAM technology offers path to differentiate,compete with higher Tokens-per second-per wattCompression enabled MRAM memory chiplet subsystems for LLM Inference AcceleratorsAngelos Arelakis,Nilesh Shah,Evangelos VasilakisZeroPoint Technologies Gothenberg,SwedenMax Simmons,B Nataraj,Rama GortiNumem,Sunnyvale,CA