ViDA：基于差分近似和自适应数据流的视频扩散变换加速.pdf

1、ViDA:Video Diffusion Transformer Acceleration with Differential Approximation and Adaptive DataflowLi Ding1,Jun Liu12,Shan Huang1,Guohao Dai12*1Shanghai Jiao Tong University 2Infinigence AIEqual Contribution*Corresponding to:ASP-DAC 20252025.1.21Li Ding Shanghai Jiao Tong UniversityBackgroundBackgro

2、undPage 22025/3/7 Current video generation models excel due to the diffusionparadigm and the Video Diffusion Transformer backbone.1 https:/www.vidu.studio/2 https:/Sora2:Clifftop Waves and WaterfallsVidu1:Toy boat on silkParadigm:Diffusion ProcessBackbone:Video Diffusion TransformerSpatial Attention

3、Temporal Attention&Li Ding Shanghai Jiao Tong UniversityBackgroundBackgroundPage 32025/3/7 The Paradigm:What is diffusion model?Frames are iteratively denoised,with noise predicted by VDiT.Multi-stepsMulti-stepsInference:reverse diffusionDenoisingDenoisingSpatial attentionTemporal attentionMLPVideoC

4、leaner videoNoise predictor:VDiTAdd nosieAdd nosieTraining:forward diffusionSpatial attentionTemporal attentionMLPNoisier videoVideoNoise predictor:VDiT+Li Ding Shanghai Jiao Tong UniversityBackgroundBackgroundPage 42025/3/7 The Backbone:What is Video Diffusion Transformer(VDiT)?A transformer-based

5、noise predictor with a spatial and temporal patching structure.Spatial attentionTemporal attentionMLPNoise predictor:VDiTPatchifyTokenizingPatchesToken matrices 0-2Frames 0-2 layersActivation 0 Activation 1Activation 2Li Ding Shanghai Jiao Tong UniversityBackgroundBackgroundPage 52025/3/7 The Backbo

6、ne:What is Video Diffusion Transformer(VDiT)?A transformer-based noise predictor with a spatial and temporal patching structure.QKVprojectionQ OutputprojectionSpatial AttentionTemporal attentionMLPAttention blockNoise framesActivation 0 Activation 1Activation 2Li Ding Shanghai Jiao Tong UniversityBa