刘树杰_基于大语言模型的零样本语音合成.pdf

1、Shujie LiuMSRA Hong KongDecoder-Only LLM:GPT(Generative Pre-Training)Large Language Model Large computing:Fast parallel computations on many GPUs(thousands of GPUs)Large data:Training on large amounts of data(terabytes of training data)Large model:Many layers with large parameters in the model(billi

2、ons of parameters)Why Large Language Model Scaling the models,compute,and data leads in increase in performance Shows emergent abilities:can perform new task in few-shot and zero-shot scenarioRadford et al.Improving Language Understanding by Generative Pre-Training.arXiv,2018How to Build a Speech LL

3、M?Speech has a much longer sequence without segment boundaries(only 2-3 words in text,but 16K scalars in speech).Speech contains continuous signals without a fixed dictionary of units for self-supervised learning as in text.Speech processing tasks need more information,such as temporal information,s

4、peaker information,emotion information,background noise.From Continuous Signals to Discrete TokensI am from ChinaASRTTSI am from China9711216124352388846759.Tokenize Speech:CodecSender9711216124352388846759.Codec EncoderReceiverCodec Decoder9711216124352388846759.Speech Codec Encoder Discrete tokens

5、 Codec Decoder SpeechTokenize Speech:SoundStream/EncodecZeghidour et al.,SoundStream:An End-to-End Neural Audio Codec.2021Defossez et al.,High Fidelity Neural Audio Compression.2022-EncoderDecoderVQ 1VQ 8residual 1-VQ 2residual 2.+residual 79.16528471.21386712.43 859+Encodec improves SoundStream wit

6、h some network modification.Tokenize Speech:SoundStream/EncodecZeghidour et al.,SoundStream:An End-to-End Neural Audio Codec.2021Defossez et al.,High Fidelity Neural Audio Compression.2022-EncoderDecoderVQ 1VQ 8residual 1-VQ 2residual 2.+residual 79.16528471.21386712.43 859+DiscriminatorEncodec impr