鲁蔚征-面向AI应用的文件系统与评测基准.pdf

1、鲁蔚征 中国人民大学Weizheng LuRenmin University of ChinaFile Systems and Benchmark Tools for AI StorageOutlinesML/AL WorkloadsDistributed File Systems for AIBenchmark Tools&ResultsML/AI WorkflowsTrainingPreprocessingData Loadersmall or big filesTFRecord or raw fileCheckpointInferenceTypical AI DatasetsImages

2、 and VideosImageNet:14M small filesyoutube-8M:1.53TBTextC4The PileFalcon-RefinedWebRecommendation SystemsML/AI Training CharacterizationsBig DataSame Data Multiple Training Jobshyper-parameter tuningdifferent model archietcure,different parameters(learning rate,loss function)Fluid,Microsoft QuiverCo

3、mpute Nodes SSD or RAMglobal storage pute nodes local storageLustre PCC,Alluxio,JuiceFS,Preprocessing:Falcon-RefinedWeb2.8TB extractedFalcon 180B LLMPipelinesPreparationFilteringDeduplicationhttps:/arxiv.org/abs/2306.01116Timeline of Distributed File Systemsl20032003Lustre Initial Releasel20062006Ha

4、doopHDFSl20062006AmazonIntroduced S3ll20032003-20062006CephShown on OSDI06 and SC06l20072007BeeGFSbeta on ISC07Timeline of Distributed File Systems(cont.)l20122012AlexNetImageNetl20142014Alluxiol20162016YRCloudFilel20172017JuiceFSl20202020GPT3l20222022ChatGPTl20142014Wekal20152015Kubernetes1.0 Relea

5、sedPOSIX or notProsConsFile SystemsPOSIXdevs,ops,and software rely on POSIXPortableoverheadLustre,JuiceFSnon-POSIXlow costlimited abilitiesadditional codeHDFS,S3Case Study:AlluxioGoal:Data OrchestrationUnder StoreS3,HDFS,POSIX FSWorkersCache on RAM or SSDClientfuseCase Study:JuiceFSGoal:high-perform

6、ance,cloud nativedata is chunked on S3,HDFSmetadata is in redis,MySQL,PostgreSQLclient mount fuseCommon Benchmark Toolstraditional tools IOPS&BandWidth(BW)fiomdtestiozonereal-world workloadsML benchmarkMLPerfMLPerfa suite contains mainstrea