赖斯遥-Lustre文件系统技术路线图_Lustre_2.16_and_beyond.pdf

1、Andreas DilgerLai SiyaoLustre 2.16 and B2Trends in High Performance Storage Ever increasing demand for high-performance storage to feed data pipelines AI/ML/ChatGPT/LLM driving surge in new users of parallel(flash)storage Existing computation models(weather,finance,)increasing resolution,data source

2、s,historical data Initial demands can be met by all-flash storage,but not everyone has the budget to scale flash Need to increase capacity,reduce costs,transparently access multiple storage types/tiers/hybrid Lustre already allows transparent data migration between tiers,hybrid storage within files

3、Disk and QLC,combined with compression,heading to lowest$/TB and more accessible than tape Meta/data redundancy improves availability above hardware,simplifies hardware requirements Security,multi-tenancy,data isolation demands always increasing(medical,privacy,IP,legislative,)3Planned Feature Relea

4、se Highlights2.16 approaching feature completion LNet IPv6 addressing must-have functionality for future deployments(SuSE,ORNL)Optimized Directory Traversal(WBC1)improve efficiency for accessing many files(WC)2.17 has major features already well underway Client-side data compression reduce network a

5、nd storage usage,costs(WC,UHamburg)Metadata Writeback Cache(WBC2)order of magnitude better metadata speed(WC)2.18 feature proposals in early stages File Level Redundancy-Erasure Coding(FLR-EC)reduce cost,improve availability(ORNL)Lustre Metadata Redundancy(LMR1)improve availability for large DNE sys

6、tems Client Container Image(CCI)improved handling of aggregations of many small 4LNet ImprovementsDemand for IPv6 in new deployments as IPv4 is exhausted Relatively few external-facing Lustre systems means 10.x.y.z is still viable for nowIPv6 large NID support(LU-10391 SuSE,ORNL)Variable-sized NIDs(