以太网交换平台和架构的演进以满足 Meta 的 AI 训练集群横向扩展的 AI 架构需求.pdf

1、Lingjun Wu,Meta HW EngineerParvez Shaikh,Meta SW EngineerMetas 51T NSF Platform HW and SWMetas 51T NSF Platform HW and SWLingjun Wu,Meta HW EngineerParvez Shaikh,Meta SW EngineerNETWORKINGNetwork traffic patternLong running AI jobs exchanging large data Low entropy trafficHash collisions lead to poo

2、r link utilizationLink failures can be expensive and disruptive Metas AI NetworkMaximize utilization of network bandwidthMinimize link failure impactEvolution of AI networks at MetaDeterministic Path Routing(static routing)+EECMPFirst Hop Traffic Engineering(controller based routing)DSFDisaggregated

3、 Scheduled FabricNSFShallow Buffer Ethernet Switch Fabric with Adaptive RoutingMetas AI NetworksFacebook Open Switch SystemMetas own Network Operating system that supports both non-AI as well as AI networkFBOSS is open source and uses OCP SAI or Switch Abstraction InterfaceSAI enables FBOSS to onboa

4、rd new SDK vendors as long as SAI API used by FBOSS is implementedFBOSSNon-Scheduled Fabric for Metas AI Network216 GPUs/L120736 GPUs/L2Non-Scheduled Fabric for Metas AI NetworkSTSW(96x400G downlink)8 x spine plane,54 STSW/plane A spine plane connects 4 data halls(or 96 pods)(or upto 20736 GPUs)FTSW

5、(9 x 6 x 400G downlink and 54 x 1 x 400G uplink)4 x 24 x fabric pods,8 FTSW/podPod connects upto 9 RTSW (or upto 216 GPUs)RTSW(24 x 2 x 400G downlink and 8 x 6 x 400G uplink)72 GPU/AI Rack,interconnected by 3 RTSWEach RTSW connects upto 24 GPUs RTSW/FTSW/STSW Same Merchant silicon,51.2T Ethernet swi

6、tches across all rolesSimpler deployment and configurationUses Shallow Buffer Switches=160MB bufferStandard congestion control mechanism to support lossless trafficECN+PFCAdaptive Routing Scheme for optimal link utilizationSAI_ARS_MODE_FLOWLET_QUALITY or SAI_ARS_MODE_PER_PACKET_QUALITYAR-Capable NIC