协调和工作负载算法以释放多态“系统之系统”架构在大规模人工智能领域的潜力.pdf

1、Jascha Achterberg,Universityof Oxford&CallosumAlgorithms to unlock the potential of polymorphic System-of-Systems architecture for large scale AISERVER:AI HW SW CO-DESIGNThe promise of polymorphic System of systemStorageHPCASICsLarge scale AIPhysics simulationsmapping ontoHeterogeneous and composabl

2、e hardware stackAs part of“AI HW SW Co-Design”Workstream we asked:Which algorithmic innovations can help us bridge this gap?What do we need algorithms for in Co-Design?The classic view:Physics simulationsHPCProblem:mapping is not dynamic and responsive with regards toChanging ratios of available har

3、dware resourcesChanging nature of active workloads(complexity,modality)Interdependent relationship of algorithm and hardware to achieve performant deploymentThe Co-Design view:StorageHPCASICsDynamic algorithm architectureDeployment algorithmHardware selection and assignment1st2nd3rd4thVarying updati

4、ng speeds:Interdependence:Our work across the Lifecycle of Co-DesignStorageHPCASICsAlgorithm architectureDeployment algorithmHardware selection and dynamic assignmentMixture of PathwaysPathway FlowAstraSim for Co-DesignNeural network architecture that can adapt its computational complexity in respon

5、se to the current task.Deployment strategy for complexity-adaptive network architectures on heterogeneous hardware.Evolutionary algorithms for optimal resource allocation for heterogeneous compute stacks.Complexity adaptive network architectureThe idea of Mixture-of-Experts(MoE)models:Scan for more

6、detailed project summary:Could MoEsadapt to task complexity?Our heterogeneous Mixture of Pathways architecture:We train these on 82 different temporal prediction tasks with varying task complexityNow accepted at Neurips!Complexity adaptive network architectureScan for more detailed project summary:O