何颖鹏-唯品会如何基于Volcano与AI训练场景提高集群利用率_0708（GOTC上海会场）（15页）.pdf

1、CNCF分论坛专场何颖鹏 2021年07月10日唯品会如何基于Volcano与AI训练场景提高集群利用率概况1.使用helm发布tfjobs，由tf-operator管理训练任务流程2.使用volcano对tfjobs的任务容器进行批量调度3.对tf-operator进行改造，按业务划分volcano队列，并支持任务优先级4.改造kubelet实现资源超卖，提高容器部署密度5.通过自研的vpa，实现在容器运行时动态调整资源HelmVolcanoTf-operatorKube-apiservertfjobpodGroupqueuekubeletkubeletps0workerps1workerw

2、orkerworkerworkerworkervpavpaTf-operator部署与使用apiVersion:kubeflow.org/v1kind:TFJobmetadata:namespace:_NAMESPACE_ name:_NAME_“spec:runPolicy:schedulingPolicy:queue:priorityClass:tfReplicaSpecs:Chief:replicas:1 template:Evaluator:replicas:1 template:PS:replicas:1 template:Worker:replicas:1 template:Hel

3、mTf-operatorKube-apiservertfjob3tfjob2tfjob1informercontrollerpsevaluatorchiefworkerspsevaluatorchiefworkerspsevaluatorchiefworkerspodgroup3podgroup2podgroup11.tf-operator管理训练任务流程，同步tfjob状态到我们的valyria管理平台2.参考社区实现，修改tf-operator，支持创建podgroup时指定queue和priorityClassValyriaVolcano原理与部署Volcano架构https:/volc

4、ano.sh/en/docs/architecture/使用Volcano实现对tfjobs容器的批调度https:/volcano.sh/en/docs/schduler_introduction/Allocate具体流程初始化namespaceQueue和JobQueueMapnamespacesOrderFnNamespaces.empty()从当前namespace中挑选queue!queueOverUsed,queueOrderFnPop namespaceNo queue 从当前queue中挑选job jobOrderFnget queuePush namespaceget job

5、 构建待调度的tasks队列 taskOrderFnNo jobtasks.empty()为task选取节点 predicatesFn,nodeOrderFnPop taskjob.ready()emptyemptycommitdiscardYesNoend任务队列apiVersion:kubeflow.org/v1kind:TFJobmetadata:spec:runPolicy:schedulingPolicy:queue:priorityClass:tfReplicaSpecs:apiVersion:scheduling.volcano.sh/v1beta1kind:Queuemetad

6、ata:name:rec-queuespec:weight:400apiVersion:scheduling.volcano.sh/v1beta1kind:Queuemetadata:name:search-queuespec:weight:200 actions:enqueue,allocate,backfill tiers:-plugins:-name:drf -name:priority -name:gang -name:conformance -plugins:-name:proportion -plugins:-name:nodeorder -name:predicatestfjob