8-1 Apache Flink 流批一体的规划和在快手的进展.pdf

1、FLINK 流批一体的规划和在快手的进展张静快手基础技术部Apache Flink Commiter|01社区在流批一体社区在流批一体方向的演进方向的演进02快手在流批一体快手在流批一体方向的进展方向的进展03社区在流批一体社区在流批一体方向的规划方向的规划04快手在流批一体快手在流批一体方向的未来工作方向的未来工作目录目录CONTENT|社区在流批一体社区在流批一体方向的演进方向的演进01|为什么要做流批一体计算引擎|业务侧引擎侧p 降低开发成本一套业务代码p 提高数据质量口径一致p 降低资源开销避免重复计算p 降低运维成本维护一套引擎p 降低学习成本技术聚焦Apache Flink 流批一

2、体架构|SQL&Table APIRelationalDataStreamPhysicalDAG SchedulerDAG APIUnified Source API&Sink API&Operator APIPluggable Shuffle APIUnified API(Done in 1.101.14 version)|SQL&TableRelationalDataSetbatch DataStreamstreaming Batch Plan Stream Transformation Batch Runtime Streaming RuntimeSQL&TableRelationalS

3、tream Graph RuntimeDataStream PhysicalQuery ProcessorPipeline Region Scheduling(Done in 1.12 version)|Pipelined Regionp 以 Pipelined 边连接的 Task 集合p 基本的调度单元Pipeline Region Schedulerp 流作业GlobalAggregateRegion1Region2Source -Filter-LocalAggregateGlobalAggregatep 批作业Source -Filter-LocalAggregateAdaptive B

4、atch Scheduling(Introduced in 1.15 version)|根据已经结束的 producer 的数据量动态确定下游并发度GlobalAggregateRegion1Region2Source -Filter-LocalAggregate优点p 不需要用户手动设置批作业的并发度p 自动适应每天变化的数据量Adaptive Batch Scheduling(Future plan)|N:number of consumer execution verticesP:number of sub partitions.For the kth consumer executio

5、n vertex,the consumed sub partition range should be:range=floor(P*(k-1)/N),floor(P*k/N)-1A0RS0RS1A1RS0RS1RS2RS2Range0,0Range1,2B0B1TODOp 下游并发之间的负载均衡p 动态调整下游算子Pluggable Shuffle API|p Blocking ShuffleShuffle 数据落盘TaskManager Shufflep write to local diskp read from upstream TaskManagerExternal Shuffle S

6、ervicep write to local diskp read from upstream shuffle serviceFLINK-13247Remote Shuffle Servicep write to remote ShuffleWorkerp read from remote shuffleWorkergithubPluggable Shuffle API|p Pipeline ShuffleShuffle 数据不落盘p Hybrid Shuffle(规划中)根据资源情况，动态决定落盘还是不落盘快手在流批一体快手在流批一体方向的进展方向的进展02|还有哪些痛点|痛点 资源浪费Ta