高赟-面向流批一体的Flink Runtime新进展.pdf

1、高赟（云骞）面向流批一体的面向流批一体的FlinkFlink RuntimeRuntime新进展新进展The New Features of the Runtime for Unified Batch and Stream Processing流批一体流批一体性能优化性能优化RemoteRemoteShuffleShuffle总结与展望总结与展望#1#2#3#4#5语义完善语义完善与增强与增强#1#1流批一体流批一体什么是流批一体什么是流批一体What is batch/stream unificationWhat is batch/stream unificationUnboundedBou

2、ndedDataStream APITable/SQL APIBoundedUnbounded为无限数据和有限数据的处理提供统一的APIUnified API for processing unbounded and bounded data为什么需要流批一体为什么需要流批一体Why batch/stream unificationWhy batch/stream unification 减少流处理与批处理的学习与维护成本 避免流批融合作业数据的不一致性-Backfill流批一体流批一体APIAPIUnified API for Batch/Stream ProcessingUnified A

3、PI for Batch/Stream ProcessingTable/SQL APIDataStream APIOperatorsEvent-timer/WatermarkProcessing TimerStateStandard SQL/Table如何实现流批一体如何实现流批一体APIAPIHow to implement batch/stream unificationHow to implement batch/stream unificationState流执行模式批执行模式基于State的增量计算逻辑上可以支持任意操作-保留所有历史数据+Retraction一些场景下存在一定性能问

4、题-特定算子缺少有效优化-增量计算导致State随机I/O-定时Checkpoint算子逐级运行依赖于数据有限性算子可以进行特定的优化-数据预先Sort，按Key逐个处理当给定相同的输入数据时，保证两种执行模式的一致性。FlinkFlink流批一体流批一体架构架构Unified Batch/Stream Processing in Unified Batch/Stream Processing in FlinkFlink统一PipelinedRegion调度统一ShuffleService插件接口统一ConnectorStream ModeBatch ModeDataStream APITab

5、le/SQL APIRuntimeExecution ModeAPI#2 2语义增强与完善语义增强与完善部分部分TaskTask结束后结束后CheckpointCheckpointSupportSupport CheckpointsCheckpoints afterafter SomeSome TasksTasks FinishedFinished如果所有Source都是有限的，那么作业最终会结束如果有些Source是无限的，用户可能通过stop-with-savepoint-drain终止作业-stop-with-savepoint 不属于终止1 FLIP-147:Support Check

6、points after Tasks Finished,https:/cwiki.apache.org/confluence/display/FLINK/FLIP-147%3A+Support+Checkpoints+After+Tasks+Finished部分部分TaskTask结束后结束后CheckpointCheckpointSupportSupport CheckpointsCheckpoints afterafter SomeSome TasksTasks FinishedFinished有限数据Source和无限数据Source的混合作业无法支持CheckpointStream模式