夏畅-Apache Flink在斗鱼的应用与实践.pdf

1、夏畅斗鱼实时计算负责人Apache Flink在斗鱼的应用与实践背景介绍实时平台建设实时数仓探索未来发展与展望#1#2#3#4#1背景介绍实时计算发展之路2018201820192020spark streaming jarSparkstrom taskStormflink1.9、flink1.10、flink1.11、flink1.12玄武计算flink jarFlink业务场景广告大屏推荐风控监控数据分析实时标签应用规模100Billions500+Jobs2000+Vcores100+Users#2实时平台建设玄武计算平台Flink JARFlink JAR玄武计算平台玄武计算平台开发方

2、式代码编程SQL/代码编程开发门槛高低部署成本高低监控告警无有版本管理无有玄武平台架构Xuanwu Real-time PlatformXuanwu Cluster ScheduleJMTMFlink 1.11JMTMFlink 1.12YZ K8sCP K8sK8s.Flink Job ServerFlink Gateway ServerK8s Cluster作业开发Flink on K8szookeeperJob ManagerPod(Master)Task ManagerPodTask ManagerPodTask ManagerPod.Job ManagerPod(Standby)Fl

3、ink on K8sPod Resource&Flink Resource SyncEntrypoint.shFetch Job SettingFlink Yaml Replace状态跟踪K8s MasterNginx IngressPod（JM）Pod（JM）NodePod（TM）Pod（TM）NodeACCEPTRUNNINGFAILED/CANCELFINISHJob SubmitMessage QueueCatalog注入Flink Job RuntimeJob AssemblySDK GetFile GetContext InitConf SetCatalogSql ExecuteS

4、ql SplitSql RunCatalog注入CREATE CATALOG hive WITH(type=hive,hive-conf-dir=/opt/hive-conf/,default-database=default);Catalog Sql ParseCatalog Factory LoaderCatalog Create Register实时调试Web debug Flink Cluster日志采集结果采集语法校验Sql Mock状态跟踪数据源FlinkGatewayServer实时调试ParserValidateOptimizeExecuteRelationRewrite实时调

5、试CREATE TABLE kafka_source_test(.)WITH(connector.type=kafka,connector.topic=ocean_log_acclog_json,.);CREATE TABLE console(.)WITH(connector.type=jdbc,connector.table=ads_dy_monitor_xxx_1d,.);INSERT INTO console.;CREATE TABLE kafka_source_test(.)WITH(connector.type=datagen,.);CREATE TABLE console(.)WI

6、TH(connector.type=console,update-mode=retract,.);INSERT INTO console.;监控报警Flink JobMetrics ReporterKafkaFlink Metrics ETLPushGatewayGrafanaAlarm监控报警#3 3实时数仓探索实时数仓探索实时数仓Kafka-ODSDIMDWDDWSADSDBLOGCDCTRACKData Services实时数仓odsdwddwsads挑战kafka数据