Kafka Forwarder：简化 OpenAI 的 Kafka 消费.pdf

1、Kafka ForwarderSimplifying Kafka Consumption at OpenAIJigar BhatiMember of Technical StaffAgenda1.Early Streaming Infrastructure at OpenAI2.Decoupling users from Kafka infrastructure3.uForwarder Architecture4.Kafka-Forwarder at OpenAI5.Writing new Kafka Consumers6.Misc Setups and Improvements7.Kafka

2、-Data Lake System8.Future Work2Early Streaming Infrastructure at OpenAI34Streaming infrastructure at OpenAI(circa H1 2024)Lots of user demandLots of user demandMany services using Kafka data warehouse ingestion,async.processing,inter-service communication.More and more critical use cases on the hori

3、zon.but a very immature platform.but a very immature platform.5No standardizationNo standardizationMany Kafka clusters with varying configurations for different use cases.Streaming infrastructure at OpenAI(circa H1 2024)6No resilience to cluster outagesNo resilience to cluster outagesKafka cluster a

4、s a SPOF.Vulnerable to regional failures and network partitions.Streaming infrastructure at OpenAI(circa H1 2024)7Integration challengesIntegration challengesManual per-service setup(secrets,firewall).“Which cluster has this topic?”Streaming infrastructure at OpenAI(circa H1 2024)8Scalability limita

5、tionsScalability limitationsKafka brokers overloaded with too many connections.Streaming infrastructure at OpenAI(circa H1 2024)9Streaming infrastructure at OpenAI(circa H1 2024)Many tactical challenges;one big strategic issue.Many tactical challenges;one big strategic issue.10Streaming infrastructu

6、re at OpenAI(circa H1 2024)Infrastructure inflexibility:Infrastructure inflexibility:tight coupling of clients to individual Kafka clusters made changes and maintenance impossible.Can we decouple producers and consumers from Kafka clusters?1112Why decouple?Achieve high availability by routing around