Kafka

About Kafka

Apache Kafka is an open-source distributed event streaming platform, originally created at LinkedIn and open-sourced in 2011. Kafka functions as a high-throughput, fault-tolerant, horizontally scalable message bus that decouples data producers from consumers via a publish-subscribe model. Producers write events to named topics; consumers read from those topics at their own pace — topics are retained on disk for configurable periods (hours to weeks), enabling replay and late-joining consumers. Kafka processes trillions of events per day at LinkedIn, Netflix, Uber, Airbnb, and Goldman Sachs. Kafka Connect (source/sink connectors for 200+ systems), Kafka Streams (stream processing library), and ksqlDB (SQL over Kafka streams) extend Kafka into a complete streaming platform. Confluent (founded by Kafka's creators) offers Confluent Cloud — a managed Kafka service — and Confluent Platform with enterprise features. Amazon MSK and Azure Event Hubs provide cloud-managed Kafka-compatible services. KRaft mode (Kafka 3.3+) removes the ZooKeeper dependency, simplifying operations. Kafka's log-based architecture and offset-based consumption make it uniquely suited for event sourcing, CQRS, CDC (Change Data Capture from databases), and real-time analytics pipelines. Alternatives include RabbitMQ (traditional message queue), AWS SQS/SNS (managed cloud messaging), and Redpanda (Kafka-compatible engine written in C++ with 10x lower latency).