Explore the intricacies of exactly-once semantics (EoS) in Apache Kafka in this comprehensive conference talk from Strange Loop. Dive deep into the recent additions to Kafka that enable EoS, including idempotence and transactions in Kafka clients. Gain insights into the specific semantics of Kafka distributed transactions and the underlying mechanics that allow them to scale efficiently. Learn about Kafka's unique approach to standard two-phase commit protocols, the maintenance and replication of transaction state, and how various failure scenarios are handled. Discover the future improvements planned to enhance exactly-once stream processing with Kafka. Led by Jason Gustafson, a top contributor to Kafka and part of the team that designed and implemented EoS support, this talk covers topics such as Kafka producers and consumers, processing models and semantics, idempotent producers, Chandy-Lamport snapshots, transaction log availability, zombie fencing, consumer read isolation, broker and consumer fetch handling, concurrent transactions, performance observations, and end-to-end EoS. Read more