Kafka is often introduced as a message broker. That description is useful for five minutes and misleading for the next five years.
The better mental model is a distributed, replicated append-only log. A producer appends records. Kafka keeps them for a configured period or size. Consumers track their own positions and can move those positions backward to replay history.
Once you see the log, most Kafka behavior stops feeling mysterious.
A topic is a collection of ordered partitions
A topic is split into partitions. Each partition is an ordered sequence of
records, and every record receives an offset: 0, 1, 2 and so on.
Kafka guarantees order inside one partition. It does not provide a single total order across all partitions of a topic. If two events must be observed in order, they need a key that routes them to the same partition — an account ID, order ID or device ID, for example.
That key is an architecture decision, not a client detail. It determines the unit of ordering, the distribution of traffic and the maximum consumer parallelism. A celebrity account, global tenant or empty key can turn one partition into a hot spot while the rest of the cluster waits politely.
Choose a key by asking: which events must be processed sequentially, and how evenly will that key distribute real traffic?
The producer finds the leader and sends a batch
Before sending, the producer asks the cluster for metadata: which broker is the leader for each partition? It serializes records, groups them into batches and sends each batch directly to the appropriate leader.
Batching is central to Kafka’s throughput. A larger batch amortizes network and storage overhead, and compression works across multiple records instead of one tiny payload at a time. The trade-off is latency: waiting briefly for a fuller batch may improve throughput while delaying an individual event.
The leader appends the batch to its partition log. Kafka leans heavily on sequential filesystem operations and the operating system page cache rather than building a giant object cache inside the JVM. This is why a durable log can still move data very quickly.
Acknowledgement is a durability contract
Each partition has a leader replica and follower replicas on other brokers. The followers continuously copy the leader’s log. Replicas that are sufficiently caught up form the in-sync replica set, usually called the ISR.
Producer acknowledgement settings decide what “send succeeded” means.
With acks=all, the producer waits for the record to satisfy the partition’s
in-sync replication requirement. Pair that with a sensible replication factor
and min.insync.replicas to refuse writes when the cluster cannot meet the
desired durability level.
For example, replication factor three and minimum ISR two can keep a single broker failure from turning an acknowledged write into a single-copy promise. The cost is intentional: if too few replicas are healthy, the partition rejects writes instead of quietly weakening durability.
Idempotent production protects another boundary. Network timeouts are ambiguous — the write may have committed even though the response never reached the client. Kafka’s idempotent producer uses producer identity and sequence numbers so a retry does not append a duplicate copy. It is enabled by default when the producer configuration does not conflict with its requirements.
KRaft manages metadata, not your event payload
Modern Kafka clusters use KRaft instead of ZooKeeper. A quorum of controllers maintains cluster metadata: brokers, topics, partition assignments, leaders and configuration. One controller is active while the others are ready to take over.
This is the control plane. The broker replicas remain the data plane that stores topic records.
Production clusters normally separate controller and broker roles so metadata consensus is isolated from heavy data traffic. Three controllers tolerate one controller failure; five tolerate two, because a majority must remain available.
When a partition leader fails, the controller chooses an eligible replacement. The important invariant is that an acknowledged, committed record must exist on the new leader. That is why replica health, ISR behavior and leader-election policy matter more than the comforting number of brokers on a dashboard.
Consumers pull and own their offsets
A Kafka consumer pulls batches from partition leaders. Its position is simply the offset of the next record it intends to read. That small piece of state is what makes replay cheap: move the offset backward and the same consumer logic can process history again.
Consumers usually join a group. Within a conventional consumer group, one partition is assigned to at most one consumer at a time. Ten partitions allow up to ten active consumers for that topic in the group; adding an eleventh does not create more partition-level parallelism.
When membership changes, partitions are reassigned. A cooperative rebalance can reduce disruption, but application correctness still depends on what happens between processing a record and committing its offset.
Commit first and crash before processing: the event may be skipped. Process first and crash before committing: the event will be delivered again. Most applications choose the second failure mode and make their side effects idempotent.
Exactly once has a boundary
Kafka transactions can atomically write to multiple Kafka partitions and commit consumed offsets with the produced results. Kafka Streams can use this to offer exactly-once processing for read-process-write flows that remain inside Kafka. Consumers must use the appropriate isolation level to hide aborted transaction records.
The promise does not automatically extend to an email provider, payment API or ordinary database. Kafka cannot atomically commit its offset and a third-party side effect unless that system participates in the protocol.
At those boundaries, use patterns such as idempotency keys, an inbox table, transactional outbox or a connector that stores output and source position together. “Exactly once” is not a property you toggle globally. It is a claim about a defined path and failure model.
Retention is not acknowledgement
Reading a record does not delete it. Topic retention is independent of consumer progress, which allows several consumer groups to read the same events at their own pace.
Time- or size-based retention removes old log segments. Log compaction serves a different purpose: for a keyed topic, it preserves at least the latest known value for each key while older superseded records become eligible for cleanup. Compaction is useful for rebuilding state, but it is not immediate and it is not a substitute for understanding tombstones and retention settings.
Operate the invariants
Useful Kafka dashboards follow the end-to-end model:
- producer error rate, retry rate and request latency;
- under-replicated partitions and shrinking ISR sets;
- unavailable partitions and leader-election activity;
- consumer lag per partition, not only a cluster-wide average;
- rebalance frequency and processing time relative to poll settings;
- disk use, retention pressure and skew between brokers;
- dead-letter volume and the age of the oldest blocked event.
Consumer lag is a symptom, not a diagnosis. It may mean slow processing, a hot partition, a downstream outage, repeated poison records or a consumer that is alive but no longer polling correctly.
Kafka becomes easier when every operational question can be mapped to a record, a partition, a replica and an offset. It is still a distributed system. It just is not magic.
For deeper reference, read the current Apache Kafka documentation on core concepts, design and KRaft operations.