Skip to main content

Kafka Didn't Fail. Your Configuration Did.

Updated 5 min readMarkdown
Part of seriesDeep Stack
Kafka Didn't Fail. Your Configuration Did.
๐Ÿ’ก
  • Consumer lag + rebalance storms kill pipelines during traffic spikes

  • Disk fills silently until a broker dies and partitions go offline

  • One schema change without a default field breaks every consumer overnight

  • Kafka didn't fail in any of these. The configuration did.

Kafka runs LinkedIn's activity pipeline, Uber's trip events, and Robinhood's order stream. It processes trillions of messages a day without blinking. And yet, it brings down production systems regularly โ€” not because Kafka is unreliable, but because engineers ship it like a managed service when it is closer to a loaded weapon. Here is what actually goes wrong.


Where Does Kafka Actually Run in Production?

Kafka sits in the middle of systems that cannot afford gaps: real-time fraud detection at payments companies, event sourcing for microservices, CDC pipelines feeding data warehouses, activity feeds at social platforms, and log aggregation at scale. The common thread is that every downstream system trusts Kafka as the source of truth. When it misbehaves, everything downstream breaks simultaneously.

Why Do Kafka Pipelines Fail in Production?

Three failure patterns appear repeatedly across teams. None of them are Kafka bugs. All of them are configuration and operational gaps that looked fine until they weren't.

Real Failure #1: Consumer Lag That Never Recovers

A fintech company processing card transactions had a Kafka consumer group fall behind during a Black Friday traffic spike. Lag hit 4 million messages. The team scaled up consumers to catch up. The result: a rebalance storm. Every time a new consumer joined, Kafka triggered a full group rebalance, pausing all consumption. More consumers made it worse. Lag kept climbing.

What went wrong: max.poll.interval.ms was set too low. Consumers were being marked dead before finishing processing, triggering constant rebalances.

Fix: Tune max.poll.interval.ms to match your actual processing time. Separate slow processing into async workers. Add consumer lag monitoring before you need it, not after.

๐Ÿ’ก

Never scale Kafka consumers during a rebalance storm. Pause, let the group stabilize, then scale.

Real Failure #2: Disk Fills Up, Broker Dies

A logistics company ran Kafka on three brokers with default 7-day retention. A downstream consumer went down for 10 days during a migration. Messages accumulated. One broker's disk hit 95%, writes slowed, then the broker went offline. With replication factor 2 and one broker down, one partition lost its leader. Producers started throwing NotLeaderForPartitionException. The pipeline stopped.

What went wrong: No disk alerting. Replication factor too low. Retention policy had no relationship to actual consumer SLA.

Fix: Set retention as min(disk capacity รท daily throughput, consumer recovery SLA). Use replication factor 3 for anything production. Alert at 70% disk, not 95%.

๐Ÿ’ก

Replication factor 2 is not redundancy. It is one broker failure away from data unavailability.

Real Failure #3: Schema Change Breaks Every Consumer

A data platform team added a required Avro field without a default value. Deployed the producer. Every consumer running the old schema started throwing deserialization exceptions. The pipeline went silent. No messages were lost โ€” nothing was being processed. The error was not obvious because consumers were technically running.

What went wrong: No schema registry enforcement. No backward compatibility check before deploy. A required field without a default is a breaking change.

Fix: Use Confluent Schema Registry with BACKWARD compatibility mode enforced. Every new field ships with a default. Run a schema diff in CI before any producer deploy.

๐Ÿ’ก

Treat schema changes like database migrations. Reviewed, versioned, and rollback-tested before they touch production.

What Actually Prevents Kafka Failures?

The engineers who avoid these failures are not smarter. They have runbooks, they set alerts before incidents, and they treat Kafka as infrastructure that needs as much care as a database. Three non-negotiables:

  • Lag alerting at the consumer group level, not the broker level
  • Replication factor 3 for every production topic
  • Schema registry with compatibility enforcement on every deploy

Kafka itself held in all three cases above. The data was intact. What failed was observability, configuration discipline, and schema governance.

Frequently Asked Questions

What causes Kafka consumer lag to spike suddenly?

Consumer lag spikes when processing time per message exceeds max.poll.interval.ms, causing Kafka to mark consumers as dead and trigger group rebalances. Scaling consumers during a rebalance worsens it.

How do you prevent Kafka broker disk failures in production?

Set log retention based on your consumer recovery SLA, not a fixed default. Alert at 70% disk usage. Use replication factor 3 so a single broker failure does not take partitions offline.

What is a Kafka rebalance storm?

A rebalance storm happens when consumers continuously join and leave a group faster than Kafka can stabilize assignments. Each rebalance pauses all consumption, creating a feedback loop where lag worsens as you scale.

How do you safely evolve Kafka schemas without breaking consumer?

Use a schema registry with BACKWARD compatibility enforced. Every new field must have a default value. Run schema compatibility checks in CI before deploying any producer change.

What is the right replication factor for Kafka in production?

Minimum replication factor 3 for production topics. Replication factor 2 means one broker failure can make a partition unavailable if the remaining replica is not caught up.

Practice These Concepts

Kafka failure modes are a standard question in data engineering and ML infrastructure system design interviews โ€” consumer lag, partition leadership, schema evolution, exactly-once semantics.

If you are preparing for roles at product companies, KnowEaseAI covers Kafka and distributed systems in the Data Engineering track (https://knoweaseai.com) โ€” real interview questions, structured answers, and graded practice so you know where you actually stand.

No fluff. No FAANG templates. Just what interviewers actually ask.

Start practicing free โ†’ (https://knoweaseai.com/practice)

About the author

  • Building agent first ecosystem for learners KnowEaseAI. Free audit and personalized guide here knoweaseai.com/audit

Deep Stack

Part 1 of 1

Real failures. Real systems. Real fixes. A technical series for software engineers, data engineers, and ML practitioners who want depth over tutorials. Each post covers a production failure, architectural decision, or engineering concept that actually matters โ€” with the specifics most articles skip.