Kafka — when · when not

Powerful, but operationally heavy. Check if you really need it first.

1. Kafka's real wins

Durable event log with replay
Fan-out to multiple consumers
High-throughput writes with per-partition ordering
Producer / consumer decoupling

2. When you don't need Kafka

Single-consumer job queue → BullMQ · Redis Streams · RabbitMQ
Ephemeral pub/sub → Redis Pub/Sub
Async DB writes → outbox pattern + single worker
Traffic < 100 msg/s → Kafka ops eat your win

3. Topics · partitions · offsets

Topic: user-events
├── Partition 0 ── 0, 1, 2, 3 ...
├── Partition 1 ── 0, 1, 2, 3 ...
└── Partition 2 ── 0, 1, 2, 3 ...

Partition count = max parallel consumers.

4. Producer

import { Kafka } from "kafkajs";
const kafka = new Kafka({ clientId: "app", brokers: ["kafka:9092"] });
const producer = kafka.producer();

await producer.connect();
await producer.send({
  topic: "user-events",
  messages: [
    { key: String(userId), value: JSON.stringify({ type: "signup", userId, at: Date.now() }) },
  ],
});

Same key → same partition (ordering).

5. Consumer

const consumer = kafka.consumer({ groupId: "notifier" });
await consumer.connect();
await consumer.subscribe({ topic: "user-events", fromBeginning: false });
await consumer.run({
  eachMessage: async ({ message }) => {
    const payload = JSON.parse(message.value!.toString());
    await handleEvent(payload);
  },
});

6. Topic naming

user.signed-up
order.placed
payment.succeeded

Keep naming consistent across languages and teams.

7. Schemas

JSON — easy, no enforcement
Avro + Schema Registry — versioned
Protobuf — strong types, cross-language

Start JSON, move to Schema Registry as the platform matures.

8. Backpressure

lag = producer offset - consumer offset

Add consumers (up to partition count)
Offload heavy work to side workers
Quarantine bad messages in DLQ

9. Ops cost

Confluent Cloud — $50–200/mo minimum
Self-host — ZooKeeper/KRaft + 3 nodes
AWS MSK — managed, hourly

Small teams should prefer managed.

10. Gotchas

Adding partitions later breaks ordering → overprovision
Sending without key → round-robin, no ordering
Reused groupId → messages split unexpectedly
Commit timing — commit after successful processing

Closing

First messaging? Start with BullMQ or Redis Streams. Reach for Kafka only when you need fan-out and replay for real.

07-pipeline-idempotency