Carry an HTTP request ID through a Kafka consumer
An HTTP request ends when it returns a response, while a Kafka job may finish later. Pass a separate correlation header instead of changing the business JSON, so the flow remains traceable without breaking the payload c…
Table of contents
An HTTP request ends when it returns a response, while a Kafka job may finish later. Pass a separate correlation header instead of changing the business JSON, so the flow remains traceable without breaking the payload contract.
Keep the two IDs separate
Idempotency-Keyconverges retried business commands.X-Request-Idconnects this transport attempt to response, logs, and messages.
The producer adds a valid request ID to a Kafka record header and keeps the old message-only path when there is no ID. The consumer prefers the valid header and retains a messageId fallback for older messages. Never copy raw payloads or user data into logs.
Observe asynchronous failure
Kafka send() failures are asynchronous, so try/catch alone is insufficient. Observe null futures, callback failures, and DLQ transitions with bounded error details. A success log and a failure state must not describe the same result.
ID lifetime comparison
| Value | Created at | Lifetime | Prevents duplicates | Logging rule |
|---|---|---|---|---|
| Request ID | HTTP attempt start | Response, producer, consumer | No | Validate format and length |
| Idempotency key | Business command creation | All retries | Yes | Prefer a hash |
| Message ID | Event creation | Topic, DLQ, replay | Event duplicate check | Allowed |
| User ID | Authentication | User lifetime | No | Never a metric label |
HTTP ──X-Request-Id──▶ producer ──record header──▶ consumer MDC
└── Idempotency-Key ──▶ command store (different lifetime) ─┘
The consumer clears MDC in finally so one message ID cannot leak into the next. When a header is absent or too long, use a bounded ID or the message ID; never promote arbitrary user text into log context.
Check before shipping
Ask whether each ID prevents duplicate business effects or merely helps investigation, whether it survives consumer restart, and whether the failure can be investigated without logging the payload.
Related course: HTTP retries and request correlation