Carry an HTTP request ID through a Kafka consumer

An HTTP request ends when it returns a response, while a Kafka job may finish later. Pass a separate correlation header instead of changing the business JSON, so the flow remains traceable without breaking the payload contract.

• Idempotency-Key converges retried business commands.
• X-Request-Id connects this transport attempt to response, logs, and messages.

The producer adds a valid request ID to a Kafka record header and keeps the old message-only path when there is no ID. The consumer prefers the valid header and retains a messageId fallback for older messages. Never copy raw payloads or user data into logs.

Kafka send() failures are asynchronous, so try/catch alone is insufficient. Observe null futures, callback failures, and DLQ transitions with bounded error details. A success log and a failure state must not describe the same result.

Ask whether each ID prevents duplicate business effects or merely helps investigation, whether it survives consumer restart, and whether the failure can be investigated without logging the payload.

Related course: HTTP retries and request correlation