Outbox Pattern Architecture (Copy)

Architecture

Adding Idempotency

rest-service writes an order and relays it to email-service without a direct call and without the producer ever writing to a broker. This page traces that path end-to-end — the request, the transactional write, the CDC relay, the idempotency check, and the trace that survives all of it — and then shows how it's actually scheduled across a Kind cluster.

Request & event flow

One HTTP request produces one transaction. Everything after that — the Kafka message, the email, the trace — is a side effect CDC discovers later, not a call rest-service makes directly.

POST /orders Idempotency-Key: <k> INSERT Order + Outbox (one transaction) logical WAL (Debezium CDC) outbox.event.order consume (rate-limited) traceparent header (W3C) — same trace continues across the CDC hop Client curl / k6 load test rest-service 2 pods · :8081 Postgres SAME TRANSACTION Order Outbox + trace_context (W3C traceparent) Kafka Connect Debezium CDC + Outbox Event Router SMT Kafka 3 brokers (Strimzi) topic: outbox.event.order email-service 2 pods · rate-limited GET / SET Idempotency-Key → orderId Redis no persistence POSSIBLE OUTCOMES 201 — genuine create 200/201 — replay (same key+body) 409 — same key, different body 425 — concurrent race on same key Tempo OTLP spans · service graph
primary request path idempotency check (Redis) async CDC relay (Kafka) trace propagation carried value, not a call
Figure 1. rest-service never calls Kafka, and email-service never calls rest-service — Debezium discovers the Outbox row by tailing Postgres's write-ahead log and republishes it as a plain Kafka message (ADR 0001). The idempotency check (ADR 0006) and the trace handoff (ADR 0005) are the two concerns riding alongside that one transaction.

Components

What each piece in the diagram above actually is.

rest-service

Spring Boot · 2 replicas · NodePort 30081

Writes the Order and Outbox row in one transaction; owns the Idempotency-Key check against Redis.

Postgres

StatefulSet · single instance

wal_level=logical for Debezium. Holds Order and Outbox — the Outbox table has no reader but the CDC connector.

Redis

Deployment · single instance · no persistence

Caches Idempotency-Key → orderId. Losing it on restart just reopens the dedup window.

Kafka Connect + Debezium

KafkaConnect · single instance

Tails the Postgres WAL via pgoutput and republishes each Outbox row with the Outbox Event Router SMT.

Kafka

Strimzi · 3 dual-role broker/controller pods

Replication factor 3, one broker per worker node. Single topic: outbox.event.order.

email-service

Spring Boot · 2 replicas · NodePort 30082

Consumes the topic and logs a confirmation, throttled by a blocking resilience4j rate limiter — lag growth under load is the intended effect.

Tempo

single binary · local storage

Receives OTLP spans from both services and derives Grafana's service graph.

Prometheus + Grafana

Prometheus Operator · NodePort 30390 / 30300

Scrapes broker, Connect, and app metrics. Grafana ships two pre-provisioned dashboards.

kafka-ui

single instance · NodePort 30080

Browse topics, consumer lag, and connector status.

Deployment topology

Built on Kind: 1 control-plane + 3 worker nodes. Pod anti-affinity is declared for exactly three things — rest-service, email-service, and the Kafka broker pool — everything else is a single instance the scheduler places freely.

Kind cluster — kind-config.yaml (1 control-plane + 3 workers) EXTERNAL ACCESS (NODEPORT) rest-service :30081    email-service :30082    kafka-ui :30080    postgres :30432 (debug) control-plane (no workloads) ns: kafka pod anti-affinity — one replica per node (one valid placement; the scheduler decides the actual nodes) worker worker2 worker3 rest-service pod 1/2 kafka broker 1/3 rest-service pod 2/2 email-service pod 1/2 kafka broker 2/3 email-service pod 2/2 kafka broker 3/3 single instance — no anti-affinity rule Postgres StatefulSet Redis Deployment Kafka Connect KafkaConnect kafka-ui Deployment ns: monitoring Prometheus NodePort 30390 Grafana NodePort 30300 Tempo OTLP traces
app pod · anti-affinity constrained Kafka broker pod · anti-affinity constrained single instance · scheduler's choice
Figure 2. The anti-affinity rules and PodDisruptionBudgets exist ahead of any test that exercises them — the HA shape was built deliberately, not grown reactively (ADR 0004). Postgres and Kafka Connect stay single-instance: there's no Postgres HA story yet, so a warm-standby Connect wouldn't have anything to fail over to.

Access

Once ./up.sh finishes, every service is reachable at $NODE_IP:<port>.

ServicePort
rest-service:30081
email-service:30082
kafka-ui:30080
grafana:30300
prometheus:30390
postgres (debug):30432