bluetape4k-workshop
한국어

Kafka Outbox Fallback · Visual Companion

Reduce Outbox Writes in Order Transactions While Recovering Event Publication After Kafka Failures

The normal path writes only orders and publishes directly to Kafka. A publish failure stores a replayable event_publications row; if that write also fails, reconciliation rebuilds publication data from the order.

1normal-path DB write
3direct Kafka attempts
6failure and recovery scenarios

Removing the Outbox row from every order adds explicit recovery responsibilities

Transactional outbox stores the order and publication data in one transaction. Kafka-first fallback reduces the normal-path database writes, but must recover timeouts and fallback-store failures separately.

Normal-path cost

Writing and indexing an Outbox row for every order increases database work in the order transaction.

Uncertain timeout

Kafka may have stored the event while only its response was delayed. The application cannot declare non-delivery.

Second write failure

When both Kafka publication and the event_publications write fail, only the order remains.

Duplicate publication

Relay or reconciliation can publish an eventId that Kafka already accepted.

Brainstorming: separate normal-path write cost from failure-recovery risk

Rejected

Use Redis Streams as the fallback store

A second durable system would obscure the core PostgreSQL and Kafka trade-off in the first example.

Rejected

Keep an Outbox row on the normal path

This retains the strongest storage semantics but does not reduce normal-order database writes.

Selected

Store only failed publications in PostgreSQL

This reuses the current Exposed and PostgreSQL patterns while separating normal and recovery costs.

Selected

Use a stable eventId and idempotent consumers

This handles duplicate publication caused by timeouts and reconciliation.

Transactional outbox and Kafka-first fallback use different normal paths

Switching the approach changes database writes, Kafka waiting, and the recovery reference together. The write count is a structural fact, not a benchmark result.

normal-order DB writes
wait for Kafka in API handling
publication data
missing-row reconstruction
consumer deduplication

Separate components own order storage, direct publication, relay, and reconstruction

The architecture view shows responsibilities and data access, not detailed timing. Use the recovery simulator below for ordered behavior.

API layer

OrderControllerorder API

Application layer

PlaceOrderUseCasecoordinate store then publish

Persistence layer

TransactionalOrderWriterorders transaction
PostgreSQL.ordersorder data
EventPublicationRepositoryreplay rows and claims
PostgreSQL.event_publicationsfailed and repaired publications

Messaging layer

OrderEventPublisherdirect Kafka publish and fallback
Kafka.order-eventsOrderPlaced events

Recovery and operations

EventPublicationRelayclaim and republish
PublicationReconcilerreconstruct missing publication data

Select a failure condition and observe database, Kafka, and API state

Advancing the scenario changes the active layer and cumulative state. A Kafka timeout uses unknown, not zero, for the received-event count.

relay-aCLAIMED
relay-bEMPTY
API result
orders
event_publications
Kafka events
direct attempts
relay retries
publication state
claim owner
API layerApplication layerPersistence layerMessaging layerRecovery and operations

Keep publication states separate from the claim-processing state

Select a state to inspect its entry condition and next action. CLAIMED is not an EventPublicationStatus enum value.

Meaning

Entry or next action

Concrete classes implement each simulator step

The class view includes only public methods and use relationships that map to the scenarios.

API layer

OrderController

  • placeOrder(request): ResponseEntity<OrderResponse>
Application layer

PlaceOrderUseCase

  • placeOrder(request): OrderResponse
Persistence layer

TransactionalOrderWriter

  • saveOrder(...): OrderRecord
  • getOrder(orderId): OrderRecord
Messaging layer

OrderEventPublisher

  • publishDirectOrFallback(event): OrderPublicationStatus
Persistence layer

EventPublicationRepository

  • upsertNotPublished(...)
  • claimNextBatch(...)
  • markPublished(...)
  • markRelayFailure(...)
  • findOrdersWithoutPublicationsCreatedOnOrBefore(...)
Recovery and operations

EventPublicationRelay

  • scheduledRelay()
  • relayOnce(): RelayResult
Recovery and operations

PublicationReconciler

  • scheduledReconcile()
  • reconcileOnce(): ReconcileResult
Application layer

PublicationQueryService

  • findAll(): List<PublicationResponse>
Recovery and operations

OutboxMetrics

  • recordDirectPublish(result)
  • recordFallbackStored(result)
  • recordRelay(result)
  • recordReconciler(result)

Exposed separates the order transaction from replayable publication data

Order writes, eventId upserts, claims, and SQL anti-joins run through different repository methods.

Order transaction

TransactionalOrderWriter.saveOrder() changes only orders. Kafka and event_publications run after commit.

Stable eventId

OrderPlacedEvent.from(order) uses order-placed:{orderId}:v1. The unique event_id upsert collapses retries and reconstruction into one row.

SQL claim

claimNextBatch() applies state, nextAttemptAt, claim expiry, ordering, and batch limits in the database.

Missing-row query

findOrdersWithoutPublicationsCreatedOnOrBefore() uses a cutoff and anti-join to find orders without publication data.

Start the related services, then inspect the normal and recovery paths

The application uses PostgreSQL and Kafka. Demo admin endpoints are disabled by default.

Tests cover the normal path, recovery, concurrent claims, and safe query output

MeaningKafkaOutboxFallbackFlowTest
order transaction writes only orderstransactional writer stores only order row
direct success without a fallback rowplaceOrder stores only order row and returns PUBLISHED_DIRECT when direct Kafka publish succeeds
NOT_PUBLISHED after three failuresdirect publish retries three times then stores NOT_PUBLISHED fallback row
bounded response time and fallback storagedirect publish timeout stores NOT_PUBLISHED fallback row
visible FALLBACK_STORE_FAILED resultfallback insert failure returns FALLBACK_STORE_FAILED and records safe metric and log
PUBLISHED after relayrelay publishes fallback row and marks it PUBLISHED
DEAD_LETTER after the third failurerelay failure increments retry and moves to DEAD_LETTER
one claim winner under concurrencyconcurrent relay calls cannot claim the same row twice
expired claims become eligiblestale relay claim becomes eligible after claim ttl
ordering and batch limits in SQLclaimNextBatch applies SQL eligibility ordering and limit
stable eventId reconstructionreconciler reconstructs deterministic fallback row and documents duplicate risk
SQL cutoff and anti-joinreconciler uses SQL cutoff and anti join for missing publications
no raw payload or sensitive error textpublication endpoint never exposes raw payload or raw exception text
direct, fallback, relay, and reconciler metricsmetrics record direct failure fallback relay and reconciler outcomes

The normal path writes less data, but duplicate and recovery risks remain

Handled

Normal-order Outbox write

The normal path writes only orders and does not persist publication data.

Handled

Republish after Kafka failure

A relay claims NOT_PUBLISHED rows and republishes them to Kafka.

Handled

Fallback-store failure

Reconciliation rebuilds publication data from orders after the grace period.

Caveat

Delivery after timeout

The application cannot determine whether Kafka stored the event.

Caveat

Exactly-once delivery

Claims and stable eventIds do not provide exactly-once delivery.

Caveat

Consumer deduplication

Consumers must prevent duplicate business effects for the same eventId.