Leader lease only
It reduces concurrent workers but cannot reject a resumed stale worker.
Leader Job Safety Lab · Visual Companion
Only the newest worker result reaches PostgreSQL after an expired worker resumes
Leader election only selects who may attempt the work. Redis must issue an ordering token and PostgreSQL must decide whether a delayed mutation is still current.
6operational scenarios
2SAFE / UNSAFE paths
41 → 42fencing tokens during takeover
01 · Problem framing
Successful leader election does not guarantee a safe database mutation
Lease expiry does not stop the old worker thread. After a GC pause, network partition, or long I/O, the old worker can resume and write after the takeover.
Lock by job name
Two differently named jobs can still mutate the same business resource.
Blind external retry
A new request can duplicate an earlier request whose result is unknown.
Brainstorming: separate the guarantees before selecting mechanisms
| Mutual exclusion | Redis leader and resource leases | Reduce concurrent execution at one moment. |
|---|---|---|
| Failover | Lease expiry and reacquisition | Allow a new worker to take over. |
| Stale-write rejection | Fencing token + PostgreSQL | Reject an older generation at commit. |
| Replay safety | OperationId + idempotency | Look up and retry the same external operation. |
| Durable completion | Exposed transaction + outbox | Commit business state and follow-up work together. |
02 · Solution design
The design separates Redis ordering from PostgreSQL commit decisions
No single component provides all guarantees. Coordination, durable mutation, and external outcome recovery remain separate.
Reuse the leader owner token
The owner token is an opaque identifier with no ordering semantics.
Let Redis make the final commit decision
Redis lease state can diverge from PostgreSQL business state.
Create a new request ID after timeout
Both the unknown first request and the retry may be applied.
Monotonic fencing token
Every takeover receives a higher generation.
PostgreSQL conditional update
Mutate only when `incomingFence > lastAcceptedFence`.
Outbox and original OperationId lookup
Query the same request instead of guessing the result.
03 · Architecture
Redis coordinates execution order and PostgreSQL decides whether to commit
No single component provides all guarantees. Coordination, durable mutation, and external outcome recovery remain separate.
APIAPI layer
Scenario selection · SAFE/UNSAFE isolation · authorization
CoordinationCoordination layer
Leader lease · resource lease · fencing-token allocation
PostgreSQLDatabase layer
Current-state recheck · conditional mutation · checkpoint/execution/outbox
Outbox / ProviderExternal-effect layer
Provider call · receipt · reconciliation
04 · Interactive flow
Select a scenario and inspect where the safeguard changes the outcome
Switch the same scenario between SAFE and UNSAFE to find the exact step where the final state diverges.
05 · Source mapping
Concrete classes and tests prove each safeguard
The visible steps are reconstructed from the actual services and deterministic scenario implementation.
JobRunCoordinatorAcquires and releases leader and resource leases in order.
coordination/JobRunCoordinator.ktRedisJobFencingLeaseAdapterUses Redis Lua to allocate monotonic fencing tokens.
coordination/redis/RedisJobFencingLeaseAdapter.ktFencedJobExecutionServiceRechecks tenant, region, version, and namespace before mutation.
execution/FencedJobExecutionService.ktJobSafetyRepositoriesOwns the conditional mutation, checkpoint, execution, and outbox writes.
persistence/JobSafetyRepositories.ktOutboxEffectWorkerCalls the provider outside the transaction and persists uncertain outcomes for reconciliation.
effect/OutboxEffectWorker.ktJobSafetyScenarioServiceProduces deterministic SAFE and UNSAFE results for all six scenarios.
scenario/JobSafetyScenarioService.ktTests that verify the implemented behavior
LeaseOverrunScenarioTestReject token 41 and preserve the token-42 resultFencedMutationPostgresIntegrationTestReject a stale conditional mutation in PostgreSQLJobSafetyEndToEndIntegrationTestCommit business state, checkpoint, execution, and outbox atomicallyNonFenceableEffectScenarioTestAvoid duplicate effects with a stable OperationId and reconciliationUnsafeJobSafetyControllerConditionTestIsolate the UNSAFE API behind both profile and property
06 · Run the example
Start the related services and verify the SAFE result first
The example uses JDK 25, a PostgreSQL 18-compatible server, and a Redis 8-compatible server. Run the UNSAFE API only as an isolated comparison.
07 · Boundaries
Problems addressed by the example and remaining limits
Delayed database mutation
PostgreSQL rejects a conditional mutation carrying an old fencing token.
Configuration changes during execution
The transaction reloads tenant, region, version, and namespace state.
Unknown external result
The worker queries the original OperationId and records a receipt.
Provider has no idempotency key or query API
Exactly-once external effects cannot be guaranteed; manual review or compensation is required.
Redis counter history is lost
Preserve the counter in backups or advance the namespace epoch.
Schema migration
The example uses SchemaUtils at startup; production needs Flyway or Liquibase.