Bluetape4k Leader Part 3: Multi-leader and Strategic Election

Robotic service nodes comparing multi-leader semaphore slots and strategic candidate scoring on a 3D workbench — Some work is too slow with one node and too risky when every node joins. Group election sits in the middle.

2026-06-01 · bluetape4k-leader series · Part 3

This is Part 3 of the bluetape4k-leader series. Part 1 covered the repository overview and mental model, while Part 2 covered LeaderElector, SuspendLeaderElector, LeaderElectionOptions, and LeaderRunResult.

This post asks a slightly different question: must there always be exactly one leader? Not always. Some work, such as schema migration, should truly run once. Other work, such as aggregating hundreds of tenants, is too slow with one worker and too risky when every replica joins at once. LeaderGroupElector handles that middle ground by allowing at most N concurrent leaders.

There is another question too: what if the best node is not the one that acquired a lock first? You may want to prefer a node with fewer recent failures, more available capacity, or a longer idle time. Strategic election handles that case with a candidate registry and a pluggable strategy.

When One Leader Is Too Few

Single-leader election is simple and strong. runIfLeader("job") runs the action on one node only. The tradeoff is throughput. These jobs often want “some, but not too many” workers:

aggregate tenant workloads with a cluster-wide concurrency limit;
warm cache partitions without starting every replica at once;
poll external APIs while respecting a rate limit;
run long maintenance work with at most N cluster participants.

LeaderGroupElector assigns a fixed number of slots for one lockName. A node that acquires a slot runs the action. A node that cannot acquire a slot waits or skips. The name still says leader, but the mechanism is closer to a distributed semaphore.

LeaderGroupElector slot-token semaphore with maxLeaders, active slots, wait or skip path, release and expiry path — `LeaderGroupElector` attaches up to `maxLeaders` slot tokens to one `lockName`; only callers with a token run the action.

Group Election as a Slot-token TTL Semaphore

The core option set is small.

LeaderGroupElectionOptions(
    maxLeaders = 3,
    waitTime = 5.seconds,
    leaseTime = 60.seconds,
)

maxLeaders is the number of actions that may run at the same time. waitTime controls how long a caller waits for a slot. leaseTime controls how long the slot token lives. If the action finishes normally, the token is released. If the node dies or fails to release it, TTL expiry is the safety net.

The blocking API mirrors the single-leader shape from Part 2.

val options = LeaderGroupElectionOptions(maxLeaders = 3)
val groupElection = ExposedJdbcLeaderGroupElector(dataSource, options)

val result: AggregationReport? = groupElection.runIfLeader("tenant-aggregation") {
    aggregateNextTenantBatch()
}

Coroutine services use the suspend variant.

val options = LeaderGroupElectionOptions(maxLeaders = 3)
val groupElection = ExposedR2DbcSuspendLeaderGroupElector(database, options)

val result = groupElection.runIfLeader("tenant-aggregation") {
    aggregateNextTenantBatchSuspend()
}

The null return has the same ambiguity as single-leader runIfLeader: either the action did not run, or the action itself returned null. Group electors also provide the LeaderRunResult API when the caller needs to distinguish those cases explicitly.

when (val result = groupElection.runIfLeaderResult("tenant-aggregation") { drainQueue() }) {
    is LeaderRunResult.Elected      -> meter.mark("slot.acquired")
    LeaderRunResult.Skipped         -> meter.mark("slot.skipped")
    is LeaderRunResult.ActionFailed -> meter.mark("slot.failed")
}

There is one important operational boundary. Extending a single leader’s lease is different from extending a group slot. Each slot carries its own owner, TTL, release, and expiry semantics. The README draws this boundary explicitly: single-leader autoExtend is opt-in, but @LeaderGroupElection does not support auto-extension yet. Streaming Flux<T> / Flow<T> returns are also outside the 0.3.1 scope because group lease extension has no per-slot stream contract yet.

For group election, choose a leaseTime that comfortably covers the action and split long or irregular work into smaller chunks where possible. TTL is a model of expected work time, not a free pass to hold slots forever.

Sources: LeaderGroupElector.kt, SuspendLeaderGroupElector.kt, LeaderGroupElectionOptions.kt, LeaderGroupElectionState.kt

Strategic Election: Candidate Registry Instead of Lock Race

Group election answers “how many nodes may run?” Strategic election asks a different question: “which node should run now?”

StrategicLeaderElector does not treat the first lock acquirer as the node that should run. Each node registers a CandidateInfo. On runIfLeader, the elector loads the candidate list and applies an ElectionStrategy. The single selected node runs the action; all other nodes immediately receive null.

Strategic election flow from CandidateInfo registration through candidate registry, strategy selection, selected-node execution, skipped nodes, and result update — Strategic election registers candidates first, then applies the same strategy to the same candidate list to compute the single selected node.

CandidateInfo is the small history object the strategy uses for that decision.

CandidateInfo(
    nodeId = "node-b",
    registeredAt = Instant.now(),
    lastCompletionTime = previousCompletion,
    successCount = 14,
    failureCount = 1,
    metadata = mapOf(
        "healthPercent" to "86",
        "availableCapacityPercent" to "88",
    ),
)

Strategies are pluggable.

Strategy	Criterion
`FifoElectionStrategy`	Earliest registered candidate; ties by `nodeId`
`RandomElectionStrategy(seed)`	Candidate list sorted by `nodeId`, then seeded or system-random pick
`ScoredElectionStrategy(scorer)`	Candidate with the highest `CandidateScorer` score

Scored election includes built-in scorers.

Scorer	Preferred candidate
`SuccessRateScorer`	Node with the better success rate
`IdleTimeScorer`	Node that has been idle longer
`RecentSuccessScorer`	Node with stronger recent success history
`WeightedScorer`	Node with the best weighted sum across scorers

The strategic-election example combines readiness, success rate, and idle time to select a maintenance node.

val strategy = ScoredElectionStrategy(
    WeightedScorer(
        ServiceReadinessScorer to 0.50,
        SuccessRateScorer to 0.35,
        IdleTimeScorer to 0.15,
    )
)

val result = election.runIfLeader("service-maintenance", strategy) {
    runMaintenance()
}

The main caveat is consistency. When all nodes apply a deterministic strategy to the same candidate list, they compute the same selected node. If candidate registration or lookup timing differs, selected nodes may diverge. In distributed deployments, candidate TTL, heartbeat cadence, and registry read consistency must be treated as part of the design. RandomElectionStrategy also needs a shared seed in distributed use; otherwise each node makes its own random choice.

Sources: StrategicLeaderElector.kt, StrategicSuspendLeaderElector.kt, CandidateInfo.kt, ElectionStrategy.kt, ScoredElectionStrategy.kt, WeightedScorer.kt, examples/strategic-election

Which One Should You Use?

The three models answer different questions.

Situation	Choose	Why
Schema migration, daily settlement, or webhook polling must run once	`LeaderElector` / `SuspendLeaderElector`	A single lock gives the simplest skip-on-contention model
Several workers may run, but cluster-wide concurrency needs a hard cap	`LeaderGroupElector` / `SuspendLeaderGroupElector`	`maxLeaders` slots bound concurrent execution
The execution node should depend on recent success, idle time, health, or capacity	`StrategicLeaderElector` / `StrategicSuspendLeaderElector`	Candidate registry plus strategy selects one node instead of first-arrival lock contention
Tenants need independent leaders	Tenant-scoped elector plus single or group API	Namespacing turns `tenantId:lockName` into separate elections
Metrics or post-processing need to distinguish skipped from action-null	`LeaderRunResult` API	`Elected(null)` and `Skipped` stay separate

In practice, start with single-leader election. Move to group election when throughput becomes the pressure. Move to strategic election when “any node” is no longer good enough and the selected node must reflect health, capacity, or history. Strategic election can look attractive early, but it also means operating candidate heartbeats, TTL, read consistency, and score interpretation from day one.

References

Repository: bluetape4k-leader
README: README.md
Korean README: README.ko.md
Group implementations: Redis Lettuce, Redis Redisson, Exposed JDBC, Exposed R2DBC, MongoDB, Hazelcast
Strategic implementations: LocalStrategicLeaderElector.kt, LettuceStrategicLeaderElector.kt, RedissonStrategicLeaderElector.kt

Closing

Part 2 was about matching the caller’s execution model. Part 3 is about choosing the form of execution authority: one node, up to N nodes, or the node selected by a strategy.

Part 4 moves this API into Spring Boot and Ktor integration. It covers @LeaderElection, @LeaderGroupElection, and Ktor route/scheduler helpers: what they automate, and what still deserves explicit control.

Series Links

Comments

Leave a note or reaction with your GitHub account.