Coroutine Observability: The Boundaries Micrometer Tracing and Readiness Actually Prove

Tracing and readiness are not decoration for dashboards. They define what operators can trust during a failure.

2026-06-30 · bluetape4k-workshop / exposed-workshop · issue #185

A coroutine-based WebFlux or Ktor service can work perfectly in local tests and still become confusing in production. Traces appear to split across suspend boundaries. A readiness probe says UP while real requests stall on the database. Adding another dashboard does not fix that. The first job is to decide which signal proves which part of the code path.

This post uses the observability examples in bluetape4k-workshop and the readiness examples in exposed-workshop. The contract is small:

• parent spans must survive suspend boundaries;
• downstream WebClient calls must carry traceparent;
• readiness must prove database reachability, not just endpoint availability.

Separate the work path from the observe path

Prometheus, Grafana, Zipkin, Actuator, and registries do not trigger business work. They read the meters and spans left behind by the request path. If that boundary is unclear, both diagrams and operational decisions drift.

The business request moves through the handler, observation scope, suspend service, and downstream clients. Prometheus, Grafana, and Zipkin observe the signals after that work is recorded.

In micrometer-tracing-coroutines, CoroutineService wraps suspend blocks with withObservationSuspending. The scope must stay alive across delay(...) and WebClient.awaitBodyOrNull() suspension points.

private suspend fun getTodoById(id: Int): Todo? {
    return withObservationSuspending("get-todo-by-id", observationRegistry) {
        client.get()
            .uri("/todos/${id}")
            .retrieve()
            .awaitBodyOrNull<Todo>()
    }
}

withObservationSuspending itself is only a thin wrapper. The important shape is the coroutine context created by withObservationContextSuspending: the current observation is stored in Reactor context, and a ThreadContextElement reopens the Micrometer scope whenever the coroutine resumes on a carrier thread.

suspend fun <T: Any> withObservationContextSuspending(
    name: String,
    observationRegistry: ObservationRegistry,
    block: suspend CoroutineScope.() -> T?,
): T? {
    name.requireNotBlank("name")
    val observation = observationRegistry.start(name)

    return try {
        withContext(observation.asCoroutineObservationContext()) {
            block()
        }
    } catch (e: CancellationException) {
        throw e
    } catch (e: Throwable) {
        observation.error(e)
        throw e
    } finally {
        observation.stop()
    }
}

The bridge is asCoroutineObservationContext().

private class ObservationScopeContextElement(
    private val observation: Observation,
): ThreadContextElement<Observation.Scope>, AbstractCoroutineContextElement(Key) {
    companion object Key: CoroutineContext.Key<ObservationScopeContextElement>

    override fun updateThreadContext(context: CoroutineContext): Observation.Scope =
        observation.openScope()

    override fun restoreThreadContext(context: CoroutineContext, oldState: Observation.Scope) {
        oldState.close()
    }
}

private suspend fun Observation.asCoroutineObservationContext(): CoroutineContext {
    val reactorContext = (currentReactiveContext() ?: Context.empty())
        .put(ObservationThreadLocalAccessor.KEY, this)

    return reactorContext.asCoroutineContext() + ObservationScopeContextElement(this)
}

The important part is not the helper name. It is how the current span is opened, restored, and stopped inside suspend code. Coroutines may resume on another carrier thread. If the trace scope is only thread-local, a span tree can suddenly look flat.

Tests pin down trace propagation

observability-basic keeps the path intentionally small. A WebFlux suspend endpoint creates the HTTP server span, OrderService creates order.service.fetch, and the Spring-managed WebClient.Builder injects W3C traceparent into the downstream inventory call.

@AutoConfigureTracing
class TracePropagationTest : AbstractBasicTest() {
    @Test
    fun `GET orders id - traceparent header propagated to downstream`() = runSuspendIO {
        webTestClient.get()
            .uri("/orders/1")
            .exchange()
            .expectStatus().isOk

        val request = mockServer.takeRequest(2, TimeUnit.SECONDS)
        request.headers["traceparent"].shouldNotBeNull()
    }
}

observability-advanced adds cache-aside behavior. On a cache miss, user.cache.get, user.db.find, and user.cache.put must be child spans under user.service.get. On a cache hit, the DB span must not exist.

http.server.requests
  └─ user.service.get
       ├─ user.cache.get
       ├─ user.db.find
       └─ user.cache.put

UserServiceTest verifies the parent-child relationship with hasParentObservationContextSatisfying. It does not only check that span names exist; it checks that the cache-miss spans attach to the same service span.

CancellationException is not a failed span

A common coroutine observability mistake is wrapping suspend calls in a broad helper such as runCatching {}. The examples rethrow CancellationException first and record only non-cancellation failures as observation errors.

return try {
    withContext(ObservationScopeContextElement(observation)) {
        block()
    }
} catch (e: CancellationException) {
    throw e
} catch (e: Throwable) {
    observation.error(e)
    throw e
} finally {
    observation.stop()
}

That keeps observability code from breaking structured concurrency. A cancellation should not pollute error dashboards. Real failures such as 5xx responses or Redis errors should still become span errors or explicit fallback signals.

Readiness means “ready to receive”, not merely “alive”

Liveness and readiness differ first in the action a platform should take when they fail. Liveness asks whether the process is still worth keeping alive. If it fails, an orchestrator such as Kubernetes normally restarts the container. Readiness asks whether this instance should receive traffic right now. If it fails, the instance should be removed from the load balancer or Service endpoints without necessarily killing the process.

That distinction matters not only when the database is down, but also when a saturated connection pool causes database timeouts while the HTTP server and coroutine dispatcher are still healthy. The process can be alive, so liveness can still pass. The service cannot safely process orders without usable database access, so readiness must fail. That is the gap between “alive” and “ready to receive”.

This distinction also shows up in incident language. Backend developers often describe connection-pool exhaustion and database timeouts as “the DB is down” rather than “the service cannot use the DB.” Infrastructure engineers can fairly answer, “the database process is healthy.” Both statements can be true within their own observation boundary. The problem is that without separate liveness and readiness semantics, “the process is alive,” “the database process is alive,” and “this service can use the database” collapse into one overloaded sentence.

The Spring Boot and Ktor examples express this operational contract with different framework shapes.

The readiness endpoint checks both example state and repository/database reachability. A degraded dependency must return 503, not a misleading 200.

The Spring example plugs a custom health contributor into Actuator readiness.

override fun health(): Health =
    when {
        !state.isDatabaseAvailable() ->
            Health.down()
                .withDetail("database", "degraded by example state")
                .build()
        else -> databaseHealth()
    }

The Ktor example implements /readyz directly.

get("/readyz") {
    val databaseReady = readinessState.isDatabaseAvailable() && repository.ping()
    call.respond(
        status = if (databaseReady) HttpStatusCode.OK else HttpStatusCode.ServiceUnavailable,
        message = ReadinessResponse(
            status = if (databaseReady) "UP" else "DOWN",
            database = if (databaseReady) "reachable" else "degraded",
            requestId = call.callId.orEmpty(),
        ),
    )
}

Spring gives you Actuator conventions and health groups. Ktor makes the response shape and route behavior explicit in application code. Neither is universally better. The choice depends on the operational contract your platform expects.

Missing signals usually mean missing wiring

Missing signal	Likely wiring issue	Example evidence
No traceparent on downstream calls	Tracing bridge is missing, or code bypasses the managed WebClient.Builder	TracePropagationTest
DB span is not a child of the service span on cache miss	Current observation scope is not restored after coroutine resume	ObservationScopeContextElement
Cancellation appears as a failed span	CancellationException is swallowed by a broad catch	local observed() helper
Readiness returns 200 during DB degradation	Readiness state and repository ping are not connected to the endpoint	Spring DatabaseReadinessHealthIndicator, Ktor /readyz
Request id differs between logs and responses	Missing sanitize/echo boundary	Spring RequestCorrelationFilter, Ktor CallId

Observability is less about what you installed and more about which failure is fixed to which signal. When traces stay connected, readiness does not lie, and request ids line up between responses and logs, incident analysis moves from guessing to checking.

Coroutines do not change the operational contract

Coroutine-based services do not need a completely different observability model. The broad shape is the same as a thread-per-request service: create the HTTP server span, attach service and downstream spans as children, record real errors, and make readiness report whether the instance can safely receive traffic.

Virtual Threads fit the same rule. When Spring MVC or blocking clients run on virtual threads, the code shape is closer to classic thread-per-request, and ThreadLocal-based observation scope is easier to reason about than in coroutine suspend/resume code. The operational contract still does not change. More virtual threads do not create more database connections. Connection-pool exhaustion, downstream timeouts, and readiness returning 503 remain the same decisions.

The difference is that coroutine code must be more explicit about what still belongs to the same request. Execution can suspend and resume on another carrier thread. If the current observation lives only in a thread-local scope, it can disappear at exactly that boundary. That is why the example restores scope through ThreadContextElement and also keeps the observation in Reactor context.

The practical cautions are narrow:

1. Pin context propagation with tests. Verify both downstream traceparent and parent-child span relationships.
2. Do not turn cancellation into a failed span. Rethrow CancellationException first, and record only real failures with observation.error(e).
3. Readiness must describe whether requests can be handled, not whether coroutines are still running. If a required dependency such as the database is degraded, the process can be alive while readiness returns 503.

In short, coroutine observability is not about memorizing a separate model. It is about preserving the existing observability contract across suspend boundaries without leaking context or misclassifying cancellation.

Source notes

• micrometer-tracing-coroutines
• observability-basic
• observability-advanced
• 09-spring-observability-readiness
• 10-ktor-observability-readiness