Temporal Observability in Production: The Gap Nobody Talks About Until It Costs You

At Xgrid, we design, run, and scale Temporal in real production environments; from high-throughput payment platforms to AI agent orchestration pipelines to healthcare data routing systems where a missed event isn’t a metric blip, it’s a compliance liability. What follows is what operating in those environments has taught us about observability: not what the documentation says to do, but what production actually demands.

TL;DR

• Temporal can be healthy while your system is not: workflows often stall in RUNNING waiting on signals/timers/dependencies with no errors or latency spikes.
• Standard APM tracing breaks at async boundaries—use OpenTelemetry interceptors + span links to connect request → workflow → activities without lying about causality.
• Replay will duplicate logs and inflate metrics unless you use replay-safe logging and guard side effects with !Workflow.isReplaying().
• Make long-running work observable: put most telemetry in activities, use structured heartbeats for progress/soft errors, and use Search Attributes + Visibility queries to monitor “stuck” workflows.

It’s a Tuesday morning. A support ticket arrives. An enterprise customer’s order hasn’t moved in seven hours.

The on-call engineer checks the monitoring stack. Green. APM service map: healthy. P99 latency: nominal. Error rates: zero. Every dashboard, every alert threshold, every carefully tuned signal is calmly and confidently telling the same story: everything is fine.

Then someone opens the Temporal UI.

One workflow execution. RUNNING. Last event: six hours and forty-three minutes ago. The workflow is sitting in perfect Temporal health; durable, persistent, waiting faithfully; for a signal from a payment service that started silently dropping webhooks forty-eight minutes after its last deployment.

Nothing paged. Nobody knew.

This scenario plays out across the Temporal community with enough regularity to have its own pattern name. It is not a Temporal failure. It is not an infrastructure failure. It is an observability failure; a gap between what your monitoring stack can see and what is actually happening inside a durable workflow execution. Closing that gap requires understanding something that most APM tooling was simply never designed to handle.

Temporal Health ≠ System Health: The Production Observability Trap

There is a mental model that feels reasonable when you first deploy Temporal and will eventually cost you a production incident if you hold it long enough: if Temporal is healthy, your system is healthy.

It is seductive because it is almost true. Temporal’s operational metrics are genuinely good. temporal_activity_schedule_to_start_latency tells you whether your workers are keeping up with demand. Task queue depths tell you whether you need to scale. Workflow completion rates give you a broad failure signal. If these metrics are healthy, Temporal is healthy.

But a healthy Temporal cluster is a faithful executor. It will faithfully execute a broken workflow. It will faithfully sit in RUNNING state waiting for a signal that will never arrive. It will faithfully retry an activity against a dependency that started returning HTTP 200s on failure three deployments ago. It will do all of this without generating a single error, triggering a single alert, or showing any anomaly in any operational metric, for as long as you let it.

The gap between Temporal is healthy and my system is working correctly is a gap Temporal cannot close for you. It is the gap this post is about.

We do not believe in ripping out your existing observability stack to solve this. If you have Datadog, Prometheus, Grafana, or New Relic already running, those tools are perfectly capable. The work is wiring Temporal into that ecosystem correctly; understanding exactly where the seams are, and instrumenting across them with intention.

Orphaned Traces in Temporal: When APM Can’t Follow Workflows

Open any distributed tracing system–Jaeger, Grafana Tempo, Datadog APM– and you will find they share the same foundational assumption: a request comes in, it fans out, spans are created and closed within seconds or minutes, the trace assembles, and you get a picture.

That model describes a portion of what Temporal does. The rest of it, i.e. long-running workflows that pause for timers, wait on signals, and execute activities across workers that may not even exist yet, lives in an asynchronous, stateful dimension that standard span-based tracing was never designed to represent.

The result, when teams wire Temporal into an existing observability stack without accounting for this, is predictable: traces look truncated. An HTTP request span ends at the line where workflowClient.start() is called. 

Everything the workflow does; every activity it executes, every decision it makes, every external call it orchestrates; happens in a disconnected trace tree with no relationship to the request that caused it. In your tracing backend, the workflow is invisible. The request looks clean. The causal chain is severed.

This is one of the most consistently reported pain points in the Temporal community. Engineers instrumenting Temporal for the first time discover that their traces contain the API layer and nothing downstream. The workflow execution and all its children are orphaned; technically present in the tracing backend, but unreachable from any trace that would lead an engineer to look for them.

Fix Temporal Tracing with OpenTelemetry Span Links (Not Parent-Child Spans)

The instinct is to inject the traceparent header into workflow input and reconstruct a child span on the worker side. It works at the surface. But it lies about causality; it tells your tracing backend that the workflow was synchronously nested inside the originating HTTP request. It was not. The request returned 202 Accepted. The workflow ran for four hours. A parent-child span relationship that implies synchronous nesting across a four-hour boundary is worse than no relationship at all.

The correct semantic is a span link. A link says: this execution is causally downstream of that one, but not synchronously nested within it. OpenTelemetry supports this natively, and it produces an honest picture of asynchronous causality.

The battle-tested way to configure your Temporal client so traces span across the boundary, on both client and worker, uses the OTel interceptor:

Apply the same interceptor configuration to your workers. This ensures that when a workflow wakes up hours or days later on any worker in your fleet, it still reports back to the correct distributed trace context in your APM.

For explicit causal linking where you want to propagate context from an upstream HTTP request into workflow telemetry:

Temporal Schedules and Tracing: Avoid One Giant “Mega-Trace” Across Runs

Symptom: Scheduled workflow runs show up as one giant trace in your APM (Datadog/Tempo/Jaeger), making it hard to isolate a single run.

Cause: The default OpenTelemetry interceptor behavior can reuse the same parent trace context across schedule executions, so spans from many runs accumulate under one trace.

Fix: Break trace context at the schedule boundary and start a new root span per scheduled run. Each scheduled execution should be its own trace, with links only where you intentionally model causality.

Why it matters: When a specific scheduled run fails or slows down, a weeks-long “mega-trace” is effectively unusable during incident response.

Temporal Replay: Why Your Logs Duplicate and Metrics Overcount

Replay is what makes Temporal reliable. It is also, if your instrumentation does not account for it, what makes your metrics wrong, your logs misleading, and your dashboards quietly useless.

Here is the mechanism. When a workflow worker processes a workflow task and the execution is not in its local cache because the worker restarted, the cache was evicted under memory pressure, or the execution was migrated, the worker replays the entire event history from the beginning to reconstruct the current state. Your workflow function re-executes from line one. Every branch re-evaluates. Every variable is reassigned. Every line of code runs again.

Any instrumentation with external side effects inside a workflow function fires again on every replay. Log statements, span creation, metric counters; all of it.

Metric overcounting is the most operationally dangerous replay side effect. Without a replay guard, a workflow.started counter increments on the first execution and again on every replay.

• What happens: the counter is re-emitted whenever workflow code replays, not just when the workflow actually starts.
• When it shows up: worker restarts, scale-outs, cache eviction/pressure from long-running executions, and routine deployments.
• Impact: a single workflow can be overcounted 5–10× during peak periods, making dashboards and alerts unreliable exactly when load is highest.

This isn’t theoretical—it shows up frequently in real production deployments and community discussions about Temporal metric inconsistency.

The log duplication version presents as a specific symptom documented across multiple community forum threads: an engineer queries a running workflow in the Temporal UI, and log lines from the workflow function reappear in the log stream, including lines emitted hours ago when the workflow started. The workflow is replaying to answer the query. Any log statement not using the SDK’s replay-safe context logger fires again. The duplicate logs are the symptom of an unguarded logger, not a malfunction.

The architectural principle that eliminates most replay-related instrumentation problems: keep workflow functions thin on instrumentation and let activities carry the observability weight. Activities do not replay; they execute exactly once per attempt. Standard OpenTelemetry instrumentation inside an activity works identically to any other service code. Moving detailed logging, business metrics, and operational spans into activities removes the entire replay constraint from your instrumentation design. The workflow function becomes an orchestration skeleton. The activities become fully observable service endpoints.

How to Make Long-Running Temporal Activities Observable with Heartbeat Payloads

The most common heartbeat pattern across Temporal codebases is activity.RecordHeartbeat(ctx, nil); an empty payload, called periodically, used to tell Temporal that a long-running activity is still alive.

This is correct and necessary. It is also the most significant missed observability opportunity in the average Temporal deployment.

The heartbeat payload is the only native mechanism in Temporal for making the internal state of a running activity visible to the outside world in real time. Not to the workflow function; workflow code cannot read heartbeat payloads while an activity is running. But to the Temporal UI, to monitoring scripts, to the on-call engineer staring at a workflow that has been in the same state for three hours trying to understand whether it is making progress or silently stuck.

Consider a data processing activity running against a large dataset; tens of millions of records, expected runtime of several hours. With an empty heartbeat payload, what you know is: the activity is alive. What you do not know is whether it is processing ten thousand records per minute or nine hundred, whether it is ten percent done or ninety, whether it encountered transient lock contention an hour ago that slowed it to a crawl. The activity is alive. You have no idea what it is doing.

With a structured payload, the same activity becomes an instrument:

The LastSoftError field deserves particular attention. Activities frequently absorb transient errors internally, such as rate-limited API calls that succeed on retry, database deadlocks that clear within the activity’s own retry loop, network timeouts that resolve before the activity-level timeout fires. These errors never appear in Temporal’s event history because they are resolved before the activity returns. But a sustained pattern of soft errors is often the earliest available signal that a downstream dependency is degrading; before it shows up in error rates, before it triggers retries visible in Temporal, before anyone files a support ticket.

In payment processing workflows, this matters acutely. A gateway that starts returning TIMEOUT on twenty percent of auth attempts but succeeds on immediate retry looks perfectly healthy from Temporal’s perspective; the activity is succeeding. The heartbeat payload surfacing a LastSoftError: “gateway_timeout” at a rising frequency is the degradation signal you can act on before it becomes a failure rate. That early-warning gap is not covered by any other Temporal primitive.

Search Attributes + Temporal Visibility

Most teams discover Search Attributes as a debugging tool: find a specific workflow by business ID when a support ticket arrives. This use case is valuable. It is also roughly one percent of what Search Attributes are capable of in production.

Temporal Search Attributes are indexed fields you can set and upsert from workflow code, making workflow state queryable via Temporal Visibility and directly filterable in the Temporal UI.

In production, treat them as an operational state index, not just a search feature. They can replace (or drastically simplify):

• status tables in your application database
• custom “workflow status” services you build just for support/Ops
• repeated DB lookups and manual cross-referencing during incidents

With this in place, Temporal Visibility answers operational questions that would otherwise require a purpose-built status service:

Why it matters: you get a queryable source of truth for workflow progress without building and maintaining a separate synchronization layer.

One constraint to design around: Visibility is eventually consistent. Under normal load, updates often appear quickly, but can lag under heavy load or if visibility processing backs up.

Do not use Search Attributes for synchronous control flow. For SLA alerting, add buffer to thresholds and verify by fetching the workflow execution (not only by relying on Visibility query results).

Temporal Metrics in Production: Cluster Health vs Business Health

Temporal’s SDK and server metrics answer one question reliably: is Temporal working?

temporal_activity_schedule_to_start_latency is the primary scaling signal; activities sitting in task queues longer than expected means your worker pool cannot keep up. temporal_workflow_task_execution_latency tells you whether workflow logic itself is a bottleneck. These should be your baseline operational monitoring, and they should be owned by your platform or SRE team; the people responsible for the Temporal cluster infrastructure.

They do not answer the question that matters to your business: is my system behaving correctly?

This second question belongs to your application engineers, and the answers only come from instrumentation you build deliberately inside your activities. Generic dashboards will not save you during a complex incident. Your metrics, traces, and alerts must match the specific domain logic you are orchestrating.

For payments and money movement: The golden alert is not a workflow failure; it is a compensation failure. When you are using sagas for partial rollbacks (gateway authorised but ledger failed to post, for example), the moment a compensation workflow fails to execute completely is a high-priority, escalation-required event. Standard gateway timeout retries should not page anyone. A failed saga compensation absolutely should. Wire your alerting to distinguish between them.

There is also a specific metric labelling gap worth knowing: the server-level workflow_failed metric does not include workflow type as a label. If multiple workflow types run in your namespace and one starts failing, the server metric alone will not tell you which one. The SDK-level temporal_workflow_failed metric; emitted by your workers, not the server; does include workflow_type. Use the SDK metric for workflow-type-specific alerting.

For AI agents and multi-agent orchestration: Standard HTTP monitoring does not cover what happens inside a long-running agent workflow. You need custom instrumentation specific to agent behaviour; dedicated spans for task routing decisions, tool invocations, model calls, human-in-the-loop approval steps, and safe retries when tools or external APIs fail. The trace for an agent workflow should tell the story of why the agent made each decision, not just that it made one. Without this, debugging a hallucinating or misbehaving agent in production is archaeology against a wall of opaque activity completions.

For business SLA tracking across all verticals: Temporal’s execution duration metric measures Temporal’s internal overhead. It does not measure your business outcome time. Emit a custom duration metric at workflow completion, tagged with outcome and business dimensions, and alert on the P95 and P99 of that distribution:

That timer; business.order.total_duration P95 > threshold; is the alert that tells you whether your system is meeting its commitments. It will fire before customers file support tickets if you build it before production. And it will never exist unless you build it, because nothing in Temporal creates it for you.

How to Query Temporal Workflow Status in Production

For any workflow that progresses through meaningful stages, a status query handler should be a default part of the workflow design—built in the first sprint, before production.

Without a query handler, answering “what is this workflow doing right now?” during a support interaction usually forces one of two bad options:

• Event-history archaeology: interpreting raw Temporal event history (accurate, but slow and hard to do live).
• A separate status database: building and maintaining a parallel status table/service in your application DB (a synchronization liability).

A query handler exposes workflow state directly from the workflow’s execution thread, evaluated against real in-memory values, and always accurate:

Queryable from support tooling, dashboards, or the CLI:

Important note: Queries can trigger workflow replay on the worker if the execution isn’t in cache. If your workflow uses non-replay-safe logging, this can surface as duplicate logs. The query result is still correct—the fix is to use replay-safe logging (or guard side effects), not to avoid queries.

Stuck RUNNING Workflows in Temporal: Monitoring for Missing Signals

There is a category of Temporal production failure with a specific and dangerous profile: no error rate, no latency spike, no failed health check. The workflow is RUNNING. Temporal is healthy. Your monitoring stack shows nothing.

These are workflows waiting on signals that will never arrive.

A checkout workflow listening for a PaymentConfirmedSignal from a payment service that started silently failing. A document verification workflow waiting for a callback from a third-party provider that changed their webhook payload format and is now sending events your handler drops without an error. In healthcare data routing environments; where workflows orchestrate the movement of patient records across legacy systems; a missed signal is not a metric anomaly. It is a compliance event. The workflow waits, faithfully and indefinitely, while a patient record sits unprocessed in a queue nobody is monitoring.

Temporal cannot see upstream. It will wait for signals that are never coming without complaint, without error, for as long as you let it.

Fix: Monitor Signal-Wait Timeouts via Visibility Queries 

Closing this gap requires external monitoring. A scheduled job that queries Visibility for signal-driven workflow types that have been in Running state past their expected maximum completion time:

When this returns results, you have a signal-routing problem in the upstream service, not a Temporal problem. Route the alert to the team that owns the signal source. Build this monitoring before production. Not having it is how that 11:47 AM support ticket happens.

From Trace IDs to Workflow IDs: The Correlation Shift Temporal Requires

The question every mature engineering organisation asks when introducing Temporal: do we need new monitoring infrastructure?

No. Datadog, Grafana, New Relic, Elastic APM; all of them work. The SDK metrics interface wires into StatsD or Prometheus. Activity spans appear in your existing APM alongside traces from your other services. Logs ship to your existing aggregation. Nothing needs to be replaced.

What changes is the atomic unit of correlation.

In a synchronous service-oriented request, that unit is the trace ID. One trace, one request, everything that happened in service of it. Clean and self-contained.

In Temporal, the unit is the workflow ID. A single workflow execution produces activity traces across multiple workers, metrics emitted over hours or days, and logs on worker processes that did not exist when the workflow started. The workflow_id and run_id are the thread that binds all of it together, and every instrumentation artifact; span, metric, log line; must carry them as first-class fields.

The operational investment is building a correlation layer: dashboards parameterised by workflow_id that pull together traces, metrics, and logs without manual joins. Build it once, and it becomes the first artifact opened in any workflow-related incident.

Best Practice: Deterministic, Human-Readable Workflow IDs for Faster Incident Response

One design choice that pays compounding returns: make workflow IDs human-readable and deterministic. order-${orderId} instead of a UUID. When a support ticket arrives about order 98765, the workflow ID is already known. No database lookup, no cross-reference, no manual step. Open the dashboard, paste order-98765, and the full execution timeline is in front of you in ten seconds.

That single design decision; deterministic, readable workflow IDs; is the difference between a five-minute diagnosis and a twenty-minute one. Across hundreds of incidents and a team operating at scale, it compounds into a measurable reduction in on-call fatigue.

Ownership: Who Gets Paged and Why

A mature Temporal observability strategy requires a clear ownership model. Mixing cluster metrics with application metrics creates alert noise, slows incident response, and trains engineers to ignore pages. The separation is straightforward:

Platform and SRE teams own the Temporal cluster. Their signals are persistence latencies, CPU on matching nodes, sync match rates, and frontend service health. An alert firing here means the Temporal infrastructure needs attention.

Application engineers own worker fleet health and workflow execution metrics. The primary signal is temporal_activity_task_schedule_to_start_latency_seconds. If this metric spikes, the cluster is healthy; the worker fleet needs to scale. Routing this alert to infrastructure teams wastes everyone’s time and delays the actual fix.

This separation also determines who responds to business SLA breaches. A business.order.total_duration P95 alert is not a platform alert. It fires in application engineer territory, it routes to the team that owns the order fulfilment workflow, and it resolves through application-layer investigation, not infrastructure intervention. Having this ownership model documented and agreed before production is the difference between an incident that routes correctly on the first page and one that bounces between teams while a customer waits.

Temporal Observability Best Practices: A Production Checklist

The patterns that consistently produce reliable observability across production Temporal deployments converge on a small set of disciplines that are worth making explicit:

• Activities are instrumented like first-class service endpoints. Every outcome, every latency, every error category is a metric; tagged with business-relevant dimensions and carrying the workflow ID. There are no invisible activities.
• Long-running activities have heartbeat payloads that carry structured operational state: progress rate, completion estimate, and the category of the last non-fatal error handled internally. That payload is mirrored as an alertable metric. The activity is an instrument, not a black box.
• Search Attributes track business-meaningful stage transitions; updated at boundaries, not just start and end. They form the operational query surface for real-time system state and are designed with eventual consistency in mind.
• Workflow query handlers exist for every workflow type with observable internal state. They are built before production. They are used by support tooling and monitoring scripts, not just manual debugging sessions.
• A monitoring job runs frequently against signal-driven workflow types, alerting when executions remain in Running state past their expected maximum duration. It routes to the team that owns the upstream signal source.
• The correlation dashboard is parameterised by workflow ID. It is the first thing opened in any workflow-related incident. It surfaces traces, metrics, and logs in a single view without requiring a human to perform the join.
• Ownership is documented: platform teams own cluster signals, application teams own worker and execution signals, business outcome alerts route to the engineers who own the business logic.
• And in every workflow function, consistently, the replay guard: !Workflow.isReplaying(); seven words that represent the accumulated cost of every overcounting metric, every duplicate alert, and every misleading dashboard that came before them.

Closing the Temporal Observability Gap (Before It Becomes an Incident)

Temporal is an exceptional tool. It gives you durable execution, automatic retries, saga compensation, and exactly-once activity semantics—reliability guarantees most teams used to rebuild from scratch.

And when something goes wrong, Temporal gives you something rare: a complete execution history. It’s an immutable, chronological record of what the workflow did and why. That audit trail is often the difference between guessing and knowing.

But observability is the honest cost of operating in an asynchronous, stateful runtime. It doesn’t come “for free.” And it won’t map cleanly onto the request/response dashboards you’ve relied on for years. The good news is that it’s not magic—it’s engineering. Once you implement it well, Temporal systems can become more diagnosable than most distributed architectures.

The teams that get into trouble aren’t the ones who don’t care. They’re the ones who assume a green dashboard means the system is working—because that’s what it has always meant before.

In Temporal, a healthy Temporal dashboard usually means the Temporal platform is healthy. It doesn’t automatically tell you whether work is progressing inside workflows. You have to make that visible:

• Business progress: workflows moving through stages
• Signals: signal-driven workflows receiving what they’re waiting for
• Activity progress: long-running activities making measurable forward progress (not just “alive”)

Make it visible before the 11:47 AM support ticket does it for you.

Where Xgrid Can Help

If your team is at one of two points on this journey, we have a specific engagement designed for where you are.

Pre-production: Temporal Launch Readiness Review. If you are evaluating Temporal or in-pilot stages, this engagement reviews your architecture diagrams, observability and SLO readiness, and failure handling strategy. You leave with concrete, prioritised recommendations; the specific things to build in the next thirty days before you go live, ranked by production risk.

In production: Temporal 90-Day Production Health Check. If Temporal is already running and you are experiencing incidents, alert fatigue, or the creeping suspicion that your visibility is incomplete, this engagement analyses your logs, metrics, workflow failure types, retry patterns, and stuck workflow inventory. You leave with quick wins that reduce risk immediately and a longer-term instrumentation roadmap built around your specific domain and stack.

In either case, the starting point is the same: understanding what your system is actually doing, not what your dashboards are telling you it is doing.

Temporal Cloud and self-hosted Temporal have meaningful differences in visibility model, metric emission, and cluster-level monitoring. The patterns in this post apply primarily to self-hosted deployments. 

To talk through your Temporal observability architecture, reach out to the engineering team at Xgrid.