Tempo on Kubernetes: Distributed Tracing on Nutanix Objects

We moved. Two states, one truck, a new house, and a two-month gap in the publishing schedule that I’m not going to pretend was intentional. The stack didn’t take the summer off — it kept ingesting, alerting, and quietly filling its S3 buckets — but the writing stopped while the boxes were open. The boxes are (mostly) closed now.

And we’re back with unfinished business. The LGTM series gave Loki two full articles, Mimir one, Grafana two — and Tempo got cameo appearances: a line in the architecture table, a bucket in the storage article, a drain war story in the retrospective. The T in LGTM never got its own deep-dive, because when the series wrapped, tracing was the newest and least-settled piece of the platform.

It’s settled now. This is the article Tempo was owed.

Table of Contents

The Honest History: We Cut Tracing First

In February, when the platform was young and every component had to justify its RAM, we cut Tempo from scope. The reasoning in the decision log was blunt: the workloads we were onboarding — switch syslog, Windows EventLog, storage array metrics — cannot benefit from distributed tracing. A trace needs a request that travels; a syslog line doesn’t travel anywhere, it just arrives.

The note said “revisit post-implementation.” The revisit happened in April (ADR-042), for two reasons. First, the core stack had stabilized — Loki and Mimir were boring in the best way, and the team had capacity. Second, the workloads changed: ArgoCD, Grafana, Mimir, and our own collectors all speak OTLP natively, and application teams were starting to ask for more. The question stopped being “do we need tracing?” and became “do we want the answer to be somebody else’s SaaS?”

We did not.

Why Tempo

Tempo is the same operational model we already run twice. S3 blocks on Nutanix Objects. Ingesters that flush to object storage. A compactor. A query-frontend. Memberlist. A wrapper Helm chart with values-common.yaml plus per-DC overrides, credentials via External Secrets Operator, deployed by ArgoCD at the same sync wave as Loki and Mimir. When the on-call gets paged at 2 AM, TraceQL reads like LogQL reads like PromQL. A different-shaped trace store — however good — would mean learning a second operational pattern for the same job.

There’s a bonus that turned out to be the sleeper feature: Tempo’s metrics-generator derives RED metrics (traces_spanmetrics_*) and service-graph edges (traces_service_graph_*) from the span stream and remote-writes them to Mimir. Grafana’s Service Graph tab lights up without instrumenting a single application for metrics. The traces pay for themselves twice.

The Shape of the Thing

Apps (on-prem, Azure, edge)          Stack components (in-cluster)
        |                                     |
        | OTLP over TLS                       | OTLP, plain gRPC
        v                                     v
traces.conveyor.internal  (DNS CNAME, 60s TTL)
        |
        v
+---------------------------+     +---------------------------+
|  alloy-traces (EastCoast) |     |  alloy-traces (WestCoast) |
|  StatefulSet x2, VIP .81  |     |  deployed, healthy, IDLE  |
+------+-----------+--------+     +---------------------------+
       |           |         \
       | LAN       | WAN      +--> spanmetrics + servicegraph --> Mimir
       v           v
+-------------+  +-------------+
| Tempo EAST  |  | Tempo WEST  |
| (VIP .82 <--+--+ cross-DC)   |
+------+------+  +------+------+
       |                |
       v                v
 tempo-traces-east  tempo-traces-west   (Nutanix Objects, one bucket per DC)

Tempo runs in full distributed microservices mode — the one component of the stack where we didn’t trim to singletons everywhere, though we still trimmed:

Retention is 30 days, not the 365 we run for logs and metrics. Traces are the highest-volume, shortest-relevance signal we store — after the incident window closes, almost nobody goes back to a specific trace, while logs and metrics carry compliance weight for a year. Thirty days on one S3 bucket per DC keeps the cost conversation short. And note where the knob lives, because it isn’t where Loki taught you to look: retention is a compactor setting (compactor.compaction.block_retention: 720h), not a storage or limits setting.

Storage is the pattern from article 2, minus the complexity: Tempo wants exactly one bucket per DC — WAL, blocks, and compacted blocks all live under one prefix. Don’t copy Loki’s multi-bucket ExternalSecret sprawl here; the credentials even alias the same Key Vault entries the Loki buckets use, since the S3 account already had rights on the new buckets.

A Third Alloy? Really?

Yes. The trace path gets its own Alloy deployment — the third in the cluster, after the DaemonSet (pod logs, node metrics) and the network receiver (syslog, SNMP). ADR-043 spends most of its length on why we didn’t just add OTLP ports to the network receiver, and the answer is traffic profiles:

• Syslog is a steady drip of small UDP packets. Its failure mode is dropped packets, and its scaling trigger is network I/O.
• OTLP is bursty, batch-oriented gRPC/HTTP. Its failure mode is queued spans eating memory, and its scaling trigger is CPU — batching and Kubernetes-metadata enrichment are compute, not I/O.

Put both on the same pod and a trace spike can starve the syslog listener — and the syslog listener is carrying firewall and audit logs with 365-day compliance retention. The decision record’s summary line: “Bundling them means every OTLP config change risks a syslog rollout. The operational coupling is not worth the ~200 lines of YAML saved.”

So: alloy-traces, two replicas behind a MetalLB VIP, required pod anti-affinity so the replicas sit on different nodes (MetalLB L2 failover is only useful if the survivor isn’t on the failed node — same lesson as the network receiver). One detail that took a production incident to get right: it’s a StatefulSet, and it wasn’t always.

Until early July it was a Deployment, and the cross-DC export queue (more on that below) lived in an emptyDir. Pod restarts kept the queue; pod replacements — chart upgrades, node drains — silently discarded it. Which means every routine upgrade threw away exactly the WAN backlog the entire HA design exists to protect, and nothing logged a complaint. A health check caught it, not an alert. Now each replica has a 10 Gi local-path PVC mounted for the queue, claimed through volumeClaimTemplates, and the buffer survives everything short of losing the node.

The Collector Pipeline

The Alloy config is a straight line with a fan-out at the end:

otelcol.receiver.otlp "default" {
  grpc {
    endpoint         = "0.0.0.0:4317"   // NOT the default localhost —
    include_metadata = true             // external traffic is silently
    tls { ... }                         // dropped otherwise
  }
  http {
    endpoint         = "0.0.0.0:4318"
    include_metadata = true
    tls { ... }
  }
  output {
    traces = [otelcol.processor.memory_limiter.default.input]
  }
}

// FIRST processor, always — drop spans before the OOM killer drops the pod
otelcol.processor.memory_limiter "default" {
  check_interval         = "1s"
  limit_percentage       = 80
  spike_limit_percentage = 25
  output {
    traces = [otelcol.processor.attributes.source_ip.input]
  }
}

// Forensic attribution for no-auth ingest — the TCP peer from the TLS
// handshake, which a sender cannot spoof
otelcol.processor.attributes "source_ip" {
  action {
    key            = "source_ip"
    from_context   = "client.address"
    action         = "insert"
  }
  action {
    key    = "received_dc"
    value  = sys.env("DC_LABEL")
    action = "insert"
  }
  output {
    traces = [otelcol.processor.k8sattributes.default.input]
  }
}

Then k8sattributes (namespace, pod, deployment, node for in-cluster senders; external spans pass through unenriched), batch (1024 spans or 5 seconds, whichever first), and the fan-out:

• tempo_local — plain gRPC to the local Tempo distributor over the LAN. Small in-memory queue; the LAN doesn’t blip.
• tempo_remote — the peer DC’s Tempo, over the WAN, through that persistent file-storage queue (10,000 spans deep, max_elapsed_time = "0" — retry forever until the queue fills). WAN blips buffer instead of dropping.
• spanmetrics + servicegraph connectors — RED metrics and graph edges to the local Mimir. Only trace data crosses the WAN; the derived metrics are regenerated on each side.

The pipeline order is not aesthetic. memory_limiter first is the difference between shedding load gracefully and losing both replicas to OOM during a burst. source_ip before k8sattributes because attribution must survive even when enrichment has nothing to add.

No Auth on Ingest — On Purpose

Here’s the part that raises audit eyebrows, so let’s do it properly: the OTLP endpoint has no application-layer authentication. Any host on either DC’s network can send us spans. That’s a decision (ADR-045), not an oversight, and it stands on four legs:

1. NetworkPolicy allow-list. Ingress to 4317/4318 is limited to the two DC CIDRs. Everything else is dropped by the CNI before Alloy sees it.
2. TLS on the listener. Traces carry request paths, SQL fragments, user IDs — they’re encrypted in transit even inside the network. The cert is one corporate-issued cert with three SANs (the CNAME plus both per-DC names), synced to both clusters from Key Vault, so one rotation covers everything and the cert validates mid-failover.
3. Source-IP attribution. Every span gets source_ip stamped from the TCP peer — cryptographically tied to the TLS handshake, not spoofable by the sender. service.name is claimed; source_ip is verified. When garbage shows up, we know which host to visit.
4. Rate limits at the distributor. 20 MB/s with 30 MB burst. A runaway sender inflates a graph, not a bill.

The reasoning that carried the decision: every other ingest path on this platform — syslog on three ports, SNMP, streaming telemetry — is already unauthenticated and NetworkPolicy-controlled. An attacker who can reach the trace endpoint can already spoof syslog. A token check on traces moves the weakest link by one step without improving the overall posture. Meanwhile, the onboarding cost of no-auth is unbeatable, and adoption velocity was a design goal:

OTEL_EXPORTER_OTLP_ENDPOINT=https://traces.conveyor.internal:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

Two env vars and a restart. That’s the entire onboarding contract for an application team.

And there’s an escape hatch we wrote down in advance: Alloy can run TLS-with-client-auth and TLS-without side by side, so per-app mTLS is a non-breaking upgrade if traces ever start carrying regulated data classes or an audit finding forces the issue.

Cross-DC HA: Where We Broke Our Own Pattern

Logs in this platform are dual-written: every Alloy ships each log line to both DCs’ Loki, so either DC has a full copy at all times. If you’ve read the series, you’d expect traces to work the same way.

They don’t, and the reason lives in the client SDKs: OTel SDKs honor exactly one OTEL_EXPORTER_OTLP_ENDPOINT. Dual-writing from the producer side would mean asking every application team to configure multi-endpoint export — code changes, inconsistent support across runtimes, and the death of the two-env-vars onboarding story. The log path only looks producer-side; syslog devices also send to one place, and the receiving Alloy does the fan-out. Traces apply the same principle — collect once, fan out underneath — but the fan-out crosses DCs, and DNS steers the producers:

• traces.conveyor.internal is a CNAME with a 60-second TTL, pointing at the active DC’s VIP.
• Both DCs run identical alloy-traces deployments. The active one exports to both Tempos (local via LAN, peer via WAN + persistent queue). The standby is deployed, healthy, and idle.
• Failover is a human flipping the CNAME — deliberately. This platform has a standing rule against automated cross-DC failover; a human confirms the target is healthy, flips, waits out the TTL, and verifies with a TraceQL query.

The honest costs, straight from the decision record: the active DC’s collector is a hard single point for ingest until the CNAME flips; spans sent to a dead VIP during an outage are gone (only the surviving side’s queue persists); the standby can rot (mitigated by synthetic probes hitting both the CNAME and the standby VIP directly, every five minutes — and the probe module accepts any HTTP response, because OTLP answers GET with a 405, and a 405 proves the listener is alive); and the team now operates two different HA models for two data types. We pay for operational consistency where the client lets us, and accept divergence where it doesn’t.

One trap deserves its own paragraph because it’s rated the likeliest failure in the whole design: JVM DNS caching. Default Java behavior caches DNS resolutions forever (networkaddress.cache.ttl = -1). A Java app that resolved the CNAME once will keep sending traces to the dead DC through the entire failover and beyond, and nothing on our side can fix that. The onboarding doc makes a TTL of ≤60 mandatory for JVM apps, with a pre-prod smoke test that failover becomes visible within two minutes. If you run Java and take nothing else from this section, take that.

The Helm Values We Run

The wrapper-chart pattern, same as every other component — trimmed to the parts that earned their place:

# values-common.yaml
tempo-distributed:           # wrapper chart dependency key
  tempo:
    image:
      tag: 2.10.5            # pinned above the chart default

  global:
    dnsService: rke2-coredns-rke2-coredns   # RKE2, not kube-dns
    extraEnvFrom:
      - secretRef:
          name: tempo-s3-credentials
    # Do NOT put extraArgs here — the chart appends global extraArgs to the
    # memcached container too, and memcached greets -config.expand-env=true
    # with "Maximum connections must be greater than 0"

  ingester:
    replicas: 3
    persistence:
      enabled: true
      size: 10Gi
      storageClass: local-path       # WAL needs a PVC — emptyDir loses
    config:                          # in-flight traces on restart
      replication_factor: 2
      max_block_duration: 30m
    # The chart consumes ingester.affinity as a STRING passed through tpl —
    # a normal YAML mapping is silently dropped with a "destination is a
    # table" coalesce warning. Hence the block scalar:
    affinity: |
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app.kubernetes.io/component: ingester
                app.kubernetes.io/name: tempo    # don't collide with
            topologyKey: kubernetes.io/hostname  # mimir-ingester

  compactor:
    replicas: 1
    config:
      compaction:
        block_retention: 720h        # 30 days — retention lives HERE,
                                     # not in storage config
  metricsGenerator:
    replicas: 1
    config:
      registry:
        collection_interval: 15s
      processor:
        service_graphs:
          max_items: 10000
        span_metrics:
          enable_target_info: true

  overrides:
    defaults:
      metrics_generator:
        # Tempo 2.x: THIS list activates the processors. The config block
        # above only tunes processors that are already on this list.
        processors: [service-graphs, span-metrics, local-blocks]
      ingestion:
        rate_limit_bytes: 20000000   # 20 MB/s
        burst_size_bytes: 30000000

  minio:
    enabled: false                   # Nutanix Objects, not bundled MinIO
  gateway:
    enabled: false                   # no gateway hop — Alloy writes to the
                                     # distributor, Grafana queries the
                                     # query-frontend, directly

Three things in there have stories.

The affinity: | block scalar is the fix for the drain catch-22 from the retrospective — two of three tempo-ingesters had drifted onto the same node, and local-path PVC pinning plus a maxUnavailable: 1 PDB deadlocked an OS-patch drain for ten minutes before timing out. Required anti-affinity makes it structurally impossible. But the tempo-distributed chart takes ingester.affinity as a string through tpl, unlike Loki and Alertmanager which take normal mappings — override it with a mapping and the chart silently keeps its default. The failure mode of the fix was the fix not applying.

local-blocks in the processors list is there because Grafana’s Traces Drilldown feature runs TraceQL metrics queries ({kind=server} | rate() by (...)), which hit a query-range endpoint that calls the metrics-generator’s local-blocks processor. Without it, Drilldown fails with localblocks processor not found, which Grafana helpfully surfaces as “gRPC streaming call failed.” Two layers of indirection between the symptom and the missing config line.

The processors list itself is the Tempo 2.x activation gotcha: configuring a processor under metricsGenerator.config does nothing unless the processor’s name also appears in overrides.defaults.metrics_generator.processors. Config tunes; the list enables.

Wiring Grafana: Traces to Everything

The payoff for running the whole stack in one place is correlation, and it’s all datasource config:

• Traces → logs: tracesToLogsV2 maps service.name, namespace, and pod to Loki labels, with a ±5-minute window around the span and filterByTraceID on. Click a span, land in its logs.
• Logs → traces: a Loki derived field regex-matches trace IDs in log lines (trace_id, traceID, or traceid, 64- or 128-bit hex) and renders a “View trace” link.
• Metrics → traces: exemplars. The metrics-generator sends them with the span metrics, Mimir stores them (only if you set max_global_exemplars_per_user — exemplar storage is off by default and Mimir drops them silently otherwise), and latency panels grow clickable dots that jump straight to the trace behind the outlier.
• The service graph: rendered from the traces_service_graph_* series in Mimir. Node graph on, whole-fleet topology from span data nobody hand-drew.

One wiring gotcha cost an afternoon: Grafana’s Tempo datasource has a streaming toggle, and streaming runs over gRPC. The datasource reuses the HTTP URL — port 3200 — for the gRPC connection, and the handshake fails with http2: frame too large, note that the frame header looked like an HTTP/1.1 header, which is HTTP/2’s poetic way of saying you pointed gRPC at an HTTP port. We disabled streaming; search and TraceQL metrics work fine over HTTP.

Dogfooding War Stories

The platform traces itself — Alloy, Mimir, Grafana, and ArgoCD all ship their own spans to Tempo at a 10% sample (application ingest is unsampled at launch; tail sampling is a future problem gated on span volume). Dogfooding found real bugs, which is the point:

Every Alloy looked identical. Alloy’s tracing {} block hardcodes service.name=alloy and ignores OTEL_SERVICE_NAME entirely. Three very different Alloy deployments — DaemonSet, network receiver, traces collector — were indistinguishable in Tempo. The fix routes each config’s self-traces through a transform processor that overwrites service.name from the env var before export. Now the span metrics show three services, as they should.

ArgoCD’s traces died twice. First in config: setting the OTLP endpoint via raw env: blocks collided with the chart’s own valueFrom entries for the same variable names, and Kubernetes rejected the duplicate env keys — ArgoCD’s own sync stuck on a ComparisonError about itself. (The chart has a proper configs.params path for OTLP; use it.) Then in the network: the observability namespace’s ingress policy allowed 4317 from the collector namespace and the DC CIDRs, but not from argocd. The spans dialed, timed out, and went nowhere until the policy grew a rule.

Loki still doesn’t trace itself — the dskit schema-URL conflict from the retrospective is still unfixed in the Loki release we run, so Loki is deliberately excluded from the tracing dashboards while everything else dogfoods. The deferral comment in the values file explains the mechanism so the next person doesn’t spend the same afternoon we did.

And the pettiest one: Tempo phones home usage stats by default, our egress policy blocks it, and seven components retrying politely produced ~44,000 “failed to send usage report” log lines per day — in the log system we run. reportingEnabled: false.

Troubleshooting at the Door

Wrapping Up

The T in LGTM, finally paid in full. What’s worth keeping:

• Tempo won on operational sameness, not features. Same S3-backed shape as Loki and Mimir, same wrapper chart, same ESO credentials, same on-call mental model. The best new component is the one that isn’t new.
• Traces get their own collector. Bursty OTLP and steady syslog don’t share a pod; the ~200 lines of saved YAML aren’t worth the coupling.
• No-auth ingest can be a real decision — with a network allow-list, TLS, unforgeable source attribution, and rate limits doing the work a token wouldn’t do better. Write down the escape hatch before you need it.
• HA followed the client’s constraints, not our preferences. One endpoint per SDK means DNS steering and collector-tier fan-out — a deliberate, documented divergence from the log dual-write.
• The queue is only as durable as its volume. Our cross-DC buffer survived restarts and quietly died on upgrades until it moved from emptyDir to a per-replica PVC. If a buffer protects something important, check what happens on pod replacement, not just restart.
• 30-day retention on the compactor, one bucket per DC, and exemplars switched on in Mimir — the three settings you’d otherwise find the hard way.

It’s good to be back.

Happy automating!