[TakGN]

Important

Draft until #30's rollout is done. Do not merge this until needle v1.1.0 is running and settled on all three boxes. Once this is on main, any routing.yml run from main applies it, so a box still on v1.0.0 would take the new image and this config in one restart. That would undo the point of #30 being digest-only. It also needs v1.1.0: NEEDLE_TRACING_TRUST_TRACEPARENT doesn't exist before it.

What changes

Part of ipni/infra#163. needle starts sending traces, which join the ones Envoy already produces.

k8s/needle/kustomization.yaml adds two env vars to all four instances:

Variable	Why
OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318	Exports to the alloy-traces collector on the box's loopback, over OTLP/HTTP. This is the endpoint ipni/infra's collector config and monitoring docs name for services.
NEEDLE_TRACING_TRUST_TRACEPARENT=true	Joins Envoy's trace and follows its 5% sampling decision, so needle samples the same requests.

NEEDLE_SAMPLING_FRACTION stays at its default of 0, so a request with no Traceparent, such as a tool run on the box, isn't traced.

k8s/route-origin/envoy.yaml.j2 gets a comment-only fix: it still said v1.0.0 drops the traceparent. It is a separate commit, because of the Envoy restart described below.

Is trusting Traceparent safe?

needle's docs allow it only when needle is reachable solely through a proxy that removes the client's headers and sends its own. Checked in this repo:

• The needle API listens on loopback only.
• envoy.yaml.j2 has one listener and one connection manager, and its early header mutation removes traceparent and tracestate before Envoy reads them.
• Nothing else here sends routing traffic to needle, apart from scripts/routing-rate-test.sh, which an operator runs on the box and which sends no Traceparent.

What this costs

• Every needle instance restarts. The env change alters the ConfigMap hash on all four Deployments, so this is a full warmed rollout again: hours per box.
• Envoy restarts too. The comment is rendered into envoy.yaml, so even this edit changes the envoy-config hash. The Deployment uses Recreate, so each box's site has a brief outage when its Envoy is replaced.
• More spans and possibly more stored traces. At Envoy's ~7 traces/s per box, each trace gains a request span, one span per router and any /dnsaddr lookups. The collector keeps any trace containing an error, and needle marks a router span as ERROR when that router returned error results. If that is common, far more than one in ten traces could reach Tempo. I have no measurement of this, so it is the main thing to watch.

Span names are low cardinality (endpoint, router operation and router name), so the span metrics stay small. If the collector is down, spans are dropped.

Merging does not deploy this

k8s/needle/ and k8s/route-origin/envoy.yaml.j2 match none of the path patterns in the changes job of .github/workflows/ci.yml, so neither this PR nor its merge triggers a CI dry run or apply. Only the checks run. routing.yml doesn't run from CI.

Rollout plan, for a person to run

After #30's rollout is finished and this is merged, from an up-to-date main:

ansible-playbook routing.yml --check --diff -l sing-1   # dry run first
ansible-playbook routing.yml -l sing-1                  # one box, then check it as below
ansible-playbook routing.yml                            # the remaining boxes, one at a time, warmed

• Never use -e production_rollout=false. It rolls all three boxes at once, which is an outage.
• A warmed rollout takes hours per box.
• To put off the Envoy restart, use --tags needle on those commands. The comment change then waits for the next routing.yml run that includes the origin role, and restarts Envoy at that point.

After sing-1, before going on:

• Tempo has traces for {resource.service.name="needle", resource.cluster="sing1"}, joined to route-origin spans in the same trace.
• traces_spanmetrics_calls_total{service_name="needle", cluster="sing1"} is non-zero, and its rate looks like about 5% of requests rather than all of them.
• TraceSpansRefused, TracesNotDelivered and TempoDiscardingSpans stay quiet, and Tempo's storage growth on lith-1 looks reasonable.
• Once warmed, needle's latency matches the other boxes.

To roll back, revert this PR and run the same steps, which costs another warm-up and another Envoy restart.

Checks run locally

All pass:

• yamllint . and ansible-lint (Passed: 0 failure(s), 0 warning(s), profile production)
• ansible-playbook --syntax-check on all seven playbooks
• scripts/render-manifests.yml, plus kubectl kustomize on every overlay
• The other CI checks: the needle instance count at 1, 2 and 4, the cert-manager checksum, libp2p_identity.py unit tests, dns.txt and dns-libp2p.txt against their generator, and shellcheck

Render diff against main:

• needle: the two new variables in the ConfigMap, and its new hash on the ConfigMap and the four Deployments.
• route-origin: the comment lines and the envoy-config hash.
• All other overlays are identical.

Differences from CI: these ran on macOS with Python 3.14, and ansible-lint used ansible-core 2.22, while CI uses Ubuntu with Python 3.12. The syntax checks and renders used ansible-core 2.19.4. The cert-manager checksum used shasum -a 256.

Not verified

• Nothing was read from the boxes. A read-only query was denied in the session that wrote this, so the state of Pin ghcr.io/ipni/needle:v1.1.0 #30's rollout, the running collector and live span volume are all unconfirmed. The collector details come from ipni/infra's config.
• Envoy's header behaviour is from needle's docs, not observed. They say the proxy sends a Traceparent on every request. If Envoy omitted it on unsampled requests, those still wouldn't be traced, because the fraction is 0. The first check after sing-1 covers this.
• No routing.yml dry run, which stays with the person doing the rollout.

🤖 Generated with Claude Code