Ja/jr/auto instrumentation in server

[aug24]

What does this change?

This PR is half of Jorge's excellent #16453

It implements request tracing in tag-page-rendering in ECS using Open Telemetry (OTEL), which has been the official AWS recommendation since February 2026.

At a high level this PR

• Implements tracing in the server code using the OTEL SDK for Node and sends the metric requests to the sidecar.

If the sidecar is unavailable (it's marked as essential: false) then we log the failure.

Collecting traces

The OTEL library is structured in a way that is somewhat counter intuitive. It's based around two packages:@opentelemetry/sdk-node and @opentelemetry/api

@opentelemetry/api works like you'd expect. We can import the module anywhere in our application code, and we get a rich interface to add traces and other telemetry data. However - by default, all the functions are just no-op placeholders.

To have its methods perform useful operations, we have to initialise the SDK. This is done with @opentelemetry/sdk-node in a separate module that has to be passed as a --require argument tonode and thus runs before the main application code.

From the package's README

Before any other module in your application is loaded, you must initialize the SDK. If you fail to initialize the SDK or initialize it too late, no-op implementations will be provided to any library which acquires a tracer or meter from the API.

The initialiser code lives in instrumentation.ts, which is mostly boilerplate from the SDK docs, plus a bit of configuration to drop traces for healthchecks and for outgoing requests (the app makes requests to cloudwatch), as they tend to be quite high volume.

To gain some insight into server.prod.ts internals, this PR adds a custom trace to the express layer of the application that grabs two spans

• One covering the express.json middleware, which is responsible for loading the incoming JSON from the network and parsing it into an object
• Another covering the request handler that performs the server-side rendering to html

The end result is the following trace for every POST to tag-page rendering

We can enrich this trace with more data as needed (for example, we may want to break down the request handler into multiple child spans to expose its inner workings)

How it tracing enabled in ECS, but not on EC2?

The separation of the API and its initialisation is key to understand why it's possible to have traces in ECS, but not on EC2.

In EC2 the application starts in user data by running node app/server.js. server.js has the new tracing code, but because the SKD is not initialised it will do nothing.

In ECS, the application starts with node --require /app/instrumentation.js app/server.js, so the SDK is initialised and traces will be collected.

A note on auto instrumentation

One the great features of OTEL enabled by this split architecture is auto-instrumentation - automatically collecting tracing for our application without having to manually write code ourselves. This is only supported for about 40 node libraries (including express), where the initialisation code monkey patches the import statement of a library to add rich telemetry data to imported methods.

Because this project uses webpack however, the imports statements are removed from the final bundle, so auto-instrumentation mostly doesn't work.

It is possible to get it to work by using a webpack plugin developed by a volunteer which performs the monkey patching at build time, but I've opted to write simple spans in server.prod.ts rather than introduce this dependency.

How has this change been tested?

Deployed to CODE and POSTed a JSON to both the ec2 and ecs target groups from the SOCKS proxy instance. Only ECS traces show up in x-ray.

[github-actions]

Hello 👋! When you're ready to run Chromatic, please apply the run_chromatic label to this PR.

You will need to reapply the label each time you want to run Chromatic.

Click here to see the Chromatic project.

[github-actions]

🚀 Image pushed to AWS ECR

Image digest: sha256:fbd6570fe2c0bf05418c3c9e1a7ef2543b342e97ce83b7373ddb9e1b67c727b6

🐛 Run the image locally

The following can be used to run the image locally:

# Refer to image using the immutable digest. Find alternatives below.
IMAGE_IDENTIFIER="@sha256:fbd6570fe2c0bf05418c3c9e1a7ef2543b342e97ce83b7373ddb9e1b67c727b6"

# Refer to image using branch tag
# IMAGE_IDENTIFIER=":branch-ja-jr-auto-instrumentation-in-server"

# Refer to image using build tag
# IMAGE_IDENTIFIER=":build-30323"

# Refer to image via the GitHub commit SHA tag
# IMAGE_IDENTIFIER=":sha-3129ba35124184862f1a06c8593ee7f0947b80bf"

# Set environment variables for the AWS CLI
AWS_PROFILE="<A_PROFILE_FROM_JANUS>"
AWS_DEFAULT_REGION="eu-west-1"

IMAGE_ACCOUNT_ID=$(aws ssm get-parameter --name /organisation/accounts/deployTools --query "Parameter.Value" --output text)
REGISTRY="${IMAGE_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com"
IMAGE="${REGISTRY}/guardian/dotcom-rendering${IMAGE_IDENTIFIER}"

# Login to AWS ECR https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html
aws ecr get-login-password | docker login --username AWS --password-stdin $REGISTRY

# Pull the image
docker pull $IMAGE

# Run the image. You'll likely need to set additional flags. See https://docs.docker.com/reference/cli/docker/container/run.
docker run $IMAGE

---

From guardian/actions-publish-image.

[github-actions]

Deploy build 2753 of dotcom:ab-testing to CODE

All deployment options

• Deploy build 2753 of dotcom:ab-testing to CODE
• Deploy parts of build 2753 to CODE by previewing it first
• What's on CODE right now?

---

From guardian/actions-riff-raff.

[github-actions]

Deploy build 30323 of dotcom:rendering-all to CODE

All deployment options

• Deploy build 30323 of dotcom:rendering-all to CODE
• Deploy parts of build 30323 to CODE by previewing it first
• What's on CODE right now?

---

From guardian/actions-riff-raff.