From 57b143c075bb1b7a3f4e606a0e533c22613d9236 Mon Sep 17 00:00:00 2001 From: Arkadiusz Komarzewski Date: Thu, 16 Jul 2026 13:49:29 +0100 Subject: [PATCH 1/6] Fix outdated live ping tables description in pipeline overview The Decoded topic is read by the Kubernetes Live Sink (part of the ingestion-sink service on GKE), not a Dataflow job. Update the prose and relabel the diagram node accordingly. Also drop the inaccurate 'guaranteed to contain a few days of history' caveat: live tables are for low-latency access to recent data, not a supported analysis interface. Point readers to the stable tables instead. --- src/concepts/pipeline/gcp_data_pipeline.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/src/concepts/pipeline/gcp_data_pipeline.md b/src/concepts/pipeline/gcp_data_pipeline.md index f3a0e88c1..a0aedcc6b 100644 --- a/src/concepts/pipeline/gcp_data_pipeline.md +++ b/src/concepts/pipeline/gcp_data_pipeline.md @@ -22,7 +22,7 @@ p1 --> d2(fa:fa-exchange-alt Decoder) d2 -->|success| p2(fa:fa-stream Decoded Topic) d2 -.->|fail| p3(fa:fa-stream Errors Topic) p3 --> d4(fa:fa-exchange-alt Errors Sink) -p2 --> d3(fa:fa-exchange-alt BigQuery Sink) +p2 --> d3(fa:fa-exchange-alt Live Sink) d3 --> b2(fa:fa-database Live Tables BQ) d4 --> b3(fa:fa-database Error Tables BQ) @@ -129,11 +129,13 @@ and the [schema generator]. The result are tables that contains SQL-friendly field names for all known measures, as implemented in the [probe scraper]. -A [Dataflow] job reads from the Decoded topic and writes out to +A Kubernetes `Live Sink` job (part of the [`ingestion-sink`][ingestion-sink] +service) reads from the Decoded topic and writes out to **[live ping tables][table layout]**. These tables are updated frequently, and typically reflect data within a few -minutes of it being ingested. They are optimized for accessing recent data, -but are only guaranteed to contain a few days of history. +minutes of it being ingested. They are intended for low-latency access to +recent data and are not the supported interface for analysis; for that, use the +**[historical ping tables][table layout]** (stable tables) described below. Historical raw ping data is stored in **[historical ping tables][table layout]**, also known as **stable tables**. @@ -234,4 +236,5 @@ build and work with aggregate datasets rather than individual-level data. [norm]: ../channels/channel_normalization.md [bqe]: https://github.com/mozilla/bigquery-etl [gcp-ingestion]: https://github.com/mozilla/gcp-ingestion +[ingestion-sink]: https://github.com/mozilla/gcp-ingestion/tree/main/ingestion-sink [looker]: https://mozilla.cloud.looker.com From 466a5ea4cc29a598878da324c3dab77a2239ca59 Mon Sep 17 00:00:00 2001 From: Arkadiusz Komarzewski Date: Thu, 16 Jul 2026 14:23:26 +0100 Subject: [PATCH 2/6] Fix outdated Landfill sink description in pipeline overview The Raw topic is read by the Kubernetes Raw Sink (part of the ingestion-sink service), not a Dataflow job, and writes to the payload_bytes_raw dataset. The 'Landfill' name is historical. Update the prose and relabel the diagram nodes; keep a parenthetical for the old name. Match gcp-ingestion's own terminology ('Kubernetes Raw Sink job') rather than the stale 'custom Java application' phrasing from a 2020 transition note. --- src/concepts/pipeline/gcp_data_pipeline.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/src/concepts/pipeline/gcp_data_pipeline.md b/src/concepts/pipeline/gcp_data_pipeline.md index a0aedcc6b..9522ff6cc 100644 --- a/src/concepts/pipeline/gcp_data_pipeline.md +++ b/src/concepts/pipeline/gcp_data_pipeline.md @@ -16,8 +16,8 @@ graph TD f1(fa:fa-firefox Firefox) -->|HTTP Post| d0(fa:fa-filter Ingestion Edge) d0 --> p1(fa:fa-stream Raw Topic) -p1 --> d1(fa:fa-exchange-alt Landfill Sink) -d1 --> b1(fa:fa-database Landfill BQ) +p1 --> d1(fa:fa-exchange-alt Raw Sink) +d1 --> b1(fa:fa-database Raw BQ) p1 --> d2(fa:fa-exchange-alt Decoder) d2 -->|success| p2(fa:fa-stream Decoded Topic) d2 -.->|fail| p3(fa:fa-stream Errors Topic) @@ -76,8 +76,10 @@ message body of optionally-gzipped JSON. These messages are forwarded to a PubSub message queue with minimal processing, and made available in a **Raw** topic. -A [Dataflow] job reads this topic and writes the raw messages to a BigQuery **Landfill** sink. -This Landfill data is not used for analysis, but is stored in its raw form for +A Kubernetes `Raw Sink` job (part of the [`ingestion-sink`][ingestion-sink] +service) reads this topic and writes the raw messages to a BigQuery +`payload_bytes_raw` sink (historically called _Landfill_). +This raw data is not used for analysis, but is stored in its raw form for recovery and backfill purposes. If there is a processing error or data-loss downstream in the pipeline, this is an important fail-safe. From 8d653e61d90c051a0d1db8d7c73eb5f700959d1b Mon Sep 17 00:00:00 2001 From: Arkadiusz Komarzewski Date: Mon, 20 Jul 2026 15:34:16 +0200 Subject: [PATCH 3/6] Fix outdated edge server behavior in HTTP edge spec Verified against gcp-ingestion ingestion-edge (create_app.py, publish.py, util.py, dockerflow.py): - GET /status does not exist. Health is served via Dockerflow endpoints (/__lbheartbeat__, /__heartbeat__, /__version__). Rewrite the GET request and response-code sections accordingly. - The ?v=4 query parameter is not validated or rejected at the edge; the edge routes on path only and forwards the body opaquely. Describe the actual behavior. - POST/PUT response codes: add 431 (attribute exceeds PubSub's 1024-byte limit) and 507 (local retry queue full), which the edge actually returns; drop 411 and 414, which are not enforced. - Define the previously dangling [decoder] and [schemas repository] reference-style links. --- src/concepts/pipeline/http_edge_spec.md | 32 +++++++++++++++++-------- 1 file changed, 22 insertions(+), 10 deletions(-) diff --git a/src/concepts/pipeline/http_edge_spec.md b/src/concepts/pipeline/http_edge_spec.md index 3ea511fde..a37441f37 100644 --- a/src/concepts/pipeline/http_edge_spec.md +++ b/src/concepts/pipeline/http_edge_spec.md @@ -23,14 +23,23 @@ documents described by a [JSONSchema]. ### GET Request -Accept GET on `/status`, returning `OK` if all is well. This can be used to -check the health of web servers. +The edge server does not serve application data over GET. It exposes +[Dockerflow] health-check endpoints: + +- `/__lbheartbeat__` - returns 200 unconditionally; used by the load balancer + and Kubernetes probes to confirm the server is up. +- `/__heartbeat__` - returns 200 when internal checks pass (enough free disk + for the local queue, and the queue is accessible) and 500 otherwise. +- `/__version__` - returns build and version information. + +[dockerflow]: https://github.com/mozilla-services/Dockerflow ### GET Response codes -- _200_ - OK. `/status` and all's well -- _404_ - Any GET other than `/status` -- _500_ - All is not well +- _200_ - a health check passed (`/__lbheartbeat__`, or `/__heartbeat__` when + all checks pass), or version info was returned (`/__version__`) +- _404_ - any other GET path +- _500_ - a `/__heartbeat__` check failed ### POST/PUT Request @@ -67,8 +76,9 @@ A specific example: Here the `namespace` is fixed as "telemetry", and there is no `docVersion` in the URL. This means that incoming JSON documents must be parsed to determine the schema version to apply for validation. This logic is part of the downstream [decoder] job. -Also note the required query parameter suffix `?v=4`. -Documents sent under `/submit/telemetry` without `v=4` will be rejected at the edge. +Also note the query parameter suffix `?v=4`, a historical convention sent by +Firefox Desktop. The edge server routes on the URL path only and forwards the +request body opaquely; the query string is preserved as request metadata. ### POST/PUT Response codes @@ -76,10 +86,10 @@ Documents sent under `/submit/telemetry` without `v=4` will be rejected at the e - _400_ - Bad request, for example an un-encoded space in the URL. - _404_ - not found - POST/PUT to an unknown namespace - _405_ - wrong request type (anything other than POST/PUT) -- _411_ - missing content-length header -- _413_ - request body too large (Note that if we have badly-behaved clients that retry on `4XX`, we may opt to send back 202 on body/path too long). -- _414_ - request path too long (See above) +- _413_ - request body too large +- _431_ - a request header (or other captured metadata attribute) exceeds PubSub's 1024-byte attribute limit - _500_ - internal error +- _507_ - insufficient storage - the edge's local retry queue is full (for example during an extended PubSub outage) ### Supported HTTP Headers @@ -132,3 +142,5 @@ is restored and messages can be flushed to the queue. Based on [past outages], this is typically a few hours or less. [past outages]: https://status.cloud.google.com/incident/cloud-pubsub +[schemas repository]: https://github.com/mozilla-services/mozilla-pipeline-schemas +[decoder]: gcp_data_pipeline.md#decoding From 1964330a5aac3237e542fbdff9a658834a3afd6a Mon Sep 17 00:00:00 2001 From: Arkadiusz Komarzewski Date: Mon, 20 Jul 2026 15:36:37 +0200 Subject: [PATCH 4/6] Fix outdated transform location in schemas doc Verified against gcp-ingestion: ingestion-beam's PubsubMessageToTableRow is now a thin wrapper that adds Beam metrics and converts to a BigQuery TableRow; the transformation logic (map types, nested arrays, tuples, coercion, additional_properties) moved to ingestion-core's PubsubMessageToObjectNode. Point the doc there and fix the 'Thee'/'telemtry' typos in the paragraph. --- src/concepts/pipeline/schemas.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/src/concepts/pipeline/schemas.md b/src/concepts/pipeline/schemas.md index c1513104f..176e4acbd 100644 --- a/src/concepts/pipeline/schemas.md +++ b/src/concepts/pipeline/schemas.md @@ -236,10 +236,11 @@ LIMIT 1 #### Data Structure Normalization -Thee decoder is also responsible for transforming the data to -accommodate BigQuery limitations in data representation. All transformations are defined in -[`ingestion-beam` under -`com.mozilla.telemtry.transforms.PubsubMessageToTableRow`](https://github.com/mozilla/gcp-ingestion/blob/main/ingestion-beam/src/main/java/com/mozilla/telemetry/transforms/PubsubMessageToTableRow.java). +The decoder is also responsible for transforming the data to +accommodate BigQuery limitations in data representation. Most transformations are implemented in +[`ingestion-core` under +`com.mozilla.telemetry.ingestion.core.transform.PubsubMessageToObjectNode`](https://github.com/mozilla/gcp-ingestion/blob/main/ingestion-core/src/main/java/com/mozilla/telemetry/ingestion/core/transform/PubsubMessageToObjectNode.java); +`ingestion-beam`'s `PubsubMessageToTableRow` is a thin wrapper that adds Beam metrics and converts the result to a BigQuery `TableRow`. The following transformations are currently applied: From 3aee1e53a6a20e00dd0bfd311c426d92b026e9e6 Mon Sep 17 00:00:00 2001 From: Arkadiusz Komarzewski Date: Mon, 20 Jul 2026 15:52:11 +0200 Subject: [PATCH 5/6] Fix spelling --- .spelling | 1 + 1 file changed, 1 insertion(+) diff --git a/.spelling b/.spelling index b16501a71..015458855 100644 --- a/.spelling +++ b/.spelling @@ -117,6 +117,7 @@ DNS DNSSEC docs.telemetry.mozilla.org Dockerfile +Dockerflow dropdown DSRE DTMO From 90d9b856f1e5ccbe5417e424acc46d197bbbe1e5 Mon Sep 17 00:00:00 2001 From: Arkadiusz Komarzewski Date: Mon, 20 Jul 2026 15:57:09 +0200 Subject: [PATCH 6/6] Fix spelling --- src/concepts/pipeline/http_edge_spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/concepts/pipeline/http_edge_spec.md b/src/concepts/pipeline/http_edge_spec.md index a37441f37..cf8673691 100644 --- a/src/concepts/pipeline/http_edge_spec.md +++ b/src/concepts/pipeline/http_edge_spec.md @@ -87,7 +87,7 @@ request body opaquely; the query string is preserved as request metadata. - _404_ - not found - POST/PUT to an unknown namespace - _405_ - wrong request type (anything other than POST/PUT) - _413_ - request body too large -- _431_ - a request header (or other captured metadata attribute) exceeds PubSub's 1024-byte attribute limit +- _431_ - a request header (or other captured metadata attribute) exceeds PubSub's 1024 byte attribute limit - _500_ - internal error - _507_ - insufficient storage - the edge's local retry queue is full (for example during an extended PubSub outage)