diff --git a/.spelling b/.spelling index b16501a71..015458855 100644 --- a/.spelling +++ b/.spelling @@ -117,6 +117,7 @@ DNS DNSSEC docs.telemetry.mozilla.org Dockerfile +Dockerflow dropdown DSRE DTMO diff --git a/src/concepts/pipeline/gcp_data_pipeline.md b/src/concepts/pipeline/gcp_data_pipeline.md index f3a0e88c1..9522ff6cc 100644 --- a/src/concepts/pipeline/gcp_data_pipeline.md +++ b/src/concepts/pipeline/gcp_data_pipeline.md @@ -16,13 +16,13 @@ graph TD f1(fa:fa-firefox Firefox) -->|HTTP Post| d0(fa:fa-filter Ingestion Edge) d0 --> p1(fa:fa-stream Raw Topic) -p1 --> d1(fa:fa-exchange-alt Landfill Sink) -d1 --> b1(fa:fa-database Landfill BQ) +p1 --> d1(fa:fa-exchange-alt Raw Sink) +d1 --> b1(fa:fa-database Raw BQ) p1 --> d2(fa:fa-exchange-alt Decoder) d2 -->|success| p2(fa:fa-stream Decoded Topic) d2 -.->|fail| p3(fa:fa-stream Errors Topic) p3 --> d4(fa:fa-exchange-alt Errors Sink) -p2 --> d3(fa:fa-exchange-alt BigQuery Sink) +p2 --> d3(fa:fa-exchange-alt Live Sink) d3 --> b2(fa:fa-database Live Tables BQ) d4 --> b3(fa:fa-database Error Tables BQ) @@ -76,8 +76,10 @@ message body of optionally-gzipped JSON. These messages are forwarded to a PubSub message queue with minimal processing, and made available in a **Raw** topic. -A [Dataflow] job reads this topic and writes the raw messages to a BigQuery **Landfill** sink. -This Landfill data is not used for analysis, but is stored in its raw form for +A Kubernetes `Raw Sink` job (part of the [`ingestion-sink`][ingestion-sink] +service) reads this topic and writes the raw messages to a BigQuery +`payload_bytes_raw` sink (historically called _Landfill_). +This raw data is not used for analysis, but is stored in its raw form for recovery and backfill purposes. If there is a processing error or data-loss downstream in the pipeline, this is an important fail-safe. @@ -129,11 +131,13 @@ and the [schema generator]. The result are tables that contains SQL-friendly field names for all known measures, as implemented in the [probe scraper]. -A [Dataflow] job reads from the Decoded topic and writes out to +A Kubernetes `Live Sink` job (part of the [`ingestion-sink`][ingestion-sink] +service) reads from the Decoded topic and writes out to **[live ping tables][table layout]**. These tables are updated frequently, and typically reflect data within a few -minutes of it being ingested. They are optimized for accessing recent data, -but are only guaranteed to contain a few days of history. +minutes of it being ingested. They are intended for low-latency access to +recent data and are not the supported interface for analysis; for that, use the +**[historical ping tables][table layout]** (stable tables) described below. Historical raw ping data is stored in **[historical ping tables][table layout]**, also known as **stable tables**. @@ -234,4 +238,5 @@ build and work with aggregate datasets rather than individual-level data. [norm]: ../channels/channel_normalization.md [bqe]: https://github.com/mozilla/bigquery-etl [gcp-ingestion]: https://github.com/mozilla/gcp-ingestion +[ingestion-sink]: https://github.com/mozilla/gcp-ingestion/tree/main/ingestion-sink [looker]: https://mozilla.cloud.looker.com diff --git a/src/concepts/pipeline/http_edge_spec.md b/src/concepts/pipeline/http_edge_spec.md index 3ea511fde..cf8673691 100644 --- a/src/concepts/pipeline/http_edge_spec.md +++ b/src/concepts/pipeline/http_edge_spec.md @@ -23,14 +23,23 @@ documents described by a [JSONSchema]. ### GET Request -Accept GET on `/status`, returning `OK` if all is well. This can be used to -check the health of web servers. +The edge server does not serve application data over GET. It exposes +[Dockerflow] health-check endpoints: + +- `/__lbheartbeat__` - returns 200 unconditionally; used by the load balancer + and Kubernetes probes to confirm the server is up. +- `/__heartbeat__` - returns 200 when internal checks pass (enough free disk + for the local queue, and the queue is accessible) and 500 otherwise. +- `/__version__` - returns build and version information. + +[dockerflow]: https://github.com/mozilla-services/Dockerflow ### GET Response codes -- _200_ - OK. `/status` and all's well -- _404_ - Any GET other than `/status` -- _500_ - All is not well +- _200_ - a health check passed (`/__lbheartbeat__`, or `/__heartbeat__` when + all checks pass), or version info was returned (`/__version__`) +- _404_ - any other GET path +- _500_ - a `/__heartbeat__` check failed ### POST/PUT Request @@ -67,8 +76,9 @@ A specific example: Here the `namespace` is fixed as "telemetry", and there is no `docVersion` in the URL. This means that incoming JSON documents must be parsed to determine the schema version to apply for validation. This logic is part of the downstream [decoder] job. -Also note the required query parameter suffix `?v=4`. -Documents sent under `/submit/telemetry` without `v=4` will be rejected at the edge. +Also note the query parameter suffix `?v=4`, a historical convention sent by +Firefox Desktop. The edge server routes on the URL path only and forwards the +request body opaquely; the query string is preserved as request metadata. ### POST/PUT Response codes @@ -76,10 +86,10 @@ Documents sent under `/submit/telemetry` without `v=4` will be rejected at the e - _400_ - Bad request, for example an un-encoded space in the URL. - _404_ - not found - POST/PUT to an unknown namespace - _405_ - wrong request type (anything other than POST/PUT) -- _411_ - missing content-length header -- _413_ - request body too large (Note that if we have badly-behaved clients that retry on `4XX`, we may opt to send back 202 on body/path too long). -- _414_ - request path too long (See above) +- _413_ - request body too large +- _431_ - a request header (or other captured metadata attribute) exceeds PubSub's 1024 byte attribute limit - _500_ - internal error +- _507_ - insufficient storage - the edge's local retry queue is full (for example during an extended PubSub outage) ### Supported HTTP Headers @@ -132,3 +142,5 @@ is restored and messages can be flushed to the queue. Based on [past outages], this is typically a few hours or less. [past outages]: https://status.cloud.google.com/incident/cloud-pubsub +[schemas repository]: https://github.com/mozilla-services/mozilla-pipeline-schemas +[decoder]: gcp_data_pipeline.md#decoding diff --git a/src/concepts/pipeline/schemas.md b/src/concepts/pipeline/schemas.md index c1513104f..176e4acbd 100644 --- a/src/concepts/pipeline/schemas.md +++ b/src/concepts/pipeline/schemas.md @@ -236,10 +236,11 @@ LIMIT 1 #### Data Structure Normalization -Thee decoder is also responsible for transforming the data to -accommodate BigQuery limitations in data representation. All transformations are defined in -[`ingestion-beam` under -`com.mozilla.telemtry.transforms.PubsubMessageToTableRow`](https://github.com/mozilla/gcp-ingestion/blob/main/ingestion-beam/src/main/java/com/mozilla/telemetry/transforms/PubsubMessageToTableRow.java). +The decoder is also responsible for transforming the data to +accommodate BigQuery limitations in data representation. Most transformations are implemented in +[`ingestion-core` under +`com.mozilla.telemetry.ingestion.core.transform.PubsubMessageToObjectNode`](https://github.com/mozilla/gcp-ingestion/blob/main/ingestion-core/src/main/java/com/mozilla/telemetry/ingestion/core/transform/PubsubMessageToObjectNode.java); +`ingestion-beam`'s `PubsubMessageToTableRow` is a thin wrapper that adds Beam metrics and converts the result to a BigQuery `TableRow`. The following transformations are currently applied: