diff --git a/src/cookbooks/bigquery/querying.md b/src/cookbooks/bigquery/querying.md index f11dc2100..ec36f8b4e 100644 --- a/src/cookbooks/bigquery/querying.md +++ b/src/cookbooks/bigquery/querying.md @@ -60,20 +60,20 @@ Under the single `moz-fx-data-shared-prod` project, each document namespace (corresponding to folders underneath the [schemas directory of `mozilla-pipeline-schemas`](https://github.com/mozilla-services/mozilla-pipeline-schemas/tree/master/schemas)) has four BigQuery datasets provisioned with the following properties: - _Live datasets_ (`telemetry_live`, `activity_stream_live`, etc.) contain live ping tables (see definitions of table types in the next paragraph) -- _Stable datasets_ (`telemetry_stable`, `activity_stream_stable`, etc.) contain historical ping tables +- _Stable datasets_ (`telemetry_stable`, `activity_stream_stable`, etc.) contain stable ping tables - _Derived datasets_ (`telemetry_derived`, `activity_stream_derived`, etc.) contain derived tables, primarily populated via nightly queries defined in [BigQuery ETL](https://github.com/mozilla/bigquery-etl) and managed by Airflow - _User-facing (unsuffixed) datasets_ (`telemetry`, `activity_stream`, etc.) contain user-facing views on top of the tables in the corresponding stable and derived datasets. The table and view types referenced above are defined as follows: -- _Live ping tables_ are the final destination for the [telemetry ingestion pipeline](https://mozilla.github.io/gcp-ingestion/). Dataflow jobs process incoming ping payloads from clients, batch them together by document type, and load the results to these tables approximately every five minutes, although a few document types are opted in to a more expensive streaming path that makes records available in BigQuery within seconds of ingestion. These tables are partitioned by date according to `submission_timestamp` and are also clustered on that same field, so it is possible to make efficient queries over short windows of recent data such as the last hour. They have a rolling expiration period of 30 days, but that window may be shortened in the future. Analyses should only use these tables if they need results for the current (partial) day. -- _Historical ping tables_ have exactly the same schema as their corresponding live ping tables, but they are populated only once per day (`12:00:00am` to `11:59:59pm` UTC) via an Airflow job and have a 25 month retention period. These tables are superior to the live ping tables for historical analysis because they never contain partial days, they have additional deduplication applied, and they are clustered on `sample_id`, allowing efficient queries on a 1% sample of clients. It is guaranteed that `document_id` is distinct within each day of each historical ping table, but it is still possible for a document to appear multiple times if a client sends the same payload across multiple UTC days. Note that this requirement is relaxed for older telemetry ping data that was backfilled from AWS; approximately 0.5% of documents are duplicated in `telemetry.main` and other historical ping tables for 2019-04-30 and earlier dates. +- _Live ping tables_ are the final destination for the [telemetry ingestion pipeline](https://mozilla.github.io/gcp-ingestion/). Incoming ping payloads from clients are decoded by Dataflow jobs and published to a Pub/Sub topic; the `ingestion-sink` service then batches the decoded payloads by document type and loads the results to these tables approximately every 10 minutes, though a delay of up to 30 minutes is normal. These tables are partitioned by date according to `submission_timestamp` and are also clustered on that same field, so it is possible to make efficient queries over short windows of recent data such as the last hour. They have a rolling expiration period of 30 days, but that window may be shortened in the future. Because the pipeline delivers each message at least once and clients retry on transient failures, live ping tables can contain the same `document_id` more than once; this duplication is not removed until the data is copied to the stable tables (see below). The rate is not bounded or guaranteed and varies by app, ping type, and day, but as a rough guide, spot checks over a one-month window in 2026 found within-day duplication around 0.05-0.1% for Firefox Desktop, ~0.2-0.45% for Firefox for Android, and ~0.7-1.1% for Firefox for iOS. Analyses should only use these tables if they need results for the current (partial) day, and should deduplicate on `document_id` if exact counts matter. +- _Stable ping tables_ (also called _historical ping tables_) have exactly the same schema as their corresponding live ping tables, but they are populated only once per day (`12:00:00am` to `11:59:59pm` UTC) via an Airflow job and have a 25 month retention period. These tables are superior to the live ping tables for historical analysis because they never contain partial days, they have additional deduplication applied, and they are clustered on `sample_id`, allowing efficient queries on a 1% sample of clients. It is guaranteed that `document_id` is distinct within each day of each stable ping table, but it is still possible for a document to appear multiple times if a client sends the same payload across multiple UTC days. - _Derived tables_ are populated by nightly [Airflow](https://workflow.telemetry.mozilla.org/home) jobs and are considered an implementation detail; their structure may change at any time at the discretion of the data platform team to allow refactoring or efficiency improvements. - Tables (unsuffixed) may contain `client_id` or other id-level columns, e.g. [clients_daily_v6](https://github.com/mozilla/bigquery-etl/blob/main/sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_v6/metadata.yaml). - Tables without `client_id`-level information use the suffix `_aggregates`, e.g. `addon_aggregates_v2` (https://github.com/mozilla/bigquery-etl/blob/main/sql/moz-fx-data-shared-prod/telemetry_derived/addon_aggregates_v2/metadata.yaml). -- _User-facing views_ are the schema objects that users are primarily expected to use in analyses. Many of these views correspond directly to an underlying historical ping table or derived table, but they provide the flexibility to hide deprecated columns or present additional calculated columns to users. These views are the schema contract with users and they should not change in backwards-incompatible ways without a version increase or an announcement to users about a breaking change. +- _User-facing views_ are the schema objects that users are primarily expected to use in analyses. Many of these views correspond directly to an underlying stable ping table or derived table, but they provide the flexibility to hide deprecated columns or present additional calculated columns to users. These views are the schema contract with users and they should not change in backwards-incompatible ways without a version increase or an announcement to users about a breaking change. -Spark and other applications relying on the BigQuery Storage API for data access need to reference derived tables or historical ping tables directly rather than user-facing views. Unless the query result is relatively large, we recommend instead that users run a query on top of user-facing views with the output saved in a destination table, which can then be accessed from Spark. +Spark and other applications relying on the BigQuery Storage API for data access need to reference derived tables or stable ping tables directly rather than user-facing views. Unless the query result is relatively large, we recommend instead that users run a query on top of user-facing views with the output saved in a destination table, which can then be accessed from Spark. ### Structure of Ping Tables in BigQuery diff --git a/src/cookbooks/live_data.md b/src/cookbooks/live_data.md index b28debda3..b50bc76ae 100644 --- a/src/cookbooks/live_data.md +++ b/src/cookbooks/live_data.md @@ -12,7 +12,7 @@ Use cases, such as real-time monitoring, dashboards, or personalized user experi | 4. Dataflow | Very low-latency streaming for large datasets | High | <10 min | High | | 5. Cloud function with Pub/Sub trigger | Low-latency for smaller subsets of data | Medium | <10 min | Medium to High | -Live ping tables are the final destination for the telemetry ingestion pipeline. Incoming ping data is loaded into these tables approximately every 10 minutes, though a delay of up to 30 minutes is normal. Data in these tables is set to expire after 30 days. +Live ping tables are the final destination for the telemetry ingestion pipeline. Incoming ping data is loaded into these tables approximately every 10 minutes, though a delay of up to 30 minutes is normal. Data in these tables is set to expire after 30 days. Live tables are not deduplicated, so the same `document_id` can appear more than once (usually well under 1%, but up to ~1-1.5% on some platforms such as iOS); deduplicate on `document_id` if you need exact counts for the current day. See [_Table Layout and Naming_](./bigquery/querying.md#table-layout-and-naming) for how live and stable tables compare. ### 1. Querying Live Tables Directly @@ -117,7 +117,7 @@ Google Cloud Functions can process incoming messages in Pub/Sub and stream the d ## Tools for Visualizing Live Data -Currently, live data in BigQuery could be considered "near real time" or nearline (as opposed to online), with latency usually below 1 hour to each live table in our ingestion-sink code base. +Currently, live data in BigQuery could be considered "near real time" or nearline (as opposed to online), with data typically loaded within 10 minutes to each live table in our ingestion-sink code base, though a delay of up to 30 minutes is normal. For near real time product monitoring use cases, it is recommended to use Looker for hosting dashboards that refresh on a regular interval such as every 10 minutes.