diff --git a/hugo/config/_default/menus/main.en.yaml b/hugo/config/_default/menus/main.en.yaml index 7bb7b156ff0..2d0651ee91a 100644 --- a/hugo/config/_default/menus/main.en.yaml +++ b/hugo/config/_default/menus/main.en.yaml @@ -5492,6 +5492,11 @@ menu: identifier: jobs_monitoring_azure_data_factory parent: data_jobs weight: 660000 + - name: Snowflake Task Monitoring + url: data_observability/jobs_monitoring/snowflake + identifier: jobs_monitoring_snowflake + parent: data_jobs + weight: 670000 - name: Custom Jobs (OpenLineage) url: data_observability/jobs_monitoring/openlineage identifier: openlineage_integrations diff --git a/hugo/content/en/data_observability/jobs_monitoring/_index.md b/hugo/content/en/data_observability/jobs_monitoring/_index.md index 1f25fa2a0b5..0d1599c2e3f 100644 --- a/hugo/content/en/data_observability/jobs_monitoring/_index.md +++ b/hugo/content/en/data_observability/jobs_monitoring/_index.md @@ -39,6 +39,7 @@ Data Observability: Jobs Monitoring supports multiple job technologies. To get s {{< image-card href="/data_observability/jobs_monitoring/dbt" src="integrations_logos/dbt-cloud_large.svg" alt="dbt cloud" >}} {{< image-card href="/data_observability/jobs_monitoring/glue/" src="integrations_logos/amazon_glue.png" alt="AWS Glue" >}} {{< image-card href="/data_observability/jobs_monitoring/azure_data_factory/" src="integrations_logos/azure_data_factory.png" alt="Azure Data Factory" >}} + {{< image-card href="/data_observability/jobs_monitoring/snowflake/" src="integrations_logos/snowflake.png" alt="Snowflake" >}} {{< /card-grid >}} Data Observability: Jobs Monitoring also supports Apache Spark jobs on the following platforms: diff --git a/hugo/content/en/data_observability/jobs_monitoring/snowflake.md b/hugo/content/en/data_observability/jobs_monitoring/snowflake.md new file mode 100644 index 00000000000..f6727e6d3bc --- /dev/null +++ b/hugo/content/en/data_observability/jobs_monitoring/snowflake.md @@ -0,0 +1,69 @@ +--- +title: "Snowflake Task Monitoring" +description: "Monitor Snowflake task run history in Datadog Data Observability." +further_reading: + - link: '/data_observability/quality_monitoring/data_warehouses/snowflake/' + tag: 'Documentation' + text: 'Snowflake integration setup' + - link: '/data_observability/jobs_monitoring/' + tag: 'Documentation' + text: 'Data Observability: Jobs Monitoring' + - link: '/data_observability/lineage/' + tag: 'Documentation' + text: 'Data Observability: Lineage' +--- + +
Snowflake Task Monitoring is in Preview.
+ +## Overview + +Datadog reads the run history of your scheduled Snowflake [tasks][1]. Each task run appears in Data Observability as a span, grouped into a trace for its task graph (DAG). + +This covers task **run history** only. To monitor Snowflake data quality and lineage, enable Data Observability for Snowflake in the [Snowflake integration setup][2]. + +## Prerequisites + +Connect the [Snowflake integration][2] to Data Observability. The standard setup script already grants everything this feature needs. The grant specific to task monitoring is **`MONITOR EXECUTION ON ACCOUNT`**, which lets Datadog read your Snowflake task run history. + +## Enable Task History Traces + +1. In Data Observability, go to your Snowflake account configuration. +2. Under **Data Jobs Monitoring**, turn on **Task History Traces**. +3. Choose a **Traces collection period** (`5 Min`, `15 Min`, `30 Min`, `Hourly`, or `Daily`). + +Task runs are read from Snowflake's `ACCOUNT_USAGE.TASK_HISTORY` view. When you first enable the feature, Datadog backfills task runs from roughly the last 18 hours. Because that view has a latency of up to 45 minutes, new runs may take that long to appear. + +## What Datadog collects + +For each task run, Datadog creates a span with: + +- **Timing** — scheduled time, completed time, and duration. +- **Status** — success or failure, with the error code and message on failed runs. +- **The SQL statement** the task ran. + +Runs are grouped into a **task graph (DAG)** trace, so a root task and its dependent tasks appear together. + +## View your task runs + +- In the Data Observability catalog, the same **Task Runs** view appears in two places: + - the **Task Runs** tab on the task's entity page, and + - the **side panel** that opens when you click a task in the Lineage map. +- In **Trace Explorer**. + +{{< img src="data_jobs/snowflake_task_runs.png" alt="Task Runs on a Snowflake task's entity page in Data Observability" style="width:100%;" >}} + +{{< img src="data_jobs/snowflake_task_runs_lineage.png" alt="Task runs in the side panel of a Snowflake task in the Data Observability Lineage map" style="width:100%;" >}} + +## Troubleshooting + +If no task runs appear: + +- Confirm the task has actually run. A suspended task, or one that hasn't reached its next scheduled time, produces no runs to collect. +- New task runs can take up to 45 minutes to appear (`ACCOUNT_USAGE.TASK_HISTORY` latency), so allow time after a task completes. + +## Further reading + +{{< partial name="whats-next/whats-next.html" >}} + +[1]: https://docs.snowflake.com/en/user-guide/tasks-intro +[2]: /data_observability/quality_monitoring/data_warehouses/snowflake/ diff --git a/hugo/static/images/data_jobs/snowflake_task_runs.png b/hugo/static/images/data_jobs/snowflake_task_runs.png new file mode 100644 index 00000000000..4175b4585db Binary files /dev/null and b/hugo/static/images/data_jobs/snowflake_task_runs.png differ diff --git a/hugo/static/images/data_jobs/snowflake_task_runs_lineage.png b/hugo/static/images/data_jobs/snowflake_task_runs_lineage.png new file mode 100644 index 00000000000..cda0a81e3e3 Binary files /dev/null and b/hugo/static/images/data_jobs/snowflake_task_runs_lineage.png differ