Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions hugo/config/_default/menus/main.en.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5492,6 +5492,11 @@ menu:
identifier: jobs_monitoring_azure_data_factory
parent: data_jobs
weight: 660000
- name: Snowflake Task Monitoring
url: data_observability/jobs_monitoring/snowflake
identifier: jobs_monitoring_snowflake
parent: data_jobs
weight: 670000
- name: Custom Jobs (OpenLineage)
url: data_observability/jobs_monitoring/openlineage
identifier: openlineage_integrations
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ Data Observability: Jobs Monitoring supports multiple job technologies. To get s
{{< image-card href="/data_observability/jobs_monitoring/dbt" src="integrations_logos/dbt-cloud_large.svg" alt="dbt cloud" >}}
{{< image-card href="/data_observability/jobs_monitoring/glue/" src="integrations_logos/amazon_glue.png" alt="AWS Glue" >}}
{{< image-card href="/data_observability/jobs_monitoring/azure_data_factory/" src="integrations_logos/azure_data_factory.png" alt="Azure Data Factory" >}}
{{< image-card href="/data_observability/jobs_monitoring/snowflake/" src="integrations_logos/snowflake.png" alt="Snowflake" >}}
{{< /card-grid >}}

Data Observability: Jobs Monitoring also supports Apache Spark jobs on the following platforms:
Expand Down
69 changes: 69 additions & 0 deletions hugo/content/en/data_observability/jobs_monitoring/snowflake.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
---
title: "Snowflake Task Monitoring"
description: "Monitor Snowflake task run history in Datadog Data Observability."
further_reading:
- link: '/data_observability/quality_monitoring/data_warehouses/snowflake/'
tag: 'Documentation'
text: 'Snowflake integration setup'
- link: '/data_observability/jobs_monitoring/'
tag: 'Documentation'
text: 'Data Observability: Jobs Monitoring'
- link: '/data_observability/lineage/'
tag: 'Documentation'
text: 'Data Observability: Lineage'
---

<div class="alert alert-info">Snowflake Task Monitoring is in Preview.</div>

## Overview

Datadog reads the run history of your scheduled Snowflake [tasks][1]. Each task run appears in Data Observability as a span, grouped into a trace for its task graph (DAG).

This covers task **run history** only. To monitor Snowflake data quality and lineage, enable Data Observability for Snowflake in the [Snowflake integration setup][2].

## Prerequisites

Connect the [Snowflake integration][2] to Data Observability. The standard setup script already grants everything this feature needs. The grant specific to task monitoring is **`MONITOR EXECUTION ON ACCOUNT`**, which lets Datadog read your Snowflake task run history.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Connect the [Snowflake integration][2] to Data Observability. The standard setup script already grants everything this feature needs. The grant specific to task monitoring is **`MONITOR EXECUTION ON ACCOUNT`**, which lets Datadog read your Snowflake task run history.
Connect the [Snowflake integration][2] to Data Observability. The standard setup script already grants everything this feature needs. The permission it grants that is specific to task monitoring is **`MONITOR EXECUTION ON ACCOUNT`**, which lets Datadog read your Snowflake task run history.

Does this make sense? I thought some more specific wording would make this a bit easier to parse


## Enable Task History Traces

1. In Data Observability, go to your Snowflake account configuration.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a link to the relevant page in the app here?

2. Under **Data Jobs Monitoring**, turn on **Task History Traces**.
3. Choose a **Traces collection period** (`5 Min`, `15 Min`, `30 Min`, `Hourly`, or `Daily`).

Task runs are read from Snowflake's `ACCOUNT_USAGE.TASK_HISTORY` view. When you first enable the feature, Datadog backfills task runs from roughly the last 18 hours. Because that view has a latency of up to 45 minutes, new runs may take that long to appear.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Task runs are read from Snowflake's `ACCOUNT_USAGE.TASK_HISTORY` view. When you first enable the feature, Datadog backfills task runs from roughly the last 18 hours. Because that view has a latency of up to 45 minutes, new runs may take that long to appear.
Data Observability reads task runs from Snowflake's `ACCOUNT_USAGE.TASK_HISTORY` view. When you first enable the feature, Datadog backfills task runs from roughly the last 18 hours. Because that view has a latency of up to 45 minutes, new runs may take that long to appear.


## What Datadog collects

For each task run, Datadog creates a span with:

- **Timing** — scheduled time, completed time, and duration.
- **Status** — success or failure, with the error code and message on failed runs.
- **The SQL statement** the task ran.
Comment on lines +40 to +42

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- **Timing** — scheduled time, completed time, and duration.
- **Status** — success or failure, with the error code and message on failed runs.
- **The SQL statement** the task ran.
- **Timing**: Scheduled time, completed time, and duration.
- **Status**: Success or failure, with the error code and message on failed runs.
- The **SQL statement** the task ran.


Runs are grouped into a **task graph (DAG)** trace, so a root task and its dependent tasks appear together.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Runs are grouped into a **task graph (DAG)** trace, so a root task and its dependent tasks appear together.
Data Observability groups runs into a **task graph (DAG)** trace, so a root task and its dependent tasks appear together.


## View your task runs

- In the Data Observability catalog, the same **Task Runs** view appears in two places:
- the **Task Runs** tab on the task's entity page, and
- the **side panel** that opens when you click a task in the Lineage map.
- In **Trace Explorer**.

{{< img src="data_jobs/snowflake_task_runs.png" alt="Task Runs on a Snowflake task's entity page in Data Observability" style="width:100%;" >}}

{{< img src="data_jobs/snowflake_task_runs_lineage.png" alt="Task runs in the side panel of a Snowflake task in the Data Observability Lineage map" style="width:100%;" >}}
Comment on lines +48 to +55

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- In the Data Observability catalog, the same **Task Runs** view appears in two places:
- the **Task Runs** tab on the task's entity page, and
- the **side panel** that opens when you click a task in the Lineage map.
- In **Trace Explorer**.
{{< img src="data_jobs/snowflake_task_runs.png" alt="Task Runs on a Snowflake task's entity page in Data Observability" style="width:100%;" >}}
{{< img src="data_jobs/snowflake_task_runs_lineage.png" alt="Task runs in the side panel of a Snowflake task in the Data Observability Lineage map" style="width:100%;" >}}
You can view task runs in the following places:
- Trace Explorer
- In the Data Observability catalog, in the Task Runs view:
- In the Task Runs tab on the task's entity page
{{< img src="data_jobs/snowflake_task_runs.png" alt="Task runs on a Snowflake task's entity page in Data Observability" style="width:100%;" >}}
- In the side panel that opens when you click a task in the Lineage map
{{< img src="data_jobs/snowflake_task_runs_lineage.png" alt="Task runs in the side panel of a Snowflake task in the Data Observability Lineage map" style="width:100%;" >}}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would also recommend adding some links to the relevant pages in the app here, in case users want to try it themselves!


## Troubleshooting

If no task runs appear:

- Confirm the task has actually run. A suspended task, or one that hasn't reached its next scheduled time, produces no runs to collect.
- New task runs can take up to 45 minutes to appear (`ACCOUNT_USAGE.TASK_HISTORY` latency), so allow time after a task completes.

## Further reading

{{< partial name="whats-next/whats-next.html" >}}

[1]: https://docs.snowflake.com/en/user-guide/tasks-intro
[2]: /data_observability/quality_monitoring/data_warehouses/snowflake/
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading