-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Add Snowflake Task Monitoring docs (Preview) #39206
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,69 @@ | ||||||||||||||||||||||||||||||||
| --- | ||||||||||||||||||||||||||||||||
| title: "Snowflake Task Monitoring" | ||||||||||||||||||||||||||||||||
| description: "Monitor Snowflake task run history in Datadog Data Observability." | ||||||||||||||||||||||||||||||||
| further_reading: | ||||||||||||||||||||||||||||||||
| - link: '/data_observability/quality_monitoring/data_warehouses/snowflake/' | ||||||||||||||||||||||||||||||||
| tag: 'Documentation' | ||||||||||||||||||||||||||||||||
| text: 'Snowflake integration setup' | ||||||||||||||||||||||||||||||||
| - link: '/data_observability/jobs_monitoring/' | ||||||||||||||||||||||||||||||||
| tag: 'Documentation' | ||||||||||||||||||||||||||||||||
| text: 'Data Observability: Jobs Monitoring' | ||||||||||||||||||||||||||||||||
| - link: '/data_observability/lineage/' | ||||||||||||||||||||||||||||||||
| tag: 'Documentation' | ||||||||||||||||||||||||||||||||
| text: 'Data Observability: Lineage' | ||||||||||||||||||||||||||||||||
| --- | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| <div class="alert alert-info">Snowflake Task Monitoring is in Preview.</div> | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| ## Overview | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| Datadog reads the run history of your scheduled Snowflake [tasks][1]. Each task run appears in Data Observability as a span, grouped into a trace for its task graph (DAG). | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| This covers task **run history** only. To monitor Snowflake data quality and lineage, enable Data Observability for Snowflake in the [Snowflake integration setup][2]. | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| ## Prerequisites | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| Connect the [Snowflake integration][2] to Data Observability. The standard setup script already grants everything this feature needs. The grant specific to task monitoring is **`MONITOR EXECUTION ON ACCOUNT`**, which lets Datadog read your Snowflake task run history. | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| ## Enable Task History Traces | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| 1. In Data Observability, go to your Snowflake account configuration. | ||||||||||||||||||||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe add a link to the relevant page in the app here? |
||||||||||||||||||||||||||||||||
| 2. Under **Data Jobs Monitoring**, turn on **Task History Traces**. | ||||||||||||||||||||||||||||||||
| 3. Choose a **Traces collection period** (`5 Min`, `15 Min`, `30 Min`, `Hourly`, or `Daily`). | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| Task runs are read from Snowflake's `ACCOUNT_USAGE.TASK_HISTORY` view. When you first enable the feature, Datadog backfills task runs from roughly the last 18 hours. Because that view has a latency of up to 45 minutes, new runs may take that long to appear. | ||||||||||||||||||||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| ## What Datadog collects | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| For each task run, Datadog creates a span with: | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| - **Timing** — scheduled time, completed time, and duration. | ||||||||||||||||||||||||||||||||
| - **Status** — success or failure, with the error code and message on failed runs. | ||||||||||||||||||||||||||||||||
| - **The SQL statement** the task ran. | ||||||||||||||||||||||||||||||||
|
Comment on lines
+40
to
+42
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| Runs are grouped into a **task graph (DAG)** trace, so a root task and its dependent tasks appear together. | ||||||||||||||||||||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| ## View your task runs | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| - In the Data Observability catalog, the same **Task Runs** view appears in two places: | ||||||||||||||||||||||||||||||||
| - the **Task Runs** tab on the task's entity page, and | ||||||||||||||||||||||||||||||||
| - the **side panel** that opens when you click a task in the Lineage map. | ||||||||||||||||||||||||||||||||
| - In **Trace Explorer**. | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| {{< img src="data_jobs/snowflake_task_runs.png" alt="Task Runs on a Snowflake task's entity page in Data Observability" style="width:100%;" >}} | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| {{< img src="data_jobs/snowflake_task_runs_lineage.png" alt="Task runs in the side panel of a Snowflake task in the Data Observability Lineage map" style="width:100%;" >}} | ||||||||||||||||||||||||||||||||
|
Comment on lines
+48
to
+55
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would also recommend adding some links to the relevant pages in the app here, in case users want to try it themselves! |
||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| ## Troubleshooting | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| If no task runs appear: | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| - Confirm the task has actually run. A suspended task, or one that hasn't reached its next scheduled time, produces no runs to collect. | ||||||||||||||||||||||||||||||||
| - New task runs can take up to 45 minutes to appear (`ACCOUNT_USAGE.TASK_HISTORY` latency), so allow time after a task completes. | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| ## Further reading | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| {{< partial name="whats-next/whats-next.html" >}} | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| [1]: https://docs.snowflake.com/en/user-guide/tasks-intro | ||||||||||||||||||||||||||||||||
| [2]: /data_observability/quality_monitoring/data_warehouses/snowflake/ | ||||||||||||||||||||||||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this make sense? I thought some more specific wording would make this a bit easier to parse