Reorganize Agent Observability docs into task-based sections

hugo/content/en/account_management/rbac/data_access.md

@@ -267,4 +267,4 @@ When querying data through Datadog APIs with restrictions enabled, users without
 [10]: /account_management/rbac/data_access/#supported-telemetry
 [11]: /logs/guide/logs-rbac/?tab=ui#restrict-access-to-logs
 [12]: /dashboards/sharing/shared_dashboards/
-[13]: /llm_observability/instrumentation/otel_instrumentation/
+[13]: /llm_observability/instrument/otel_instrumentation/

hugo/content/en/api/latest/using-the-api/_index.md

@@ -104,7 +104,7 @@ You can also use the Datadog API to manage your account programmatically:
 [33]: /api/v1/key-management/
 [34]: /api/v1/usage-metering/
 [35]: /api/v1/ip-ranges/
-[36]: /llm_observability/evaluations/export_api
+[36]: /llm_observability/investigate/export_api
 [37]: /api/latest/cloudflare-integration/
 [38]: /api/latest/fastly-integration/
 [39]: /api/latest/jira-integration/

hugo/content/en/llm_observability/_index.md

@@ -132,12 +132,12 @@ See the [Setup documentation][5] for instructions on instrumenting your LLM appl
 {{< partial name="whats-next/whats-next.html" >}}
 
 [1]: https://app.datadoghq.com/llm/traces
-[2]: /llm_observability/terms
+[2]: /llm_observability/quickstart/terms
 [3]: /llm_observability/setup/sdk
 [4]: /llm_observability/setup/api
 [5]: /llm_observability/setup
 [6]: /llm_observability/quickstart
 [7]: https://app.datadoghq.com/dash/integration/llm_operational_insights
 [8]: /llm_observability/setup/auto_instrumentation
-[9]: /llm_observability/evaluations/managed_evaluations
-[10]: /llm_observability/monitoring/patterns
+[9]: /llm_observability/configure/evaluations/managed_evaluations
+[10]: /llm_observability/investigate/patterns

hugo/content/en/llm_observability/build_with_ai/_index.md

@@ -0,0 +1,65 @@
+---
+title: Build with AI
+description: Use the Datadog MCP server, CLI, and Claude Code skills to build and analyze LLM applications from your development environment.
+---
+
+Datadog supports coding agent workflows through the Datadog MCP server, the Pup CLI, and Claude Code skills. Use them to investigate your Agent Observability data and iterate on your LLM application without leaving your development environment.
+
+## Get started
+
+### Install the skills
+
+{{< code-block lang="shell" >}}
+npx skills add datadog-labs/agent-skills/agent-observability --full-depth -y
+{{< /code-block >}}
+
+### Choose a data backend
+
+The skills read your Agent Observability data through either the Datadog MCP server or the Pup CLI. Set up one of them. Each skill detects the MCP server at startup and falls back to the Pup CLI when the MCP server is unavailable.
+
+{{% collapse-content title="Option A: Datadog MCP server" level="h4" expanded=true id="option-a-mcp-server" %}}
+
+{{< site-region region="us,us3,us5,eu,ap1,ap2,uk1" >}}
+Connect the MCP server to your Claude Code session:
+
+<pre><code>claude mcp add --scope user --transport http datadog-llmo-mcp \
+  '{{< region-param key="mcp_server_endpoint" >}}?toolsets=llmobs,core'</code></pre>
+
+To run the MCP server as a local binary instead, install the Datadog MCP CLI:
+
+{{< code-block lang="shell" >}}
+curl -sSL https://coterm.datadoghq.com/mcp-cli/install.sh | bash
+datadog_mcp_cli login
+claude mcp add datadog --scope user -- ~/.local/bin/datadog_mcp_cli
+{{< /code-block >}}
+{{< /site-region >}}
+
+{{< site-region region="gov,gov2" >}}
+<div class="alert alert-danger">This product is not supported for your selected site ({{< region-param key="dd_site_name" >}}).</div>
+{{< /site-region >}}
+
+For the full setup, including authentication options and the available tools, see [MCP Server][2].
+
+{{% /collapse-content %}}
+
+{{% collapse-content title="Option B: Pup CLI" level="h4" expanded=false id="option-b-pup-cli" %}}
+
+Install the Pup CLI with Homebrew (macOS/Linux) and authenticate:
+
+{{< code-block lang="shell" >}}
+brew tap datadog-labs/pack
+brew install datadog-labs/pack/pup
+pup auth login
+{{< /code-block >}}
+
+For other installation methods, supported commands, and authentication options, see [Pup CLI][1].
+
+{{% /collapse-content %}}
+
+{{< whatsnext desc="Build with the Datadog MCP server and skills:" >}}
+    {{< nextlink href="/llm_observability/build_with_ai/mcp_server" >}}MCP Server{{< /nextlink >}}
+    {{< nextlink href="/llm_observability/build_with_ai/claude_code_skills" >}}Claude Code Skills{{< /nextlink >}}
+{{< /whatsnext >}}
+
+[1]: /cli/
+[2]: /llm_observability/build_with_ai/mcp_server

hugo/content/en/llm_observability/guide/claude_code_skills.md → hugo/content/en/llm_observability/build_with_ai/claude_code_skills.md

@@ -1,11 +1,13 @@
 ---
 title: Analyze LLM Applications with Claude Code Skills
+aliases:
+- /llm_observability/guide/claude_code_skills/
 description: Use Datadog's Claude Code skills to classify sessions, diagnose failures, compare experiments, generate Python experiment code, and bootstrap evaluators against your live production data.
 further_reading:
-    - link: '/llm_observability/evaluations/'
+    - link: '/llm_observability/configure/evaluations/'
       tag: 'Documentation'
       text: 'Agent Observability Evaluations'
-    - link: '/llm_observability/experiments/'
+    - link: '/llm_observability/improve/experiments/'
       tag: 'Documentation'
       text: 'LLM Experiments'
     - link: '/llm_observability/guide/evaluation_developer_guide'
@@ -296,7 +298,7 @@ If you are new to evaluating an LLM application, the recommended flow is:
 
 [1]: https://claude.ai/code
 [2]: /llm_observability/setup/
-[3]: /llm_observability/experiments/
+[3]: /llm_observability/improve/experiments/
 [4]: /llm_observability/guide/evaluation_developer_guide
 [5]: https://datadoghq.atlassian.net/wiki/spaces/BITSAI/pages/5226692942/pup+CLI
 [6]: https://github.com/datadog-labs/agent-skills

hugo/content/en/llm_observability/mcp_server.md → hugo/content/en/llm_observability/build_with_ai/mcp_server.md

@@ -1,17 +1,19 @@
 ---
 title: Agent Observability MCP and Skills
+aliases:
+- /llm_observability/mcp_server/
 description: "Connect AI agents to your Agent Observability traces and experiments using the Datadog MCP Server."
 further_reading:
 - link: "mcp_server"
   tag: "Documentation"
   text: "Datadog MCP Server"
-- link: "/llm_observability/experiments"
+- link: "/llm_observability/improve/experiments"
   tag: "Documentation"
   text: "Set up and use Agent Observability Experiments"
-- link: "/llm_observability/monitoring"
+- link: "/llm_observability/investigate"
   tag: "Documentation"
   text: "Monitor your application with Agent Observability"
-- link: "/llm_observability/guide/claude_code_skills"
+- link: "/llm_observability/build_with_ai/claude_code_skills"
   tag: "Guide"
   text: "Analyze LLM Applications with Claude Code Skills"
 ---
@@ -483,4 +485,4 @@ For custom visualizations that go beyond standard Datadog widgets, like comparis
 [6]: /account_management/api-app-keys/
 [7]: /account_management/org_settings/service_accounts/
 [8]: https://github.com/datadog-labs/agent-skills
-[9]: /llm_observability/guide/claude_code_skills
+[9]: /llm_observability/build_with_ai/claude_code_skills

hugo/content/en/llm_observability/configure/_index.md

@@ -0,0 +1,12 @@
+---
+title: Configure
+description: Configure evaluations, prompt management, and automation rules for Agent Observability.
+---
+
+Configure the Agent Observability capabilities that Datadog runs for you, including evaluations, automation rules, and prompt management. Datadog hosts and scales them, so you do not operate your own evaluation, routing, or prompt-serving infrastructure.
+
+{{< whatsnext desc="Configuration options:" >}}
+    {{< nextlink href="/llm_observability/configure/evaluations/" >}}Evaluations{{< /nextlink >}}
+    {{< nextlink href="/llm_observability/configure/prompt_management" >}}Prompt Management{{< /nextlink >}}
+    {{< nextlink href="/llm_observability/configure/automation_rules" >}}Automation Rules{{< /nextlink >}}
+{{< /whatsnext >}}

hugo/content/en/llm_observability/monitoring/automation_rules.md → hugo/content/en/llm_observability/configure/automation_rules.md

@@ -1,11 +1,13 @@
 ---
 title: Automation Rules
+aliases:
+- /llm_observability/monitoring/automation_rules/
 description: Route Agent Observability traces into annotation queues or datasets automatically using filter-based rules.
 further_reading:
-  - link: /llm_observability/evaluations/annotation_queues
+  - link: /llm_observability/investigate/annotation_queues
     tag: Documentation
     text: Set up annotation queues for human review
-  - link: /llm_observability/experiments/datasets
+  - link: /llm_observability/improve/datasets
     tag: Documentation
     text: Build datasets from production traces
   - link: /api/latest/llm-observability/
@@ -89,6 +91,6 @@ The Trace Explorer supports a wider field set than automations. The button's too
 {{< partial name="whats-next/whats-next.html" >}}
 
 [1]: https://app.datadoghq.com/llm/traces
-[2]: /llm_observability/evaluations/annotation_queues/?tab=usingautomationrules
-[3]: /llm_observability/experiments/datasets/?tab=fromproductiontraces
+[2]: /llm_observability/investigate/annotation_queues/?tab=usingautomationrules
+[3]: /llm_observability/improve/datasets/?tab=fromproductiontraces
 [4]: https://app.datadoghq.com/llm/settings/automations

hugo/content/en/llm_observability/evaluations/_index.md → hugo/content/en/llm_observability/configure/evaluations/_index.md

@@ -4,6 +4,7 @@ description: Learn how to configure Evaluations for your LLM application.
 aliases:
     - /tracing/llm_observability/evaluations/
     - /llm_observability/configuration/
+    - /llm_observability/evaluations/
 
 further_reading:
   - link: https://www.datadoghq.com/blog/llm-prompt-tracking
@@ -66,15 +67,15 @@ Agent Observability offers an [Export API][9] that you can use to retrieve spans
 
 {{< partial name="whats-next/whats-next.html" >}}
 
-[1]: /llm_observability/evaluations/custom_llm_as_a_judge_evaluations
-[2]: /llm_observability/evaluations/managed_evaluations
-[3]: /llm_observability/evaluations/external_evaluations
-[5]: /llm_observability/evaluations/submit_nemo_evaluations
+[1]: /llm_observability/configure/evaluations/llm_as_a_judge_evaluations
+[2]: /llm_observability/configure/evaluations/managed_evaluations
+[3]: /llm_observability/configure/evaluations/external_evaluations
+[5]: /llm_observability/configure/evaluations/external_evaluations/nemo
 [6]: /security/sensitive_data_scanner/
 [7]: /account_management/rbac/permissions/#llm-observability
 [8]: https://app.datadoghq.com/llm/evaluations
-[9]: /llm_observability/evaluations/export_api
+[9]: /llm_observability/investigate/export_api
 [10]: /llm_observability/guide/evaluation_developer_guide
-[11]: /llm_observability/evaluations/annotation_queues
+[11]: /llm_observability/investigate/annotation_queues
 [12]: /security/sensitive_data_scanner/scanning_rules/library_rules/
-[13]: /llm_observability/evaluations/end_user_feedback
+[13]: /llm_observability/configure/evaluations/end_user_feedback

hugo/content/en/llm_observability/evaluations/evaluation_compatibility.md → hugo/content/en/llm_observability/configure/evaluations/compatibility.md

@@ -1,5 +1,7 @@
 ---
 title: Evaluation compatibility
+aliases:
+- /llm_observability/evaluations/evaluation_compatibility/
 description: Learn about the compatibility requirements for evaluations.
 ---
 
@@ -43,14 +45,14 @@ Existing templates for custom LLM-as-a-judge evaluations are supported for the f
 | [Goal Completeness][3]          | Fully supported | All third party LLM providers | LLM only |
 
 
-[1]: /llm_observability/evaluations/custom_llm_as_a_judge_evaluations/template_evaluations#tool-selection
-[2]: /llm_observability/evaluations/custom_llm_as_a_judge_evaluations/template_evaluations#tool-argument-correctness
-[3]: /llm_observability/evaluations/custom_llm_as_a_judge_evaluations/template_evaluations#goal-completeness
-[4]: /llm_observability/evaluations/custom_llm_as_a_judge_evaluations/template_evaluations#hallucination
-[5]: /llm_observability/evaluations/custom_llm_as_a_judge_evaluations/template_evaluations#failure-to-answer
-[6]: /llm_observability/evaluations/custom_llm_as_a_judge_evaluations/template_evaluations#sentiment
-[7]: /llm_observability/evaluations/custom_llm_as_a_judge_evaluations/template_evaluations#toxicity
-[8]: /llm_observability/evaluations/custom_llm_as_a_judge_evaluations/template_evaluations#prompt-injection
-[9]: /llm_observability/evaluations/custom_llm_as_a_judge_evaluations/template_evaluations#topic-relevancy
-[10]: /llm_observability/evaluations/managed_evaluations#language-mismatch
-[11]: /llm_observability/evaluations/custom_llm_as_a_judge_evaluations#define-the-evaluation-output
+[1]: /llm_observability/configure/evaluations/llm_as_a_judge_evaluations/template_evaluations#tool-selection
+[2]: /llm_observability/configure/evaluations/llm_as_a_judge_evaluations/template_evaluations#tool-argument-correctness
+[3]: /llm_observability/configure/evaluations/llm_as_a_judge_evaluations/template_evaluations#goal-completeness
+[4]: /llm_observability/configure/evaluations/llm_as_a_judge_evaluations/template_evaluations#hallucination
+[5]: /llm_observability/configure/evaluations/llm_as_a_judge_evaluations/template_evaluations#failure-to-answer
+[6]: /llm_observability/configure/evaluations/llm_as_a_judge_evaluations/template_evaluations#sentiment
+[7]: /llm_observability/configure/evaluations/llm_as_a_judge_evaluations/template_evaluations#toxicity
+[8]: /llm_observability/configure/evaluations/llm_as_a_judge_evaluations/template_evaluations#prompt-injection
+[9]: /llm_observability/configure/evaluations/llm_as_a_judge_evaluations/template_evaluations#topic-relevancy
+[10]: /llm_observability/configure/evaluations/managed_evaluations#language-mismatch
+[11]: /llm_observability/configure/evaluations/llm_as_a_judge_evaluations#define-the-evaluation-output

hugo/content/en/llm_observability/evaluations/end_user_feedback.md → hugo/content/en/llm_observability/configure/evaluations/end_user_feedback.md

@@ -1,14 +1,16 @@
 ---
 title: End-User Feedback
+aliases:
+- /llm_observability/evaluations/end_user_feedback/
 description: Submit end-user feedback to Agent Observability and connect it to spans, traces, sessions, or external entities.
 further_reading:
-    - link: '/llm_observability/instrumentation/api/#evaluations-api'
+    - link: '/llm_observability/instrument/api/#evaluations-api'
       tag: 'Documentation'
       text: 'Learn about the Evaluations API'
-    - link: '/llm_observability/evaluations/external_evaluations'
+    - link: '/llm_observability/configure/evaluations/external_evaluations'
       tag: 'Documentation'
       text: 'Learn about submitting external evaluations'
-    - link: '/llm_observability/evaluations/annotation_queues'
+    - link: '/llm_observability/investigate/annotation_queues'
       tag: 'Documentation'
       text: 'Learn about Annotation Queues'
 ---
@@ -115,9 +117,9 @@ To create a dashboard widget for feedback, create the widget as you would for an
 
 {{< partial name="whats-next/whats-next.html" >}}
 
-[1]: /llm_observability/evaluations/external_evaluations
-[2]: /llm_observability/evaluations/annotation_queues
-[3]: /llm_observability/instrumentation/api/#evaluations-api
-[4]: /llm_observability/instrumentation/sdk/?tab=python#enriching-spans
-[5]: /llm_observability/instrumentation/api/?tab=model#spans-api
-[6]: /llm_observability/monitoring/querying/#feedback-queries
+[1]: /llm_observability/configure/evaluations/external_evaluations
+[2]: /llm_observability/investigate/annotation_queues
+[3]: /llm_observability/instrument/api/#evaluations-api
+[4]: /llm_observability/instrument/sdk/?tab=python#enriching-spans
+[5]: /llm_observability/instrument/api/?tab=model#spans-api
+[6]: /llm_observability/investigate/querying/#feedback-queries

hugo/content/en/llm_observability/evaluations/evaluation_developer_guide.md → hugo/content/en/llm_observability/configure/evaluations/evaluation_developer_guide.md

@@ -2,15 +2,16 @@
 title: Evaluation Developer Guide
 aliases:
   - /llm_observability/guide/evaluation_developer_guide
+  - /llm_observability/evaluations/evaluation_developer_guide/
 description: Learn how to build custom evaluators using the Agent Observability SDK.
 further_reading:
-    - link: '/llm_observability/evaluations/external_evaluations'
+    - link: '/llm_observability/configure/evaluations/external_evaluations'
       tag: 'Documentation'
       text: 'Learn about submitting external evaluations'
     - link: '/llm_observability/setup/sdk/python'
       tag: 'Documentation'
       text: 'Learn about the Agent Observability SDK for Python'
-    - link: '/llm_observability/instrumentation/api'
+    - link: '/llm_observability/instrument/api'
       tag: 'Documentation'
       text: 'Learn about the HTTP API Reference'
 ---
@@ -645,7 +646,7 @@ except RemoteEvaluatorError as e:
 
 ## Using evaluators in production
 
-<div class="alert alert-info">This section covers evaluations you run and submit manually from your application code. To have Datadog run evaluations automatically on production traces, see <a href="/llm_observability/evaluations/custom_llm_as_a_judge_evaluations">Custom LLM-as-a-Judge Evaluations</a> instead.</div>
+<div class="alert alert-info">This section covers evaluations you run and submit manually from your application code. To have Datadog run evaluations automatically on production traces, see <a href="/llm_observability/configure/evaluations/llm_as_a_judge_evaluations">Custom LLM-as-a-Judge Evaluations</a> instead.</div>
 
 To submit evaluations from your application code, construct the `EvaluatorContext` yourself, call the evaluator, and submit the result with `LLMObs.submit_evaluation()`. You can also submit evaluations through the HTTP API.
 
@@ -775,11 +776,11 @@ When submitting evaluations for [OpenTelemetry-instrumented spans][3], include t
 
 {{< partial name="whats-next/whats-next.html" >}}
 
-[1]: /llm_observability/evaluations/external_evaluations
-[2]: /llm_observability/instrumentation/api/#evaluations-api
-[3]: /llm_observability/instrumentation/otel_instrumentation
-[4]: /llm_observability/experiments
-[5]: /llm_observability/evaluations/custom_llm_as_a_judge_evaluations
-[6]: /llm_observability/evaluations/deepeval_evaluations/
-[7]: /llm_observability/evaluations/custom_llm_as_a_judge_evaluations#configure-the-prompt
-[8]: /llm_observability/evaluations/pydantic_evaluations
+[1]: /llm_observability/configure/evaluations/external_evaluations
+[2]: /llm_observability/instrument/api/#evaluations-api
+[3]: /llm_observability/instrument/otel_instrumentation
+[4]: /llm_observability/improve/experiments
+[5]: /llm_observability/configure/evaluations/llm_as_a_judge_evaluations
+[6]: /llm_observability/configure/evaluations/external_evaluations/deepeval/
+[7]: /llm_observability/configure/evaluations/llm_as_a_judge_evaluations#configure-the-prompt
+[8]: /llm_observability/configure/evaluations/external_evaluations/pydantic