Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
12833d7
feat(inspect): add IR-based pipeline reasoning tools (inspect, graph,…
Jun 13, 2026
979dcbb
fix(inspect): use CARGO_MANIFEST_DIR for lint test fixture path
Jun 13, 2026
1da03b5
fix(inspect): address PR feedback and suggestions
Copilot Jun 14, 2026
5f25901
fix(inspect): follow up review nits
Copilot Jun 14, 2026
eda822b
fix(inspect,audit,mcp-author): address second-pass PR #998 review fee…
jamesadevine Jun 14, 2026
f64c4d5
fix(inspect): close substring-match bug in whatif + activate missing-…
jamesadevine Jun 14, 2026
160d4f8
fix(audit,mcp-author,compile): address third-round PR #998 review fee…
jamesadevine Jun 14, 2026
7df9b1a
docs(inspect,audit,main): document residual limitations flagged in PR…
jamesadevine Jun 14, 2026
7660b06
refactor(audit,inspect,mcp-author): consolidate audit cache root via …
jamesadevine Jun 14, 2026
883f106
refactor(inspect,audit): consolidate PipelineSummary::all_jobs + drop…
jamesadevine Jun 14, 2026
04be14a
fix(inspect,audit): byte-safe is_negated_call + close symlink-bypass …
jamesadevine Jun 14, 2026
21ae00d
fix(inspect,audit,mcp-author,ir): close same-job unused-output false …
jamesadevine Jun 14, 2026
216cc6e
fix(mcp-author,inspect): validate source_path + unwrap graph_dump JSO…
jamesadevine Jun 15, 2026
d1fd3a0
refactor(audit,inspect,main): gate downstream-impact finding + rename…
jamesadevine Jun 15, 2026
544c42d
refactor(compile,audit,mcp-author,inspect): extract shared source_pat…
jamesadevine Jun 15, 2026
65a180d
fix(compile,inspect): extend relative-path symlink guard + flesh out …
jamesadevine Jun 15, 2026
3e77e5a
fix(inspect,compile,mcp-author): propagate stage condition to jobs + …
jamesadevine Jun 15, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 22 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,20 @@ Every compiled pipeline runs as three sequential jobs:
│ │ │ ├── 0002_pool_object_form.rs # Legacy scalar pool → object form codemod
│ │ │ └── helpers.rs # take_key, insert_no_overwrite, rename_key, ConflictPolicy
│ │ ├── codemod_integration_test.rs # White-box rewrite-path tests (stub registry injection)
│ │ └── types.rs # Front matter grammar and types
│ │ ├── types.rs # Front matter grammar and types
│ │ └── ir/ # Typed Azure DevOps pipeline IR (see docs/ir.md)
│ │ ├── mod.rs # Pipeline / PipelineBody / PipelineShape root types
│ │ ├── ids.rs # Typed StageId / JobId / StepId newtypes
│ │ ├── step.rs # Step variants (Bash, Task, Checkout, Download, Publish, RawYaml)
│ │ ├── job.rs # Job, Pool, TemplateContext, JobVariable
│ │ ├── stage.rs # Stage + external-params wrap
│ │ ├── env.rs # Typed EnvValue (Literal, AdoMacro, PipelineVar, Secret, StepOutput, Coalesce, Concat)
│ │ ├── condition.rs # Typed Condition / Expr AST + codegen to ADO condition syntax
│ │ ├── output.rs # OutputDecl / OutputRef + location-aware lowering
│ │ ├── graph.rs # Dependency graph: validation, edge derivation, isOutput promotion, cycle detection
│ │ ├── lower.rs # IR → serde_yaml::Value lowering
│ │ ├── emit.rs # Thin `lower() + serde_yaml::to_string()` wrapper
│ │ └── summary.rs # Public, serializable PipelineSummary / GraphSummary for agent-facing tooling (see docs/ir.md Public JSON summary)
│ ├── init.rs # Repository initialization for AI-first authoring
│ ├── execute.rs # Stage 3 safe output execution
│ ├── fuzzy_schedule.rs # Fuzzy schedule parsing
Expand Down Expand Up @@ -130,6 +143,10 @@ Every compiled pipeline runs as three sequential jobs:
│ │ ├── mod.rs
│ │ ├── console.rs # Human-readable console report
│ │ └── json.rs # Machine-readable AuditData JSON
│ ├── inspect/ # `ado-aw inspect` / `graph` / (planned) `trace` / `whatif` / `lint` / `catalog` — read-only IR queries
│ │ ├── mod.rs # Module entry; public re-exports of every dispatcher
│ │ ├── cli.rs # Dispatchers (`dispatch_inspect`, `dispatch_graph`, …) and option structs
│ │ └── graph_query.rs # Text/DOT renderers for the resolved dependency graph
│ ├── detect.rs # Agentic pipeline detection — discovers compiled pipelines; used by all lifecycle commands
│ ├── update_check.rs # Version update check — queries GitHub Releases and prints advisory when newer version is available
│ ├── ndjson.rs # NDJSON parsing utilities
Expand Down Expand Up @@ -276,7 +293,7 @@ index to jump to the right page.

### Compiler internals & operations

- [`docs/ir.md`](docs/ir.md) — typed Azure DevOps pipeline IR (`Pipeline`, jobs/stages/steps, output refs, graph pass, lowering, and target builders).
- [`docs/ir.md`](docs/ir.md) — typed Azure DevOps pipeline IR (`Pipeline`, jobs/stages/steps, output refs, graph pass, lowering, target builders, and the public JSON summary consumed by agent-facing tooling).
- [`docs/cli.md`](docs/cli.md) — `ado-aw` CLI commands (`init`, `compile`,
`check`, `mcp`, `mcp-http`, `execute`, `secrets`, `enable`, `disable`,
`remove`, `list`, `status`, `run`, `audit`; `configure` is a deprecated hidden alias).
Expand All @@ -285,6 +302,9 @@ index to jump to the right page.
report shape.
- [`docs/mcp.md`](docs/mcp.md) — MCP server configuration (stdio containers,
HTTP servers, env passthrough).
- [`docs/mcp-author.md`](docs/mcp-author.md) — author-facing MCP server
(stdio); exposes `inspect`, `graph`, `whatif`, `lint`, `catalog`, `trace`,
`audit_build` over MCP for IDE/Copilot Chat agents.
- [`docs/mcpg.md`](docs/mcpg.md) — MCP Gateway architecture and pipeline
integration.
- [`docs/network.md`](docs/network.md) — AWF network isolation, default
Expand Down
24 changes: 21 additions & 3 deletions docs/audit.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ URL-encoded project segments are decoded before the ADO context is resolved. `t=

| Flag | Default | Behavior |
| --- | --- | --- |
| `-o, --output <dir>` | `./logs` | Directory under which `<dir>/build-<id>/` is written. |
| `-o, --output <dir>` | `./logs` | Directory under which `<dir>/build-<id>/` is written. Non-CLI entry points (`ado-aw trace`, the mcp-author tools) instead default to the shared `${TEMP}/ado-aw/audit` cache root so they do not scatter `./logs/` directories under arbitrary working directories. |
| `--json` | off | Emit the full `AuditData` as JSON to stdout (suppresses the trailing `Audit complete` stderr line). |
| `--org <url>` | auto | Azure DevOps organization override for bare build IDs. Full build URLs provide the host / org directly. |
| `--project <name>` | auto | Azure DevOps project override for bare build IDs. Full build URLs provide the project directly. |
Expand Down Expand Up @@ -86,9 +86,10 @@ Current top-level keys include the following. Optional sections are omitted from
| `rejected_safe_outputs` | Rollup of rejections by reason / threat flag. |
| `detection_analysis` | `threat-analysis.json`. |
| `mcp_server_health` | MCPG logs aggregated per server. |
| `pipeline_graph` | Optional typed-IR `PipelineSummary` rebuilt from local source metadata (`aw_info.json.source`) for graph correlation. |
| `mcp_tool_usage` | MCPG logs aggregated per `(server, tool)`. |
| `mcp_failures` | MCPG `tool_error` / `server_error` events. |
| `jobs` | ADO `/timeline` records filtered to `type: Job`. |
| `jobs` | ADO `/timeline` records filtered to `type: Job`; when `pipeline_graph` is available, each entry may include `upstream_jobs` and `downstream_jobs` from IR job edges. |
| `firewall_analysis` | AWF Squid proxy logs aggregated by domain. |
| `policy_analysis` | AWF policy artifacts aggregated into allow / deny summaries. |
| `missing_tools` / `missing_data` / `noops` | NDJSON entries from the corresponding SafeOutputs MCP tools. |
Expand All @@ -109,6 +110,23 @@ Additionally, exactly one severity-`high` finding is emitted summarizing the gat

Per-item detection verdicts are not currently available. `threat-analysis.md` emits an aggregate verdict only; per-item verdicts are a follow-up that should stay aligned with gh-aw.

## Pipeline graph correlation

After the standard analyzers run, `audit` looks for
`agent_outputs[_<BuildId>]/staging/aw_info.json` (falling back to the artifact
top level) and resolves its `source` path relative to the current working
directory. If that markdown source exists locally, the command rebuilds the
typed IR with the same public summary shape emitted by `ado-aw inspect --json`
and stores it under `pipeline_graph.summary`. The audit embeds the full
`PipelineSummary` rather than a reduced subset so audit, inspect, graph, and
trace consumers share one schema.

When graph correlation succeeds, `jobs[]` entries also gain optional
`upstream_jobs` and `downstream_jobs` arrays. These are omitted when empty or
when the source markdown is unavailable locally. Failed jobs with downstream
edges emit a medium-severity finding summarizing the downstream runtime
classifications.

## Cache behavior

`<output>/build-<id>/run-summary.json` is written after a successful run. On subsequent invocations against the same build:
Expand All @@ -135,7 +153,7 @@ Per-item detection verdicts are not currently available. `threat-analysis.md` em

## Related Documentation

- [CLI Commands](cli.md) — full CLI reference
- [CLI Commands](cli.md) — full CLI reference, including `trace`
- [Front Matter](front-matter.md) — agent file format
- [Safe Outputs](safe-outputs.md) — what proposals look like
- [Network](network.md) — AWF firewall configuration
25 changes: 24 additions & 1 deletion docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ Global flags (apply to all subcommands): `--verbose, -v` (enable info-level logg
- Useful for CI checks to ensure pipelines are regenerated after source changes
- `mcp <output_directory> <bounding_directory>` - Run SafeOutputs as a stdio MCP server
- `--enabled-tools <name>` - Restrict available tools to those named (repeatable)
- `mcp-author` - Run the author-facing stdio MCP server for IDE/Copilot Chat integrations. See [`mcp-author.md`](mcp-author.md) for the full tool surface and trust model.
- `mcp-http <output_directory> <bounding_directory>` - Run SafeOutputs as an HTTP MCP server (for MCPG integration)
- `--port <port>` - Port to listen on (default: 8100)
- `--api-key <key>` - API key for authentication (auto-generated if not provided)
Expand Down Expand Up @@ -134,7 +135,7 @@ Both `--all-repos` and `--source` route through `ado-aw`'s `discover_ado_aw_pipe
- `--dry-run` - Print the planned `templateParameters` body without calling the ADO API.

- `audit <build-id-or-url>` - Audit a single Azure DevOps build: download the known stage artifacts, run the audit analyzers, and render a structured console report or `AuditData` JSON.
- `-o, --output <dir>` - Output directory for downloaded artifacts and reports. Defaults to `./logs`; the run is stored under `<dir>/build-<id>/`.
- `-o, --output <dir>` - Output directory for downloaded artifacts and reports. Defaults to `./logs`; the run is stored under `<dir>/build-<id>/`. Non-CLI entry points (`ado-aw trace` and the mcp-author tools) instead anchor under `${TEMP}/ado-aw/audit` so they do not scatter directories under arbitrary working directories.
- `--json` - Emit machine-readable JSON (`AuditData`) instead of the console report. Suppresses the trailing `Audit complete` stderr line.
- `--org <url>` - Override: Azure DevOps organization (used when the input is a bare build ID). Full build URLs provide the host / org directly.
- `--project <name>` - Override: Azure DevOps project name (used when the input is a bare build ID). Full build URLs provide the project directly.
Expand All @@ -143,6 +144,28 @@ Both `--all-repos` and `--source` route through `ado-aw`'s `discover_ado_aw_pipe
- `--no-cache` - Ignore `<output>/build-<id>/run-summary.json` and re-process the build.
- See [`audit.md`](audit.md) for accepted build-reference formats, output layout, cache semantics, and the `AuditData` report shape.

- `trace <build-id-or-url> [--step <id>] [--json]` - Query audit telemetry plus local typed-IR graph correlation to explain failed-job chains and downstream skip classifications. Downloads / caches under `${TEMP}/ado-aw/audit/build-<id>/` (separate from `ado-aw audit`'s `./logs` default so the MCP server and IDE-driven traces do not scatter `./logs/` directories under arbitrary working dirs), and degrades to runtime-only output when the source markdown is not local.
- `--step <id>` - Focus the report on a named IR step and show the containing job's runtime status plus upstream/downstream job classifications.
- `--json` - Emit a structured `TraceReport`.
- `--org <url>`, `--project <name>`, `--pat <pat>` / `AZURE_DEVOPS_EXT_PAT` - Same ADO context/auth passthroughs as `audit`.

- `inspect <source> [--json]` - Build the typed IR for an agent source file and emit a terse summary (jobs, stages, steps, output decls, derived `dependsOn` edges, isOutput-promoted outputs).
- `<source>` - Path to the agent markdown file.
- `--json` - Emit the full [`PipelineSummary`](ir.md#public-json-summary-irsummary) as pretty-printed JSON instead of the human view.
- No YAML is written; this is a read-only query over the same IR the compiler builds.

- `graph dump <source> [--format text|json|dot]` - Print the resolved dependency graph (job edges, stage edges, step locations, outputs needing `isOutput=true`). The graph dump now uses an explicit `dump` subcommand so `graph deps` and `graph outputs` can share the namespace.
- `<source>` - Path to the agent markdown file.
- `--format text` - Default. Human-scannable plain text.
- `--format json` - Emit the [`GraphSummary`](ir.md#public-json-summary-irsummary) JSON.
- `--format dot` - Emit Graphviz DOT. Pipe to `dot -Tsvg -o pipeline.svg` to visualize.

- `graph deps <source> <step-id> [--direction upstream|downstream] [--json]` - Traverse transitive job and step-output dependencies for one step. If `<step-id>` names a job with no matching step, the command falls back to job-level traversal.

- `graph outputs <source> [--producer <step-id>] [--consumer <step-id>] [--json]` - Print declared step outputs and the steps that read them from `env` or `condition`.

- `whatif <source> --fail <step-id-or-job-id> [--json]` - Statically classify downstream jobs that would be skipped, or would run anyway, if a step or job failed.

### Hidden Build-Time Tools

These commands are not shown in `--help` but are available for contributors working on the ado-aw compiler itself:
Expand Down
81 changes: 81 additions & 0 deletions docs/ir.md
Original file line number Diff line number Diff line change
Expand Up @@ -263,3 +263,84 @@ The production target wrappers are:
The canonical 5-job Setup → Agent → Detection → SafeOutputs → Teardown shape itself lives in `agentic_pipeline.rs` and is reused unchanged by every wrapper above; extensions plug into it via `Declarations` (steps, env, hosts, MCPG entries, and Agent-job condition clauses — see `Declarations::agent_conditions`).

When adding a target, follow the same pattern: parse and validate front matter, collect extension `Declarations`, build typed jobs/stages/steps, set the correct `PipelineShape`, and call the shared emit path.

## Public JSON summary (`ir::summary`)

The internal IR types (`Pipeline`, `Job`, `Step`, `Graph`, …) are
intentionally tied to the compiler's lowering needs and are **not**
public API. To give agent-facing tooling a stable view of a compiled
pipeline, `src/compile/ir/summary.rs` defines a parallel
**summary tree** with `#[derive(Serialize)]` that is consumed by:

- `ado-aw inspect <source> [--json]` — top-level pipeline summary.
- `ado-aw graph dump <source> [--format text|json|dot]` — resolved
dependency graph (subset of the summary).
- `ado-aw graph deps <source> <step-id>` and `ado-aw graph outputs
<source>` — focused graph queries over step dependencies and output
declaration/reference edges.
- `ado-aw whatif <source> --fail <step-id-or-job-id>` — static
downstream skip classification from graph reachability and rendered
conditions.
- The `ado-aw audit` JSON (`AuditData.pipeline_graph`) and the
author-MCP server.

### Stability contract

`PipelineSummary::schema_version` (currently `1`) is the public schema
version. **Bump** it when the JSON shape changes in a way a downstream
consumer would notice (renamed field, removed variant, changed
semantics). Additive changes like new optional fields do not require a
bump. New enum variants currently do require a schema-version bump
because the serialized enums do not have catch-all `Unknown` variants.

The summary is the public schema. Internal IR types may change freely
without bumping the summary version, as long as the summary lowering
keeps the existing field set populated correctly.

### Shape

```jsonc
{
"schema_version": 1,
"name": "<pipeline name>",
"shape": "standalone" | "1es" | "job-template" | "stage-template",
"body": { "kind": "jobs", "jobs": [...] }
// OR
{ "kind": "stages", "stages": [...] },
"graph": {
"step_locations": [{ "step", "stage?", "job", "outputs": [...] }],
"job_edges": [{ "consumer", "producer" }], // consumer dependsOn producer
"stage_edges": [{ "consumer", "producer" }],
"outputs_needing_is_output": [{ "step", "outputs": [...] }]
}
}
```

Per-`JobSummary`: `id`, `stage?`, `display_name`, `depends_on`,
`condition?` (lowered ADO condition string), `pool`, `steps`.

Per-`StepSummary`: `id?`, `kind` (`bash` / `task` / `checkout` /
`download` / `publish` / `raw_yaml`), `display_name?`, `task?`,
`condition?`, `outputs[]` (`{name, is_secret, auto_is_output}`),
`env_refs[]` (`{step, name}`), `condition_refs[]` (`{step, name}`).

`condition?` is the lowered ADO condition string (e.g.
`"eq(dependencies.Detection.outputs['threatAnalysis.SafeToProcess'], 'true')"`),
not the typed AST — consumers don't need the AST to reason about
"would this run if X failed?".

### Construction

```rust
let (front_matter, pipeline) = ado_aw::compile::build_pipeline_ir(&source).await?;
let summary = ado_aw::compile::ir::summary::PipelineSummary::from_pipeline(&pipeline)?;
let json = serde_json::to_string_pretty(&summary)?;
```

`build_pipeline_ir` is the public read-only entry point: it parses
and sanitises front matter, runs the same target dispatch as
`compile_pipeline`, and returns the typed `Pipeline` without writing
any YAML. `PipelineSummary::from_pipeline` runs the graph pass
(reusing `graph::build_graph` for validation + edge derivation) and
populates `auto_is_output` for any output that has at least one
cross-step consumer — without mutating the input pipeline.
72 changes: 72 additions & 0 deletions docs/mcp-author.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# Author MCP Server

_Part of the [ado-aw documentation](../AGENTS.md)._

`ado-aw mcp-author` runs a local, author/debug-facing MCP server over stdio for
IDE and Copilot Chat integrations. It exposes read-only workflow inspection,
graph, lint, what-if, trace, and audit tools.

It is **not** the SafeOutputs MCP server embedded in compiled pipelines. The
pipeline SafeOutputs server records proposed mutations for Stage 3 execution;
`mcp-author` is a local helper for humans and agents authoring or debugging
workflows.

## Tool surface

| Tool | Description | Input shape |
| --- | --- | --- |
| `inspect_workflow` | Build and return the public `PipelineSummary`. | `{ "source_path": "agents/example.md" }` |
| `graph_summary` | Return the resolved `GraphSummary`. | `{ "source_path": "agents/example.md" }` |
| `graph_dump` | Render the graph as text or Graphviz DOT. | `{ "source_path": "...", "format": "text" \| "dot" }` |
| `step_dependencies` | Traverse dependencies for a step or job id. | `{ "source_path": "...", "step_id": "Agent", "direction": "upstream" \| "downstream" }` |
| `step_outputs` | List declared outputs and consumers. | `{ "source_path": "...", "producer": null, "consumer": null }` |
| `trace_failure` | Trace a build's failed-job chain using audit data plus any local IR graph. | `{ "build_id_or_url": "123", "step": null, "org": null, "project": null, "pat": null }` |
| `whatif` | Classify downstream jobs if a step or job fails. | `{ "source_path": "...", "failing_id": "Agent" }` |
| `lint_workflow` | Run structural lint checks. | `{ "source_path": "agents/example.md" }` |
| `catalog` | List safe-outputs, runtimes, tools, engines, and models. | `{ "kind": "safe-outputs" }` |
| `audit_build` | Download and analyze a build; same shape as `ado-aw audit --json`. | `{ "build_id_or_url": "123", "org": null, "project": null, "pat": null, "artifacts": null, "no_cache": false }` |

## Trust model

`mcp-author` runs as the invoking local user. It has no bounding directory,
sandbox, or pipeline-style filesystem restrictions. ADO-facing calls (`audit`,
`trace`) use the same `resolve_auth()` path as `ado-aw audit`: explicit PAT,
environment, or Azure CLI fallback depending on local configuration.

## IDE configuration

### VS Code MCP

```json
{
"mcp": {
"servers": {
"ado-aw-author": {
"command": "ado-aw",
"args": ["mcp-author"]
}
}
}
}
```

### Claude Desktop

Add this to `claude_desktop_config.json`:

```json
{
"mcpServers": {
"ado-aw-author": {
"command": "ado-aw",
"args": ["mcp-author"]
}
}
}
```

## Related references

- [`docs/ir.md#public-json-summary-irsummary`](ir.md#public-json-summary-irsummary) — public summary schema contract.
- [`docs/audit.md`](audit.md) — `audit_build` and `trace_failure` build reference and report details.
- [`docs/cli.md`](cli.md) — CLI counterparts for every MCP tool.
Loading
Loading