Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,4 @@
!tsconfig.json
!type-defs.graphql
!jest.config.ts
!mcp/
49 changes: 49 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,9 +146,58 @@ The project includes comprehensive documentation in the `docs/` directory follow
- [Integrate Blob API](docs/how-to/integrate-blob-api.md) - Frontend integration guide
- **Explanation** (`docs/explanation/`) - Conceptual deep dives
- [Query Optimization](docs/explanation/query-optimization.md) - CALL {} subquery pattern for variable-length path queries
- [Diff System](docs/explanation/diff-system.md) - max_depth semantics, the filter gotcha (Blob nodes), and the max_depth=1 rule for leaf-node types

**When working on Cypher queries or the graph data model, ALWAYS consult [docs/reference/data-model.md](docs/reference/data-model.md) first.**

## MCP Server (`mcp/`)

The MCP (Model Context Protocol) server exposes OSWatcher tools to AI assistants. It is a **separate Docker service** — not part of the `api` container.

- **Location**: `graphql-api/mcp/`, with its own `Dockerfile`, `package.json`, `tsconfig.json` and codegen. The root `tsconfig.json` excludes it.
- **Compose service**: defined in `oswatcher-hub/compose.dev.yml` as `mcp`, port 3001, built from `context: ../graphql-api`. The two repos must be siblings on disk.
- **Rebuilding**: must be rebuilt explicitly — `docker compose -f compose.yml -f compose.dev.yml up --build mcp -d`. Rebuilding `api` does NOT rebuild `mcp`.
- **GraphQL client**: calls the API at `GRAPHQL_API_URL` (`http://api:4000/graphql` in Docker). The API is unauthenticated, so no credential is forwarded and the SDK is built once at module scope.
- **Codegen**: `npm run generate` (not `codegen`) regenerates `src/graphql/generated/sdk.ts` from `src/graphql/queries.graphql`. It reads the schema from a **running** API on port 4000.

### Tools (17 total)

Task-shaped tools take `(ref, path, ...)` and resolve the chain internally. The hash-based tools are retained as an escape hatch for entity combinations the task-shaped ones do not reach.

| Tool | Description |
|------|-------------|
| `list_branches` | List OS branches with optional name filter |
| `list_commits` | List commits on a branch |
| `get_commit_capabilities` | Which data kinds were extracted for a snapshot |
| `list_tree` | List a directory's subdirectories and files |
| `list_registry_key` | List a registry key's subkeys and values, by hive name |
| `get_struct` | A struct's full field layout, by name |
| `list_structs` | Structs a PE file defines, optionally by name |
| `list_symbols` | Symbols a PE file exports, optionally by name |
| `git_log` | How one entity changed across commit history |
| `diff_versions` | Filesystem diff between two refs at a path |
| `search` | Substring search across filesystem/registry/symbols/structs |
| `search_next` | Next page of a search session |
| `search_close` | End a search session early |
| `traverse_path` | Raw: walk the filesystem from a ref to a path |
| `get_winreg_root` | Raw: follow `HAS_WINREG` from a Blob hash |
| `get_blobs_with_symbols` | Raw: PE blobs that carry PDB data |
| `diff_nodes` | Raw: diff on node hashes, any entity type |

### Key design constraints

**Filesystem diff filter**: `DiffNodesAt` **must** pass `filter: ["Tree", "Blob"]` when diffing filesystems. Without it, the Java procedure (`oswatcher.diffTreesRecursive`) defaults to filtering only `Tree` nodes and silently drops every `Blob` leaf. See `docs/explanation/diff-system.md`.

**`max_depth=1` required for Symbol, Struct and StructField diffs**: the Java procedure adds `parentLabel` to the filter and recurses into Struct/Symbol nodes, but their children are `StructField`, not `Struct`/`Symbol`. Without `max_depth=1` the diff silently returns 0 results.

**UNCHANGED status filter for a full C type layout**: to reconstruct a whole struct (e.g. `_EPROCESS`, 261 fields) pass `status_filter: ["NEW", "MOD", "DEL", "UNCHANGED"]`. Without it only the delta comes back (~5 items).

**Search sessions are process-local**: `src/search-session.ts` holds them in a `Map`, so the API cannot be horizontally scaled while sessions are in use.

**Resolution lives in the MCP layer**: `mcp/src/resolve.ts` composes existing GraphQL queries client-side. 0.1 adds no custom resolvers to the API beyond paginated search: the API is a stable open-source artifact with a second consumer (the frontend, which composes the same queries client-side), and the hops are localhost-cheap inside the compose network.

**One pagination envelope**: every paginated tool returns `{ items, has_more, next_cursor }` over three different mechanisms. `list_tree` and `list_registry_key` page two connections at once and pack both cursors into one opaque composite cursor, see `mcp/src/pagination.ts`.

## Important Patterns

### Module System
Expand Down
95 changes: 95 additions & 0 deletions docs/explanation/diff-system.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
# Diff System: Behavior and Known Gotchas

## Overview

The diff system computes changes between two filesystem snapshots. It spans three layers:

1. **GraphQL resolver** (`src/resolvers.ts` → `diffNodesAt`): validates input with a zod schema
2. **TypeScript diff logic** (`src/diff/diff.ts` → `diffNodesAtInternal`, `diffTreesIterative`): resolves paths, calls the Neo4j procedure
3. **Java Neo4j procedure** (`oswatcher-procedures` → `oswatcher.diffTreesRecursive`): recursive graph traversal

## `max_depth` Semantics

| Value | Behavior |
|-------|----------|
| `0` | Compare the root node only (no children). Returns a single MOD/NEW/DEL entry. |
| `1` | Immediate children only. No recursion into subdirectories. |
| `N > 1` | Children + N-1 levels of descendants. |
| `-1` (or omitted) | Unlimited recursive diff. Returns all changed leaf nodes. |

Note: the Java procedure's `canRecurse` logic is **shifted by 1**: `max_depth=1` means "children only, no recursion", not "recurse one level deep".

## `max_depth` Is Unrestricted

`diffNodesAt` applies no authorization to `max_depth`. Every value, including
the unlimited default, is allowed. There is no authentication model in this API
at all. `src/auth/` was removed on 2026-09-08 along with Auth0.

The `filter` and `max_depth=1` gotchas below are **not** auth rules. They are
behaviours of the Java procedure `oswatcher.diffTreesRecursive`, and they apply
identically to every caller.

## `with_intermediates` and `status_filter`

- `with_intermediates=false` (default): only leaf nodes (Blobs, WinRegValues, etc.) are returned. Intermediate directory nodes are not emitted even if they changed.
- `with_intermediates=true`: emit intermediate Tree/WinRegKey nodes as MOD/NEW/DEL in addition to their leaves.
- `status_filter`: restrict results to `NEW`, `MOD`, `DEL`, or `UNCHANGED`. Empty = all except UNCHANGED (UNCHANGED requires explicit opt-in).

## The `filter` Parameter and the Blob Gotcha

The `filter` parameter controls which child node labels are collected at each level of the recursive traversal (in `collectNodeInfo` in the Java procedure).

**Critical behavior**: if `filter=[]` (empty), the Java procedure adds `parentLabel` (e.g. `"Tree"`) to it automatically:

```java
effectiveFilter.add(parentLabel); // always adds "Tree" for filesystem diffs
```

This means without an explicit filter, only `Tree` children are collected, and `Blob` nodes are silently skipped.

**Consequence**:
- `max_depth=1`: works fine (no recursion occurs), Tree children are returned as MOD/DEL/NEW directly
- `max_depth=-1`: returns **0 results** (recursion only visits Tree→Tree chains, Blob leaves are never collected, and with `with_intermediates=false` nothing is emitted)

**Fix**: always pass `filter: ["Tree", "Blob"]` for filesystem diffs. The MCP's `DiffNodesAt` query does this by default (see `mcp/src/tools/diff.ts`).

For non-filesystem diffs (registry, symbols, structs), use the appropriate labels:
- Registry: `["WinRegKey", "WinRegValue"]`
- Symbols: `["Symbol"]`
- Structs (list): `["Struct"]`
- Struct fields: `["StructField"]`

## `max_depth=1` Rule for Leaf-Node Types

**Always pass `max_depth=1` for Symbol, Struct, and StructField diffs.** Without it, diffs return 0 results.

Root cause: the Java procedure adds `parentLabel` to `effectiveFilter` and uses `isRecursableLabel` to decide whether to recurse into a node. `Struct` and `Symbol` are recursable, so the procedure recurses into them looking for children of the same label type. But Struct children are `StructField`, not `Struct`; Symbol nodes have no children of the same type. Without `max_depth=1`, the procedure exhausts all recursion levels and emits nothing.

With `max_depth=1`, recursion is cut off after the immediate children, exactly what is needed for flat lists of Symbols/Structs/StructFields.

| Entity | `parent_label` | `filter` | `max_depth` |
|--------|---------------|----------|-------------|
| Filesystem | `Tree` | `["Tree", "Blob"]` | omit (-1) |
| Registry | `WinRegKey` | `["WinRegKey", "WinRegValue"]` | omit (-1) |
| Symbols | `Blob` | `["Symbol"]` | **1 (required)** |
| Structs list | `Blob` | `["Struct"]` | **1 (required)** |
| Struct fields | `Blob` | `["StructField"]` | **1 (required)** |

## Reconstructing Full C Type Layout

Struct field diffs with `status_filter` omitted (or `["NEW","MOD","DEL"]`) return only delta fields. To reconstruct the complete C type layout, pass `status_filter: ["NEW", "MOD", "DEL", "UNCHANGED"]`.

Example: `_EPROCESS` between win11-24h2 and win11-25h2 (5 changed fields vs 261 total fields).

## Path Resolution

Before calling the Java procedure, `diffNodesAtInternal` resolves the `at_path` argument by traversing from the filesystem root hash to the target node using `get_path_entry()`. If the path doesn't exist in either tree, the function returns early with `total_count: 0, items: []`. This is normal (e.g. a path that only exists in one version).

## MCP vs Frontend

The MCP's `diff_versions` tool (`mcp/src/tools/diff.ts`) hardcodes
`parentLabel="Tree"` and `filter=["Tree", "Blob"]`, which is correct for
filesystem diffs. `diff_nodes` (`mcp/src/tools/diff-nodes.ts`) takes both from
the caller, which is what makes it usable for registry, symbol and struct diffs.

The frontend passes the `filter` parameter explicitly depending on the view (filesystem, registry, symbols, structs). Do not change the MCP defaults without verifying the filter handles all relevant child label types.
Loading
Loading