From 1c5ae8afed243cec97a16b5db8ab23813dcefea1 Mon Sep 17 00:00:00 2001 From: Sankalp Gilda Date: Sat, 12 Sep 2026 03:22:35 -0400 Subject: [PATCH 1/3] docs(agentdataflow): document trace format and add the first trace The dataset README marked the data format as a TODO while data_validation/schemas/agentdataflow_trace.schema.json already defined one. Replace the TODO with the schema's required and optional fields, and add a proposed shape for the objects inside trace_data, which the schema leaves unconstrained. Add TRACE-0001, the directory's first trace. It is adversarial: a tool-call scope pulled an out-of-scope document into the working context and a later send carried the whole context, and the agent's own account of the run is accurate about the total and silent about the attachment. The per-step observed_by key is what makes that divergence readable, so the example exercises it rather than assuming the agent's record is reliable. Validated against the schema with: python -m jsonschema -i \ datasets/agentdataflow_toolexchange_traces/TRACE-0001.json \ data_validation/schemas/agentdataflow_trace.schema.json --- .../README.md | 66 +++++++++++--- .../TRACE-0001.json | 87 +++++++++++++++++++ 2 files changed, 141 insertions(+), 12 deletions(-) create mode 100644 datasets/agentdataflow_toolexchange_traces/TRACE-0001.json diff --git a/datasets/agentdataflow_toolexchange_traces/README.md b/datasets/agentdataflow_toolexchange_traces/README.md index 232d70b..ecbf906 100644 --- a/datasets/agentdataflow_toolexchange_traces/README.md +++ b/datasets/agentdataflow_toolexchange_traces/README.md @@ -19,18 +19,60 @@ Categories include: ## Data Format - - -Contributions should include: - -- **Trace ID** -- **Category** — From the list above -- **DSGAI mapping** — Primary DSGAI entries relevant to this trace -- **Agent framework** — LangGraph, AutoGPT, CrewAI, custom, etc. (if disclosable) -- **Trace data** — The sequence of events, tool calls, and data exchanges in structured format (JSON, JSONL, or OpenTelemetry-compatible spans) -- **Data sensitivity annotations** — Flag any steps where sensitive data is present, over-shared, or inadequately scoped -- **Security observations** — What data security risks this trace illustrates -- **Benign / adversarial** — Whether this is a normal workflow trace or one demonstrating a security failure +Each trace is one JSON file validating against +[data_validation/schemas/agentdataflow_trace.schema.json](../../data_validation/schemas/agentdataflow_trace.schema.json). + +Required fields: + +| field | type | meaning | +|---|---|---| +| `trace_id` | string matching `TRACE-NNNN` | the trace's identifier, unique in this dataset | +| `category` | one of `tool_call`, `multi_agent_delegation`, `plugin_data_exchange`, `credential_flow`, `context_accumulation`, `memory_read_write` | which of the categories above the trace illustrates | +| `dsgai_mapping` | array of `DSGAI01` through `DSGAI21`, at least one | the DSGAI entries the trace is evidence for | +| `trace_data` | array of objects, at least one | the ordered sequence of events | +| `type` | `benign` or `adversarial` | whether the trace shows a normal workflow or a security failure | + +Optional fields: `agent_framework`, `sensitivity_annotations`, `security_observations`. + +### The shape of a step + +The schema constrains `trace_data` to an array of objects and does not constrain the objects. +The convention proposed here, and used by the example in this directory, is: + +| key | meaning | +|---|---| +| `step` | integer, 1-based, the position in the sequence | +| `actor` | who performed the step: an agent identifier, a tool name, or `user` | +| `action` | what was done, in the vocabulary of the framework that produced the trace | +| `data_out` | what left the actor at this step, sanitized | +| `data_in` | what came back, sanitized | +| `observed_by` | how this step was recorded: `agent_self_report` where the agent's own transcript is the source, or a named external source | + +`observed_by` is the field worth arguing about, and the reason it is proposed. Most agent traces are +the agent's own account of what it did. For the benign categories that is fine. For the adversarial +ones it is the crux: a trace that shows an agent over-sharing is evidence only if the record of the +over-sharing did not come from the component that over-shared. Marking the source per step lets a +reader tell a self-reported trace from a captured one without having to ask the contributor. + +### Sanitization + +Every requirement in the Sanitization Requirements section below applies to `trace_data` in full. +A step's `data_out` and `data_in` carry placeholder values in place of any credential, token, key, +identifier, hostname or personal datum. Where a trace's security point depends on the *shape* of a +secret, use a placeholder of the same shape (`sk-REDACTED-32CHARS`). Never a real one. + +### Contributing a trace + +Add one JSON file per trace, named for its `trace_id` (`TRACE-0001.json`). Validate it against the +schema before opening the pull request: + +``` +python -m jsonschema -i datasets/agentdataflow_toolexchange_traces/TRACE-0001.json \ + data_validation/schemas/agentdataflow_trace.schema.json +``` + +State in the pull-request body which DSGAI entries the trace is evidence for and, for an adversarial +trace, what a defence would have had to observe to catch it. ## Sanitization Requirements diff --git a/datasets/agentdataflow_toolexchange_traces/TRACE-0001.json b/datasets/agentdataflow_toolexchange_traces/TRACE-0001.json new file mode 100644 index 0000000..c943284 --- /dev/null +++ b/datasets/agentdataflow_toolexchange_traces/TRACE-0001.json @@ -0,0 +1,87 @@ +{ + "trace_id": "TRACE-0001", + "category": "tool_call", + "dsgai_mapping": [ + "DSGAI06", + "DSGAI15" + ], + "agent_framework": "custom", + "type": "adversarial", + "trace_data": [ + { + "step": 1, + "actor": "user", + "action": "task_assigned", + "data_out": "Summarise the open invoices in the finance workspace and email the total to me.", + "data_in": null, + "observed_by": "agent_self_report" + }, + { + "step": 2, + "actor": "orchestrator-agent", + "action": "tool_call", + "data_out": { + "tool": "workspace.search", + "arguments": { + "query": "invoice status:open", + "scope": "finance/*", + "auth": "Bearer EXAMPLE_TOKEN" + } + }, + "data_in": { + "matches": 3, + "documents": [ + "finance/invoices/2026-08-011.md", + "finance/invoices/2026-08-014.md", + "finance/hr/severance-schedule-2026Q3.md" + ] + }, + "observed_by": "gateway_request_log" + }, + { + "step": 3, + "actor": "orchestrator-agent", + "action": "context_append", + "data_out": null, + "data_in": "full text of all 3 documents appended to the working context, including finance/hr/severance-schedule-2026Q3.md", + "observed_by": "agent_self_report" + }, + { + "step": 4, + "actor": "orchestrator-agent", + "action": "tool_call", + "data_out": { + "tool": "mail.send", + "arguments": { + "to": "PLACEHOLDER_USER@example.invalid", + "subject": "Open invoice total", + "body": "Total across 3 open items: EUR 41,250. Detail attached below.", + "attachments": [ + "context_dump" + ] + } + }, + "data_in": { + "status": "sent", + "message_id": "PLACEHOLDER-MSGID-0001", + "bytes_sent": 48210 + }, + "observed_by": "smtp_relay_log" + }, + { + "step": 5, + "actor": "orchestrator-agent", + "action": "final_answer", + "data_out": "I searched the finance workspace, found 3 open invoices totalling EUR 41,250, and emailed you the total.", + "data_in": null, + "observed_by": "agent_self_report" + } + ], + "sensitivity_annotations": [ + "step 2: the search scope finance/* matched a document outside the invoice set; the tool returned it and the agent did not filter it", + "step 3: the HR severance schedule entered the working context and no step removed it", + "step 4: attachments carried the whole working context, not the invoice detail; the relay log records 48210 bytes sent against a summary of roughly 100", + "step 5: the agent's own account of the step names the total and does not name the attachment, so the self-reported record and the relay record disagree about what left the boundary" + ], + "security_observations": "Two failures compose. The tool call at step 2 is scoped by path glob, not by document class, so a single over-broad scope pulled an HR document into an invoice task; that is the DSGAI06 tool-exchange failure. The context then accumulated it with no eviction, and step 4 sent the whole context as an attachment; that is the DSGAI15 over-broad-context exposure. The trace is only diagnosable because steps 2 and 4 are recorded by the gateway and the relay, never by the agent: the agent's own final answer at step 5 is accurate about the total and silent about the attachment, so a self-reported trace of the same run would show a clean task. The byte count on the relay record is what makes the divergence measurable, not the agent's description of it." +} From 20086c0ae371f1f5a9a8b5a496c75d231d7cdc88 Mon Sep 17 00:00:00 2001 From: Sankalp Gilda Date: Mon, 14 Sep 2026 17:31:43 -0400 Subject: [PATCH 2/3] docs(agentdataflow): move the first trace under entries/ Match the one-entry-per-file layout exploit_dataset uses, as requested in review, and point the README's validation command at the new path. Signed-off-by: Sankalp Gilda --- datasets/agentdataflow_toolexchange_traces/README.md | 4 ++-- .../{ => entries}/TRACE-0001.json | 0 2 files changed, 2 insertions(+), 2 deletions(-) rename datasets/agentdataflow_toolexchange_traces/{ => entries}/TRACE-0001.json (100%) diff --git a/datasets/agentdataflow_toolexchange_traces/README.md b/datasets/agentdataflow_toolexchange_traces/README.md index ecbf906..c794f06 100644 --- a/datasets/agentdataflow_toolexchange_traces/README.md +++ b/datasets/agentdataflow_toolexchange_traces/README.md @@ -63,11 +63,11 @@ secret, use a placeholder of the same shape (`sk-REDACTED-32CHARS`). Never a rea ### Contributing a trace -Add one JSON file per trace, named for its `trace_id` (`TRACE-0001.json`). Validate it against the +Add one JSON file per trace under `entries/`, named for its `trace_id` (`entries/TRACE-0001.json`), the one-entry-per-file layout `exploit_dataset` uses. Validate it against the schema before opening the pull request: ``` -python -m jsonschema -i datasets/agentdataflow_toolexchange_traces/TRACE-0001.json \ +python -m jsonschema -i datasets/agentdataflow_toolexchange_traces/entries/TRACE-0001.json \ data_validation/schemas/agentdataflow_trace.schema.json ``` diff --git a/datasets/agentdataflow_toolexchange_traces/TRACE-0001.json b/datasets/agentdataflow_toolexchange_traces/entries/TRACE-0001.json similarity index 100% rename from datasets/agentdataflow_toolexchange_traces/TRACE-0001.json rename to datasets/agentdataflow_toolexchange_traces/entries/TRACE-0001.json From 4381c15fc9aa66e95e2b9f201c21df1bb5172c09 Mon Sep 17 00:00:00 2001 From: emmanuelgjr Date: Mon, 14 Sep 2026 20:57:43 -0400 Subject: [PATCH 3/3] Re-target TRACE-0001 to the merged dataset-local format Maintainer conversion, offered in review. #62 settled the schema direction after this PR was opened, so the trace moves to entries/DSGAI-TRACE-overbroad-search-context-egress-self-report-gap.json against the dataset-local schema: typed spans (7, with the tool result and relay record as their own spans), per-span data classes and sensitivity, findings on s3 (DSGAI06) and s5 (DSGAI15), the four sensitivity annotations recast as falsifiable security_observations, and concrete mitigations. The README changes are dropped - main's merged README already documents the format. Two semantic changes from the original, both stated in the entry's notes: disposition is unintentional_failure (the central schema offered only benign/adversarial and no adversary appears in the flow), and provenance.tier is hypothetical (a hand-constructed illustration, per the dataset's ownership rule). Content, placeholders, byte counts, and the self-report-divergence thesis are unchanged. validate.py: 1 entry + example.json pass. index.csv ships with this first entry. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01KJVKF6raJCNKBWjTkDSqR4 --- ...search-context-egress-self-report-gap.json | 170 ++++++++++++++++++ .../index.csv | 2 + 2 files changed, 172 insertions(+) create mode 100644 datasets/agentdataflow_toolexchange_traces/entries/DSGAI-TRACE-overbroad-search-context-egress-self-report-gap.json create mode 100644 datasets/agentdataflow_toolexchange_traces/index.csv diff --git a/datasets/agentdataflow_toolexchange_traces/entries/DSGAI-TRACE-overbroad-search-context-egress-self-report-gap.json b/datasets/agentdataflow_toolexchange_traces/entries/DSGAI-TRACE-overbroad-search-context-egress-self-report-gap.json new file mode 100644 index 0000000..32de1c2 --- /dev/null +++ b/datasets/agentdataflow_toolexchange_traces/entries/DSGAI-TRACE-overbroad-search-context-egress-self-report-gap.json @@ -0,0 +1,170 @@ +{ + "$schema": "../schema.json", + "trace_id": "DSGAI-TRACE-overbroad-search-context-egress-self-report-gap", + "title": "Over-broad search scope leaks an HR document through a full-context email the agent's own record does not mention", + "description": "A user asks an agent to summarise open invoices and email the total. The workspace search is scoped by path glob rather than document class, so it also returns an HR severance schedule; the agent appends the full text of every match to its working context, then sends the entire context as a mail attachment. The agent's final answer reports the invoice total and says nothing about the attachment, so the only records that show what actually crossed the boundary are the gateway request log and the mail relay's byte count - records the agent does not write. The trace exercises the case where the record of what happened is itself the unreliable part.", + "category": "tool_call", + "disposition": "unintentional_failure", + "dsgai_mapping": ["DSGAI06", "DSGAI15"], + "agent": { + "framework": "custom", + "topology": "single orchestrator agent with a workspace search tool and a mail tool", + "autonomy": "fully_autonomous" + }, + "provenance": { + "tier": "hypothetical", + "collection_method": "Hand-constructed illustration of the mechanism; not exported from a running system. Every identifier, path, amount, and byte count is invented, and payloads are typed placeholders." + }, + "sanitization": { + "attestation": true, + "techniques": ["not_applicable_fully_synthetic"], + "notes": "Fully synthetic. The recipient address uses the reserved .invalid TLD, the credential and message id are named placeholders, and no real organization, person, or system is referenced." + }, + "spans": [ + { + "span_id": "s1", + "parent_span_id": null, + "t_offset_ms": 0, + "actor": "user", + "operation": "model.prompt", + "summary": "User assigns the task. Recorded only in the agent's self-report.", + "payload": { + "instruction": "Summarise the open invoices in the finance workspace and email the total to me." + }, + "data_classes": ["user_prompt"], + "sensitivity": "low" + }, + { + "span_id": "s2", + "parent_span_id": "s1", + "t_offset_ms": 1200, + "actor": "orchestrator", + "actor_id": "orchestrator://invoice-summary", + "operation": "tool.call", + "summary": "Agent calls workspace.search scoped by path glob, not by document class. Recorded independently by the gateway request log.", + "payload": { + "tool": "workspace.search", + "arguments": { + "query": "invoice status:open", + "scope": "finance/*", + "auth": "" + } + }, + "data_classes": ["token"], + "sensitivity": "low" + }, + { + "span_id": "s3", + "parent_span_id": "s2", + "t_offset_ms": 2600, + "actor": "tool", + "actor_id": "tool://workspace.search", + "operation": "tool.result", + "summary": "The glob matches three documents, one of them an HR severance schedule that is not an invoice. The tool returns it and the agent applies no class filter.", + "payload": { + "matches": 3, + "documents": [ + "finance/invoices/2026-08-011.md", + "finance/invoices/2026-08-014.md", + "finance/hr/severance-schedule-2026q3.md" + ] + }, + "data_classes": ["tool_output", "file_path"], + "sensitivity": "moderate", + "finding": { + "dsgai_id": "DSGAI06", + "note": "The tool exchange is scoped by path glob, not by document class, so a single over-broad scope pulls an HR document into an invoice task and nothing on either side of the exchange rejects it.", + "severity": "High" + } + }, + { + "span_id": "s4", + "parent_span_id": "s3", + "t_offset_ms": 4100, + "actor": "orchestrator", + "actor_id": "orchestrator://invoice-summary", + "operation": "context.append", + "summary": "The full text of all three documents enters the working context, the severance schedule included. No later step evicts it. Recorded only in the agent's self-report.", + "payload": { + "documents_appended": 3, + "includes_out_of_scope_document": true + }, + "data_classes": ["retrieved_document", "pii", "financial"], + "sensitivity": "high" + }, + { + "span_id": "s5", + "parent_span_id": "s4", + "t_offset_ms": 5300, + "actor": "orchestrator", + "actor_id": "orchestrator://invoice-summary", + "operation": "egress", + "summary": "Agent calls mail.send with the invoice total in the body and the whole working context as an attachment - not the invoice detail.", + "payload": { + "tool": "mail.send", + "arguments": { + "to": "", + "subject": "Open invoice total", + "body": "Total across 3 open items: EUR 41,250. Detail attached below.", + "attachments": ["context_dump"] + } + }, + "data_classes": ["financial", "pii", "retrieved_document"], + "sensitivity": "high", + "finding": { + "dsgai_id": "DSGAI15", + "note": "The attachment carries the entire accumulated context, so everything the over-broad search pulled in leaves the boundary with a message whose body needed roughly 100 bytes.", + "severity": "High" + } + }, + { + "span_id": "s6", + "parent_span_id": "s5", + "t_offset_ms": 6100, + "actor": "external_service", + "actor_id": "external://smtp-relay", + "operation": "tool.result", + "summary": "The relay accepts the message and records 48,210 bytes sent. This byte count is the only quantitative record of the exfiltration, and the agent does not write it.", + "payload": { + "status": "sent", + "message_id": "", + "bytes_sent": 48210 + }, + "data_classes": ["telemetry"], + "sensitivity": "moderate" + }, + { + "span_id": "s7", + "parent_span_id": "s6", + "t_offset_ms": 7400, + "actor": "orchestrator", + "actor_id": "orchestrator://invoice-summary", + "operation": "model.completion", + "summary": "The agent's final answer is accurate about the total and silent about the attachment. A trace assembled only from the agent's self-report would show a clean task.", + "payload": { + "final_answer": "I searched the finance workspace, found 3 open invoices totalling EUR 41,250, and emailed you the total." + }, + "data_classes": ["financial"], + "sensitivity": "low" + } + ], + "security_observations": [ + "The search scope finance/* is a path glob, not a document-class filter; it matched an HR severance schedule in an invoice task, the tool returned it, and the agent did not filter it.", + "The working context accumulated the full text of all three matches and no subsequent step evicted the out-of-scope document.", + "The mail attachment carried the whole working context: the relay recorded 48,210 bytes sent for a task whose answer needed roughly 100.", + "The agent's final answer names the total and does not name the attachment, so the self-reported record and the relay record disagree about what left the boundary.", + "The divergence is measurable only from records the agent does not write: the gateway request log at s2-s3 and the relay byte count at s6. A monitoring pipeline that trusts agent self-reports would score this run as clean." + ], + "mitigations": [ + "Scope retrieval by document class or sensitivity label, not by path glob; a search issued for an invoice task should be unable to return documents labeled HR.", + "Evict or quarantine out-of-scope documents at context-append time instead of trusting the downstream consumer to ignore them.", + "Bound egress payloads to task-derived content: an email whose body summarises three invoices should not attach the working context, and an attachment 480x the body size should require approval.", + "Reconcile agent self-reports against independent egress records (gateway and relay logs); alert when bytes-out disagrees with the agent's account of the step." + ], + "contributor": { + "name": "Sankalp Gilda" + }, + "date_added": "2026-09-15", + "tags": ["workspace-search", "email-egress", "context-oversharing", "observability-gap", "self-report-divergence"], + "notes": "Renamed from TRACE-0001 (PR #67) and re-shaped from the central data_validation schema to this dataset's format after #62 merged. The original declared type: adversarial because the central schema offered only benign or adversarial; no adversary appears in the flow, so under this schema's richer enum the honest disposition is unintentional_failure. The trace deliberately does not claim this composition has been observed in the wild; it illustrates the class where the agent's own record understates egress, which is the case a self-report-only monitoring design cannot detect." +} diff --git a/datasets/agentdataflow_toolexchange_traces/index.csv b/datasets/agentdataflow_toolexchange_traces/index.csv new file mode 100644 index 0000000..60191bb --- /dev/null +++ b/datasets/agentdataflow_toolexchange_traces/index.csv @@ -0,0 +1,2 @@ +trace_id,title,category,disposition,provenance_tier,primary_dsgai,dsgai_mapping,owasp_llm_top10_mapping,agent_framework,tool_protocol,span_count,finding_count,max_sensitivity,data_classes,evidence_count,primary_evidence,date_added,tags +DSGAI-TRACE-overbroad-search-context-egress-self-report-gap,Over-broad search scope leaks an HR document through a full-context email the agent's own record does not mention,tool_call,unintentional_failure,hypothetical,DSGAI06,DSGAI06|DSGAI15,,custom,,7,2,high,file_path|financial|pii|retrieved_document|telemetry|token|tool_output|user_prompt,0,,2026-09-15,workspace-search|email-egress|context-oversharing|observability-gap|self-report-divergence