Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 95 additions & 0 deletions .claude/skills/sam-automation/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
---
name: sam-automation
description: Write, validate and run Sam (.sb) automations that delegate small, well-scoped tasks to LLM agents (claude, opencode, HTTP providers). Use when the user wants a .sb file, an agent pipeline, a scripted review or triage flow, or wants to split work across cheaper agents.
---

# Sam automation

Sam is a declarative DSL. A `.sb` file declares **schemas** (expected output shapes), **agents** (who runs the work), and **flows** (the steps). Claude's job here is to write the `.sb`, check it, run it on a small input, and report what it cost.

## Workflow

1. **Scope the task.** Decide which steps need a model and which are plain logic. Model calls are the expensive part.
2. **Write the `.sb`** following the syntax below. Keep the file in the project, not in scratch.
3. **Compile** before anything else: `sam compile <file.sb>`. Fix every error first.
4. **Dry run** with `runner: mock` for the agents. The mock returns schema-shaped placeholder output, so the control flow can be checked without spending tokens.
5. **Real run** on one small input with `sam run <file.sb> --flow=<Name> "<input>"`. Keep `max_budget_usd` low while testing.
6. **Report** what ran, the cost if known, and any error code (see Pitfalls).

## Syntax

Verified with `sam compile` against this repo.

```sam
schema Verdict {
approved: boolean,
risk_level: string
}

agent Reviewer {
runner: claude,
model: "sonnet",
workspace: "./src",
system_prompt: "Return only the requested JSON.",
permission_mode: "plan",
max_turns: 4,
max_budget_usd: 0.25,
disallowed_tools: "Bash, Edit, Write",
timeout: 120
}

flow Review(change: string) {
let verdict = use Reviewer<Verdict>(change)
if (!verdict.ok) {
return verdict
}
return verdict.value
}
```

- **Schemas:** fields are `string`, `number`, `boolean`, or `array` (untyped, so no element type or length is enforced).
- **Agents:** either `runner:` (local: `claude`, `opencode`, `mock`, which need `workspace`) or `provider:` (HTTP, which needs `model`). Not both.
- **`timeout` is in seconds** in `.sb` (`timeout: 60` means 60 s). The runtime converts it to milliseconds. Writing `timeout: 120000` means about 33 hours.
- **Calls:** `use Agent<Schema>(input)` validates the output against the schema. `use unsafe Agent(input)` returns raw text and must not have `<Schema>`.
- **Results:** every `use` returns `{ ok, value }` on success or `{ ok: false, ... }` with an error code.
- **Flows:** `let`, `if (...) { } else { }`, `return`, operators `==`, `!=`, `<`, `>`, `!`. Mark helpers `private flow`.
- **Strings:** double quotes, or backtick templates with `${expr}`. There is no `+` concatenation.
- **Not available:** loops, and `parallel(...)` (the token exists but the compiler rejects it with `Unexpected token 'parallel'`).

## Claude runner rules

- `temperature`, `max_tokens` and `retries` are **rejected** for `runner: claude`. They only apply to HTTP providers.
- Use `permission_mode: "plan"` and `disallowed_tools: "Bash, Edit, Write"` unless the task explicitly needs more. Workspace writes are not configurable yet (tracked in the permissions issue).
- Use `max_turns: 4` or more. `max_turns: 2` was too low for `sonnet` in plan mode, and the run failed with `error_max_turns`.
- Every `claude` call is a separate Claude Code session with fixed overhead. Measured at about $0.08 per call, before the task itself.

## Cost rules (from measurements)

- **Batch related items in one call when you can**, because the per-call overhead dominates small tasks. In the measured run, six reviews in one `sonnet` call cost about one fifth of six separate calls. The DSL cannot type a batch result, though, since `array` has no element type. Batching then means one schema per fixed item count, which is awkward. Say so to the user instead of hiding it.
- **Do not assume `haiku` is cheaper per call** on the `claude` runner. In the measured run it cost slightly more than `sonnet` per call, because each model builds its own cache.
- Set `max_budget_usd` on every agent. Report the total you expect to spend before a large run.

## Parallel work

There is no DSL `parallel`. To run several agents in parallel, launch separate `sam run` processes in the background, one per independent input, collect each JSON result, and merge them in the orchestrating step. Keep the number of concurrent processes small. Two concurrent runs with different models were verified.

## Secrets and safety

- Never write API keys into `.sb` files or commit them. Use `env("NAME")` for values such as the workspace path, and set variables in the shell or the CI environment.
- `ANTHROPIC_API_KEY`, when set, makes Claude Code bill per token through the API instead of the user's subscription. Set it only for the run that needs it, such as `bare` mode or unattended jobs.
- `bare: true` needs an API key, according to `claude --help`.

## Pitfalls seen in testing

- **Timeouts in milliseconds** silently become enormous. Use seconds.
- **`max_turns` too low** causes `error_max_turns` when the model wants a tool call first.
- **Child process failures** return the full JSON envelope in `stderr`. Read the `subtype` and `errors` fields instead of the whole blob.
- **Windows `.cmd` shims:** on Windows the npm install of `claude` is a `.cmd`. Sam resolves the native `claude.exe` behind it. If a structured call fails with `cmd.exe special character`, check the `claude` install.
- **Unknown model names** return `EXECUTION_ERROR` with `unrecognized_model`.
- **Missing workspace** looks like a missing binary. If a run fails with `ENOENT ... uv_spawn 'bun'` (or `'claude'`), first check that the agent's `workspace` path exists relative to where `sam run` is started. The binary is usually fine.

## Done when

- `sam compile` passes.
- A `mock` run shows the flow's control path.
- One real run on a small input returned `ok: true` with a value matching the schema, and the spend is reported.
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -572,6 +572,7 @@ gh attestation verify sam-linux-x64 --repo SammyBytes/Sam
* **v0.4.0 — Direct Execution** *(Done)*: `sam run` with positional arguments and stdin pipelines, smart entry point resolution, standalone compiled binaries for Linux/macOS/Windows, tag-driven releases, and guaranteed temporary-file cleanup.
* **v0.5.0 — Providers** *(Done)*: top-level `provider` declarations, OpenAI-compatible HTTP execution, `env()` expansion, `strict_schema` structured outputs, native `retries` with `Retry-After` backoff, `max_tokens` limits, `reasoning_effort` budgets, and full backward compatibility with local `opencode` and `mock` runners.
* **v0.6.0 — Integrations & Analyzer** *(Next)*: additional provider protocols, richer `model` configuration, webhook triggers, contract validation loops, a compiler-level `sam check` analyzer for best-practice warnings, and a repository-level technical whitepaper.
* **Claude Code runner hardening** *(Planned, v0.6.0)*: the `claude` runner (`runner: claude`, branch `feat/claude-code-runner`) is functional for structured and `use unsafe` calls on Windows, but needs three follow-ups tracked as issues: [#18](https://github.com/SammyBytes/Sam/issues/18) a `RunnerPool` that reuses and cleans up processes, [#19](https://github.com/SammyBytes/Sam/issues/19) explicit DSL timeout units, and [#20](https://github.com/SammyBytes/Sam/issues/20) an explicit permissions block for models, workspace writes and authentication. A small cost benchmark found about $0.08 of fixed overhead per separate `claude -p` call; batching six reviews into one call cost about one fifth of six separate calls, while `haiku` was not cheaper per call than `sonnet` in that run.
* **v0.8.0 — Deeper Analysis** *(Planned)*: analyzer output as JSON, project-level rule configuration, inline suppressions (which require comment syntax in the lexer), an extensible model capability table, editor/CI integrations, and a website edition of the whitepaper maintained as the project's long-form history. Warnings stay non-blocking by default; `sam check --strict` promotes them to errors.
* **v1.0.0 — Self-Hosting** *(Planned)*: a compiler written in Sam itself, retiring `@sam-lang/compiler` and removing the TypeScript bootstrap toolchain.

Expand Down
46 changes: 46 additions & 0 deletions examples/workflows/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# Sam workflow examples

Small, single-purpose automations. Each `.sb` file does one job and returns a
structured result, so a person or a script can act on it without reading a
long model reply.

| File | Job | Output |
|---|---|---|
| `ticket_readiness.sb` | Checks that a ticket has an objective, out-of-scope notes and validation steps | `{ has_objective, has_out_of_scope, has_validation, missing_sections }` |
| `pr_review_gate.sb` | Reviews a diff and blocks when it finds blocking issues | `"approved"` or `"blocked"` |
| `iteration_digest.sb` | Writes a short status digest from task lines (uses `use unsafe`, free text) | plain text |
| `run_parallel.sh` | Runs one flow over every file in a folder, in parallel | one JSON file per input |

## Run one example

```bash
sam compile examples/workflows/ticket_readiness.sb
sam run examples/workflows/ticket_readiness.sb --flow=CheckTicket "$(cat examples/workflows/inputs/tickets/ticket_incomplete.txt)"
```

`inputs/` contains synthetic tickets, a task list, and a diff of public code
from this repository. Nothing from a private tracker belongs in this folder.

## Dry run without tokens

Copy an example and change `runner: claude` to `runner: mock`. The mock runner
returns placeholder output that matches the schema, so you can check the flow
logic first.

## Fan out over many tickets

```bash
./examples/workflows/run_parallel.sh examples/workflows/ticket_readiness.sb CheckTicket examples/workflows/inputs/tickets out/readiness
```

Each ticket is a separate `sam run` process. A failure in one does not stop the
others, and the script exits non-zero if any run failed. Errors go to
`<name>.err` next to each output.

## Cost notes

- Each `claude` call pays a fixed overhead of roughly $0.08 before the task
itself. Keep `max_budget_usd` set on every agent.
- `ticket_readiness` uses `haiku`. `pr_review_gate` uses `sonnet`, because a
wrong approval costs more than the extra tokens.
- The DSL has no `parallel(...)` yet. Use `run_parallel.sh` instead.
48 changes: 48 additions & 0 deletions examples/workflows/inputs/diff_claude_envelope_fix.patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
commit 9bba51e358772a1fabff1a763edbc16a0837b88a
Author: Sammy <43734971+SammyBytes@users.noreply.github.com>
Date: Tue Nov 17 09:13:45 2026 -0400

fix(runner): only parse Claude Code JSON envelopes

Always request --output-format json so the response shape is known, and
only unwrap stdout when it is an envelope (type: "result"). Plain model
replies that happen to look like JSON, such as {"result": "hola"}, are
returned unchanged instead of being rewritten or treated as errors.

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>

diff --git a/packages/runtime/src/strategies/agent/claudeStrategy.ts b/packages/runtime/src/strategies/agent/claudeStrategy.ts
index 773f945..c204678 100644
--- a/packages/runtime/src/strategies/agent/claudeStrategy.ts
+++ b/packages/runtime/src/strategies/agent/claudeStrategy.ts
@@ -77,14 +77,13 @@ export class ClaudeStrategy implements IAgentStrategy {
// Sam never resumes sessions, so do not leave transcripts behind.
args.push("--no-session-persistence");

+ // Always request the JSON envelope so extractOutput never has to guess
+ // whether stdout is plain text or an envelope.
+ args.push("--output-format", "json");
+
if (schema) {
const jsonSchema = z.toJSONSchema(schema, { target: "draft-7" });
- args.push(
- "--output-format",
- "json",
- "--json-schema",
- JSON.stringify(jsonSchema),
- );
+ args.push("--json-schema", JSON.stringify(jsonSchema));
}

args.push(inputPrompt);
@@ -106,6 +105,10 @@ export class ClaudeStrategy implements IAgentStrategy {

const envelope = parsed as Record<string, unknown>;

+ // Only treat the output as an envelope when it is one. Plain text that
+ // happens to be JSON (for example {"result": "..."}) passes through.
+ if (envelope.type !== "result") return trimmed;
+
if (envelope.structured_output !== undefined) {
return JSON.stringify(envelope.structured_output);
}
4 changes: 4 additions & 0 deletions examples/workflows/inputs/tasks.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Login redesign | done
Webhook retries | in progress
Export to CSV | blocked by legal review
Dark mode | to do
11 changes: 11 additions & 0 deletions examples/workflows/inputs/tickets/ticket_complete.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
Add retry with backoff to the webhook sender

Objective: retry failed webhook deliveries with exponential backoff so transient 5xx responses stop dropping events.

What to do:
- Add a retry policy with a maximum of 5 attempts and jittered backoff.
- Persist the attempt count so retries survive a restart.

Out of scope: do not change the webhook payload format or the signing logic.

How to validate: simulate 503 responses from a test endpoint and confirm delivery succeeds on the third attempt.
3 changes: 3 additions & 0 deletions examples/workflows/inputs/tickets/ticket_incomplete.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Login button does nothing on Safari

Users report that clicking the login button in Safari 17 does nothing. Chrome works.
19 changes: 19 additions & 0 deletions examples/workflows/iteration_digest.sb
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
agent DigestWriter {
runner: claude,
model: "haiku",
workspace: ".",
system_prompt: "You write short status digests for a development team. Use at most five sentences. Mention what is done, what is in progress, and what is blocked.",
permission_mode: "plan",
max_turns: 4,
max_budget_usd: 0.15,
disallowed_tools: "Bash, Edit, Write",
timeout: 120
}

flow Digest(tasks: string) {
let result = use unsafe DigestWriter(`Write a status digest for these tasks:\n\n${tasks}`)
if (!result.ok) {
return result
}
return result.value
}
28 changes: 28 additions & 0 deletions examples/workflows/pr_review_gate.sb
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
schema Review {
approved: boolean,
risk_level: string,
blocking_issues: string
}

agent DiffReviewer {
runner: claude,
model: "sonnet",
workspace: ".",
system_prompt: "You review code diffs for correctness, security and missing tests. Answer only with the requested JSON. blocking_issues is a short list, or the word none. Approve only when there are no blocking issues.",
permission_mode: "plan",
max_turns: 4,
max_budget_usd: 0.25,
disallowed_tools: "Bash, Edit, Write",
timeout: 180
}

flow ReviewDiff(diff: string) {
let review = use DiffReviewer<Review>(`Review this diff:\n\n${diff}`)
if (!review.ok) {
return review
}
if (review.value.approved == false) {
return "blocked"
}
return "approved"
}
40 changes: 40 additions & 0 deletions examples/workflows/run_parallel.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
#!/usr/bin/env bash
# Runs one Sam flow over every ticket file in a directory, in parallel.
# Each run is an independent `sam run` process, so one failure does not stop the others.
#
# Usage: run_parallel.sh <file.sb> <FlowName> <input-dir> <output-dir>
# Set SAM to override the command, for example:
# SAM="bun run packages/cli/src/cli.ts" ./run_parallel.sh ticket_readiness.sb CheckTicket inputs/tickets out/
set -euo pipefail

sb="$1"
flow="$2"
indir="$3"
outdir="$4"
SAM="${SAM:-sam}"

mkdir -p "$outdir"
pids=()
names=()

for f in "$indir"/*; do
[ -f "$f" ] || continue
name="$(basename "$f")"
name="${name%.*}"
$SAM run "$sb" --flow="$flow" "$(cat "$f")" > "$outdir/$name.json" 2> "$outdir/$name.err" &
pids+=($!)
names+=("$name")
done

status=0
for i in "${!pids[@]}"; do
if wait "${pids[$i]}"; then
echo "ok ${names[$i]}"
else
echo "FAIL ${names[$i]} (see $outdir/${names[$i]}.err)"
status=1
fi
done

echo "done: ${#pids[@]} runs, outputs in $outdir"
exit $status
26 changes: 26 additions & 0 deletions examples/workflows/ticket_readiness.sb
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
schema Readiness {
has_objective: boolean,
has_out_of_scope: boolean,
has_validation: boolean,
missing_sections: string
}

agent TicketChecker {
runner: claude,
model: "haiku",
workspace: ".",
system_prompt: "You check engineering tickets for completeness. Answer only with the requested JSON. missing_sections is a comma-separated list of missing items, or the word none.",
permission_mode: "plan",
max_turns: 4,
max_budget_usd: 0.15,
disallowed_tools: "Bash, Edit, Write",
timeout: 120
}

flow CheckTicket(ticket: string) {
let result = use TicketChecker<Readiness>(`Check this ticket:\n\n${ticket}`)
if (!result.ok) {
return result
}
return result.value
}
Loading
Loading