Concise skills rollout: firecrawl (3 skills, 1 model-invoked)

Part of #6476. Rewrite firecrawl's skills to the skill criteria in one draft PR, following the per-plugin runbook (/playbooks:skill-authoring, reference/rewrite-a-plugin.md). The runbook and the rewrite protocol it cites decide what ships; this issue holds only what is specific to the plugin.

The plugin today

As of origin/main on 2026-10-24:

Skills: firecrawl (model-invoked); setup, update (user-invoked).
Outcome eval case directories under the plugin: 0. Probe files under probes/: 0.

Done when

/skill-quality:check plugins/firecrawl on the PR head reports no FAIL that the run on main does not.
/evals:validate passes on plugins/firecrawl's eval suite at the PR head.
Each model-invoked skill has a probe file: plugins/firecrawl/probes/firecrawl.json.
The PR body holds a per-skill decision table, description and body decided separately, with the before and after numbers behind each row.
The PR body names every skill or case left unmeasured, with the reason.
The plugin's version or changelog entry is in the PR, per the plugin's release mode.

Eval spend and review

Eval passes within prepaid API credit are pre-approved (owner decision Q46, 2026-10-24), at the --max-cost-usd 5 per-pass ceiling set on #6476 on 2026-10-18.
No person gates the PR: the owner's 2026-10-24 decision (C12) leaves only an intent mismatch during the build or a security issue at review as stops.
Eval cases run without a Firecrawl API key; a case that needs a live third-party call is named unmeasured in the PR, never run against a paid outside service.

Generated by Claude Code

Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.

Size: Medium
Complexity: Medium
Uncertainty: Known
Flags: none
Blocked by: none

Understood: rewrite the three firecrawl skills (firecrawl, setup, update) to the skill criteria in one draft PR, following the per-plugin runbook. Done-when covers the skill-quality check, eval validation, a probe file, a decision table, a list of unmeasured items, and a version or changelog entry. The scope is clear and the owner released the earlier hold, so nothing is missing. Eval spend is prepaid and pre-approved, so no cost flag applies. I found no likely duplicates among the open issues, apart from the parent #6476.

_{Automated triage, applied by the lanes App.}

This was generated by AI during triage.

Held: on #6476 (2026-10-18) the owner said the middle group of the concise-skills rollout is decided after batch 2's numbers; this plugin is not in batch 2.

Released by the owner's HITL session (2026-10-24, C12): it waited only on a removed gate. Intake triage re-runs on it to set its fields, then admission takes it.

The triage sweep dispatched intake triage for this issue (attempt 1 of 2).

This was generated by AI.

Parked for capacity, not for the owner. This rollout's remaining work is the measured eval passes of the batch-1 protocol, which the issue lane cannot run: it cannot start claude or claude plugin eval (see the lane's escalation on #6585, https://github.com/melodic-software/claude-code-plugins/actions/runs/40186335064). It waits on the plugin eval dispatch work in #9704, now a blocked-by edge, so admission holds it until #9704 closes.

Under the owner's 2026-10-26 stop rule, a person is needed only for a change in intent or vision, an implementation finding that changes the intent or primary design, or a high or critical security issue; host, attended and eval runs are agent work. So needs-human comes off, and the issue moves to host-run with status: confirmed, which keeps the issue lane from taking it when #9704 closes: #9704 adds an eval dispatch workflow, not eval capability for the issue lane.

No description provided.

This was generated by AI.

Progress: #11939 merged (ae7183b). It adds five outcome eval cases (all graders deterministic) and 18 frozen trigger probes for the firecrawl plugin's firecrawl and setup skills. It removes one host-dependent grader and tightens the command graders on review so a call inside a shell conditional or loop is still counted. No skill text changed. Its smoke pass was INVALID (one sandbox denial of a compound Bash command), so its scores are a signal, not a measurement.

This issue stays open. The concise rewrite and its measured passes wait on the eval validity fixes in #11618 and #11628. Those runs must also unset FIRECRAWL_API_KEY and put a failing az stub first on PATH (see the PR body). The frozen suite digest for those runs is bd2409ec691bf568043c0fb0af2d9bc58ef8a16f (plugins/firecrawl/evals at e57bd17). The claim on this issue is released.

No description provided.

Progress: the firecrawl skill body shipped in #12102 (squash d5c37d3), measured no-loss: every case scored 1.000 before and after on Opus 5.5 and Sonnet 5.5, with all passes VALID.

Still open on this issue:

setup body: unmeasured, because its only case ran INVALID on both versions (permission denials in don't-ask mode on the env-presence probe). It needs a case shape or grant that lets that probe through.
update skill: it has no case that can run, since it needs network and a working-tree checkout.

No description provided.

Host run (lane 2): measured pass for the firecrawl description, plus a negative control for the body cut merged in #12102. Nothing new ships; main keeps its text.

Decision table

Skill	Part	Decision	Basis
`firecrawl`	body	Stays as merged in #12102	The negative control detects a loss, so #12102's no-loss measurement counts (below)
`firecrawl`	description	`main` kept (683 characters)	The rewrite (503) passed every trigger condition, but a meaning-diff labeler found a semantic loss (below)
`setup`	body, description	`main` kept, unmeasured	Its only case ran INVALID on both versions in #12102: the eval's don't-ask mode denies the env-presence probe. It needs a case shape or a grant that lets the probe through. User-invoked only, so it has no trigger measure
`update`	body, description	`main` kept, unmeasured	It has no runnable case: it needs network access and a working-tree checkout. User-invoked only

#12102 body control (owner rule #12222)

Before: main's firecrawl plugin (7cfcf0c59a). Broken copy: the same plugin with the contents of six procedure sections of skills/firecrawl/SKILL.md emptied (Core pattern, Commands, Configuration & defaults, Prerequisites, Updating, Gotchas). Frontmatter and every heading were kept: 173 to 93 lines. diff -rq lists only that file.
Run: claude-sonnet-5-5 with an Opus judge, 5 runs, threshold 0.8, --scaffold --allow-tools WebFetch Bash Write Edit, the four fc-* outcome cases. The two passes launched 5 s apart. Both VALID.
Per-case means, before / broken: fc-blocked-routes 1.00 / 1.00; fc-init-refused 1.00 / 0.67; fc-plain-page 1.00 / 1.00; fc-spill-injection 1.00 / 1.00.
Noise report: a case drop on fc-init-refused. The deterministic no-init grader failed in 5 of 5 broken runs, because the model ran firecrawl init. Detects.

Description pass

Rewrite: a fresh agent that never saw the probes wrote it. It kept all six quoted trigger phrases, the 403/429 anti-bot clause and both skip clauses. It cut the Actions: list, "the … binary" and "instead of streaming them into context": 683 to 503 characters.
Arms launched together: before, after and a control with the description replaced by "Helper for repository maintenance tasks.". Opus 5.5 under test with an Opus judge, 3 runs. T1 is the frozen 18-query probe file. T2 is a 16-query held-out set written blind by a separate fresh agent.
Validity: stock run-validity flags every pass for Skill-only trigger denials (fix(evals): stop outcome passes failing validity on a required user-scope read and on forked skills reporting 0 turns #11727). The plan's per-run rules were applied the same way to all three arms. One unclear class was scored both ways: a negative that held, made no Skill call and was denied only withheld tools. The outcome is the same under both readings.
Control: detects. Case drops in T1 (pos-train-01 1.00 to 0.33, pos-train-06, pos-val-02, pos-val-04) and in T2 (pos-01 1.00 to 0.00, pos-07).
After vs before:
- Pooled trigger rate: 42/47 to 42/48 (-1.9 points).
- False-trigger rate: 21/33 to 21/33 (+0.0).
- Both majority rules hold, and no family has a case drop.
- Fire counts match on every query.
Meaning diff: two fresh /docs-hygiene:compress compare labelers.
- One found a SEMANTIC LOSS: the removed Actions: sentence was the only trigger text that named monitor, developer, doctor, search-feedback and credit-usage, and no probe exercises those actions.
- The other reported no loss, but flagged two of the same actions as a residual risk.
- The runbook takes the union of the two (rewrite-a-plugin.md step 6), so main's description stays.
- A rewrite that restores those actions would save about 110 characters, but its text would be unmeasured and would need a new pass launched together with a fresh before arm. That was not run.
Key containment: every run had FIRECRAWL_API_KEY removed and a failing az stub first on PATH. A scan of every kept trace, result file and the branch found 0 key-shaped strings.

Findings

main's description already false-triggers on 64% of VALID should-not-fire probe runs (71% on the frozen set). The rewrite is identical there.
The probe set does not cover five firecrawl actions (monitor, developer, doctor, search-feedback, credit-usage). A cut that drops their trigger words passes the trigger measure. Only the meaning diff caught it.
The frozen probe file's cross-plugin competitors are now valid: since feat(evals): competitor-loaded routing harness so description cuts can be measured #12249, emit-plugin-eval loads competitor plugins (plugins/skill-quality/probes/template.md, "Competitors by purpose"). They are left as they are.

This issue stays open for the setup case shape and the update case. The lane-2 claim is released.

This was generated by AI.

Closed under the owner's Q2 decision, which stops the concise-rewrite series: #6476 (comment). Lane 2's measured description pass shipped nothing new (result above); main keeps its text.

Navigation Menu

Repository navigation

Concise skills rollout: firecrawl (3 skills, 1 model-invoked) #6606

The plugin today

Done when

Eval spend and review

6 remaining items

Decision table

#12102 body control (owner rule #12222)

Description pass

Findings

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Concise skills rollout: firecrawl (3 skills, 1 model-invoked) #6606

Description

The plugin today

Done when

Eval spend and review

Activity

github-actions commented on Oct 7, 2026

kyle-sexton commented on Oct 19, 2026

kyle-sexton commented on Oct 24, 2026

github-actions commented on Oct 25, 2026

6 remaining items

kyle-sexton commented on Oct 27, 2026

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 2, 2026

kyle-sexton commented on Nov 2, 2026

Decision table

#12102 body control (owner rule #12222)

Description pass

Findings

kyle-sexton commented on Nov 2, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions