Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .changeset/26515-automation-eval-fixtures.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
'@objectstack/skills': patch
---

fix(skills): objectstack-automation eval fixtures 5, 7 and 8 agree with their own assertions

Clause-②: no
Metadata-protocol: none

- **Before.** Three ideal answers in `evals/flows-triggers-approvals.json` failed the assertions that grade them: eval 5 spelled the negated error-label string inside its answer, eval 7 never spelled the `type: 'position'` approver it requires and spelled the negated positions check, eval 8 never spelled the `type: 'screen'` flow kind it requires. Graded against those assertions, a correct answer scored wrong.
- **Now.** The three ideal answers say each negation in words, eval 7 names `approvers: [{ type: 'position', value: 'finance_manager' }]`, and eval 8 opens with the flow kind. No fixture added or removed; the assertions and the skill's teaching are unchanged.
- No schema, key or behaviour changes.
11 changes: 10 additions & 1 deletion scripts/check-skills-token-ratchet.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -829,7 +829,16 @@ export const CEILINGS = new Map([
// = 2724; headroom was 3 bytes, 0 tokens, so the ceiling moves +297, pinned
// AT the measurement. Same maintainer ruling as the SKILL.md row above
// (2026-10-27, tap A verbatim and untranslated): 「A:先削重复,余额按实测提高」.
['skills/objectstack-automation/evals/flows-triggers-approvals.json', 2724],
// 10893 -> 10934 (#26515): no new fixture — three ideal answers rewritten
// to agree with their own assertions (eval 5 spelled the negated error-label
// string, eval 7 lacked the `type: 'position'` approver and spelled the
// negated positions check, eval 8 lacked `type: 'screen'`). Paid down first:
// evals 5 and 7 shortened by 25 and 7 bytes; the `approvers` line eval 7
// owed costs more than that. +41 bytes -> ceil(10934 / 4) = 2734; headroom
// was 3 bytes, 0 tokens, so the ceiling moves +10, pinned AT the
// measurement. Same maintainer ruling as the SKILL.md row above (2026-10-27,
// tap A verbatim and untranslated): 「A:先削重复,余额按实测提高」.
['skills/objectstack-automation/evals/flows-triggers-approvals.json', 2734],
// 894 -> 1095 (#26179): one regression case for the field permission `when`
// (the contract prompt: lock the legal terms for finance from approval on,
// legal keeps the edit) — expected: `when` on the finance set's entries with
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@
{
"id": 5,
"prompt": "Add a step that calls our `pricing.recalc` function to compute the discount and saves it on the quote, then posts the quote to our tax service over HTTP; if the tax call fails, route to a handler node instead of failing the run.",
"expected_output": "Emits a `script` node with `config.function: 'pricing.recalc'` (registered via `defineStack({ functions })`), `inputs`, and `outputVariable: 'recalc'`; the function is a pure compute step, so a following `update_record` persists it with `fields: { discount: { dialect: 'cel', source: 'recalc.discount' } }`. The `http` node sets `timeoutMs`. Failure routing is a `type: 'fault'` edge — a `label: 'error'` alone routes nothing.",
"expected_output": "Emits a `script` node with `config.function: 'pricing.recalc'` (registered via `defineStack({ functions })`), `inputs`, and `outputVariable: 'recalc'`; a pure compute step, so a following `update_record` persists it with `fields: { discount: { dialect: 'cel', source: 'recalc.discount' } }`. The `http` node sets `timeoutMs`. Failure routing is a `type: 'fault'` edge; a bare error label routes nothing.",
"files": [],
"assertions": {
"must_contain": ["function:", "outputVariable", "type: 'fault'", "timeoutMs", "update_record", "dialect: 'cel'"],
Expand All @@ -64,7 +64,7 @@
{
"id": 7,
"prompt": "An expense approval only holders of the `expense.approve_request` capability may start; route to the `finance_manager` position, and the submitter must never approve their own expense.",
"expected_output": "Flow-level `requiredPermissions: ['expense.approve_request']` (checked at every door that starts the flow, not a `current_user.positions` check inside it); the `approval` node sets `excludeSubmitter: true`, and its `onEmptyApprovers` is anything but `auto_approve`, which is refused beside it.",
"expected_output": "Flow-level `requiredPermissions: ['expense.approve_request']`, checked at every door that starts the flow (never a run-user positions check inside it); the `approval` node names `approvers: [{ type: 'position', value: 'finance_manager' }]`, sets `excludeSubmitter: true`, and its `onEmptyApprovers` is anything but `auto_approve`, refused beside it.",
"files": [],
"assertions": {
"must_contain": ["requiredPermissions", "excludeSubmitter: true", "type: 'position'"],
Expand All @@ -74,7 +74,7 @@
{
"id": 8,
"prompt": "A screen flow: the user uploads the signed contract PDF on the last step, then the flow creates the `contract` record with the file in its `signed_copy` field.",
"expected_output": "A `screen` field `{ name: 'signed_copy', type: 'file', required: true }`; the screen resumes with the uploaded `sys_file` id bound to `signed_copy`, and a `create_record` after the last screen writes `fields: { signed_copy: { dialect: 'cel', source: 'signed_copy' } }`, which claims the file for the record.",
"expected_output": "A `type: 'screen'` flow whose last `screen` carries `{ name: 'signed_copy', type: 'file', required: true }` and resumes with the uploaded `sys_file` id bound to `signed_copy`; a `create_record` after it writes `fields: { signed_copy: { dialect: 'cel', source: 'signed_copy' } }`, which claims the file for the record.",
"files": [],
"assertions": {
"must_contain": ["type: 'screen'", "type: 'file'", "create_record", "signed_copy"],
Expand Down
Loading