Skip to content

Define an acceptance criterion once #63

Description

@daniel-kindl

Problem

Two modules parse "an acceptance criterion" from an issue body. They disagree.

lib/agentqueue/prompts.py:51-69 scopes the count to the ## Acceptance heading. It falls
back to the whole body only when that section is absent:

if stripped.startswith("#"):
    inside = "acceptance" in stripped.lower()
    continue
if inside and re.match(r"^[-*] \[[ xX]\]\s+", stripped):
    out.append(re.sub(r"^[-*] \[[ xX]\]\s+", "", stripped))

lib/agentqueue/effort.py:41,300-302 counts every checkbox anywhere in the body:

_ACCEPTANCE_ITEM = re.compile(r"^\s*[-*]\s*\[[ xX]\]", re.MULTILINE)

def acceptance_criteria(body: str) -> int:
    """How many acceptance-criteria items the issue body states."""
    return len(_ACCEPTANCE_ITEM.findall(body or ""))

Reproduction

An issue body with a two-item ## Acceptance section and a five-item ## Out of scope
checklist:

$ cd ~/projects/dk-devkit
$ python3 - <<'EOF'
import sys
sys.path.insert(0, "lib")
from agentqueue import prompts, effort

body = """## Acceptance
- [ ] one
- [ ] two

## Out of scope
- [ ] a
- [ ] b
- [ ] c
- [ ] d
- [ ] e
"""
print("prompts:", prompts.acceptance_criteria(body))
print("effort: ", effort.acceptance_criteria(body))
EOF
prompts: ['one', 'two']
effort:  7

Effect

docs/agentq.md:185 states the rule:

Six or more acceptance criteria in the body make the task hard.

The count of 7 clears the acceptance_hard threshold. The issue routes to the hard tier,
so the run uses Opus and Sol. The escalation comes from five checkboxes that the prompt
never hands to the implementer, so the reason is invisible in the artefact the agent
receives. agentq effort --issue N prints the issue states 7 acceptance criteria for an
issue that states two.

A checklist under a heading such as ## Out of scope, ## Tasks or ## Notes is ordinary
in this repository, so the divergence is reachable with a normal issue.

This issue body is itself an example. prompts.acceptance_criteria reads 7 items from it,
and effort.acceptance_criteria reads 14. The regex has no fenced-code-block exclusion, so
it also counts the checkboxes inside the console block above.

Why no test catches it

Every body in verify/probes/agentqueue-effort.test.py has only an acceptance section, so
the two definitions return the same number in every fixture.

Goal

Define an acceptance criterion once. Let the tier rule and the prompt read the same
definition.

Proposed direction

  • Move the issue-body parsers into one module. referenced_issues belongs with them,
    because it has the same character: a total, deterministic parser over an issue body.
    lib/agentqueue/model.py already holds parsers of that kind, so it is a candidate home.
  • prompts.py and effort.py both read that module.
  • Decide which reading is correct, and state it in docs/agentq.md. Scoping to the
    ## Acceptance section matches what the prompt shows the implementer, so it is the
    reading this repository already acts on.
  • Add adversarial bodies to the test fixtures: a nested checklist, a second checklist under
    another heading, an empty box, and a body with no acceptance section at all.

Acceptance criteria

  • One function answers how many acceptance criteria an issue body states.
  • prompts.py and effort.py read that one function.
  • A body with a two-item acceptance section and a five-item second checklist reports the same count to the tier rule and to the prompt.
  • A test covers a checklist outside the acceptance section.
  • A test covers a body that has no acceptance section.
  • docs/agentq.md states which checkboxes rule 4 counts.
  • ./verify.sh remains green.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething is not working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions