Conversation
…eclarable profile set (agentrust-io#116) Verifier behaviour and conformance vectors for the two obligations agentrust-io#116's thread settled as landable without a sponsor: a verifier declares the set of profile URIs it supports and refuses anything outside it, and the verification statement names the profile it ran under. The normative `docs/verification.md` text for those obligations is not here and is not proposed; a maintainer carries it. What a verifier can now declare. `DEFAULT_ACCEPTED_PROFILES` is the rule `spec/trace-v0.2.md` already states, made addressable: accept v0.2, refuse the superseded identifier, never both. `validate.profiles_with_schema()` reads the packaged schemas for their `eat_profile` const and is the ceiling on any declared set, so a verifier cannot declare support for a shape it has no way to check. The superseded schema is read for that const alone and is still never validated against, which is why the exemption in `tests/test_safe_integer_range.py` is rewritten rather than dropped: it rested on a grep for the filename, that proxy became false the moment anything read the directory, and the property it stands for is now measured instead. What the statement carries. `VerificationResult` gains `profile` and `accepted_profiles`. The declared set is there because a declaration that lives only in the verifier's documentation cannot be recovered from a result read years later, which is the condition the issue opens with. The guard, which is the part asked for ahead of more vectors. Nine wrong implementations run against the whole set and the module records what each is caught by, so the figure fails loudly instead of expiring quietly the next time a gate lands over the same inputs. Measured here: the set separates 4 of its 10 vectors against a verifier implementing none of the obligations. One recorded shortfall stands, a verifier satisfying the statement obligation with a literal, uncatchable while exactly one profile is admissible in this build. Found while writing this and fixed here: `accepted_profiles` was materialised with `tuple()`, so a bare profile URI iterated as characters and was refused for naming each of them as an unschemaed profile, the right verdict for the wrong reason, and an `int` or `None` left `verify_record` raising an undocumented `TypeError`. Same shape as the `trusted_key_identifiers` case closed in `revocation.py`, and found the same way, by the keyword-argument sweep once the parameter was swept. 1638 passed, 1 skipped; ruff and mypy clean. Signed-off-by: LouieLuNZ <48041247+lywinged@users.noreply.github.com>
|
❔ Contributor Check: UNKNOWN
Automated check by AgenTrust Contributor Check. |
`tools/check_dashes.py` is a CI step and this branch did not run it, so three dashes carried in from the fork went out with the pull request: one in `verify_record`'s docstring and two in `test_sign.py`. Replaced in the house style, a full stop in two cases and a comma in the third. No behaviour changes and no test changes. All five CI steps now run clean locally, with the same arguments the workflow uses: ruff, the dash check, mypy on `src/agentrust_trace`, and pytest with coverage. 1638 passed, 1 skipped. Signed-off-by: LouieLuNZ <48041247+lywinged@users.noreply.github.com>
The documentation moved its example model from a real provider and model id to `example-provider` / `example-model-1`, and these fixtures still carried the old pair because they were written against the adapter, which that change did not touch. Regenerated rather than edited: the records are signed, so a hand edit to a signed field would have left ten invalid signatures behind. The generator carries the new values and re-signs; `test_every_fixture_signature_is_genuine` passes on the result. Fixture data only. Every figure recomputed and none moved: ten vectors, four separating (01, 04, 06, 09), six inert, nine panel rows, one wrong implementation no vector catches. All five CI steps clean, 1638 passed and 1 skipped. Signed-off-by: LouieLuNZ <48041247+lywinged@users.noreply.github.com>
imran-siddique
left a comment
There was a problem hiding this comment.
On the two open questions:
- Make
failureinformative for portable conformance. Keep tests for this library’s diagnostic messages separately. A verifier applying the agreed rules and returning a generic refusal should pass the portable suite. Add that control and remeasure the separation figures; the current panel records three affected vectors while the README says four. - Move vectors 03 and 08 into the cutover’s own coverage, following the same ownership decision as vector 10. Preserve their coverage.
Please also resolve the changelog conflict against current main, regenerate the fixtures, and rerun the checks. The current head passes 200 focused tests and hosted CI.
Could you update this by September 18?
Resolves the CHANGELOG conflict under Unreleased/Added by keeping all three entries: this branch's, and the two that landed on main while it was in review. Signed-off-by: LouieLuNZ <48041247+lywinged@users.noreply.github.com>
… keeps its own coverage Both changes were asked for in review on the pull request. `expected.failure` is informative. The portable adapter asserts the verdict and the statement and says nothing about which rule refused, because the draft text for agentrust-io#116 asks a verifier to report refusal-for-an-unimplemented-profile distinguishably from a verification failure, which is coarser than a rule name and is a SHOULD. Asserting the label was this set asking more of a foreign implementation than the text it encodes. Two controls hold it open: test_a_generic_refusal_passes_this_set runs every vector through a verifier that applies every rule and names none of them, and test_a_generic_refusal_is_separated_by_nothing does the same at the panel. Both fail if an assertion on the cause comes back, measured by putting one back. This library's refusal messages are still worth pinning and are now pinned separately, in tests/test_verifier_compatibility_diagnostics.py, which says in its first paragraph that nothing in it is a conformance requirement. The panel loses the near miss that only existed because the comparison looked at the cause, and gains it back as a control. One recorded figure moves with that revert: the membership reading no longer fails vector 06, which it refused for the right verdict under a different rule, and a different rule is not a wrong answer any more. Vectors 03 and 08 are retired into the cutover's own coverage, the same ownership decision that moved vector 10: the rule they pin is merged normative text, not an obligation this issue proposes, and coverage inside a proposal's set becomes unowned when the proposal closes. Their ids are retired rather than reused, and the explicit configurations they stated are pinned in tests/test_sign.py beside the cutover's other tests. Coverage preserved and measured rather than asserted: removing the record-side guard fails four tests there and removing the set-side guard fails three, and the new test fails under both. The set is eight vectors and still separates four, so the two that left were inert. The rules carried by a single vector drop from three to one. Also merges current main, resolving the changelog conflict by keeping all three Unreleased entries, and regenerates the vectors from the generator. ruff, the dash check and mypy clean; 1715 passed and 6 skipped. Signed-off-by: LouieLuNZ <48041247+lywinged@users.noreply.github.com>
…sserts the label A cold read of the reply this branch is about to be handed back with found that the change announces one thing and the files say another, in the module a reviewer opens to check it. Everything here is prose and comments except one control and one guard, and it is all the same defect: text that was true before the review. `_v_wrong_reason`'s docstring still said the vector set is stricter than the text it encodes, that the adapter asserts `expected.failure`, and that resolving it is the maintainer's call. It was resolved, by the review, in the direction that docstring named as one of the two options. It now says so and says why this verifier sits outside the panel. The counts that moved with the retirement and were left behind: ten near misses where the panel holds eight, 4 of 10, all ten vectors, vectors 04 09 and 10 where 10 left the set two days ago, three thin rules where the table shows two, eleven correctly signed records where there are eight, and a 932-test figure from a suite that is now 1717. The new control's own docstring carried a number from the wrong instrument: it said four vectors refused the generic verifier, which was the panel's figure. Measured at the adapter, which is what that control runs, it is every one of the seven refusal vectors. Re-run, not recalled. The diagnostics module carried markers and a lambda for two labels no vector uses any more, left behind when 03 and 08 were retired. Both are gone, and the test that checked every used label has a marker now checks the other direction too, which is the one that went quiet. The README told an adapter author to look in `FAILURE_MARKERS`, which is no longer in the adapter, and claimed every signature is asserted independently. Four of the eight records are re-verified; the other four are refused by the record schema before the signature is read, and the file that two places sent the reader to for them has never existed in this repository. The README now says what is covered, and the two pointers are gone rather than left promising a file. The retired-vectors test had a control whose comment described a call it did not make. The call is there now, and the control below it says what it controls for. Also merges main again, which had moved one commit further. Signed-off-by: LouieLuNZ <48041247+lywinged@users.noreply.github.com>
… third Three assertions survive in test_nothing_validates_a_record_against_the_superseded_schema and the comment recording what was taken out called it the third. The pull request description had it right and the file had it wrong, which is the wrong way round. Signed-off-by: LouieLuNZ <48041247+lywinged@users.noreply.github.com>
… unique to it" was false Checking the pull request's quotations against issue 116 turned up one more claim of the same shape as the rest, in the same module: the docstring said vector 09 had nothing unique to it while 09 has had the "declared set checked, last member only" column to itself throughout. Every one of the four live vectors is the sole entry in some column, so on this measurement none of them is deletable. Written into a test rather than into the paragraph that was wrong, because that paragraph is the third claim in this module to have gone stale without anything noticing. test_no_live_vector_is_redundant fails in both directions: a vector that stops being anyone's only catch, and a vector that becomes one. Signed-off-by: LouieLuNZ <48041247+lywinged@users.noreply.github.com>
…ger here A wider grep than the last one, over every file the review's commits touch rather than the five a private checklist happened to name. `gen_vectors.py` told a reader of vector 04 to "see vector 08" for why the v0.1 identifier is excluded for a different reason. 08 was deleted by the same change. That file is published with the set, so the pointer breaks in somebody else's hands rather than in ours; it now names the cutover and where its coverage lives. The separation module described 10 and 08 in the present tense as one separating and the other not, written when 10 had left and 08 had not. Both have, and the paragraph says so. `test_the_published_artifacts_name_no_vector_that_is_not_here` is the guard: every vector id the README or the generator names has to be a file in the directory. History belongs in the test modules, which are read by us; these two describe the set as it stands. Proven by putting "see vector 08" back and watching it fail. The fixture for 04 is regenerated, since the text that changed is inside it, and test_generator_reproduces_its_committed_fixtures is what caught that rather than a reading of the diff. Signed-off-by: LouieLuNZ <48041247+lywinged@users.noreply.github.com>
…f one per defect Three defects of the same shape shipped inside this change, and each was answered with a guard aimed at that one shape: a constant that had moved modules, a test file that never existed, a vector id that had been deleted. This is their general form, so the fourth does not need its own. test_every_name_this_set_cites_resolves recovers the citations from the files rather than from a list: every repository path, every backticked test name and every backticked constant the set's own surface names has to exist. Proven three ways, by pointing a path, a test name and a constant at something that is not there. The first thing it caught was a false positive of its own making. A repository name ending in one of the path words, followed by a slash, yields a match starting mid-word, and the comment explaining that spelled the bad match out, so the guard tripped on its own explanation. The lookbehind fixes the class and the comment is worded rather than quoted, because a guard whose explanation trips it teaches the reader to weaken it. The two guards added earlier were mutated to check they still guard: an unused marker in the diagnostics table fails its test, and giving vector 09's column a second entry fails the redundancy test. Signed-off-by: LouieLuNZ <48041247+lywinged@users.noreply.github.com>
…ve label once A reader grepping the portable adapter for `expected.failure` finds one use and could reasonably conclude the contract is looser than the module's own docstring claims. It is test_the_precondition_check_fires_and_covers_every_vector_that_needs_one, which uses the label to derive which fixtures must declare a premise: an assertion about the fixtures, which no run of any implementation can change. Said where that reader lands rather than in a reply they will not have. Signed-off-by: LouieLuNZ <48041247+lywinged@users.noreply.github.com>
|
Done, on
On the four and the three, both numbers were wrong for where they sat, and the disagreement was 03 and 08 are retired into the cutover's own coverage, ids retired rather than reused, following The set is eight and still separates four, so the two that left were inert, and the rules carried
Six further passes found the files still saying the old thing, and that is worth naming The ones a test can recognise are pinned by one now, since a paragraph is what failed each time. The counts and that docstring are pinned by nothing, and I would rather say so than imply the Seven commits are mine: One thing I have added to the description rather than assumed: the The description carries the same figures. |
Part of #116, and deliberately not closing it.
This lands the half the ruling released. The other half stays with that issue: "Held: the
normative
verification.mdtext for obligations 2 and 3 ...I carry it." Closing on merge would leave the held text, the deferral of obligations 1 and 4, and
the
policy: take-internallabel sitting on a closed issue with nothing tracking them, which isthe same shape as the reason row 4 was taken out of this set: coverage that becomes unowned when
the issue closes. So there is no closing keyword here, and the issue stays open for the half its
author carries.
What this changes
Verifier behaviour and conformance vectors for obligations 2 and 3 of #116, quoted from that
issue so this page stands on its own:
These are the two the issue thread settled as landable without a sponsor. Obligations 1 and 4
are deferred there; the ruling says only that they stay deferred, and the reason developed
earlier in the thread was that no vector written against the current schema can separate them
from the
eat_profileconst.No normative text is proposed here. The draft
docs/verification.mdsection for theseobligations is deliberately not in this branch. It is on
lywinged/trace-specwork/116-proposal-text, one file off this repository'smain, for the maintainer who said he carries it, to takeor discard. The enumeration of three grounds the ruling said it would keep in whatever text lands
is in that file rather than here, under "Declared support" and "Statement content", because that
is the text it was asked to land in. Nothing in it is proposed upstream by this pull request.
The diff
Twelve commits, nine of them mine. The first three were the branch as opened, rebased onto
maineach of the three times it moved while it was being prepared. Then
mainmerged in rather thanrebased onto, twice, because it moved again while the review was in hand, so nothing of mine has
moved underneath what has already been read. Then the review's own changes:
67c3ef1is what itasked for, and
6f63406,e8b1c45,97232b7,591dee4,412dfedand963b392are passes overprose that still described the behaviour before it, with the guards that came out of them. The changelog conflict is
resolved in the first merge,
7803701, by keeping all three entries under Unreleased. Every figurebelow is recomputed on the current base rather than carried over.
23 files, +3110 −52.
The ruling released three things: the guard, the table and the vectors. All three are here.
The guard is
tests/test_verifier_compatibility_separation.py. The vectors are the eight fixtures.The table is the one in
examples/verifier-compatibility/README.md, one row per fixture givingthe outcome and the rule it pins, which is the five-row table from the issue rewritten
against what was committed rather than against what was proposed, minus the row the ruling took
out. It runs to eight rows because it lists every fixture in the directory rather than only the
rows that were proposed; the row the ruling took out was vector 10, and 03 and 08 have since
followed it to the cutover's own coverage.
examples/verifier-compatibility/(eight fixtures, README with the table, generator),tests/test_verifier_compatibility_fixtures.py,tests/test_verifier_compatibility_separation.py,tests/test_verifier_compatibility_diagnostics.pysrc/agentrust_trace/sign.py,validate.py,revocation.py,__init__.pytests/test_sign.py,test_adequacy_all_sets.py,test_public_functions_raise_what_they_document.py,test_safe_integer_range.py,CHANGELOG.mdWhat it adds
DEFAULT_ACCEPTED_PROFILESis exported rather than left as a private default becauseobligation 2 requires a verifier to declare the set it supports, and a caller cannot declare
what it cannot name. It makes addressable a rule
spec/trace-v0.2.mdalready states under"Changes from v0.1", quoted because the whole non-breaking claim rests on it:
verify_record()takes anaccepted_profilesset, refuses a record whoseeat_profileisoutside it, and refuses the configuration itself where the declared set names the superseded
identifier or a profile this build carries no schema for.
validate.profiles_with_schema()reads the packaged schemas for their
eat_profileconst and is that second ceiling.VerificationResultgainsprofileandaccepted_profiles, both defaulted: a declarationthat lives only in the verifier's documentation cannot be recovered from a result read years
later, which is the condition the issue opens with.
One process note
require-maintainer-approval.ymlreturns early when the author is in itsMAINTAINERSlist, andthis account is in it, so the
gatecheck passes on this pull request by skipping rather than byanyone approving. It can merge on CI alone. I am not merging it: it is mine, it carries a
changes-requested review, and the order the next section states, that verifier behaviour and
vectors land while the normative text follows with a maintainer carrying it, is a maintainer's to
agree rather than mine to assume.
Type of change
Neither box fits, and the template has none for code and tests, so both are ticked and the
position is stated instead.
Non-breaking is exact: no signature, no schema, no wire format, and no caller has to
change.
verify_record()gains a parameter and it is defaulted. Every call site in this repository keeps its behaviour, becauseverify_record()already accepted the v0.2 profile alone; a caller outside it that passesnothing sees no change for the same reason.
"Non-normative" needs qualifying rather than claiming. No RFC 2119 keyword is added to any
document under
spec/, and none is added todocs/. But a conformance vector set is what implementers getmeasured against, and this one still encodes the stronger reading of obligation 2: a declared
set naming an unusable profile is refused, which the draft text does not say in those words. It
was stricter in a second way until the review, in asserting which rule refused, and that half is
retracted:
failureis informative and nothing portable reads it. It is normative in effect andinformative in status, which is the order the ruling set deliberately: verifier behaviour and vectors land, the text follows
with a maintainer carrying it. Merging this is agreeing to that order, not only to the code.
Spec section
None modified. The behaviour implements the cutover paragraph quoted above, which is at line 24
of
spec/trace-v0.2.mdunder "Changes from v0.1", not inside a numbered section. The gap thevectors point at is in
docs/verification.md, which this change does not edit.The separation guard
The part the thread asked for ahead of more vectors. Eight wrong implementations run against the
whole vector set and the module records what each one is caught by, so a figure that stops
being true fails loudly rather than expiring quietly.
The failure it exists to catch already happened once here: the previous version of this set
stopped separating when #156 made
verify_record()validate against the packaged v0.2 schema,a second gate over the same inputs, and nothing inside the set could see it because its own
tests run the real verifier and compare verdicts, and the verdicts did not move. That is a
different failure from the one the issue opens with, verifiers accepting artifacts under
semantics they do not implement; this one is the measurement of that failure going stale.
Measured on this branch:
The effective set is four. The other four carry records whose profile is not v0.2, so the schema
refuses them before any profile rule runs, and they demonstrate nothing about these obligations
today. They are kept because they become live the day both gates come off, and deleting them
now would mean rediscovering them then; they are not counted as evidence that the set tests
anything.
What no vector catches, recorded with its expiry rather than left implicit: a verifier that
satisfies the statement obligation by writing the profile in as a literal. The argument that no
vector written against this build can catch it is a claim about the vectors this build admits,
not about vectors in general: the build packages a schema for two profiles and a conformant
declared set must exclude the superseded one, so exactly one profile can appear in a conformant
statement, and a literal and an observation print the same string. Varying the declared set does
not help, because the set that would distinguish them is the non-conformant one. The entry
expires when a second admissible profile schema ships, and a test asserts that condition
directly.
One refusal rule is carried by a single vector, against #124's standard of at least two
independent vectors per rule:
no_accepted_profiles, by 06. It was three until the review;retiring 03 and 08 into the cutover's coverage took the other two with them, which is a better
record than the one this page carried, and it is measured rather than claimed:
KNOWN_THINrecords the one that is left, and the adequacy test fails if the measured set of thin rules stops
matching it in either direction. Obligation 3's statement rule is carried by one vector too, 01,
and will not appear in that table: the framework counts refusal boundaries and 01 is the set's
only positive vector.
The panel is not independent of the vectors. It was written after them by the same hand, so
it agrees with them by construction, which is exactly the caveat #124 makes about a vector set
and the checker that runs it. A reader who stops at the table above should not take the guard
for an outside instrument. The cross-run that issue proposes, a second set through a verifier
its author did not write, is what would fix it, and nothing here substitutes for it.
A defect introduced here and fixed here
accepted_profileswas materialised withtuple(). A bare profile URI passed instead of asequence iterated as characters and was refused for naming each of them as a profile this build
carries no schema for: the right verdict reached for the wrong reason. An
intorNoneleftverify_record()raisingTypeError, which its docstring does not list. Same shape as thetrusted_key_identifierscase closed inrevocation.pyby #325, and found the same way, bythat sweep once this parameter was swept.
The contract now: anything that is not an iterable of
str, and anything iterable whose membersare not all
str, raisesValueErrornaming the type it got, which is what the docstringalready promises for every other rejection. A bare
stris refused as a type rather thaniterated. Six cases pin it: a bare string, an
int,None, adict,bytes, and a listwhose members are not strings, plus a separate control that a real tuple gets past the shape
check and is refused for what it contains.
The question this diff asked, and the answer it was given
The ruling on the v0.1 row was that a vector for the cutover tests an existing rule rather than
an obligation this issue proposes, and that when the issue closes the coverage becomes unowned.
That vector went out, its id retired rather than reused, and its rule got a test in
tests/test_sign.pybeside the cutover's own.This page asked whether the same reasoning reaches 03, which refuses a record carrying the
superseded identifier, and 08, which refuses a verifier declaring both. It does, and the review
said so. Both are retired the same way, ids retired rather than reused, with the explicit
configuration each one stated pinned in
tests/test_sign.py. Coverage preserved is measuredrather than asserted: removing the record-side guard fails four tests there, removing the
set-side guard fails three, and the test that carries both retired configurations fails under
either mutation.
Both were inert, so the set still separates four of what are now eight vectors, and two of the
three single-vector rules left with them.
Two changes in files this issue did not ask for
The superseded schema's exemption in
tests/test_safe_integer_range.pyis rewritten. Itexempted
trace-v0.1.jsonfrom the integer-bound rule on the ground that nothing loads it, andchecked that by asserting no Python file mentions the filename.
profiles_with_schema()readsevery packaged schema's
eat_profileconst, so that proxy became false.The third option is worth naming because it is the one that would have avoided touching the
file: have
profiles_with_schema()skip the superseded schema. It does not work. The ceiling iswhat makes a declared set naming an unusable profile refusable, and if the superseded identifier
were outside the ceiling, a set naming it would be refused as a profile this build cannot check
rather than as the cutover violation it is. The right verdict for the wrong reason, which is the
defect class this set exists to separate.
So the exemption's reason is rewritten to the property it actually rests on, that no record is
ever validated against the file, and three assertions in the test now measure it rather than
proxy it: the profile is in the ceiling
profiles_with_schema()returns, the live validator'seat_profileconst is the current profile and not this one, and that reader returns stringsrather than schema objects. The alternative was to stop naming the file in a docstring, which
would have dodged the grep and left a reason standing that is no longer true.
An assertion was removed from that same test. A fourth leg asserted that a record carrying
the superseded profile is rejected on the
eat_profilepath. Three mutations were run againstit: dropping the live schema's const, setting it to the superseded identifier, and setting the
superseded file's const to the live one. All three fail an earlier assertion in the same test
first, so no positive control reaches the fourth, and it reported nothing. It is recorded in a
comment where it stood rather than kept. This is a coverage deletion in a file this issue did
not ask us to touch, decided on the above and open to being reversed.
_TRACE_PROFILE_V0_1stays private. Its comment says no caller should be able to spell itwithout reading the file. This change does not export it: the tests import the private name
in-repo. The fixtures do print the identifier as a JSON literal, which is spelling it, and there
is no way to publish a vector for a rule about an identifier without naming the identifier. The
tension is real and resolved in favour of not widening the public API.
What the review settled
expected.failureis informative. The portable adapter asserts the verdict and the statement andnothing about which rule refused, because the draft text asks for a coarser distinction and asks
for it as a SHOULD, so asserting a rule name was this set asking more of a foreign implementation
than the text it encodes. Two controls hold it open,
test_a_generic_refusal_passes_this_setinthe adapter and
test_a_generic_refusal_is_separated_by_nothingin the separation module, and bothwere checked by putting an assertion on the cause back and watching them fail. This library's own
refusal messages are pinned in
tests/test_verifier_compatibility_diagnostics.py, whose docstringcloses by saying that nothing in that file may be read as a conformance requirement.
Still open
agentrust-io/trace-testsat660cf84contains no reference to these vectors or theirfailure names, so removing one breaks nothing there. That is the one consumer measured; no
claim is made about any other.
Checklist
git commit -s)CHANGELOG.mdupdated (for any normative change)<!-- CHANGED: #NNN: description -->in spec textThe changelog entry is under Added, written because the public API grows even though no
normative text changes. The last two do not apply: nothing breaks.
1720 passed and 6 skipped:
test_versiondeclining to compare an installed distributionthat is not this source tree, and five diagnostics cases whose refusal is not a complaint
about a member of the declared set.
ruff check src tests scripts,tools/check_dashes.pyandmypy src/agentrust_trace, which are the scopes CI runs, are clean.