fix: ground public scoring claims in the current measurement, generate NOTICE mechanically - #460
Merged
runyourempire merged 3 commits intoAug 15, 2026
Conversation
The published "92% rejection / 98% noise accuracy" figures were real when d956d46 measured them (2026-04-26), but nothing regenerated or enforced them afterwards, and three things had drifted since: - The corpus grew from 215 to 245 labeled items, so "215 labeled items" was stale everywhere it appeared. - README said the claims were "tested, not asserted" and pointed at `cargo test scoring::benchmark`. That suite has 2 profiles, not 9 personas, and asserts avg precision/rejection >= 0.75. The 9-persona numbers come from `scoring::simulation`, and what it asserts are floors (aggregate precision >= 0.70, F1 >= 0.40, >= 80% noise rejection per persona) — not the headline percentages. Nothing failed the build if rejection fell from 92% to 81%. - de/es/fr/tr locales were missed by d956d46 entirely and still shipped the unmeasured "over 99% rejection" claim it was written to remove. Re-ran the suite on the current pipeline: 245 items x 9 personas = 1,997 scored evaluations, TP 119 / FP 19 / TN 1,646 / FN 213, giving 93.1% rejection, 98.9% noise accuracy, 86.2% precision, 35.8% blended recall (71.3% on strongly-relevant items). Every surface now carries those numbers, and the long-form docs carry the confusion matrix, the command that regenerates them, and an explicit statement that CI enforces the floors rather than the headline. Also corrected specs/ARCHITECTURE.md's unbacked "99.9% rejection rate", and repointed 9 locales from "shown in the Evidence tab" to the Signal tab — Evidence is in the removed-views list, so that string sent users to a tab that no longer exists. No scoring code touched: the claims were wrong, not the thresholds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AUeKTKwNmdow8yUk3q8RB2
NOTICE was hand-maintained at 92 direct dependencies against a graph of 828 shipping Rust crates and 42 production npm packages, and it had drifted the way hand-maintained inventories always do: - ts-rs listed as 10, lockfile resolves 12.0.1 - scraper listed as "0.23 - MIT", actually 0.27.0 and ISC-licensed - chacha20poly1305 listed as 0.10, actually 0.11.0 - ammonia 4.1.4 absent entirely - all 8 shipping MPL-2.0 crates unattributed (cssparser x2, cssparser-macros x2, selectors x2, dtoa-short, option-ext) even though MPL-2.0 s3.2 requires notice - both OFL-1.1 fonts unattributed, and the OFL text — which the licence requires to travel with the font software — reached no build output scripts/generate-notice.cjs now derives NOTICE from `cargo metadata` (non-dev closure: normal + build edges, all platforms) and `pnpm licenses list --prod`. It emits a dedicated reciprocal-licence section with a source-availability statement, embeds the full OFL text for each bundled font, and hard-fails on a strong-copyleft dependency or an OFL package missing its licence file. Dual licences with a permissive branch are not misclassified — r-efi (MIT OR Apache-2.0 OR LGPL-2.1-or-later) neither blocks nor lands in the reciprocal section. `--check` compares the committed file against a fresh render. It runs in the repo-guards job, which is the only CI job with no `if:` — a Rust-only dependency bump therefore cannot slip past it. That costs the job a Rust toolchain and a dependency install; the alternative was a gate that skipped on exactly the PRs that move dependencies. `--require` makes a missing toolchain a failure there so the gate cannot pass by skipping. 19 self-tests added to test:scripts, per the convention that every guard verifies its own claims. Licence compatibility itself was already clean: no GPL/AGPL/SSPL anywhere in the graph. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AUeKTKwNmdow8yUk3q8RB2
main's quick-xml advisory fix bumped calamine 0.25 -> 0.36.1, zip -> 8.6.0, quick-xml (3 versions) -> 0.41.0, docx-rs 0.4.20 -> 0.4.22 and plist 1.9 -> 1.10, pulling in atoi_simd, debug_unsafe, fast-float2 and typed-path. 828 -> 832 shipping crates. The generator caught this on its first CI run: NOTICE was rendered from the pre-rebase lockfile and `--check` failed against the merge result. That is the gate doing its job — under the old hand-maintained file this dependency bump would have silently desynced the attribution users see in the Third-Party Licenses modal, exactly as the previous four version errors did. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AUeKTKwNmdow8yUk3q8RB2
runyourempire
force-pushed
the
fix/truthful-scoring-claims-and-generated-notice
branch
from
August 15, 2026 15:32
59a5a67 to
733c636
Compare
runyourempire
enabled auto-merge (squash)
August 15, 2026 15:51
runyourempire
deleted the
fix/truthful-scoring-claims-and-generated-notice
branch
August 15, 2026 17:03
runyourempire
added a commit
that referenced
this pull request
Aug 16, 2026
…nt main, replaces #419) (#469) ## AD-030: retire the "gets sharper every day" promise — rebuilt onto current main Replaces **#419**, which was CONFLICTING, 31 commits behind, with CI dead since 08-11 (self-hosted runner timeouts). Rather than rebase a 46k-line diff dominated by whole-file locale churn, this PR **merges #419's branch onto current `main`** and resolves every conflict in favour of main's newer facts. Net diff: **62 files, +848/−677** (vs #419's +23,646/−22,838). Operator decision on record (2026-08-14): the promise is completely retired, on every surface. Decision record: `.ai/DECISIONS.md` **AD-030** (in this PR). ### What this ships - **New canonical line** on every copy surface: *"4DA reads the internet for developers — privately, locally. Your codebase decides what's relevant."* — README, site (title/meta/OG/JSON-LD/hero/FAQ), docs, scan page, streets module, writing footers, installer metadata (`tauri.conf.json`), mcp README, RELEASE-NOTES. - **"Yesterday's noise becomes tomorrow's signal" KEPT** — re-attributed to corpus re-judging (verdict epochs), never to engagement. - **13 locales, surgically**: #419's translations ported **per-leaf** onto main's current files — 312 rewordings + 130 deletions of dead affinity-surface keys. Its 752 key *additions* were dropped: they belong to surfaces #421 already deleted from `en` (momentum, wisdom, comparison, toolkit…), and porting them would have resurrected orphans. `validate-translations`: 0 errors, warnings identical to main. - **v19.1 hotfix**: auto-detected `anti_topics` removed from both `build_negative_stack` call sites — the last structurally-live behavioral scoring path. No `PIPELINE_VERSION` bump; justification (live probe: `anti_topics` = 0 rows → byte-identical priors) recorded at the constant. - **Enforcement gate**: `scripts/check-retired-claims.cjs` + 9 tests. Violations on main measured before: **57 across 24 files**; after this PR: **2, both in CLAUDE.md** (see residuals). - **Public handling**: `site/src/writing/retiring-a-claim-we-could-not-measure.njk` — the engineering record, now linked from /writing and the sitemap. Site builds clean (11ty, 127 files). ### Conflict-resolution calls worth reviewing - Kept main's newer copy where it was already promise-free and more current: the 93%/98.9%/245-item benchmark (#460), Precision Ledger references, BYOK never-paywalled rationale. - **Fixed 3 fact regressions #419 would have reintroduced**: stale 92%/98%/215 benchmark figures (2 files) and a RELEASE-NOTES tier table listing Developer DNA / Signal Chains / Score Autopsy as paid (they are free per AD-025/026). - `apply-translations.cjs`, `missing-keys.json`, `BadgeRow.tsx` stay deleted (deleted on main since #419's base). ### Residuals (tracked in `.claude/plans/PENDING-DECISION.md`) 1. **`CLAUDE.md` tagline + `package.json` gate wiring** — both files are claimed by live peer lanes (#466; agent-a875). A follow-up commit on this branch lands the moment the claims release. Until then the gate exists but is not yet wired into `validate`, so CI on this PR does not self-fail on CLAUDE.md. 2. **npm README** — `@4da/mcp-server` live README still serves the old table; needs a republish (different terminal, tag `mcp-v5.0.0`). 3. **Live 4da.ai** — CF Pages is direct-upload; needs `wrangler pages deploy --branch main` after merge. 4. **Close #419** after this merges. ### Verification - Frontend: **1249/1249** pass, `tsc --noEmit` clean, ESLint clean, i18n types regenerate in sync. - Rust: `cargo check --lib` clean; targeted tests (negative_stack / context / blind_spots) green. - Gate self-test: 9/9; full-tree scan = only the 2 deferred CLAUDE.md hits. - Site: eleventy build clean, post renders. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01MMarGhXjbKyNJzsm3JG1jw --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
runyourempire
added a commit
that referenced
this pull request
Aug 17, 2026
…rod dep bump past its own gate `generate-notice.cjs --check` fails on main: `@tanstack/react-virtual` moved 3.14.5 -> 3.14.9 (and `@tanstack/virtual-core` 3.17.3 -> 3.17.7) without NOTICE being regenerated. Both are in the `--prod` closure, so they belong in the attribution file. Not this branch's doing, but it blocks this branch — and everyone else's, since the NOTICE step lives in `repo-guards`, which has no path filter. HOW IT GOT PAST THE GATE, because the mechanism matters more than the two lines: #460 added the NOTICE step to repo-guards 2026-08-15 17:03Z #450 (@tanstack/react-virtual) MERGED 2026-08-17 13:39Z #450's repo-guards: pass, in 13 SECONDS, on run 31889094566 Thirteen seconds cannot contain a Rust toolchain install, `pnpm install --frozen-lockfile`, `cargo metadata` and `pnpm licenses list`. That run was created before #460 existed, so it validated against a workflow with no NOTICE step — and then auto-merge landed it two days later on that stale green without re-running anything. The ruleset sets `strict_required_status_checks_policy: false`, so "up to date with base" is not required before merging. A PR can therefore merge on checks that passed days earlier against a different workflow, and a gate added in the meantime never sees it. The gate did not fail here; it was never invoked, and the next unrelated PR inherits the breakage and the blame. Regenerating is the instance fix. The class fix is enabling strict required status checks, which is a repository setting and an operator action. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
runyourempire
added a commit
that referenced
this pull request
Aug 18, 2026
…ce key (#485) ## A browser tab was the only thing delivering a paid licence key Buying Signal minted a key into Stripe customer metadata and **emailed nothing**. The only way a customer ever saw it was `/signal/success` rendering it — and that page polls the webhook 4 times over ~8 seconds before giving up. Close the tab inside that window, or hit a slow webhook, and the key existed but the buyer never had it. The advertised fallback was worse than absent. The **"Email me my key"** button on that page called an endpoint answering **503**, because `RESEND_API_KEY` / `RESEND_FROM_EMAIL` were never set on the Pages project. Verified against production: it has `STRIPE_SECRET_KEY`, `STRIPE_WEBHOOK_SECRET`, `LICENSE_PRIVATE_KEY_HEX`, `SITE_URL` and the three price IDs — and neither `RESEND_*`. That 503 is *correct code*. #427 closed a real vulnerability — the endpoint used to return any address's key to anyone who asked — and deliberately refuses to fall back to returning the key. But closing the hole replaced a working-if-unsafe recovery path with one that was never switched on, and nothing reported the difference. ## What changes **1 · The key is emailed at purchase and renewal.** The success page becomes a convenience instead of the delivery mechanism. It **never throws**. This runs off a paid Stripe webhook and `generateAndStoreLicense` is not idempotent: a throw returns non-2xx, Stripe retries, and the retry **mints a second valid key** with a fresh expiry. A failed email must degrade to "use the page or recovery", never to "issue another entitlement". Awaited rather than backgrounded precisely *because* it cannot throw, so the log line is guaranteed written before we answer 200. An unprovisioned mailer logs at **error level on every single sale** rather than skipping silently. That is the whole lesson of the 503: correct behaviour nobody can see is indistinguishable from no behaviour at all. **2 · Renewals say so.** A renewal silently replaces the key the customer is holding — the old one dies at its original expiry. Previously the first they learned of it was the app rejecting their key. **3 · The footer matches why the mail arrived.** There was one shared footer reading *"someone asked 4DA to recover the licence for this address"*, which on a purchase confirmation lands as an account-compromise warning at the happiest moment of the funnel. Purchase / renewal / recovery now each say the true thing, with matching subjects. **4 · `streets_*` metadata keys become `signal_*`.** STREETS was retired in June 2026, but every entitlement key still carried its name — so the *live* licensing path read as a dead feature. That is not hypothetical: this endpoint was believed retired for exactly that reason, and its 503 was read as expected behaviour of something switched off. Reads accept **either** prefix, preferring `signal_`, so no customer record is orphaned and no migration is needed. Contained entirely to `site/` — nothing in `src-tauri/`, `src/` or `mcp-4da-server/` ever read these keys. **5 · `scripts/check-pages-secrets.cjs` — a gate for the class, not the instance.** A missing runtime variable is invisible to every test (it isn't a code property) and to every CI job (CI can't read a Pages secret). The only thing that can catch it is asking the live project, so that's what this does — names only; Cloudflare never returns values. *"Could not check" exits 2, never 0.* Conflating unknown with fine is the same defect the script exists to catch. Run against production it currently **exits 1** and names both missing variables with the fix command. Deliberately **not** in `validate`/CI — it needs a token, so there it would be permanently inconclusive, which would either fail every build or get read as a pass. It's `pnpm run ops:pages-secrets`. ## The delivery fix had this bug in it, twice over **A `Date` expiry would have sent no email at all.** `buildLicenseEmail` did `expiresAt.slice(0, 10)`, but the webhook path holds a `Date` — `generateAndStoreLicense` returns `new Date(...)` — while only the recovery path reads an ISO string out of Stripe metadata. `.slice()` on a `Date` throws, and the no-throw contract above would have caught that throw and degraded it to a logged `'error'`. So the headline path of this PR would have silently sent **nothing**, in exactly the failure mode the PR exists to end, and the tests written first would not have caught it because they all passed strings. `formatExpiry()` now normalises both shapes, and a junk expiry omits the line rather than costing the buyer their key. The first version of the maintenance-timestamp fix in this same batch had the mirror-image defect — `unwrap_or(0)` would have written "never ran" instead of failing to write — which is why both are called out rather than quietly corrected. ## The activate button did nothing, in the most-used mail client there is Real delivered mail proved it: the email linked straight to `4da://activate?key=...`, and **Gmail removes custom-scheme hrefs outright** — browser and mobile apps alike. The button rendered as a button and carried no href at all. Nothing was wrong with the app. The `4da` scheme is declared in `tauri.conf.json`, the plugin is initialised in `lib.rs`, `app_setup.rs` validates and emits `deep-link-activate`, and on a machine with 4DA installed the handler is registered correctly (`HKCU\Software\Classes\4da` → `fourda.exe "%1"`, `URL Protocol` set). The link simply never reached the OS. The email now points at **`https://4da.ai/activate`**, which no mail client sanitises, and that page performs the `4da://` handoff from an ordinary click on an ordinary web page — where the restriction does not apply, because it is a mail-client sanitiser and not a browser policy. The same desktop-handoff bridge Slack and Zoom use. Shipped in this PR so the button is never live against a 404. **The key travels in the fragment, not the query string.** A fragment is never transmitted, keeping the licence key out of Cloudflare's request logs and out of any Referer header. That mattered more than it first appears: `base.njk` injects PostHog into *every* page unconditionally, and a client-side script can read `location.hash` even though the server cannot — so `/activate` would have handed licence keys to a third-party analytics service. Added a `noAnalytics` front-matter flag and set it there. Verified both directions against the build: `activate.html` has **0** posthog references while index, privacy and success still have 14, 13 and 10. The page degrades honestly instead of looking broken: it does **not** fire the protocol navigation on load — an unprompted handoff on a machine without the app is exactly what makes these pages feel dead — and it always shows the key, a copy button, and the `Settings → License` instructions with a download link. A link carrying no key says so and names the likely cause. Links to `/activate`, not `/activate.html`: Pages 308-redirects the extension form to the clean URL, and a redirect is not somewhere to route a URL whose meaning lives in its fragment. ## Privacy policy: Stripe was not in it Checking whether the policy covered emailing at purchase (its Resend row scoped it to *"licence recovery only, when you ask for it"* — no longer true) surfaced that **Stripe appeared nowhere in `privacy.njk`**. It is the payment processor: it takes the buyer's email and card details, and the issued licence key and expiry are stored against the Stripe customer record — which is our **only** server-side copy, since the app verifies offline. Three more in the same document: - **PostHog** was absent from the third-party table, and §3.1 hedged that we *"may use privacy-respecting analytics ... if used"*. It is injected by `base.njk` on every page and `posthog.capture()` is called from `index.njk`. - The **CCPA table had two false rows**, not merely thin ones: identifiers *"Only via Store purchases"* and commercial information *"Only via Store"*. Signal checkout collects an email address and produces billing history. Added a financial-information row recording that card details reach Stripe/Shopify directly and never us. - **Retention covered App, Website and Store but not subscriptions.** Added §7.3, including the consequence a buyer needs before requesting deletion: that Stripe record is the only copy of their key we hold, so erasing it ends our ability to re-send it. **Keygen stays.** It looked stale beside offline verification, but `src-tauri/src/settings/license/keygen.rs` is live, so the row is correct. ## The sole required status check could pass a job that did not succeed Caught live on this PR, and it outweighs the delivery fix it turned up beside. `Validate Success` reported: ``` dependency results: success,success,success,success,abandoned,success ``` That `abandoned` is `Relay`. The gate was a **denylist** — it exited 1 only on `failure` or `cancelled` — so `abandoned` fell through to the success branch. Relay's conclusion was `failure`, the run's own conclusion was `failure`, and yet the one job configured as **the required status check** reported SUCCESS and the PR read `mergeable=MERGEABLE`. A human clicking merge would have landed a PR whose Relay leg never ran. GitHub emits more job results than the four that are documented, so a denylist here is unsound by construction: every value nobody thought of is a pass. Inverted to an **allowlist** — only `success` and `skipped` pass (`skipped` because the path-filtered legs legitimately skip when a change does not touch them) and everything else, known or not, blocks the merge. An empty results string now fails too, rather than being a gate that measured nothing and reported success. **Not introduced here.** The case statement dates to `bb9a843d` (#141); adding `relay` to `needs` is merely what made a job report a value outside the two it happened to check. It has been able to pass a non-succeeding job for as long as it has existed — the same failure shape as the stale-green NOTICE gate below, on the check that guards every merge. Verified against every result value: the live string above now fails naming `abandoned`, `success,skipped,success` still passes, and `failure` / `cancelled` / `timed_out` / `neutral` / empty all fail. Workflow YAML parses with the `needs` list unchanged. ## Also fixed, unrelated but blocking **`check-retired-claims` was failing on `main`.** #478's changelog quotes the retired claims verbatim to document what its republish removed — a legitimate historical quotation, which is exactly what the gate's `retired-ok:` escape hatch is for. It just never got the marker, and the violation spanned two lines so the one-line lookbehind couldn't cover it from above. Reflowed onto one line with an inline marker; HTML comments don't render, so the changelog is unchanged to a reader. **And the gate is now wired into `repo-guards`.** `.ai/RULES.md` and `AGENTS.md` have been telling every agent this is *"enforced by `scripts/check-retired-claims.cjs`"* while nothing in CI invoked it — the claim of enforcement was itself unenforced, which is how it broke on main unnoticed. **`NOTICE` was stale on `main`, and how it got there matters more than the diff.** `@tanstack/react-virtual` 3.14.5→3.14.9 and `@tanstack/virtual-core` 3.17.3→3.17.7 were missing. #460 added the NOTICE gate on 15 Aug; #450 merged on 17 Aug on a CI run **created before that gate existed**, passing in 13 seconds. `strict_required_status_checks_policy: false` permits a stale run to satisfy a required check, so the gate was never invoked and the next PR inherits both the breakage and the blame. Regenerated — but the class fix is enabling strict required status checks, which is an operator action on the ruleset. ## Verification **169 script tests pass, 0 fail** — up from 141 on `main`. All 28 new ones were confirmed to fail without their fix and pass with it: - neutering delivery back to the old silent no-op turns **7 of 8** delivery tests red - the two expiry tests fail against the original `.slice()` implementation - the three activation-link tests pin the regression rather than the implementation: the href must be https and must never be `4da://` again, the key must be in the fragment and never in the query string, and a key containing `+`, `/` and `=` must survive the round trip — a raw `+` would decode back as a space and hand the buyer a broken key - the namespace tests pin the legacy fallback rather than assuming it: a legacy-only record still resolves; a legacy *terminal* status is still terminal (reading only `signal_*` would silently re-grant revoked access); the current prefix wins when both are present; and a legacy first-seen stamp survives the rename instead of being restamped Site build clean — `privacy.html` renders at 41,384 bytes with Stripe, PostHog, the Signal retention section and the card-details disclosure all present, and both false CCPA claims gone. File-size gate clean · `check-retired-claims` clean · `node --check` clean on every changed JS file · YAML parses with the new step inside `repo-guards`, the job with no path filter. ## Still an operator action **Setting `RESEND_API_KEY` + `RESEND_FROM_EMAIL` on the Pages project.** The domain must be verified in Resend first, and `_dmarc.4da.ai` does not currently exist — worth adding in the same pass, since a licence key filtered to spam is indistinguishable to the buyer from not being sent. Current DNS is `v=spf1 include:spf.improvmx.com ~all` with MX on ImprovMX for inbound forwarding; Resend's free tier allows one verified domain. Until the variables are set, this PR changes the failure from *silent, with the page as the only copy* to *loud in the logs of every sale, with `ops:pages-secrets` exiting 1 and naming it* — but no email is sent, because there is nothing to send it with. **Enabling strict required status checks** on the ruleset, per the NOTICE finding above. ## Not done, deliberately Rate limiting on recovery, and **event-id dedup on the mint path**. Both need a KV namespace, and naming a binding that doesn't exist fails closed at request time on the live payment path. An earlier draft of this section claimed that namespace was infrastructure this PR could not conjure. That was not checked, and it is false: the available Cloudflare token carries `workers_kv (write)`, and `wrangler kv namespace list` returns `[]` — reachable, simply never created. The blocker is a missing decision, not a missing capability. It is still deliberately out of scope here, for a different and better reason: **dedup is a change to the live payment path and deserves its own tests and its own review**, not a late addition to a PR already in flight. It is also the more valuable of the two — without it a duplicate `checkout.session.completed` or `invoice.paid` delivery mints a second valid key, which is the same class of defect as the one this PR fixes. It should be the next piece of work, and there is now nothing infrastructural standing in the way. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Fq96xWyPQjx2bCCzWtsnC9 --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two truthfulness defects in public-facing claims, plus the drift that caused them.
1. Rejection-rate claims (
fix(docs))The headline numbers were real, and I found their source — commit
d956d46f, 2026-04-26, "ground rejection rate claims in measured benchmark data", which replaced an earlier unmeasured "99%+" claim with 92% / 98% / 77% measured from the 9-persona simulation over 1,997 evaluations. So they were never fabricated.What was actually broken is that nothing regenerated or enforced them afterwards:
README:345)cargo test scoring::benchmarkcited for the 9-persona numbersscoring::simulationd956d46fwas written to remove — those 4 locales were missedui-slice-navigation.test.ts:46); repointed to SignalOn "tested, not asserted": the suite asserts floors, not the headline.
reality.rsrequires aggregate precision >= 0.70, aggregate F1 >= 0.40, and >= 80% noise rejection per persona, with per-persona floors as low as P>=0.10 for bootstrap. Rejection could have fallen from 92% to 81% without failing a single test. Every doc now says so explicitly.Re-measured on the current pipeline (
cargo test --lib scoring::simulation -- --nocapture, 159 passed):The dashboard only prints P/R/F1, so rejection rate and noise accuracy have to be derived from the confusion matrix — that derivation is now written into the README rather than left as tribal knowledge.
Sanity check under real embeddings (
--features calibrated-sim, real fastembed fixtures rather than the synthetic CI baseline): 94.0% rejection / 98.8% noise accuracy / 83.2% precision. The headline claims hold under both embedding modes.Also fixed
specs/ARCHITECTURE.md's unbacked "99.9% rejection rate".No scoring code touched. The claims were wrong, not the thresholds.
2. Third-party attribution (
fix(legal))NOTICEwas hand-maintained at 92 direct dependencies against 828 shipping Rust crates + 42 production npm packages, and it is raw-imported byThirdPartyLicensesModal.tsx— so it is what users actually see. Confirmed drift:ts-rslisted as 10 -> 12.0.1scraperlisted as "0.23 - MIT" -> 0.27.0, and ISC (licence was wrong too)chacha20poly13050.10 -> 0.11.0ammonia4.1.4 -> absent entirelycssparserx2,cssparser-macrosx2,selectorsx2,dtoa-short,option-ext) — MPL-2.0 §3.2 requires noticenode_modules, but nothing copied it intodist/— OFL-1.1 requires the licence to travel with the font software.Route taken: mechanical generation, so it cannot drift again.
scripts/generate-notice.cjsderives NOTICE fromcargo metadata(non-dev closure: normal + build edges, all platforms) andpnpm licenses list --prod. It also:r-efi(MIT OR Apache-2.0 OR LGPL-2.1-or-later) neither blocks nor lands in the reciprocal sectionLicence compatibility was already clean: no GPL/AGPL/SSPL anywhere in the graph.
Why 29 old entries disappeared
Worth checking during review, since a shrinking NOTICE looks alarming. All 29 are correct removals:
criterion,eslint,vitest,playwright,sharp,tailwindcss, …) — the old file had explicit "Dev Dependencies" sections. Dev tooling is not distributed, so it needs no attribution.ocrs,rten— behind the non-defaultocrfeature (default = ["fastembed-local"]), so they are not linked into the shipped binary. Release builds pass no--featuresoverride.uhlc,tokio-tungstenite— genuinely gone; neither is inCargo.lockandtokio-tungsteniteis not even declared inCargo.toml. The hand-maintained file was attributing crates that had already been removed.Gate placement
--checkruns inrepo-guards, the only CI job with noif:— so a Rust-only dependency bump cannot slip past it. That costs the job a Rust toolchain and a dependency install it did not previously need (~2-3 min); the alternative was a gate that skipped on exactly the PRs that move dependencies.--requiremakes a missing toolchain a failure there, so the gate cannot pass by doing nothing. Also wired intopnpm run validateasvalidate:notice.19 self-tests added to
test:scripts, per the repo convention that every guard verifies its own claims.The gate caught real drift on its first run
Worth recording, because it is the whole argument for the change. This branch was cut before #433 landed. CI builds the merge commit, so
--checkrendered against a lockfile where #433 had bumpedcalamine0.25 -> 0.36.1,zip-> 8.6.0,quick-xml(three versions) -> 0.41.0,docx-rs0.4.20 -> 0.4.22 andplist1.9 -> 1.10, pulling inatoi_simd,debug_unsafe,fast-float2andtyped-path— 828 -> 832 shipping crates. It failed, named the exact crates, and the branch was rebased and regenerated.Under the hand-maintained file that bump would have silently desynced the attribution users see in the Third-Party Licenses modal — which is precisely how the four pre-existing version errors got there.
Verification
cargo test --lib scoring::simulation— 159 passedpnpm run test— 1249 passed (111 files)pnpm run typecheck,pnpm run build— cleanpnpm run test:scripts— 88 passed (69 + 19 new)check-file-sizes,check-doc-location,check-vanity-metrics,check-llm-gate-honesty,validate-translations— clean (14 translation warnings, all pre-existing and unchanged)generate-notice.cjs --check— idempotentFollow-ups (not done here, deliberately)
--features calibrated-simcurrently fails 9 tests (4 persona reality tests, 2 enrichment, golden snapshot, registry precision, version comparison). Pre-existing — this PR touches no Rust. The feature is off by default and not in CI, but its thresholds have drifted from its fixtures.scoring/, which was out of bounds for this PR.assert_qualityvalues inreality.rs. If someone lowers a threshold, the docs go stale silently.tauri.conf.jsonbundlessrc-tauri/models(text-detection.rten2.4 MB,text-recognition.rten9.3 MB, fetched byfetch-ocr-models.cjs, untracked in git). These are third-party artifacts that ship in the installer, and this generator covers source dependencies only. Separately worth a look: the models ship even though theocrfeature that reads them is off by default.🤖 Generated with Claude Code
https://claude.ai/code/session_01AUeKTKwNmdow8yUk3q8RB2