Add the generated data-audit dashboard (gh-pages): audit + migration tracker

[mmcky]

Delivers #20 — the one-time, hand-built dataset audit becomes a generated, self-updating dashboard deployed to this repo's GitHub Pages site, extended into a first-class migration tracker showing exactly which data has moved here, via which PRs, and what's queued.

What's in the box

scripts/build_audit.py — two stages, kept separate so the data is reusable. scan greps each of the 8 synced Python-family repos on origin/main (clone + grep; gh search code cannot find URLs), resolves string-built URLs (f-strings, continuation lines, var + 'file' concatenation), reads only executed {code-cell} blocks (skipping skip-execution cells), classifies every reference into the hosting-pattern taxonomy, and emits audit.json. render writes the 3-page site.

Three pages (design system evolved from the 2026-07-15 artifact — same theme-aware palette and pills, plus nav, a migration meter with dataviz-validated series colors, and per-dataset lifecycle steppers):

Page	Contents
Overview	stat tiles, migration progress meter, hosting-pattern distribution, the what-moved-since-2026-07-15 story
Migration tracker	per-dataset lifecycle pending → landed → repointed → final with the PRs that moved it, verified against a fresh scan; pending waves (P3/P4); consistency-check panel
Full audit	the regenerated registries — hosting patterns, static files, %%file-embedded, live-API (with pedagogy classification), cross-series reuse, committed-but-unreferenced files, provenance classes, draft styleguide rules feeding QuantEcon.manual#108

Three sources of truth, reconciled on every build. Manifests (lectures/*.yml) own the migrated datasets; migration.yml (new) owns lifecycle + PR provenance — deliberately separate from the manifests so it can be archived when the migration programme completes; scripts/audit_annotations.yml (new) owns curated judgment for not-yet-migrated references. The strict build fails on an unannotated reference or a migration.yml status the scan contradicts — the dashboard cannot rot into the stale snapshot it replaces.

Pages workflow (.github/workflows/audit-dashboard.yml): push to main / weekly / manual dispatch; serves the dashboard at / and the published lectures/ tree at /lectures/ (checkout lfs: true), pre-staging Phase 4 so only the custom domain changes when the DNS question resolves. On PRs touching audit inputs it runs the strict build as a guardrail without deploying.

Fresh numbers (2026-07-17, vs the 2026-07-15 hand audit)

• 36 distinct static files referenced (was 34): +2 genuinely new local-path datasets from lecture-python-intro#790 (us_adult_heights.csv, japan_population_by_age.xlsx), +graph.txt promoted from "shadowed" to load-bearing (lecture-wasm fetches intro's copy by URL), −test_table.csv (an exercise prose link, now annotated as such rather than a code read)
• Legacy-repo URLs 8 → 0, branch-pinned refs 1 → 0 — the flags the last two days of PRs resolved, confirmed from live state
• 4 datasets fully migrated (P1 lingcod, P2 pandas_panel trio), each verified: every consumer reads the data-lectures interim URL
• 8th hosting pattern discovered (sibling repo URL — the wasm mirror), 35 committed-but-unreferenced files including wasm's 14 mirror-orphans

Notes for review

• Merging this deploys Pages on the default quantecon.github.io/data-lectures/ domain (configure-pages with enablement: true); data.quantecon.org remains gated on the Phase 4 DNS investigation (Phase 4 cutover: swap interim raw URLs → data.quantecon.org/lectures #15 stays open)
• PLAN.md updates ride along: Phase 4 Pages checkbox (default domain), the Phase 5 workflow entry, and pilot P2 marked complete (P2 pilot: add the pandas_panel trio (realwage, countries, employ) with manifests #17, P2 pilot: read pandas_panel data from data-lectures lecture-python-programming#578, P2 pilot: read pandas_panel data from data-lectures lecture-python.myst#973)
• site/ and audit.json are generated and gitignored; CI builds them fresh

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR adds a generated, CI-built data-audit dashboard (with a migration tracker) for the QuantEcon Python-family lecture repos, and deploys it via GitHub Pages alongside the published lectures/ dataset tree. It introduces a strict “truthfulness” contract across lectures/*.yml manifests, migration.yml, and scripts/audit_annotations.yml, and updates project docs/plans to reflect the new automation.

Changes:

• Add scripts/build_audit.py (scan + strict reconciliation) and scripts/render_audit.py (HTML renderer) plus curated scripts/audit_annotations.yml.
• Introduce migration.yml as a lifecycle/provenance tracker separate from manifests, and document the strict drift checks in AGENTS.md / README.md.
• Add a GitHub Actions Pages workflow to rebuild and deploy the dashboard and lectures/ tree on main/schedule/dispatch (and run as a PR guardrail).

Reviewed changes

Copilot reviewed 9 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
scripts/render_audit.py	New HTML renderer for the dashboard pages (overview, migration tracker, full audit).
scripts/build_audit.py	New scanner/reconciler producing audit.json and invoking the renderer; supports strict drift detection.
scripts/audit_annotations.yml	New curated annotations for non-manifested datasets + live-API pedagogy notes + orphan classifications.
scripts/README.md	Documents which scripts are builders vs generators/metadata inputs.
README.md	Documents the audit dashboard purpose and how to run the strict build.
PLAN.md	Marks Pages deploy and audit-dashboard automation as completed items.
migration.yml	New migration lifecycle/provenance tracker for migrated datasets and pending waves.
AGENTS.md	Adds repository rule: keep dashboard inputs consistent; don’t commit generated outputs.
.gitignore	Ignores generated site/ and audit.json.
.github/workflows/audit-dashboard.yml	New Pages workflow to build (strict) and deploy dashboard + lectures/ tree.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.