Add the generated data-audit dashboard (gh-pages): audit + migration tracker

.github/workflows/audit-dashboard.yml

@@ -0,0 +1,82 @@
+# The generated data-audit dashboard (data-lectures#20): scan the 8 Python-
+# family lecture repos, verify migration.yml + the manifests against what the
+# lectures actually read, render the dashboard, and deploy it to GitHub Pages
+# together with the published lectures/ tree.
+#
+# The scan is strict: an unannotated data reference or a migration.yml status
+# that disagrees with reality fails the build — the dashboard cannot rot into
+# a hand-tended snapshot. On PRs the build runs as a guardrail; deploys happen
+# only from main (push / weekly schedule / manual dispatch).
+
+name: audit-dashboard
+
+on:
+  push:
+    branches: [main]
+  schedule:
+    - cron: "17 5 * * 1"    # weekly — lecture repos move under us
+  workflow_dispatch:
+  pull_request:
+    paths:
+      - migration.yml
+      - lectures/*.yml
+      - scripts/build_audit.py
+      - scripts/render_audit.py
+      - scripts/audit_annotations.yml
+      - .github/workflows/audit-dashboard.yml
+
+permissions:
+  contents: read
+
+concurrency:
+  group: pages
+  cancel-in-progress: false
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          lfs: true    # published files must be bytes, never LFS pointers
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install pyyaml
+      - name: Clone the 8 lecture repos (shallow, main only)
+        run: |
+          mkdir -p repos
+          for repo in lecture-python-intro lecture-python-programming \
+                      lecture-python.myst lecture-python-advanced.myst \
+                      lecture-jax lecture-dp lecture-wasm continuous_time_mcs; do
+            git clone --quiet --depth 1 --branch main \
+              "https://github.com/QuantEcon/$repo" "repos/$repo"
+          done
+      - name: Build the dashboard (strict — fails on unannotated refs or migration drift)
+        run: python scripts/build_audit.py all --strict --repos-dir repos -o site
+      - name: Assemble the Pages tree (dashboard at /, data at /lectures/)
+        run: |
+          mkdir -p _site
+          cp -r site/. _site/
+          cp -r lectures _site/lectures
+          cp audit.json _site/audit.json
+      - uses: actions/upload-pages-artifact@v3
+        with:
+          path: _site
+
+  deploy:
+    if: github.event_name != 'pull_request'
+    needs: build
+    runs-on: ubuntu-latest
+    permissions:
+      pages: write
+      id-token: write
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    steps:
+      - uses: actions/configure-pages@v5
+        with:
+          enablement: true
+      - id: deployment
+        uses: actions/deploy-pages@v4

.gitignore

@@ -0,0 +1,5 @@
+# generated by scripts/build_audit.py — built fresh in CI, never committed
+site/
+audit.json
+
+__pycache__/

AGENTS.md

@@ -78,6 +78,16 @@ For the public data sources most snapshots come from (World Bank, FRED, Eurostat
 
 One exception: a `restricted` file the lectures have already served publicly may be cached here if it is marked `redistribution: restricted` with a `note` and logged for licence review in the migration-licensing tracker ([workspace-lectures#20](https://github.com/QuantEcon/workspace-lectures/issues/20)) — resolve it (permission, an open replacement, or removal) before `data.quantecon.org` is promoted as a public open-data host.
 
+### The audit dashboard stays truthful
+
+The generated dashboard (`scripts/build_audit.py`, [#20](https://github.com/QuantEcon/data-lectures/issues/20)) verifies its three inputs against a fresh scan of the lecture repos, and the strict build **fails** when they drift. Keep them current in the same PR as the change that moves reality:
+
+- **Landing or repointing a dataset** → update its `migration.yml` record (status, PR refs, dates). A dataset marked `repointed` whose consumers still read an old URL — or the reverse — is a build failure, by design.
+- **A new manifest** (`lectures/*.yml`) → add its `migration.yml` record; delete any stale entry for the same file in `scripts/audit_annotations.yml` (manifested datasets must not be annotated there).
+- **The weekly scan flags an unannotated reference** (a lecture repo started reading a new file) → classify it and add an entry to `scripts/audit_annotations.yml`; that file holds judgment (description, provenance, why-live), never facts the scan can derive.
+
+`site/` and `audit.json` are generated — never commit them; CI rebuilds and deploys on every push to `main`.
+
 ## Cross-repo hygiene
 
 - Changes here often pair with PRs in lecture repos and issues in `QuantEcon/meta`. In commit messages and PR bodies, **never place a GitHub closing keyword (`fixes`, `closes`, `resolves`, …) immediately before a cross-repo reference** like `QuantEcon/meta#336` — GitHub will auto-close the referenced issue when the commit lands on the default branch. Write "See QuantEcon/meta#336" or "Part of QuantEcon/meta#336".
@@ -89,8 +99,13 @@ One exception: a `restricted` file the lectures have already served publicly may
 lectures/            # the published tree — flat, served at data.quantecon.org/lectures/
                      #   9 datasets + business_cycle's upstream metadata dumps
                      #   manifests live here as sidecars: <filename>.yml
-scripts/             # builders — NOT published
+scripts/             # builders + generators — NOT published
   business_cycle.py  #   writes business_cycle_data.csv into lectures/
+  build_catalog.py   #   generates CATALOG.md from the manifests
+  build_audit.py     #   the audit dashboard: scan lecture repos → audit.json → site/
+  render_audit.py    #   its render stage
+  audit_annotations.yml  # curated judgment for not-yet-migrated data refs
+migration.yml        # migration lifecycle tracker (status + PR provenance per dataset)
 manifest-schema.yml  # per-dataset manifest schema (strawman)
 requirements.txt
 PLAN.md              # roadmap — start here

PLAN.md

@@ -70,7 +70,7 @@ The sidecar naming uses the **full filename** (`mpd2020.xlsx.yml`, not `mpd2020.
 
 ### Phase 4 — Publishing
 
-- [ ] GitHub Pages deploy of the published tree, **`lfs: true` at checkout** (else pointer files publish)
+- [x] GitHub Pages deploy of the published tree, **`lfs: true` at checkout** (else pointer files publish) — landed 2026-07-17 with the audit dashboard (`.github/workflows/audit-dashboard.yml`, [#20](https://github.com/QuantEcon/data-lectures/issues/20)): the default `quantecon.github.io/data-lectures/` site serves the dashboard at `/` and the published tree at `/lectures/`. The custom domain below stays open
 - [ ] `data.quantecon.org` DNS + custom domain (an old NestJS box on AWS Sydney currently answers this name — investigate before repointing)
 - [ ] Verify `access-control-allow-origin: *` on served files (pyodide/JupyterLite, meta#143)
 - [ ] Monitor Pages soft limits (~1 GB site, 100 GB/month)
@@ -84,6 +84,7 @@ The sidecar naming uses the **full filename** (`mpd2020.xlsx.yml`, not `mpd2020.
 
 Full automation:
 
+- [x] Audit dashboard workflow ([#20](https://github.com/QuantEcon/data-lectures/issues/20), added 2026-07-17): `.github/workflows/audit-dashboard.yml` rebuilds the full-universe data audit + migration tracker from the 8 lecture repos' `main` (push to main / weekly / dispatch) and deploys it with the published tree to Pages. Strict mode fails the build on an unannotated data reference or a `migration.yml` status the scan contradicts
 - [ ] PR validation: manifest schema check + per-dataset invariant tests (expected columns/dtypes, row-count floor, date-range recency, no all-NaN columns, overlap-window agreement with the previous vintage) on every PR touching data. The schema decisions these tests force — column patterns for wide files, `known_nulls` exact-vs-ceiling, a canonical dtype vocabulary — are researched in [#14](https://github.com/QuantEcon/data-lectures/issues/14)
 - [ ] Retrofit `scripts/business_cycle.py` to the four-stage builder contract — it has fetch/transform/write today but **no validate stage**. Builder architecture and a copy-able template: [#14](https://github.com/QuantEcon/data-lectures/issues/14)
 - [ ] Scheduled refresh workflow for dynamic datasets — cron per cadence class, runs the builder (fetch → pre-process → validate → write), lands the result as a PR whose diff summary (rows added, date-range delta, overlap-window changes) is the review surface; low-risk series may auto-merge on green (first consumer: the UNRATE pilot, meta#338 P4)
@@ -116,7 +117,7 @@ Verify that what this repo holds is actually the data it claims to be — agains
 The first end-to-end deployment: one dataset per hosting pattern, each the hardest representative of its class, carried through the full chain — layout, manifest, integrity check, publish, lecture repoint. Validates the convention empirically before anything is written into a standard. Sequence P1 → P2 → P3 → P4, each a small PR set (data repo + consuming lecture repos).
 
 - [x] **P1 — local-path static**: `lingcod_msy_recovery.csv` (`msy_fishery`, intro). Tests: single-PR green build under `-nW`, Colab-unchanged download, catalog metadata for an author-assembled file. **Complete 2026-07-17** — data half #12, repoint QuantEcon/lecture-python-intro#792 (lecture build green in the single repoint PR); served URL verified byte-identical to the manifest `sha256`; Colab holds by construction (the lecture now reads a public URL, where the old relative path was exactly what broke downloaded notebooks); metadata findings recorded in meta#338. **The repo is live from this merge** — the pyodide/CORS check below remains open
-- [ ] **P2 — cross-series shared static**: the `pandas_panel` trio (`realwage.csv`, `countries.csv`, `employ.csv`), consumed by programming **and** python.myst. Tests: flat namespace with two consuming series, one data PR updating two lecture repos; retires 5 of the 8 legacy-repo references as a side effect
+- [x] **P2 — cross-series shared static**: the `pandas_panel` trio (`realwage.csv`, `countries.csv`, `employ.csv`), consumed by programming **and** python.myst. Tests: flat namespace with two consuming series, one data PR updating two lecture repos; retires 5 of the 8 legacy-repo references as a side effect. **Complete 2026-07-17** — data half #17, repoints QuantEcon/lecture-python-programming#578 and QuantEcon/lecture-python.myst#973; both lecture repos' own stale copies deleted in the repoint PRs. Lifecycle recorded in `migration.yml`
 - [ ] **P3 — external-repo static with LFS**: the `heavy_tails` set (Forbes ×2, cities ×2) plus the SCF pair from `high_dim_data`. Tests: served URL makes the raw-vs-media LFS trap invisible, Pages handles LFS objects (`lfs: true`), builders (`webscrape_forbes.ipynb`, `generating_mini.md`) migrate alongside their data
 - [ ] **P4 — dynamic snapshot twin**: `UNRATE`, consumed today by 4 lectures across 3 repos via 2 access methods. Tests: the full dynamic template — manifest, four-stage builder, refresh-as-PR, canary catching an induced failure — plus the documented live-call ↔ snapshot switch mechanism
 - [ ] Verify each migrated URL with a pyodide/JupyterLite fetch (CORS, meta#143)

README.md

@@ -41,9 +41,31 @@ See the [draft convention](https://github.com/QuantEcon/QuantEcon.manual/pull/10
 | Path | What | Published |
 | --- | --- | --- |
 | `lectures/` | the published tree — flat. Every dataset lives here, directly. No folder implies ownership by a lecture series: any lecture may consume any file | yes |
-| `scripts/` | builders for constructed and dynamic datasets | no |
+| `scripts/` | builders for constructed and dynamic datasets, plus the audit-dashboard generator | no |
 | `manifest-schema.yml` | the per-dataset manifest schema (strawman — see [`PLAN.md`](PLAN.md) Phase 2) | no |
+| `migration.yml` | the migration lifecycle tracker — which PRs landed and repointed each dataset (transitional; archivable when the migration programme completes) | rendered |
 
 The tree is flat because the URL is the interface: `lectures/<filename>` maps to
 `data.quantecon.org/lectures/<filename>`, so a file can never be re-filed under a
 new owner and break its consumers. Anything outside `lectures/` is not served.
+
+## The audit dashboard
+
+A **generated** dashboard covering all data referenced by the 8 synced
+Python-family lecture repos — the full-universe audit plus a per-dataset
+migration tracker — deploys to this repo's GitHub Pages site alongside the
+published `lectures/` tree ([data-lectures#20](https://github.com/QuantEcon/data-lectures/issues/20)).
+
+```
+python scripts/build_audit.py all --strict     # scan + render into site/
+```
+
+The scan greps each lecture repo's `main` (clones under `--repos-dir`; defaults
+to this repo's parent, matching the workspace-lectures layout), classifies every
+data reference, and reconciles three sources of truth: the manifests
+(`lectures/*.yml`, migrated datasets), `migration.yml` (lifecycle + PR
+provenance), and `scripts/audit_annotations.yml` (curated judgment for
+not-yet-migrated references). A new data reference with no annotation, or a
+migration status the scan contradicts, **fails the build** — the dashboard
+cannot silently rot. CI rebuilds it on push to `main`, weekly, and on demand
+(`.github/workflows/audit-dashboard.yml`).

migration.yml

@@ -0,0 +1,105 @@
+# migration.yml — the migration lifecycle tracker for data-lectures.
+#
+# One record per dataset moving into this repo, holding the PROCESS facts the
+# manifests deliberately do not: which PRs landed the file, which PRs repointed
+# each consumer, and where the dataset sits in the lifecycle. Manifests
+# (lectures/*.yml) are the permanent record of a dataset; this file documents
+# a transition and can be archived once the migration programme completes,
+# without touching a single manifest.
+#
+# The audit dashboard (scripts/build_audit.py) renders this file and VERIFIES
+# it against reality on every build: a dataset marked `repointed` whose
+# consumers still read an old URL (or vice versa) fails the consistency check.
+#
+# Lifecycle (status):
+#   pending    identified for migration; nothing landed here yet
+#   landed     file + manifest merged in data-lectures; consumers unchanged
+#   repointed  every consumer reads this repo's interim raw URL
+#   final      every consumer reads https://data.quantecon.org/lectures/…
+#              (gated on the Phase 4 DNS work — data-lectures#15)
+
+datasets:
+  lingcod_msy_recovery.csv:
+    pilot: P1
+    status: repointed
+    prior_pattern: local-path
+    landed:
+      pr: QuantEcon/data-lectures#12
+      date: 2026-07-15
+    repoints:
+      - repo: QuantEcon/lecture-python-intro
+        pr: QuantEcon/lecture-python-intro#792
+        date: 2026-07-17
+    cutover: null
+
+  realwage.csv:
+    pilot: P2
+    status: repointed
+    prior_pattern: legacy
+    landed:
+      pr: QuantEcon/data-lectures#17
+      date: 2026-07-17
+    repoints:
+      - repo: QuantEcon/lecture-python-programming
+        pr: QuantEcon/lecture-python-programming#578
+        date: 2026-07-17
+      - repo: QuantEcon/lecture-python.myst
+        pr: QuantEcon/lecture-python.myst#973
+        date: 2026-07-17
+    cutover: null
+
+  countries.csv:
+    pilot: P2
+    status: repointed
+    prior_pattern: legacy
+    landed:
+      pr: QuantEcon/data-lectures#17
+      date: 2026-07-17
+    repoints:
+      - repo: QuantEcon/lecture-python-programming
+        pr: QuantEcon/lecture-python-programming#578
+        date: 2026-07-17
+      - repo: QuantEcon/lecture-python.myst
+        pr: QuantEcon/lecture-python.myst#973
+        date: 2026-07-17
+    cutover: null
+
+  employ.csv:
+    pilot: P2
+    status: repointed
+    prior_pattern: legacy   # programming read the legacy repo; python.myst its own copy
+    landed:
+      pr: QuantEcon/data-lectures#17
+      date: 2026-07-17
+    repoints:
+      - repo: QuantEcon/lecture-python-programming
+        pr: QuantEcon/lecture-python-programming#578
+        date: 2026-07-17
+      - repo: QuantEcon/lecture-python.myst
+        pr: QuantEcon/lecture-python.myst#973
+        date: 2026-07-17
+    cutover: null
+
+# Planned waves that have not landed anything here yet. `datasets` names the
+# files as the audit sees them today, so the dashboard can join the two views.
+pending:
+  - pilot: P3
+    scope: the LFS case — heavy_tails Forbes/cities set + SCF extracts (fold in high_dim_data)
+    datasets:
+      - forbes-global2000.csv
+      - forbes-billionaires.csv
+      - cities_us.csv
+      - cities_brazil.csv
+      - SCF_plus_mini.csv
+      - SCF_plus_mini_no_weights.csv
+    tracking:
+      - QuantEcon/data-lectures#2
+      - QuantEcon/meta#337
+  - pilot: P4
+    scope: the dynamic-snapshot case — a maintained UNRATE twin for the incidental FRED consumers
+    datasets: []          # produces a NEW file here; no existing static file moves
+    tracking:
+      - QuantEcon/meta#338
+
+# Interim → final URL cutover (status: repointed → final) is tracked as one
+# sweep, not per dataset: QuantEcon/data-lectures#15, gated on Phase 4 DNS.

scripts/README.md

@@ -14,3 +14,12 @@ bug.
 `business_cycle.py` is run by hand today and has no validate stage; retrofitting
 it to the four-stage contract (fetch → pre-process → validate → write) is
 PLAN Phase 5.
+
+## Not builders
+
+| Script | What |
+| --- | --- |
+| `build_catalog.py` | generates `CATALOG.md` from the manifests (`lectures/*.yml`) |
+| `build_audit.py` | the audit dashboard: scans the 8 lecture repos, writes `audit.json`, renders `site/` |
+| `render_audit.py` | the render stage of `build_audit.py` (HTML generation) |
+| `audit_annotations.yml` | curated judgment for not-yet-migrated data references — the strict scan fails when a new reference has no entry here and no manifest |