Skip to content

Serve public projection media from object storage instead of the git-tracked tree #301

Description

@alexeygrigorev

Product outcome

Public projection images keep working exactly as they do today at exactly the same URLs, while the
1,253 image objects (≈154 MB, 1,254 files on disk) stop being carried in the git working tree. A
visitor sees no difference: /images/... still returns 200 with the same Content-Type, the same
bytes, and the same page rendering. A contributor gets a repository whose checkout — and every one
of the many .tmp/ worktrees — no longer materialises 150 MB of binaries per copy.

This issue delivers Phase 1 only: Django reads each object from a pluggable media store and
serves it at the unchanged public path. Phase 2 (CloudFront serving /images/* straight from an S3
origin with Django out of the request path) is a named non-goal here — see "Serving strategy".

Normative authority

  • _docs/PROCESS.md — lifecycle, evidence, [HUMAN] criteria, no-PR local merge.
  • _docs/specs/03-github-content-and-people.mdContentAsset carries "release, source path,
    stable public path, immutable S3 key, content type, size, and checksum", and "The Django asset
    resolver selects the active release while keeping the original public URL; edge caching makes the
    stable path efficient." This is the normative basis for Phase 1.
  • _docs/specs/02-url-link-seo-compatibility.md — preserve-first URL contract and the route cache
    classes.
  • _docs/specs/08-aws-development-terraform.md — "S3 bucket for immutable release assets, exports,
    and controlled operational artifacts …"; "CloudFront/application resolution keeps public asset
    paths unchanged."
  • _docs/specs/07-security-privacy-operations.md — fail-closed behaviour, redacted logging, no
    secret or credential exposure.
  • _docs/specs/10-verification-strategy.md and _docs/ci/change-selective-ci.md — verification plan
    and evidence rules.
  • _docs/architecture/app-boundaries.mdcontent owns "versioned GitHub-owned read models and
    public content presentation".
  • _docs/compatibility/generated-path-baseline.jsonl — 2,937 rows, every one
    classification: preserve / expected_status: 200.

Verified current behaviour (re-checked at 55f6743)

Fact Evidence
1,253 media records, 815 from DataTalksClub/content@1375c50…, 438 from DataTalksClub/datatalksclub.github.io@ee43d3f… content/public_projection/media.json
Every record already carries provenance.checksum (sha256), repository, revision, source_path, source_url media.json
1,254 files under content/public_projection/media/, 154,154,378 bytes; median object 33.9 KB, largest 3,022,797 bytes on-disk census
Exactly one orphan file with no record: media/podcast/s24e06-how-to-build-ai-that-actually-ships-in-production.jpg; it returns 404 today on-disk census, request check; owned by #253
Serving path: content/public_views.py:1151 media()FileResponse(path.open("rb"), content_type=record["content_type"]), routed at content/public_urls.py:235 code
Live response headers on a hit: Content-Type, Content-Length, Content-Disposition: inline; filename="…", plus the usual security headers. No Cache-Control. GET /images/authors/%20aashishnair.jpg
Runtime integrity today: content/public_data.py:625 compares manifest["tree_sha256"] with _tree_sha256(PROJECTION_ROOT) (:339-352), which reads every byte of every file under the projection root, media included code
Two records contain a literal space: /images/authors/ aashishnair.jpg and /images/podcast/production-ml-search-vector-search-embeddings-hybrid search.jpg; the builder has a deliberate one-line whitelist at scripts/build_public_projection.py:1745 code, media.json
1,281 baseline rows start with /images/; 1,245 of them match a projection record, 36 do not and return 404 today (site chrome: /images/logo.svg, /images/partners/*, /images/landing/*, …). 8 records are not in the baseline. baseline vs media.json, request checks
content_sync/dtc_content/contract.py:34 PROJECTION_TREE_SHA256 = 9baae604… and PROJECTION_MANIFEST_SHA256 = 6e09a551… already differ from the checked manifest (29a1819f… / 266d8daa…). The parity gate at content_sync/dtc_content/parity.py:345-346 is opt-in (DTC_CONTENT_ACCEPTED_CHECKOUT) and is already red for reasons this issue does not cause. code, digests
scripts/validate_github_editorial_source_projection_inventory.py is already failing on current main (manifest preferred content source drift), and so is scripts/tests/test_github_editorial_source_projection_inventory.py::test_inventory_matches_the_exact_checked_in_snapshot. Its pinned media.json digest at :180 (6b6670d0…) is stale versus the live 199c7860…. uv run python scripts/validate_…py; uv run pytest -k snapshot
No Playwright test depends on real projection image bytes (the only naturalWidth assertions use data: URLs and static fixtures) playwright_tests/

Scope

  1. A media store abstraction in the content app with three backends: local (filesystem
    directory), memory (deterministic offline fixture for tests/CI), s3 (boto3, already a
    dependency). Selected by settings; default requires no AWS credentials and no network.
  2. content/public_views.py::media() reads through the store instead of the projection tree, with
    fail-closed per-object integrity and unchanged observable success responses.
  3. Integrity contract change: the projection complete-tree digest covers the JSON artifacts and wiki
    assets only; per-object integrity moves to the provenance.checksum already recorded per media
    record. The manifest states the digest scope explicitly so its meaning cannot drift silently.
  4. Operator tooling: hydrate (materialise the objects locally from pinned upstream, credential
    free), publish (upload to the bucket), verify (compare store contents to media.json).
  5. Repository change: content/public_projection/media/ stops being git-tracked and is ignored,
    with .dockerignore updated so a local image build matches a CI build.
  6. Documentation of the local developer flow and the operator publish/verify flow.

Design decisions

D1 — Tree digest scope (resolves the tree_sha256 blocker)

_tree_sha256 in both content/public_data.py:339 and scripts/build_public_projection.py:2690
walks the projection root and skips, in addition to manifest.json, everything under media/. The
symlink rejection stays exactly as it is: a symlink anywhere under the root, including under
media/, is still a hard failure.

manifest.json gains an explicit, machine-readable scope declaration next to the digest, for
example:

"tree_sha256": "<recomputed, media-free>",
"tree_digest_scope": "projection artifacts and wiki assets; excludes manifest.json and media/",
"media_storage": {
  "location": "object-store",
  "records": "media.json",
  "count": 1253,
  "integrity": "per-record provenance.checksum"
}

content/public_data.py verifies the scope declaration as well as the digest, so a manifest produced
by an older builder (whole-tree scope) fails closed rather than being silently accepted.

Because the checked projection is currently not reproducible from upstream (#253), the engineer
must not re-run the full builder to obtain the new digest. Provide a narrow, deterministic re-pin
utility (a --repin-digests mode of the existing builder, or a small dedicated script) that
recomputes only the derived digest fields from the already-checked artifacts and rewrites
manifest.json. The manifest diff must contain only the digest/scope/media-storage fields; a test
must recompute the field from the tree so the value can never be hand-typed.

D2 — Per-object integrity

Every served object is verified against its record's provenance.checksum before any byte reaches
the client. Verification may use the store's own server-side SHA-256 (S3 ChecksumSHA256) when the
store supplies one, otherwise it is computed over the retrieved bytes. Objects are size-bounded
(configurable, default above the 3,022,797-byte largest known object) so a response is never
unbounded in memory. Unverified bytes, partial bytes, and placeholder substitutes are never served.

D3 — Serving strategy: Django-mediated origin read (Phase 1)

Decision: Django reads the object from the store and returns it at the same URL. Reasons:

  • the production CloudFront distribution does not exist yet, and decision 20 in
    _docs/specs/open-decisions.md governs its cache/WAF plan; nothing here may pre-empt it;
  • spec 03 already prescribes exactly this shape ("The Django asset resolver … keeping the original
    public URL");
  • it preserves all 2,937 baseline rows and today's response header shape with zero edge
    configuration, so the compatibility gate stays measurable;
  • one fail-closed integrity check stays in code rather than being dissolved into edge config;
  • it is a prerequisite for Phase 2 anyway: the bucket must exist, be populated, and use
    path-mirrored keys before any distribution can be pointed at it.

The view must not redirect to a presigned or public S3 URL. A 302 would change the response
contract, leak the bucket name, and break the baseline's expected_status: 200.

Phase 2 (a CloudFront behaviour for /images/* with an S3 origin and Django out of the path) is a
non-goal of this issue and needs its own issue once the production distribution exists.

D4 — Backends, settings, and credential-free defaults

Setting Default Meaning
PUBLIC_MEDIA_STORE_BACKEND local local | memory | s3
PUBLIC_MEDIA_LOCAL_ROOT content/public_projection/media filesystem root for local
PUBLIC_MEDIA_S3_BUCKET "" bucket name for s3
PUBLIC_MEDIA_S3_PREFIX public-projection key prefix for s3
PUBLIC_MEDIA_S3_REGION "" region for s3
PUBLIC_MEDIA_S3_ENDPOINT_URL "" optional; lets a developer point at a local/faked endpoint
PUBLIC_MEDIA_S3_TIMEOUT_SECONDS small bounded value connect/read timeout; at most one retry
PUBLIC_MEDIA_MAX_OBJECT_BYTES 8 MiB fail-closed size bound

Rules:

  • The default configuration in base, development, local, and test settings is local. No
    AWS credential, network call, or new dependency is required to run the site or the test suite.
  • memory is a deterministic offline fixture store derived from media.json: for each record it
    serves a minimal valid image of the recorded content type whose sha256 is used as that
    record's expected checksum inside the fixture. It refuses to activate under production settings.
  • CI test jobs set PUBLIC_MEDIA_STORE_BACKEND=memory in the workflow environment (not in the
    Makefile), so CI is offline and deterministic while a local developer or tester still renders the
    real hydrated bytes with the default local backend.
  • s3 is required under production settings: manage.py check --deploy fails when
    DTC_ENVIRONMENT=production and the backend is not s3 or the bucket is empty.
  • The S3 client is constructed from the ambient role/credential chain. This issue introduces no
    credential handling, no static keys, and no secret in settings, logs, or reports.

D5 — Keys, bucket population, and orphans

  • Key = f"{PUBLIC_MEDIA_S3_PREFIX}/{record['record_key']}", i.e. path-mirrored
    (public-projection/images/authors/ aashishnair.jpg). Path-mirrored keys are what a future
    CloudFront origin needs; a content-addressed layout would require an edge key-value lookup, which
    spec 08 forbids.
  • The key is derived only from the matched record, never from the raw request path. Traversal,
    encoding, and the two spaced filenames therefore cannot reach the store as attacker-controlled
    input.
  • hydrate materialises exactly the 1,253 records into the local root from the pinned upstream
    revisions recorded in each record's provenance. It needs no AWS credentials, is idempotent and
    resumable, verifies every object against provenance.checksum, and never writes an object whose
    digest does not match. It may also hydrate from the configured store (--from-store) once the
    bucket exists.
  • publish uploads exactly the 1,253 records from a local root, sets Content-Type from the record
    and ChecksumAlgorithm=SHA256, skips objects already present with a matching checksum, and
    refuses to upload any file that has no record (the known orphan stays unpublished and keeps
    returning 404). It prints a report of added/changed/skipped/orphan counts.
  • verify compares the configured store against media.json and exits non-zero on any missing,
    extra, or checksum-mismatched object.
  • Release-scoped or immutable per-revision key layouts belong to the direct-sync ContentAsset work
    (Expand direct-sync schema and reconcile historical active content #273Cut public content reads and projections over to direct-sync authority #276) and are out of scope here.

D6 — Failure semantics

  • Path with no media.json record → 404, exactly as today (including the known orphan file and
    the 36 not-yet-implemented baseline chrome paths).
  • Record exists but the object is missing, oversized, unreadable, times out, or fails its checksum →
    502 with Cache-Control: no-store, no upstream detail in the body, no bucket/key/credential in
    the response, and exactly one redacted structured log event. Never a 404 (that would let the
    edge cache an outage as "not found"), never a placeholder, never unverified bytes.
  • local backend with an empty root → the same 502 path, plus a manage.py check warning
    naming the hydrate command, so a fresh clone is told what to run instead of silently rendering
    broken images.

D7 — Count canaries stay put

The record set does not change. content/public_data.py:66 ("media": 1_253),
scripts/build_public_projection.py:117-121 (815 / 438 / 1,253 / fallbacks),
content_sync/dtc_content/parity.py:346,867-868, and
scripts/validate_github_editorial_source_projection_inventory.py:102,182 all keep their current
values. A diff that changes any media count is a defect in this issue.

D8 — Already-red gates this issue must not adopt

  • content_sync/dtc_content/contract.py:34 PROJECTION_TREE_SHA256 and PROJECTION_MANIFEST_SHA256
    are already stale versus the checked manifest, and the parity gate is already red. Do not re-pin
    the hex values here
    — that would silently accept the face8e4-era drift that Reconcile non-reproducible public projection after face8e4 #253 exists to
    reconcile. Add a comment beside the pin recording that the digest scope is now media-free so Reconcile non-reproducible public projection after face8e4 #253
    re-pins correctly when it regenerates, and reference this issue from that comment.
  • scripts/validate_github_editorial_source_projection_inventory.py and its test are already red on
    main against the 2026-08-14 audit snapshot. Fixing them requires reconciling that audit, which is
    Reconcile non-reproducible public projection after face8e4 #253's territory, and is a non-goal here. This issue must not make them worse: after the change
    they must fail with the same message they fail with today.

Non-goals

  • Any change to a public URL, redirect, canonical, or the two legacy spaced filenames.
  • CloudFront/edge delivery of /images/*, new cache headers on media, or anything in decision 20.
  • Rewriting git history to reclaim the 303 MB .git. Whether history is rewritten stays an owner
    decision.
  • The 36 baseline /images/* chrome paths that 404 today (/images/logo.svg, /images/partners/*,
    /images/landing/*, /images/cover.jpg, …). They stay 404.
  • The orphan media/podcast/s24e06-…jpg file and projection reproducibility (Reconcile non-reproducible public projection after face8e4 #253).
  • Re-pinning contract.py / parity digests, or repairing the already-red 2026-08-14 inventory
    validator.
  • Moving /wiki/assets/*, content/faq_assets, content/docs_assets, or /static/ to object
    storage.
  • The DNS/registrar migration and the GitHub Pages → S3 rehost of the legacy site.
  • Any Terraform in this repository. The bucket is owned by DataTalksClub/aws-infra.

Dependencies and merge gate

  • Infrastructure dependency (external, not yet applied): a new main/dtc-website Terraform state
    root in DataTalksClub/aws-infra provides the bucket in production account 387546586013 with
    cross-account read for sandbox 817685572750. The PR is in progress and requires main-account
    credentials. Implementation, tests, tester verification, and PM acceptance do not wait for it
    every gate is satisfiable against the local, memory, and stubbed-s3 backends.
  • Merge/deploy gate. Untracking the media directory means the container image no longer contains
    it, so the deployed environment must have a working s3 backend at that moment. The approved
    commit is therefore held until the [HUMAN] gate below passes. The engineer commits with
    Refs #301; the issue keeps the human label and stays open until the gate is closed.
  • Related, not blocking: Reconcile non-reproducible public projection after face8e4 #253 (projection reproducibility, the orphan asset, the stale audit pins).
    Whichever of Reconcile non-reproducible public projection after face8e4 #253 / Serve public projection media from object storage instead of the git-tracked tree #301 lands second recomputes the derived digests under the media-free scope.
  • Recommended follow-up (do not fix here): the already-red
    scripts/tests/test_github_editorial_source_projection_inventory.py needs its own issue or must be
    absorbed by Reconcile non-reproducible public projection after face8e4 #253.

Acceptance criteria

Public contract and compatibility

  • All 1,253 media records return 200 with the record's exact Content-Type
    (content/tests/test_review_skeleton.py:410-416 still passes, run against the deterministic
    offline store so it needs no hydrated tree and no network).
  • Content-Length is present and correct, and Content-Disposition: inline; filename="…" is
    byte-identical to today's value for a sampled set that includes both spaced filenames; a test
    asserts the header set explicitly.
  • /images/authors/%20aashishnair.jpg and
    /images/podcast/production-ml-search-vector-search-embeddings-hybrid%20search.jpg return
    200 with the correct bytes; no filename is renamed and the builder whitelist at
    scripts/build_public_projection.py:1745 is untouched.
  • HEAD on a media path behaves exactly as it does today; a test pins it.
  • GET /images/../../manage.py and any unknown /images/... path return 404 with the ordinary
    site 404 page; the known orphan podcast/s24e06-…jpg still returns 404.
  • make test-compatibility, make check-links, and make compatibility-artifacts-check stay
    green, and the report states explicitly that no row of
    _docs/compatibility/generated-path-baseline.jsonl changed classification or expected status;
    the 36 currently-404 chrome paths are unchanged.
  • No Cache-Control header is added to media responses on success in this issue.

Integrity contract

  • _tree_sha256 in content/public_data.py and scripts/build_public_projection.py produce the
    same media-free digest for the same tree, proven by a test that computes both.
  • The digest ignores everything under media/: adding, changing, or deleting a file under
    media/ does not change the digest, while changing any JSON artifact or wiki asset does. A
    symlink anywhere under the root, including under media/, still fails closed.
  • manifest.json declares the digest scope and the media-storage block; public_data rejects a
    manifest whose scope declaration is missing or different, with a test for the rejection.
  • The manifest.json diff contains only the derived digest/scope/media-storage fields — no
    record, count, artifact digest, source pin, or projection rule changes.
  • Every served object is checked against provenance.checksum; a test with a deliberately
    corrupted object proves the response is 502, not corrupted bytes.
  • EXPECTED_COUNTS, the builder count constants, the parity counts, and the inventory validator
    counts are unchanged (D7).

Storage backends

  • local, memory, and s3 backends each have focused tests, including the s3 backend
    against a stubbed boto3 client (no new test dependency, following the existing
    courses/tests/test_development_content_import.py pattern). No test performs a real network
    call.
  • make test, make test-content, and the applicable Playwright tier pass with no AWS
    credentials configured and with AWS_* environment variables absent.
  • The S3 key is derived from the matched record only; a test proves a crafted request path
    cannot influence the key.
  • Timeout, missing object, oversized object, and client error each map to the 502 contract with
    one redacted structured log event, and no bucket name, key, credential, or stack trace appears
    in any response body.
  • manage.py check --deploy under production settings fails when the backend is not s3 or the
    bucket is unset; it passes with a configured bucket.

Tooling

  • hydrate materialises all 1,253 objects into the local root from the pinned upstream
    provenance without AWS credentials, verifies every checksum, is idempotent, and refuses to
    write a mismatching object. Its behaviour is covered by tests using local fixtures rather than
    live network.
  • publish uploads only recorded objects with the recorded content type and a SHA-256 checksum,
    skips already-matching objects, refuses orphans, and reports counts. Covered against a stubbed
    client.
  • verify reports missing/extra/mismatched objects and exits non-zero on any of them. Covered
    against a stubbed client and a local root.
  • None of the three commands is reachable from any public, Studio, or admin API request path.

Repository and operations

  • content/public_projection/media/ is no longer git-tracked; .gitignore ignores it and
    .dockerignore excludes it so a local image build matches a CI build.
  • git ls-files content/public_projection | wc -l drops by 1,254 and a fresh git worktree add
    no longer materialises ≈150 MB of images. The engineer records the measured before/after
    worktree size.
  • The container image build succeeds from a clean checkout with no media directory.
  • make lint, make typecheck, make terminology-check, make database-portability-check, and
    the migration-drift check stay green.
  • scripts/validate_github_editorial_source_projection_inventory.py fails with the same message
    as on current main — no better, no worse — and the report says so explicitly.
  • Developer and operator documentation is updated: how to hydrate a fresh clone, what the
    backends are, how to publish and verify, and what a 502 on an image means.

[HUMAN] deploy gate — closed by the owner after the infrastructure lands

  • [HUMAN] The main/dtc-website Terraform root in DataTalksClub/aws-infra is applied and the
    bucket exists with public access blocked, encryption, and versioning.
  • [HUMAN] publish has uploaded all 1,253 objects and verify reports 1,253 matching, 0
    missing, 0 mismatched, 0 extra.
  • [HUMAN] The deployed environment is configured with PUBLIC_MEDIA_STORE_BACKEND=s3 and the
    bucket name, and a deployed spot check of at least one article cover, one portrait, one podcast
    artwork, and one 404 path returns the expected statuses.

Django/integration scenarios

  1. All 1,253 records: 200 plus exact Content-Type against the offline store.
  2. Both spaced-filename records via their percent-encoded spelling: 200, correct bytes, correct
    content type.
  3. The largest object (/images/posts/2025-09-23-ai-dev-tools-zoomcamp…/course-cover.png,
    3,022,797 bytes): 200, correct Content-Length.
  4. Unknown path, traversal attempt, and the orphan file: 404 with the ordinary 404 page.
  5. Corrupted object (checksum mismatch): 502, Cache-Control: no-store, redacted log, no leak.
  6. Store timeout and store client error: 502 with the same contract.
  7. Oversized object beyond PUBLIC_MEDIA_MAX_OBJECT_BYTES: 502, and memory is never unbounded.
  8. local backend with an empty root: 502 for a known record, and manage.py check warns with the
    hydrate command name.
  9. Manifest with the old whole-tree scope, or with a missing scope declaration: ImproperlyConfigured
    at projection load.
  10. Media-free digest equality between runtime and builder; digest insensitivity to media/ changes;
    symlink still fails closed.
  11. s3 backend key derivation, content type, checksum verification path, and single bounded retry
    against a stubbed client.
  12. Production settings check: missing bucket or non-s3 backend fails check --deploy.

Browser scenarios (desktop 1280×720 and mobile 390×844)

Run these with the local backend and the real hydrated objects so the screenshots show real
artwork; state the backend used in the report.

  1. /blog/ai-dev-tools-zoomcamp.html — the 3 MB cover renders, is bounded by its column, and there
    is no horizontal overflow or broken-image icon.
  2. /people/aashishnair.html — the portrait whose filename begins with a space renders inside the
    profile card, and the page shows no broken image.
  3. /podcast/production-ml-search-vector-search-embeddings-hybrid-search.html — the artwork whose
    filename contains a space renders.
  4. /blog and /podcast hubs — every card image loads; no placeholder, no layout shift, no
    console/network 4xx or 5xx for an /images/ request.
  5. /images/does-not-exist.jpg — the ordinary site 404 page, correct copy, working navigation, no
    stack trace and no store detail.
  6. Fault-injected store failure — an image-bearing page still renders its text, headings, and
    navigation; the failing image degrades to its alt text; nothing on screen or in the response
    exposes a bucket name, key, or credential.

Evidence and reporting

  • The engineer produces the versioned verification plan per _docs/ci/change-selective-ci.md with
    exact base/head, graph and plan digests, and every rerun / reused / skipped /
    not_applicable disposition, and leaves the worktree uncommitted and frozen.
  • The tester independently recomputes the plan, runs focused content Django tests plus
    make test-playwright-core (template/render impact is limited, but every image-bearing public page
    is affected), and stores desktop and mobile screenshots under .tmp/screenshots/, reading each one
    and reporting whether it shows the expected page with real artwork.
  • Reports must state the store backend used for each run, the measured worktree/image size change,
    and the exact statuses observed for the six browser scenarios.
  • No bucket name that is not already public, no account credential, no presigned URL, and no
    registration or production data appears in any log, screenshot, comment, or report.

Risks

  • Deploying before the bucket exists breaks every image. Mitigated by the [HUMAN] merge/deploy
    gate above; do not merge around it.
  • Silent digest-scope drift. Mitigated by the explicit tree_digest_scope declaration, the
    fail-closed rejection of an undeclared scope, and the test that recomputes the digest.
  • A stale object in the bucket after a future content update. Fails closed as 502 for that one
    path rather than serving wrong bytes; publish before deploying a new media.json.
  • Origin cost/latency of per-request object reads. Acceptable at 34 KB median with a 3 MB
    ceiling, and superseded by Phase 2. Any in-process caching added here must be bounded, keyed by
    record checksum, off by default in tests, and covered by tests.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P0Must-have or release-blockingcontentArea: contentdata-migrationArea: data-migrationenhancementNew feature or requesthumanCode complete; manual verification requiredinfraArea: infraoperationsArea: operationstestingArea: testing

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions