You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Public projection images keep working exactly as they do today at exactly the same URLs, while the
1,253 image objects (≈154 MB, 1,254 files on disk) stop being carried in the git working tree. A
visitor sees no difference: /images/... still returns 200 with the same Content-Type, the same
bytes, and the same page rendering. A contributor gets a repository whose checkout — and every one
of the many .tmp/ worktrees — no longer materialises 150 MB of binaries per copy.
This issue delivers Phase 1 only: Django reads each object from a pluggable media store and
serves it at the unchanged public path. Phase 2 (CloudFront serving /images/* straight from an S3
origin with Django out of the request path) is a named non-goal here — see "Serving strategy".
Normative authority
_docs/PROCESS.md — lifecycle, evidence, [HUMAN] criteria, no-PR local merge.
_docs/specs/03-github-content-and-people.md — ContentAsset carries "release, source path,
stable public path, immutable S3 key, content type, size, and checksum", and "The Django asset
resolver selects the active release while keeping the original public URL; edge caching makes the
stable path efficient." This is the normative basis for Phase 1.
_docs/specs/02-url-link-seo-compatibility.md — preserve-first URL contract and the route cache
classes.
_docs/specs/08-aws-development-terraform.md — "S3 bucket for immutable release assets, exports,
and controlled operational artifacts …"; "CloudFront/application resolution keeps public asset
paths unchanged."
_docs/specs/07-security-privacy-operations.md — fail-closed behaviour, redacted logging, no
secret or credential exposure.
_docs/specs/10-verification-strategy.md and _docs/ci/change-selective-ci.md — verification plan
and evidence rules.
_docs/architecture/app-boundaries.md — content owns "versioned GitHub-owned read models and
public content presentation".
_docs/compatibility/generated-path-baseline.jsonl — 2,937 rows, every one classification: preserve / expected_status: 200.
Verified current behaviour (re-checked at 55f6743)
Fact
Evidence
1,253 media records, 815 from DataTalksClub/content@1375c50…, 438 from DataTalksClub/datatalksclub.github.io@ee43d3f…
content/public_projection/media.json
Every record already carries provenance.checksum (sha256), repository, revision, source_path, source_url
media.json
1,254 files under content/public_projection/media/, 154,154,378 bytes; median object 33.9 KB, largest 3,022,797 bytes
on-disk census
Exactly one orphan file with no record: media/podcast/s24e06-how-to-build-ai-that-actually-ships-in-production.jpg; it returns 404 today
Serving path: content/public_views.py:1151media() → FileResponse(path.open("rb"), content_type=record["content_type"]), routed at content/public_urls.py:235
code
Live response headers on a hit: Content-Type, Content-Length, Content-Disposition: inline; filename="…", plus the usual security headers. No Cache-Control.
GET /images/authors/%20aashishnair.jpg
Runtime integrity today: content/public_data.py:625 compares manifest["tree_sha256"] with _tree_sha256(PROJECTION_ROOT) (:339-352), which reads every byte of every file under the projection root, media included
code
Two records contain a literal space: /images/authors/ aashishnair.jpg and /images/podcast/production-ml-search-vector-search-embeddings-hybrid search.jpg; the builder has a deliberate one-line whitelist at scripts/build_public_projection.py:1745
code, media.json
1,281 baseline rows start with /images/; 1,245 of them match a projection record, 36 do not and return 404 today (site chrome: /images/logo.svg, /images/partners/*, /images/landing/*, …). 8 records are not in the baseline.
baseline vs media.json, request checks
content_sync/dtc_content/contract.py:34PROJECTION_TREE_SHA256 = 9baae604… and PROJECTION_MANIFEST_SHA256 = 6e09a551…already differ from the checked manifest (29a1819f… / 266d8daa…). The parity gate at content_sync/dtc_content/parity.py:345-346 is opt-in (DTC_CONTENT_ACCEPTED_CHECKOUT) and is already red for reasons this issue does not cause.
code, digests
scripts/validate_github_editorial_source_projection_inventory.py is already failing on current main (manifest preferred content source drift), and so is scripts/tests/test_github_editorial_source_projection_inventory.py::test_inventory_matches_the_exact_checked_in_snapshot. Its pinned media.json digest at :180 (6b6670d0…) is stale versus the live 199c7860….
uv run python scripts/validate_…py; uv run pytest -k snapshot
No Playwright test depends on real projection image bytes (the only naturalWidth assertions use data: URLs and static fixtures)
playwright_tests/
Scope
A media store abstraction in the content app with three backends: local (filesystem
directory), memory (deterministic offline fixture for tests/CI), s3 (boto3, already a
dependency). Selected by settings; default requires no AWS credentials and no network.
content/public_views.py::media() reads through the store instead of the projection tree, with
fail-closed per-object integrity and unchanged observable success responses.
Integrity contract change: the projection complete-tree digest covers the JSON artifacts and wiki
assets only; per-object integrity moves to the provenance.checksum already recorded per media
record. The manifest states the digest scope explicitly so its meaning cannot drift silently.
Operator tooling: hydrate (materialise the objects locally from pinned upstream, credential
free), publish (upload to the bucket), verify (compare store contents to media.json).
Repository change: content/public_projection/media/ stops being git-tracked and is ignored,
with .dockerignore updated so a local image build matches a CI build.
Documentation of the local developer flow and the operator publish/verify flow.
Design decisions
D1 — Tree digest scope (resolves the tree_sha256 blocker)
_tree_sha256 in bothcontent/public_data.py:339 and scripts/build_public_projection.py:2690
walks the projection root and skips, in addition to manifest.json, everything under media/. The
symlink rejection stays exactly as it is: a symlink anywhere under the root, including under media/, is still a hard failure.
manifest.json gains an explicit, machine-readable scope declaration next to the digest, for
example:
"tree_sha256": "<recomputed, media-free>",
"tree_digest_scope": "projection artifacts and wiki assets; excludes manifest.json and media/",
"media_storage": {
"location": "object-store",
"records": "media.json",
"count": 1253,
"integrity": "per-record provenance.checksum"
}
content/public_data.py verifies the scope declaration as well as the digest, so a manifest produced
by an older builder (whole-tree scope) fails closed rather than being silently accepted.
Because the checked projection is currently not reproducible from upstream (#253), the engineer
must not re-run the full builder to obtain the new digest. Provide a narrow, deterministic re-pin
utility (a --repin-digests mode of the existing builder, or a small dedicated script) that
recomputes only the derived digest fields from the already-checked artifacts and rewrites manifest.json. The manifest diff must contain only the digest/scope/media-storage fields; a test
must recompute the field from the tree so the value can never be hand-typed.
D2 — Per-object integrity
Every served object is verified against its record's provenance.checksum before any byte reaches
the client. Verification may use the store's own server-side SHA-256 (S3 ChecksumSHA256) when the
store supplies one, otherwise it is computed over the retrieved bytes. Objects are size-bounded
(configurable, default above the 3,022,797-byte largest known object) so a response is never
unbounded in memory. Unverified bytes, partial bytes, and placeholder substitutes are never served.
Decision: Django reads the object from the store and returns it at the same URL. Reasons:
the production CloudFront distribution does not exist yet, and decision 20 in _docs/specs/open-decisions.md governs its cache/WAF plan; nothing here may pre-empt it;
spec 03 already prescribes exactly this shape ("The Django asset resolver … keeping the original
public URL");
it preserves all 2,937 baseline rows and today's response header shape with zero edge
configuration, so the compatibility gate stays measurable;
one fail-closed integrity check stays in code rather than being dissolved into edge config;
it is a prerequisite for Phase 2 anyway: the bucket must exist, be populated, and use
path-mirrored keys before any distribution can be pointed at it.
The view must not redirect to a presigned or public S3 URL. A 302 would change the response
contract, leak the bucket name, and break the baseline's expected_status: 200.
Phase 2 (a CloudFront behaviour for /images/* with an S3 origin and Django out of the path) is a
non-goal of this issue and needs its own issue once the production distribution exists.
D4 — Backends, settings, and credential-free defaults
Setting
Default
Meaning
PUBLIC_MEDIA_STORE_BACKEND
local
local | memory | s3
PUBLIC_MEDIA_LOCAL_ROOT
content/public_projection/media
filesystem root for local
PUBLIC_MEDIA_S3_BUCKET
""
bucket name for s3
PUBLIC_MEDIA_S3_PREFIX
public-projection
key prefix for s3
PUBLIC_MEDIA_S3_REGION
""
region for s3
PUBLIC_MEDIA_S3_ENDPOINT_URL
""
optional; lets a developer point at a local/faked endpoint
PUBLIC_MEDIA_S3_TIMEOUT_SECONDS
small bounded value
connect/read timeout; at most one retry
PUBLIC_MEDIA_MAX_OBJECT_BYTES
8 MiB
fail-closed size bound
Rules:
The default configuration in base, development, local, and test settings is local. No
AWS credential, network call, or new dependency is required to run the site or the test suite.
memory is a deterministic offline fixture store derived from media.json: for each record it
serves a minimal valid image of the recorded content type whose sha256 is used as that
record's expected checksum inside the fixture. It refuses to activate under production settings.
CI test jobs set PUBLIC_MEDIA_STORE_BACKEND=memory in the workflow environment (not in the
Makefile), so CI is offline and deterministic while a local developer or tester still renders the
real hydrated bytes with the default local backend.
s3 is required under production settings: manage.py check --deploy fails when DTC_ENVIRONMENT=production and the backend is not s3 or the bucket is empty.
The S3 client is constructed from the ambient role/credential chain. This issue introduces no
credential handling, no static keys, and no secret in settings, logs, or reports.
D5 — Keys, bucket population, and orphans
Key = f"{PUBLIC_MEDIA_S3_PREFIX}/{record['record_key']}", i.e. path-mirrored
(public-projection/images/authors/ aashishnair.jpg). Path-mirrored keys are what a future
CloudFront origin needs; a content-addressed layout would require an edge key-value lookup, which
spec 08 forbids.
The key is derived only from the matched record, never from the raw request path. Traversal,
encoding, and the two spaced filenames therefore cannot reach the store as attacker-controlled
input.
hydrate materialises exactly the 1,253 records into the local root from the pinned upstream
revisions recorded in each record's provenance. It needs no AWS credentials, is idempotent and
resumable, verifies every object against provenance.checksum, and never writes an object whose
digest does not match. It may also hydrate from the configured store (--from-store) once the
bucket exists.
publish uploads exactly the 1,253 records from a local root, sets Content-Type from the record
and ChecksumAlgorithm=SHA256, skips objects already present with a matching checksum, and refuses to upload any file that has no record (the known orphan stays unpublished and keeps
returning 404). It prints a report of added/changed/skipped/orphan counts.
verify compares the configured store against media.json and exits non-zero on any missing,
extra, or checksum-mismatched object.
Path with no media.json record → 404, exactly as today (including the known orphan file and
the 36 not-yet-implemented baseline chrome paths).
Record exists but the object is missing, oversized, unreadable, times out, or fails its checksum → 502 with Cache-Control: no-store, no upstream detail in the body, no bucket/key/credential in
the response, and exactly one redacted structured log event. Never a 404 (that would let the
edge cache an outage as "not found"), never a placeholder, never unverified bytes.
local backend with an empty root → the same 502 path, plus a manage.py checkwarning
naming the hydrate command, so a fresh clone is told what to run instead of silently rendering
broken images.
D7 — Count canaries stay put
The record set does not change. content/public_data.py:66 ("media": 1_253), scripts/build_public_projection.py:117-121 (815 / 438 / 1,253 / fallbacks), content_sync/dtc_content/parity.py:346,867-868, and scripts/validate_github_editorial_source_projection_inventory.py:102,182 all keep their current
values. A diff that changes any media count is a defect in this issue.
D8 — Already-red gates this issue must not adopt
content_sync/dtc_content/contract.py:34PROJECTION_TREE_SHA256 and PROJECTION_MANIFEST_SHA256
are already stale versus the checked manifest, and the parity gate is already red. Do not re-pin
the hex values here — that would silently accept the face8e4-era drift that Reconcile non-reproducible public projection after face8e4 #253 exists to
reconcile. Add a comment beside the pin recording that the digest scope is now media-free so Reconcile non-reproducible public projection after face8e4 #253
re-pins correctly when it regenerates, and reference this issue from that comment.
scripts/validate_github_editorial_source_projection_inventory.py and its test are already red on
main against the 2026-08-14 audit snapshot. Fixing them requires reconciling that audit, which is Reconcile non-reproducible public projection after face8e4 #253's territory, and is a non-goal here. This issue must not make them worse: after the change
they must fail with the same message they fail with today.
Non-goals
Any change to a public URL, redirect, canonical, or the two legacy spaced filenames.
CloudFront/edge delivery of /images/*, new cache headers on media, or anything in decision 20.
Rewriting git history to reclaim the 303 MB .git. Whether history is rewritten stays an owner
decision.
The 36 baseline /images/* chrome paths that 404 today (/images/logo.svg, /images/partners/*, /images/landing/*, /images/cover.jpg, …). They stay 404.
Re-pinning contract.py / parity digests, or repairing the already-red 2026-08-14 inventory
validator.
Moving /wiki/assets/*, content/faq_assets, content/docs_assets, or /static/ to object
storage.
The DNS/registrar migration and the GitHub Pages → S3 rehost of the legacy site.
Any Terraform in this repository. The bucket is owned by DataTalksClub/aws-infra.
Dependencies and merge gate
Infrastructure dependency (external, not yet applied): a new main/dtc-website Terraform state
root in DataTalksClub/aws-infra provides the bucket in production account 387546586013 with
cross-account read for sandbox 817685572750. The PR is in progress and requires main-account
credentials. Implementation, tests, tester verification, and PM acceptance do not wait for it —
every gate is satisfiable against the local, memory, and stubbed-s3 backends.
Merge/deploy gate. Untracking the media directory means the container image no longer contains
it, so the deployed environment must have a working s3 backend at that moment. The approved
commit is therefore held until the [HUMAN] gate below passes. The engineer commits with Refs #301; the issue keeps the human label and stays open until the gate is closed.
All 1,253 media records return 200 with the record's exact Content-Type
(content/tests/test_review_skeleton.py:410-416 still passes, run against the deterministic
offline store so it needs no hydrated tree and no network).
Content-Length is present and correct, and Content-Disposition: inline; filename="…" is
byte-identical to today's value for a sampled set that includes both spaced filenames; a test
asserts the header set explicitly.
/images/authors/%20aashishnair.jpg and /images/podcast/production-ml-search-vector-search-embeddings-hybrid%20search.jpg return 200 with the correct bytes; no filename is renamed and the builder whitelist at scripts/build_public_projection.py:1745 is untouched.
HEAD on a media path behaves exactly as it does today; a test pins it.
GET /images/../../manage.py and any unknown /images/... path return 404 with the ordinary
site 404 page; the known orphan podcast/s24e06-…jpg still returns 404.
make test-compatibility, make check-links, and make compatibility-artifacts-check stay
green, and the report states explicitly that no row of _docs/compatibility/generated-path-baseline.jsonl changed classification or expected status;
the 36 currently-404 chrome paths are unchanged.
No Cache-Control header is added to media responses on success in this issue.
Integrity contract
_tree_sha256 in content/public_data.py and scripts/build_public_projection.py produce the
same media-free digest for the same tree, proven by a test that computes both.
The digest ignores everything under media/: adding, changing, or deleting a file under media/ does not change the digest, while changing any JSON artifact or wiki asset does. A
symlink anywhere under the root, including under media/, still fails closed.
manifest.json declares the digest scope and the media-storage block; public_data rejects a
manifest whose scope declaration is missing or different, with a test for the rejection.
The manifest.json diff contains only the derived digest/scope/media-storage fields — no
record, count, artifact digest, source pin, or projection rule changes.
Every served object is checked against provenance.checksum; a test with a deliberately
corrupted object proves the response is 502, not corrupted bytes.
EXPECTED_COUNTS, the builder count constants, the parity counts, and the inventory validator
counts are unchanged (D7).
Storage backends
local, memory, and s3 backends each have focused tests, including the s3 backend
against a stubbed boto3 client (no new test dependency, following the existing courses/tests/test_development_content_import.py pattern). No test performs a real network
call.
make test, make test-content, and the applicable Playwright tier pass with no AWS
credentials configured and with AWS_* environment variables absent.
The S3 key is derived from the matched record only; a test proves a crafted request path
cannot influence the key.
Timeout, missing object, oversized object, and client error each map to the 502 contract with
one redacted structured log event, and no bucket name, key, credential, or stack trace appears
in any response body.
manage.py check --deploy under production settings fails when the backend is not s3 or the
bucket is unset; it passes with a configured bucket.
Tooling
hydrate materialises all 1,253 objects into the local root from the pinned upstream
provenance without AWS credentials, verifies every checksum, is idempotent, and refuses to
write a mismatching object. Its behaviour is covered by tests using local fixtures rather than
live network.
publish uploads only recorded objects with the recorded content type and a SHA-256 checksum,
skips already-matching objects, refuses orphans, and reports counts. Covered against a stubbed
client.
verify reports missing/extra/mismatched objects and exits non-zero on any of them. Covered
against a stubbed client and a local root.
None of the three commands is reachable from any public, Studio, or admin API request path.
Repository and operations
content/public_projection/media/ is no longer git-tracked; .gitignore ignores it and .dockerignore excludes it so a local image build matches a CI build.
git ls-files content/public_projection | wc -l drops by 1,254 and a fresh git worktree add
no longer materialises ≈150 MB of images. The engineer records the measured before/after
worktree size.
The container image build succeeds from a clean checkout with no media directory.
make lint, make typecheck, make terminology-check, make database-portability-check, and
the migration-drift check stay green.
scripts/validate_github_editorial_source_projection_inventory.py fails with the same message
as on current main — no better, no worse — and the report says so explicitly.
Developer and operator documentation is updated: how to hydrate a fresh clone, what the
backends are, how to publish and verify, and what a 502 on an image means.
[HUMAN] deploy gate — closed by the owner after the infrastructure lands
[HUMAN] The main/dtc-website Terraform root in DataTalksClub/aws-infra is applied and the
bucket exists with public access blocked, encryption, and versioning.
[HUMAN]publish has uploaded all 1,253 objects and verify reports 1,253 matching, 0
missing, 0 mismatched, 0 extra.
[HUMAN] The deployed environment is configured with PUBLIC_MEDIA_STORE_BACKEND=s3 and the
bucket name, and a deployed spot check of at least one article cover, one portrait, one podcast
artwork, and one 404 path returns the expected statuses.
Django/integration scenarios
All 1,253 records: 200 plus exact Content-Type against the offline store.
Both spaced-filename records via their percent-encoded spelling: 200, correct bytes, correct
content type.
The largest object (/images/posts/2025-09-23-ai-dev-tools-zoomcamp…/course-cover.png,
3,022,797 bytes): 200, correct Content-Length.
Unknown path, traversal attempt, and the orphan file: 404 with the ordinary 404 page.
Store timeout and store client error: 502 with the same contract.
Oversized object beyond PUBLIC_MEDIA_MAX_OBJECT_BYTES: 502, and memory is never unbounded.
local backend with an empty root: 502 for a known record, and manage.py check warns with the
hydrate command name.
Manifest with the old whole-tree scope, or with a missing scope declaration: ImproperlyConfigured
at projection load.
Media-free digest equality between runtime and builder; digest insensitivity to media/ changes;
symlink still fails closed.
s3 backend key derivation, content type, checksum verification path, and single bounded retry
against a stubbed client.
Production settings check: missing bucket or non-s3 backend fails check --deploy.
Browser scenarios (desktop 1280×720 and mobile 390×844)
Run these with the local backend and the real hydrated objects so the screenshots show real
artwork; state the backend used in the report.
/blog/ai-dev-tools-zoomcamp.html — the 3 MB cover renders, is bounded by its column, and there
is no horizontal overflow or broken-image icon.
/people/aashishnair.html — the portrait whose filename begins with a space renders inside the
profile card, and the page shows no broken image.
/podcast/production-ml-search-vector-search-embeddings-hybrid-search.html — the artwork whose
filename contains a space renders.
/blog and /podcast hubs — every card image loads; no placeholder, no layout shift, no
console/network 4xx or 5xx for an /images/ request.
/images/does-not-exist.jpg — the ordinary site 404 page, correct copy, working navigation, no
stack trace and no store detail.
Fault-injected store failure — an image-bearing page still renders its text, headings, and
navigation; the failing image degrades to its alt text; nothing on screen or in the response
exposes a bucket name, key, or credential.
Evidence and reporting
The engineer produces the versioned verification plan per _docs/ci/change-selective-ci.md with
exact base/head, graph and plan digests, and every rerun / reused / skipped / not_applicable disposition, and leaves the worktree uncommitted and frozen.
The tester independently recomputes the plan, runs focused content Django tests plus make test-playwright-core (template/render impact is limited, but every image-bearing public page
is affected), and stores desktop and mobile screenshots under .tmp/screenshots/, reading each one
and reporting whether it shows the expected page with real artwork.
Reports must state the store backend used for each run, the measured worktree/image size change,
and the exact statuses observed for the six browser scenarios.
No bucket name that is not already public, no account credential, no presigned URL, and no
registration or production data appears in any log, screenshot, comment, or report.
Risks
Deploying before the bucket exists breaks every image. Mitigated by the [HUMAN] merge/deploy
gate above; do not merge around it.
Silent digest-scope drift. Mitigated by the explicit tree_digest_scope declaration, the
fail-closed rejection of an undeclared scope, and the test that recomputes the digest.
A stale object in the bucket after a future content update. Fails closed as 502 for that one
path rather than serving wrong bytes; publish before deploying a new media.json.
Origin cost/latency of per-request object reads. Acceptable at 34 KB median with a 3 MB
ceiling, and superseded by Phase 2. Any in-process caching added here must be bounded, keyed by
record checksum, off by default in tests, and covered by tests.
Product outcome
Public projection images keep working exactly as they do today at exactly the same URLs, while the
1,253 image objects (≈154 MB, 1,254 files on disk) stop being carried in the git working tree. A
visitor sees no difference:
/images/...still returns200with the sameContent-Type, the samebytes, and the same page rendering. A contributor gets a repository whose checkout — and every one
of the many
.tmp/worktrees — no longer materialises 150 MB of binaries per copy.This issue delivers Phase 1 only: Django reads each object from a pluggable media store and
serves it at the unchanged public path. Phase 2 (CloudFront serving
/images/*straight from an S3origin with Django out of the request path) is a named non-goal here — see "Serving strategy".
Normative authority
_docs/PROCESS.md— lifecycle, evidence,[HUMAN]criteria, no-PR local merge._docs/specs/03-github-content-and-people.md—ContentAssetcarries "release, source path,stable public path, immutable S3 key, content type, size, and checksum", and "The Django asset
resolver selects the active release while keeping the original public URL; edge caching makes the
stable path efficient." This is the normative basis for Phase 1.
_docs/specs/02-url-link-seo-compatibility.md— preserve-first URL contract and the route cacheclasses.
_docs/specs/08-aws-development-terraform.md— "S3 bucket for immutable release assets, exports,and controlled operational artifacts …"; "CloudFront/application resolution keeps public asset
paths unchanged."
_docs/specs/07-security-privacy-operations.md— fail-closed behaviour, redacted logging, nosecret or credential exposure.
_docs/specs/10-verification-strategy.mdand_docs/ci/change-selective-ci.md— verification planand evidence rules.
_docs/architecture/app-boundaries.md—contentowns "versioned GitHub-owned read models andpublic content presentation".
_docs/compatibility/generated-path-baseline.jsonl— 2,937 rows, every oneclassification: preserve/expected_status: 200.Verified current behaviour (re-checked at
55f6743)DataTalksClub/content@1375c50…, 438 fromDataTalksClub/datatalksclub.github.io@ee43d3f…content/public_projection/media.jsonprovenance.checksum(sha256),repository,revision,source_path,source_urlmedia.jsoncontent/public_projection/media/, 154,154,378 bytes; median object 33.9 KB, largest 3,022,797 bytesmedia/podcast/s24e06-how-to-build-ai-that-actually-ships-in-production.jpg; it returns 404 todaycontent/public_views.py:1151media()→FileResponse(path.open("rb"), content_type=record["content_type"]), routed atcontent/public_urls.py:235Content-Type,Content-Length,Content-Disposition: inline; filename="…", plus the usual security headers. NoCache-Control.GET /images/authors/%20aashishnair.jpgcontent/public_data.py:625comparesmanifest["tree_sha256"]with_tree_sha256(PROJECTION_ROOT)(:339-352), which reads every byte of every file under the projection root, media included/images/authors/ aashishnair.jpgand/images/podcast/production-ml-search-vector-search-embeddings-hybrid search.jpg; the builder has a deliberate one-line whitelist atscripts/build_public_projection.py:1745media.json/images/; 1,245 of them match a projection record, 36 do not and return 404 today (site chrome:/images/logo.svg,/images/partners/*,/images/landing/*, …). 8 records are not in the baseline.media.json, request checkscontent_sync/dtc_content/contract.py:34PROJECTION_TREE_SHA256 = 9baae604…andPROJECTION_MANIFEST_SHA256 = 6e09a551…already differ from the checked manifest (29a1819f…/266d8daa…). The parity gate atcontent_sync/dtc_content/parity.py:345-346is opt-in (DTC_CONTENT_ACCEPTED_CHECKOUT) and is already red for reasons this issue does not cause.scripts/validate_github_editorial_source_projection_inventory.pyis already failing on current main (manifest preferred content source drift), and so isscripts/tests/test_github_editorial_source_projection_inventory.py::test_inventory_matches_the_exact_checked_in_snapshot. Its pinnedmedia.jsondigest at:180(6b6670d0…) is stale versus the live199c7860….uv run python scripts/validate_…py;uv run pytest -k snapshotnaturalWidthassertions usedata:URLs and static fixtures)playwright_tests/Scope
contentapp with three backends:local(filesystemdirectory),
memory(deterministic offline fixture for tests/CI),s3(boto3, already adependency). Selected by settings; default requires no AWS credentials and no network.
content/public_views.py::media()reads through the store instead of the projection tree, withfail-closed per-object integrity and unchanged observable success responses.
assets only; per-object integrity moves to the
provenance.checksumalready recorded per mediarecord. The manifest states the digest scope explicitly so its meaning cannot drift silently.
hydrate(materialise the objects locally from pinned upstream, credentialfree),
publish(upload to the bucket),verify(compare store contents tomedia.json).content/public_projection/media/stops being git-tracked and is ignored,with
.dockerignoreupdated so a local image build matches a CI build.Design decisions
D1 — Tree digest scope (resolves the
tree_sha256blocker)_tree_sha256in bothcontent/public_data.py:339andscripts/build_public_projection.py:2690walks the projection root and skips, in addition to
manifest.json, everything undermedia/. Thesymlink rejection stays exactly as it is: a symlink anywhere under the root, including under
media/, is still a hard failure.manifest.jsongains an explicit, machine-readable scope declaration next to the digest, forexample:
content/public_data.pyverifies the scope declaration as well as the digest, so a manifest producedby an older builder (whole-tree scope) fails closed rather than being silently accepted.
Because the checked projection is currently not reproducible from upstream (#253), the engineer
must not re-run the full builder to obtain the new digest. Provide a narrow, deterministic re-pin
utility (a
--repin-digestsmode of the existing builder, or a small dedicated script) thatrecomputes only the derived digest fields from the already-checked artifacts and rewrites
manifest.json. The manifest diff must contain only the digest/scope/media-storage fields; a testmust recompute the field from the tree so the value can never be hand-typed.
D2 — Per-object integrity
Every served object is verified against its record's
provenance.checksumbefore any byte reachesthe client. Verification may use the store's own server-side SHA-256 (S3
ChecksumSHA256) when thestore supplies one, otherwise it is computed over the retrieved bytes. Objects are size-bounded
(configurable, default above the 3,022,797-byte largest known object) so a response is never
unbounded in memory. Unverified bytes, partial bytes, and placeholder substitutes are never served.
D3 — Serving strategy: Django-mediated origin read (Phase 1)
Decision: Django reads the object from the store and returns it at the same URL. Reasons:
_docs/specs/open-decisions.mdgoverns its cache/WAF plan; nothing here may pre-empt it;public URL");
configuration, so the compatibility gate stays measurable;
path-mirrored keys before any distribution can be pointed at it.
The view must not redirect to a presigned or public S3 URL. A
302would change the responsecontract, leak the bucket name, and break the baseline's
expected_status: 200.Phase 2 (a CloudFront behaviour for
/images/*with an S3 origin and Django out of the path) is anon-goal of this issue and needs its own issue once the production distribution exists.
D4 — Backends, settings, and credential-free defaults
PUBLIC_MEDIA_STORE_BACKENDlocallocal|memory|s3PUBLIC_MEDIA_LOCAL_ROOTcontent/public_projection/medialocalPUBLIC_MEDIA_S3_BUCKET""s3PUBLIC_MEDIA_S3_PREFIXpublic-projections3PUBLIC_MEDIA_S3_REGION""s3PUBLIC_MEDIA_S3_ENDPOINT_URL""PUBLIC_MEDIA_S3_TIMEOUT_SECONDSPUBLIC_MEDIA_MAX_OBJECT_BYTES8 MiBRules:
base,development,local, andtestsettings islocal. NoAWS credential, network call, or new dependency is required to run the site or the test suite.
memoryis a deterministic offline fixture store derived frommedia.json: for each record itserves a minimal valid image of the recorded content type whose sha256 is used as that
record's expected checksum inside the fixture. It refuses to activate under production settings.
PUBLIC_MEDIA_STORE_BACKEND=memoryin the workflow environment (not in theMakefile), so CI is offline and deterministic while a local developer or tester still renders the
real hydrated bytes with the default
localbackend.s3is required under production settings:manage.py check --deployfails whenDTC_ENVIRONMENT=productionand the backend is nots3or the bucket is empty.credential handling, no static keys, and no secret in settings, logs, or reports.
D5 — Keys, bucket population, and orphans
f"{PUBLIC_MEDIA_S3_PREFIX}/{record['record_key']}", i.e. path-mirrored(
public-projection/images/authors/ aashishnair.jpg). Path-mirrored keys are what a futureCloudFront origin needs; a content-addressed layout would require an edge key-value lookup, which
spec 08 forbids.
encoding, and the two spaced filenames therefore cannot reach the store as attacker-controlled
input.
hydratematerialises exactly the 1,253 records into the local root from the pinned upstreamrevisions recorded in each record's provenance. It needs no AWS credentials, is idempotent and
resumable, verifies every object against
provenance.checksum, and never writes an object whosedigest does not match. It may also hydrate from the configured store (
--from-store) once thebucket exists.
publishuploads exactly the 1,253 records from a local root, setsContent-Typefrom the recordand
ChecksumAlgorithm=SHA256, skips objects already present with a matching checksum, andrefuses to upload any file that has no record (the known orphan stays unpublished and keeps
returning 404). It prints a report of added/changed/skipped/orphan counts.
verifycompares the configured store againstmedia.jsonand exits non-zero on any missing,extra, or checksum-mismatched object.
ContentAssetwork(Expand direct-sync schema and reconcile historical active content #273–Cut public content reads and projections over to direct-sync authority #276) and are out of scope here.
D6 — Failure semantics
media.jsonrecord →404, exactly as today (including the known orphan file andthe 36 not-yet-implemented baseline chrome paths).
502withCache-Control: no-store, no upstream detail in the body, no bucket/key/credential inthe response, and exactly one redacted structured log event. Never a
404(that would let theedge cache an outage as "not found"), never a placeholder, never unverified bytes.
localbackend with an empty root → the same502path, plus amanage.py checkwarningnaming the hydrate command, so a fresh clone is told what to run instead of silently rendering
broken images.
D7 — Count canaries stay put
The record set does not change.
content/public_data.py:66("media": 1_253),scripts/build_public_projection.py:117-121(815 / 438 / 1,253 / fallbacks),content_sync/dtc_content/parity.py:346,867-868, andscripts/validate_github_editorial_source_projection_inventory.py:102,182all keep their currentvalues. A diff that changes any media count is a defect in this issue.
D8 — Already-red gates this issue must not adopt
content_sync/dtc_content/contract.py:34PROJECTION_TREE_SHA256andPROJECTION_MANIFEST_SHA256are already stale versus the checked manifest, and the parity gate is already red. Do not re-pin
the hex values here — that would silently accept the
face8e4-era drift that Reconcile non-reproducible public projection after face8e4 #253 exists toreconcile. Add a comment beside the pin recording that the digest scope is now media-free so Reconcile non-reproducible public projection after face8e4 #253
re-pins correctly when it regenerates, and reference this issue from that comment.
scripts/validate_github_editorial_source_projection_inventory.pyand its test are already red onmain against the 2026-08-14 audit snapshot. Fixing them requires reconciling that audit, which is
Reconcile non-reproducible public projection after face8e4 #253's territory, and is a non-goal here. This issue must not make them worse: after the change
they must fail with the same message they fail with today.
Non-goals
/images/*, new cache headers on media, or anything in decision 20..git. Whether history is rewritten stays an ownerdecision.
/images/*chrome paths that 404 today (/images/logo.svg,/images/partners/*,/images/landing/*,/images/cover.jpg, …). They stay 404.media/podcast/s24e06-…jpgfile and projection reproducibility (Reconcile non-reproducible public projection after face8e4 #253).contract.py/ parity digests, or repairing the already-red 2026-08-14 inventoryvalidator.
/wiki/assets/*,content/faq_assets,content/docs_assets, or/static/to objectstorage.
DataTalksClub/aws-infra.Dependencies and merge gate
main/dtc-websiteTerraform stateroot in
DataTalksClub/aws-infraprovides the bucket in production account387546586013withcross-account read for sandbox
817685572750. The PR is in progress and requires main-accountcredentials. Implementation, tests, tester verification, and PM acceptance do not wait for it —
every gate is satisfiable against the
local,memory, and stubbed-s3backends.it, so the deployed environment must have a working
s3backend at that moment. The approvedcommit is therefore held until the
[HUMAN]gate below passes. The engineer commits withRefs #301; the issue keeps thehumanlabel and stays open until the gate is closed.Whichever of Reconcile non-reproducible public projection after face8e4 #253 / Serve public projection media from object storage instead of the git-tracked tree #301 lands second recomputes the derived digests under the media-free scope.
scripts/tests/test_github_editorial_source_projection_inventory.pyneeds its own issue or must beabsorbed by Reconcile non-reproducible public projection after face8e4 #253.
Acceptance criteria
Public contract and compatibility
200with the record's exactContent-Type(
content/tests/test_review_skeleton.py:410-416still passes, run against the deterministicoffline store so it needs no hydrated tree and no network).
Content-Lengthis present and correct, andContent-Disposition: inline; filename="…"isbyte-identical to today's value for a sampled set that includes both spaced filenames; a test
asserts the header set explicitly.
/images/authors/%20aashishnair.jpgand/images/podcast/production-ml-search-vector-search-embeddings-hybrid%20search.jpgreturn200with the correct bytes; no filename is renamed and the builder whitelist atscripts/build_public_projection.py:1745is untouched.HEADon a media path behaves exactly as it does today; a test pins it.GET /images/../../manage.pyand any unknown/images/...path return404with the ordinarysite 404 page; the known orphan
podcast/s24e06-…jpgstill returns404.make test-compatibility,make check-links, andmake compatibility-artifacts-checkstaygreen, and the report states explicitly that no row of
_docs/compatibility/generated-path-baseline.jsonlchanged classification or expected status;the 36 currently-404 chrome paths are unchanged.
Cache-Controlheader is added to media responses on success in this issue.Integrity contract
_tree_sha256incontent/public_data.pyandscripts/build_public_projection.pyproduce thesame media-free digest for the same tree, proven by a test that computes both.
media/: adding, changing, or deleting a file undermedia/does not change the digest, while changing any JSON artifact or wiki asset does. Asymlink anywhere under the root, including under
media/, still fails closed.manifest.jsondeclares the digest scope and the media-storage block;public_datarejects amanifest whose scope declaration is missing or different, with a test for the rejection.
manifest.jsondiff contains only the derived digest/scope/media-storage fields — norecord, count, artifact digest, source pin, or projection rule changes.
provenance.checksum; a test with a deliberatelycorrupted object proves the response is
502, not corrupted bytes.EXPECTED_COUNTS, the builder count constants, the parity counts, and the inventory validatorcounts are unchanged (D7).
Storage backends
local,memory, ands3backends each have focused tests, including thes3backendagainst a stubbed boto3 client (no new test dependency, following the existing
courses/tests/test_development_content_import.pypattern). No test performs a real networkcall.
make test,make test-content, and the applicable Playwright tier pass with no AWScredentials configured and with
AWS_*environment variables absent.cannot influence the key.
502contract withone redacted structured log event, and no bucket name, key, credential, or stack trace appears
in any response body.
manage.py check --deployunder production settings fails when the backend is nots3or thebucket is unset; it passes with a configured bucket.
Tooling
hydratematerialises all 1,253 objects into the local root from the pinned upstreamprovenance without AWS credentials, verifies every checksum, is idempotent, and refuses to
write a mismatching object. Its behaviour is covered by tests using local fixtures rather than
live network.
publishuploads only recorded objects with the recorded content type and a SHA-256 checksum,skips already-matching objects, refuses orphans, and reports counts. Covered against a stubbed
client.
verifyreports missing/extra/mismatched objects and exits non-zero on any of them. Coveredagainst a stubbed client and a local root.
Repository and operations
content/public_projection/media/is no longer git-tracked;.gitignoreignores it and.dockerignoreexcludes it so a local image build matches a CI build.git ls-files content/public_projection | wc -ldrops by 1,254 and a freshgit worktree addno longer materialises ≈150 MB of images. The engineer records the measured before/after
worktree size.
make lint,make typecheck,make terminology-check,make database-portability-check, andthe migration-drift check stay green.
scripts/validate_github_editorial_source_projection_inventory.pyfails with the same messageas on current main — no better, no worse — and the report says so explicitly.
backends are, how to publish and verify, and what a
502on an image means.[HUMAN]deploy gate — closed by the owner after the infrastructure lands[HUMAN]Themain/dtc-websiteTerraform root inDataTalksClub/aws-infrais applied and thebucket exists with public access blocked, encryption, and versioning.
[HUMAN]publishhas uploaded all 1,253 objects andverifyreports 1,253 matching, 0missing, 0 mismatched, 0 extra.
[HUMAN]The deployed environment is configured withPUBLIC_MEDIA_STORE_BACKEND=s3and thebucket name, and a deployed spot check of at least one article cover, one portrait, one podcast
artwork, and one 404 path returns the expected statuses.
Django/integration scenarios
200plus exactContent-Typeagainst the offline store.200, correct bytes, correctcontent type.
/images/posts/2025-09-23-ai-dev-tools-zoomcamp…/course-cover.png,3,022,797 bytes):
200, correctContent-Length.404with the ordinary 404 page.502,Cache-Control: no-store, redacted log, no leak.502with the same contract.PUBLIC_MEDIA_MAX_OBJECT_BYTES:502, and memory is never unbounded.localbackend with an empty root:502for a known record, andmanage.py checkwarns with thehydrate command name.
ImproperlyConfiguredat projection load.
media/changes;symlink still fails closed.
s3backend key derivation, content type, checksum verification path, and single bounded retryagainst a stubbed client.
s3backend failscheck --deploy.Browser scenarios (desktop 1280×720 and mobile 390×844)
Run these with the
localbackend and the real hydrated objects so the screenshots show realartwork; state the backend used in the report.
/blog/ai-dev-tools-zoomcamp.html— the 3 MB cover renders, is bounded by its column, and thereis no horizontal overflow or broken-image icon.
/people/aashishnair.html— the portrait whose filename begins with a space renders inside theprofile card, and the page shows no broken image.
/podcast/production-ml-search-vector-search-embeddings-hybrid-search.html— the artwork whosefilename contains a space renders.
/blogand/podcasthubs — every card image loads; no placeholder, no layout shift, noconsole/network 4xx or 5xx for an
/images/request./images/does-not-exist.jpg— the ordinary site 404 page, correct copy, working navigation, nostack trace and no store detail.
navigation; the failing image degrades to its alt text; nothing on screen or in the response
exposes a bucket name, key, or credential.
Evidence and reporting
_docs/ci/change-selective-ci.mdwithexact base/head, graph and plan digests, and every
rerun/reused/skipped/not_applicabledisposition, and leaves the worktree uncommitted and frozen.contentDjango tests plusmake test-playwright-core(template/render impact is limited, but every image-bearing public pageis affected), and stores desktop and mobile screenshots under
.tmp/screenshots/, reading each oneand reporting whether it shows the expected page with real artwork.
and the exact statuses observed for the six browser scenarios.
registration or production data appears in any log, screenshot, comment, or report.
Risks
[HUMAN]merge/deploygate above; do not merge around it.
tree_digest_scopedeclaration, thefail-closed rejection of an undeclared scope, and the test that recomputes the digest.
502for that onepath rather than serving wrong bytes; publish before deploying a new
media.json.ceiling, and superseded by Phase 2. Any in-process caching added here must be bounded, keyed by
record checksum, off by default in tests, and covered by tests.