From 7d1333310fe23d3003fa7110e153644658b49519 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 28 Sep 2026 09:52:38 +0000 Subject: [PATCH] docs(import): fix the cookie mount advice; document the Firefox export The URL-import docs said to mount a YouTube cookies file through docker-compose.override.yml. Compose reads that file only when COMPOSE_FILE is unset, and every deployment of this app sets COMPOSE_FILE for the reverse-proxy overlay, so the mount was silently ignored (reproduced with `docker compose config`). - docker-compose.cookies.yml: an opt-in overlay, named in COMPOSE_FILE like the prod one. Mounts the ./secrets directory read-only (not the file, so a refreshed file needs no restart and a missing one cannot become a directory) and defaults URL_IMPORT_COOKIES_FILE to it. - /secrets/ gitignored: a cookies file is a login. - docs/url-import.md#youtube-cookies: the full Linux procedure: a dedicated Firefox profile, finding its folder (Snap, Flatpak, XDG and legacy locations), exporting by explicit path, keeping only youtube.com lines, installing it on the server, refreshing it. Commands were run verbatim against a fake profile. A bare `--cookies-from-browser firefox` exported the most recently used profile instead, so the docs require the path. - .env.example and deployment.md point at it, and say that setting COMPOSE_FILE turns off docker-compose.override.yml. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_017KYsSofkyeywy1NNPV8NtM --- .env.example | 21 ++++-- .gitignore | 4 ++ docker-compose.cookies.yml | 27 +++++++ docs/deployment.md | 19 +++++ docs/url-import.md | 142 ++++++++++++++++++++++++++++++++++--- 5 files changed, 198 insertions(+), 15 deletions(-) create mode 100644 docker-compose.cookies.yml diff --git a/.env.example b/.env.example index 1ecf6dd..c5b9679 100644 --- a/.env.example +++ b/.env.example @@ -71,12 +71,16 @@ URL_IMPORT_MAX_HEIGHT=1080 # Most videos one playlist or channel link may queue. URL_IMPORT_MAX_PLAYLIST_ITEMS=200 -# Optional. A Netscape cookies.txt *inside the container*, for YouTube's "confirm you're -# not a bot" wall (common on VPS/datacenter IPs) and age-restricted videos. Mount it -# read-only, e.g. in docker-compose.override.yml: -# services: { backend: { volumes: ["./secrets/cookies.txt:/app/secrets/cookies.txt:ro"] } } -# and set: URL_IMPORT_COOKIES_FILE=/app/secrets/cookies.txt -# Treat that file like a password — it is a signed-in session for the account it came from. +# A signed-in YouTube session (a Netscape cookies.txt), for YouTube's "confirm you're not +# a bot" wall — common on VPS/datacenter IPs — and for age-restricted videos. +# +# Under Docker, leave this empty. Put the file at secrets/youtube-cookies.txt and add +# docker-compose.cookies.yml to COMPOSE_FILE (see Reverse proxy, below): that overlay +# mounts secrets/ read-only and sets this for you. Exporting the file from Firefox, and +# why to use a spare account: docs/url-import.md#youtube-cookies +# +# Set it here only to name a different file, or for a bare `uvicorn` dev run, where it +# is a path on your own machine. The file is a login: treat it like a password. URL_IMPORT_COOKIES_FILE= # ─── Development only ──────────────────────────────────────────────────────── @@ -111,4 +115,9 @@ TZ=UTC # Set this so `docker compose up -d` always includes the prod overlay and cannot # forget to join the proxy's network. +# +# Once COMPOSE_FILE is set, Compose no longer reads docker-compose.override.yml — every +# overlay you want must be listed here. To give the URL importer YouTube cookies (see +# URL_IMPORT_COOKIES_FILE above), append the cookies overlay: +# COMPOSE_FILE=docker-compose.yml:docker-compose.prod.yml:docker-compose.cookies.yml # COMPOSE_FILE=docker-compose.yml:docker-compose.prod.yml diff --git a/.gitignore b/.gitignore index 5edc6d4..5915a73 100644 --- a/.gitignore +++ b/.gitignore @@ -247,3 +247,7 @@ data/media/ # Backup service secrets (SSH private key, known_hosts) — never commit these. ops/backup/secrets/ + +# Signed-in session cookies for the URL importer (docker-compose.cookies.yml mounts this +# directory). A cookies file is a login: committing one publishes the account. +/secrets/ diff --git a/docker-compose.cookies.yml b/docker-compose.cookies.yml new file mode 100644 index 0000000..6459c10 --- /dev/null +++ b/docker-compose.cookies.yml @@ -0,0 +1,27 @@ +# Overlay giving the URL importer a signed-in YouTube session (a cookies file). +# +# Only needed when YouTube answers imports with "Sign in to confirm you're not a bot" +# (common on VPS/datacenter IPs), or for age-restricted videos. How to export the cookies +# from Firefox, and what they are, is in docs/url-import.md#youtube-cookies. +# +# Activate by adding it to COMPOSE_FILE in .env, after whatever is already there: +# COMPOSE_FILE=docker-compose.yml:docker-compose.prod.yml:docker-compose.cookies.yml +# +# Not docker-compose.override.yml, which is the usual home for a local addition like +# this: Compose reads that file only when COMPOSE_FILE is unset, and every deployment of +# this app sets COMPOSE_FILE for the reverse-proxy overlay. An override file would be +# ignored without a word. + +services: + backend: + volumes: + # The directory, not the file. A single-file bind mount pins the file's inode at + # container start, so a refreshed cookies file written as a new file (rsync, most + # editors) would stay invisible until the container was recreated; and if the file + # were missing, Docker would create an empty *directory* in its place. With the + # directory mounted, dropping in a new file takes effect on the next import. + - ./secrets:/app/secrets:ro + environment: + # A default rather than a fixed value, so an explicit URL_IMPORT_COOKIES_FILE in + # .env still wins. + - URL_IMPORT_COOKIES_FILE=${URL_IMPORT_COOKIES_FILE:-/app/secrets/youtube-cookies.txt} diff --git a/docs/deployment.md b/docs/deployment.md index b94744f..f37b4f4 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -135,6 +135,25 @@ Changing the model later does not invalidate what is stored — vectors record t that produced them and only matching ones are searched — so a switch quietly shrinks the searchable set until the affected assets are embedded again. +## Importing from YouTube + +Nothing to configure unless YouTube refuses the server. Many VPS and datacenter IP ranges +get *"Sign in to confirm you're not a bot"*, and age-restricted videos always need a +signed-in session. For either, export a cookies file from a Firefox profile made for the +purpose, put it at `secrets/youtube-cookies.txt`, and add `docker-compose.cookies.yml` to +`COMPOSE_FILE`: + +```env +COMPOSE_FILE=docker-compose.yml:docker-compose.prod.yml:docker-compose.cookies.yml +``` + +The full steps — the Firefox export, keeping only YouTube's cookies, and why a spare +account — are in [`url-import.md` → YouTube cookies](url-import.md#youtube-cookies). +Not `docker-compose.override.yml`: with `COMPOSE_FILE` set, Compose never reads it. + +When imports start failing across the board, YouTube has changed under yt-dlp: bump its +pin in `backend/requirements.txt` and rebuild. + ## Updating ```bash diff --git a/docs/url-import.md b/docs/url-import.md index c0d81cd..f87752a 100644 --- a/docs/url-import.md +++ b/docs/url-import.md @@ -61,6 +61,7 @@ from the source that no person here has checked — the same standing as an EXIF | `backend/app/enrichment/transcribe.py::store_transcript` | The transcript writer, split out of the Deepgram job so captions and Deepgram store identically. | | `frontend/src/components/UrlImport.tsx` | The form under the drop zone. | | `frontend/src/views/LibraryView.tsx` | Adds a finished import (and its chapter clips) to the grid, on seeing its job go from running to done. | +| `docker-compose.cookies.yml` | Opt-in overlay that mounts `secrets/` read-only and points the importer at the cookies file. See [YouTube cookies](#youtube-cookies). | `KIND_IMPORT_URL` is a library kind: the asset does not exist when the job is queued. The request rides in `EnrichmentJob.payload`, and the finished job adds `created_asset_id` @@ -94,15 +95,138 @@ installs Deno from PyPI (manylinux wheels, x86_64 and aarch64, about 40 MB) and `default` extra brings the `yt-dlp-ejs` solver scripts. The backend logs a warning at startup if `deno` is not on `PATH`. -### "Sign in to confirm you're not a bot" +### YouTube cookies -YouTube shows this to many datacenter and VPS IP ranges; home connections usually never -see it. Age-restricted videos need a signed-in session too. Export your browser's -`youtube.com` cookies in Netscape format (the -[yt-dlp FAQ](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#how-do-i-pass-cookies-to-yt-dlp) -covers how, and why a private window is best), mount the file read-only into the backend, -and point `URL_IMPORT_COOKIES_FILE` at it. yt-dlp is given a *copy*, because it writes the -jar back out when it finishes. Treat the file like a password: it is a signed-in session. +An import that fails with *"The site wants this server to sign in first"* has hit +YouTube's "Sign in to confirm you're not a bot" wall, which it shows to many VPS and +datacenter IP ranges. Age-restricted and members-only videos need a signed-in session +too. The fix is to give the importer one: a cookies file exported from a browser. + +Before starting: + +- **The file is a login.** Anyone holding it is signed in as that account. Keep it out of + chats and out of git — `/secrets/`, where it goes, is gitignored for this reason. +- **Use a spare Google account.** An account used for automated downloading can get + flagged by YouTube, and a spare one keeps your main account out of that. + +The steps use Firefox, because its cookie store is not encrypted and yt-dlp reads it +directly. Everything here assumes Linux. + +#### On your own computer + +1. **Install yt-dlp.** It is only used here for the export, so any recent version works: + + ```bash + pipx install yt-dlp # or your distro's yt-dlp package + ``` + +2. **Make a Firefox profile just for this.** Open `about:profiles`, choose *Create a New + Profile* (call it `gam-youtube`), then *Launch profile in new browser*. In that window, + sign in to YouTube with the spare account and play any video. Then close the YouTube + tab and **quit Firefox completely**. + + A separate profile, because YouTube rotates the cookies of a session left open in a + tab, which invalidates an exported copy within hours. A profile you never open YouTube + in again keeps them valid, and its export holds nothing else of yours. (yt-dlp's own + advice is a private window, but a private window's cookies are never written to disk, + so `--cookies-from-browser` cannot see them.) + +3. **Find the profile's folder.** In `about:profiles`, copy the new profile's **Root + Directory**. Its folder name is a random prefix plus the profile name, such as + `ab12cd34.gam-youtube`, under one of these depending on how Firefox is installed: + + | Firefox install | Profiles live under | + |---|---| + | Snap (Ubuntu's default) | `~/snap/firefox/common/.mozilla/firefox/` | + | Flatpak | `~/.var/app/org.mozilla.firefox/.mozilla/firefox/` or `~/.var/app/org.mozilla.firefox/config/mozilla/firefox/` | + | Profiles created before Firefox 147 | `~/.mozilla/firefox/` | + | New installs of Firefox 147 or later | `~/.config/mozilla/firefox/` | + +4. **Export**, naming that folder explicitly (this example is the Snap location): + + ```bash + yt-dlp --cookies-from-browser "firefox:$HOME/snap/firefox/common/.mozilla/firefox/ab12cd34.gam-youtube" \ + --cookies cookies.txt + ``` + + It ends with `error: You must provide at least one URL.` That is expected and + harmless — the file was written before it. **Always give the path.** A bare + `--cookies-from-browser firefox` reads whichever profile was used most recently, which + is usually your everyday one: every site you are signed in to, and the wrong YouTube + session. + +5. **Keep only the YouTube lines**, then delete the full export: + + ```bash + grep -E $'^(#|\\.?youtube\\.com\t)' cookies.txt > youtube-cookies.txt + rm cookies.txt + ``` + + The export holds every cookie in the profile, Google's own sign-in cookies included. + The importer needs only `youtube.com`'s. + +#### On the server + +1. **Put the file in `secrets/`**, beside `docker-compose.yml`: + + ```bash + # on the server, in the gam checkout + mkdir -p secrets && chmod 700 secrets + + # from your computer + scp youtube-cookies.txt you@your-server:/path/to/gam/secrets/ + + # on the server again + chmod 600 secrets/youtube-cookies.txt + ``` + +2. **Add the cookies overlay to `COMPOSE_FILE`** in `.env`: + + ```env + COMPOSE_FILE=docker-compose.yml:docker-compose.prod.yml:docker-compose.cookies.yml + ``` + + (Without the reverse-proxy overlay, `docker-compose.yml:docker-compose.cookies.yml`.) + `docker-compose.cookies.yml` mounts `secrets/` read-only at `/app/secrets` and points + `URL_IMPORT_COOKIES_FILE` at `/app/secrets/youtube-cookies.txt`. Set that variable in + `.env` only to use a different file name. + +3. **Apply it**, and check the container can see the file: + + ```bash + docker compose up -d + docker compose exec backend ls -l /app/secrets/ + ``` + + No rebuild is needed. Then retry the import that failed. + +**Not `docker-compose.override.yml`.** An earlier version of these docs said to put the +mount there. Compose reads that file only when `COMPOSE_FILE` is unset, and this +deployment sets `COMPOSE_FILE` for the reverse proxy — so the mount would be silently +ignored. The overlay is named in `COMPOSE_FILE` explicitly for that reason. + +Each import hands yt-dlp a *copy* of the file, because yt-dlp writes the cookie jar back +out when it finishes. The read-only mount is therefore fine, and your file is never +rewritten. + +#### When it stops working + +The sign-in message coming back means YouTube has expired or revoked that session. +Launch the `gam-youtube` profile, sign in again if asked, play a video, close the tab and +quit Firefox. Then repeat steps 4 and 5 on your computer and copy the new file over the +old one on the server. No restart: the directory is mounted, not the file, so the next +import reads the new one. + +If every import instead fails with *"URL_IMPORT_COOKIES_FILE is set to …, but there is no +file there"*, the overlay is on but `secrets/youtube-cookies.txt` is missing or named +differently. + +#### Other browsers + +`--cookies-from-browser` also accepts `chrome`, `chromium`, `brave` and `edge`. On Linux +those encrypt their cookie store with the desktop keyring, and yt-dlp may need the keyring +named — `chrome+gnomekeyring`, for instance; `yt-dlp --help` lists the choices. Firefox +needs none of that. ### Disk @@ -116,7 +240,7 @@ size free while that happens. |---|---|---| | `URL_IMPORT_MAX_HEIGHT` | `1080` | Tallest video fetched. | | `URL_IMPORT_MAX_PLAYLIST_ITEMS` | `200` | Most videos one playlist or channel link may queue. | -| `URL_IMPORT_COOKIES_FILE` | empty | See above. | +| `URL_IMPORT_COOKIES_FILE` | empty | Set for you by `docker-compose.cookies.yml` — see [YouTube cookies](#youtube-cookies). | ---