diff --git a/.env.example b/.env.example index 1ecf6dd..c5b9679 100644 --- a/.env.example +++ b/.env.example @@ -71,12 +71,16 @@ URL_IMPORT_MAX_HEIGHT=1080 # Most videos one playlist or channel link may queue. URL_IMPORT_MAX_PLAYLIST_ITEMS=200 -# Optional. A Netscape cookies.txt *inside the container*, for YouTube's "confirm you're -# not a bot" wall (common on VPS/datacenter IPs) and age-restricted videos. Mount it -# read-only, e.g. in docker-compose.override.yml: -# services: { backend: { volumes: ["./secrets/cookies.txt:/app/secrets/cookies.txt:ro"] } } -# and set: URL_IMPORT_COOKIES_FILE=/app/secrets/cookies.txt -# Treat that file like a password — it is a signed-in session for the account it came from. +# A signed-in YouTube session (a Netscape cookies.txt), for YouTube's "confirm you're not +# a bot" wall — common on VPS/datacenter IPs — and for age-restricted videos. +# +# Under Docker, leave this empty. Put the file at secrets/youtube-cookies.txt and add +# docker-compose.cookies.yml to COMPOSE_FILE (see Reverse proxy, below): that overlay +# mounts secrets/ read-only and sets this for you. Exporting the file from Firefox, and +# why to use a spare account: docs/url-import.md#youtube-cookies +# +# Set it here only to name a different file, or for a bare `uvicorn` dev run, where it +# is a path on your own machine. The file is a login: treat it like a password. URL_IMPORT_COOKIES_FILE= # ─── Development only ──────────────────────────────────────────────────────── @@ -111,4 +115,9 @@ TZ=UTC # Set this so `docker compose up -d` always includes the prod overlay and cannot # forget to join the proxy's network. +# +# Once COMPOSE_FILE is set, Compose no longer reads docker-compose.override.yml — every +# overlay you want must be listed here. To give the URL importer YouTube cookies (see +# URL_IMPORT_COOKIES_FILE above), append the cookies overlay: +# COMPOSE_FILE=docker-compose.yml:docker-compose.prod.yml:docker-compose.cookies.yml # COMPOSE_FILE=docker-compose.yml:docker-compose.prod.yml diff --git a/.gitignore b/.gitignore index 5edc6d4..5915a73 100644 --- a/.gitignore +++ b/.gitignore @@ -247,3 +247,7 @@ data/media/ # Backup service secrets (SSH private key, known_hosts) — never commit these. ops/backup/secrets/ + +# Signed-in session cookies for the URL importer (docker-compose.cookies.yml mounts this +# directory). A cookies file is a login: committing one publishes the account. +/secrets/ diff --git a/docker-compose.cookies.yml b/docker-compose.cookies.yml new file mode 100644 index 0000000..6459c10 --- /dev/null +++ b/docker-compose.cookies.yml @@ -0,0 +1,27 @@ +# Overlay giving the URL importer a signed-in YouTube session (a cookies file). +# +# Only needed when YouTube answers imports with "Sign in to confirm you're not a bot" +# (common on VPS/datacenter IPs), or for age-restricted videos. How to export the cookies +# from Firefox, and what they are, is in docs/url-import.md#youtube-cookies. +# +# Activate by adding it to COMPOSE_FILE in .env, after whatever is already there: +# COMPOSE_FILE=docker-compose.yml:docker-compose.prod.yml:docker-compose.cookies.yml +# +# Not docker-compose.override.yml, which is the usual home for a local addition like +# this: Compose reads that file only when COMPOSE_FILE is unset, and every deployment of +# this app sets COMPOSE_FILE for the reverse-proxy overlay. An override file would be +# ignored without a word. + +services: + backend: + volumes: + # The directory, not the file. A single-file bind mount pins the file's inode at + # container start, so a refreshed cookies file written as a new file (rsync, most + # editors) would stay invisible until the container was recreated; and if the file + # were missing, Docker would create an empty *directory* in its place. With the + # directory mounted, dropping in a new file takes effect on the next import. + - ./secrets:/app/secrets:ro + environment: + # A default rather than a fixed value, so an explicit URL_IMPORT_COOKIES_FILE in + # .env still wins. + - URL_IMPORT_COOKIES_FILE=${URL_IMPORT_COOKIES_FILE:-/app/secrets/youtube-cookies.txt} diff --git a/docs/deployment.md b/docs/deployment.md index b94744f..f37b4f4 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -135,6 +135,25 @@ Changing the model later does not invalidate what is stored — vectors record t that produced them and only matching ones are searched — so a switch quietly shrinks the searchable set until the affected assets are embedded again. +## Importing from YouTube + +Nothing to configure unless YouTube refuses the server. Many VPS and datacenter IP ranges +get *"Sign in to confirm you're not a bot"*, and age-restricted videos always need a +signed-in session. For either, export a cookies file from a Firefox profile made for the +purpose, put it at `secrets/youtube-cookies.txt`, and add `docker-compose.cookies.yml` to +`COMPOSE_FILE`: + +```env +COMPOSE_FILE=docker-compose.yml:docker-compose.prod.yml:docker-compose.cookies.yml +``` + +The full steps — the Firefox export, keeping only YouTube's cookies, and why a spare +account — are in [`url-import.md` → YouTube cookies](url-import.md#youtube-cookies). +Not `docker-compose.override.yml`: with `COMPOSE_FILE` set, Compose never reads it. + +When imports start failing across the board, YouTube has changed under yt-dlp: bump its +pin in `backend/requirements.txt` and rebuild. + ## Updating ```bash diff --git a/docs/url-import.md b/docs/url-import.md index c0d81cd..f87752a 100644 --- a/docs/url-import.md +++ b/docs/url-import.md @@ -61,6 +61,7 @@ from the source that no person here has checked — the same standing as an EXIF | `backend/app/enrichment/transcribe.py::store_transcript` | The transcript writer, split out of the Deepgram job so captions and Deepgram store identically. | | `frontend/src/components/UrlImport.tsx` | The form under the drop zone. | | `frontend/src/views/LibraryView.tsx` | Adds a finished import (and its chapter clips) to the grid, on seeing its job go from running to done. | +| `docker-compose.cookies.yml` | Opt-in overlay that mounts `secrets/` read-only and points the importer at the cookies file. See [YouTube cookies](#youtube-cookies). | `KIND_IMPORT_URL` is a library kind: the asset does not exist when the job is queued. The request rides in `EnrichmentJob.payload`, and the finished job adds `created_asset_id` @@ -94,15 +95,138 @@ installs Deno from PyPI (manylinux wheels, x86_64 and aarch64, about 40 MB) and `default` extra brings the `yt-dlp-ejs` solver scripts. The backend logs a warning at startup if `deno` is not on `PATH`. -### "Sign in to confirm you're not a bot" +### YouTube cookies -YouTube shows this to many datacenter and VPS IP ranges; home connections usually never -see it. Age-restricted videos need a signed-in session too. Export your browser's -`youtube.com` cookies in Netscape format (the -[yt-dlp FAQ](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#how-do-i-pass-cookies-to-yt-dlp) -covers how, and why a private window is best), mount the file read-only into the backend, -and point `URL_IMPORT_COOKIES_FILE` at it. yt-dlp is given a *copy*, because it writes the -jar back out when it finishes. Treat the file like a password: it is a signed-in session. +An import that fails with *"The site wants this server to sign in first"* has hit +YouTube's "Sign in to confirm you're not a bot" wall, which it shows to many VPS and +datacenter IP ranges. Age-restricted and members-only videos need a signed-in session +too. The fix is to give the importer one: a cookies file exported from a browser. + +Before starting: + +- **The file is a login.** Anyone holding it is signed in as that account. Keep it out of + chats and out of git — `/secrets/`, where it goes, is gitignored for this reason. +- **Use a spare Google account.** An account used for automated downloading can get + flagged by YouTube, and a spare one keeps your main account out of that. + +The steps use Firefox, because its cookie store is not encrypted and yt-dlp reads it +directly. Everything here assumes Linux. + +#### On your own computer + +1. **Install yt-dlp.** It is only used here for the export, so any recent version works: + + ```bash + pipx install yt-dlp # or your distro's yt-dlp package + ``` + +2. **Make a Firefox profile just for this.** Open `about:profiles`, choose *Create a New + Profile* (call it `gam-youtube`), then *Launch profile in new browser*. In that window, + sign in to YouTube with the spare account and play any video. Then close the YouTube + tab and **quit Firefox completely**. + + A separate profile, because YouTube rotates the cookies of a session left open in a + tab, which invalidates an exported copy within hours. A profile you never open YouTube + in again keeps them valid, and its export holds nothing else of yours. (yt-dlp's own + advice is a private window, but a private window's cookies are never written to disk, + so `--cookies-from-browser` cannot see them.) + +3. **Find the profile's folder.** In `about:profiles`, copy the new profile's **Root + Directory**. Its folder name is a random prefix plus the profile name, such as + `ab12cd34.gam-youtube`, under one of these depending on how Firefox is installed: + + | Firefox install | Profiles live under | + |---|---| + | Snap (Ubuntu's default) | `~/snap/firefox/common/.mozilla/firefox/` | + | Flatpak | `~/.var/app/org.mozilla.firefox/.mozilla/firefox/` or `~/.var/app/org.mozilla.firefox/config/mozilla/firefox/` | + | Profiles created before Firefox 147 | `~/.mozilla/firefox/` | + | New installs of Firefox 147 or later | `~/.config/mozilla/firefox/` | + +4. **Export**, naming that folder explicitly (this example is the Snap location): + + ```bash + yt-dlp --cookies-from-browser "firefox:$HOME/snap/firefox/common/.mozilla/firefox/ab12cd34.gam-youtube" \ + --cookies cookies.txt + ``` + + It ends with `error: You must provide at least one URL.` That is expected and + harmless — the file was written before it. **Always give the path.** A bare + `--cookies-from-browser firefox` reads whichever profile was used most recently, which + is usually your everyday one: every site you are signed in to, and the wrong YouTube + session. + +5. **Keep only the YouTube lines**, then delete the full export: + + ```bash + grep -E $'^(#|\\.?youtube\\.com\t)' cookies.txt > youtube-cookies.txt + rm cookies.txt + ``` + + The export holds every cookie in the profile, Google's own sign-in cookies included. + The importer needs only `youtube.com`'s. + +#### On the server + +1. **Put the file in `secrets/`**, beside `docker-compose.yml`: + + ```bash + # on the server, in the gam checkout + mkdir -p secrets && chmod 700 secrets + + # from your computer + scp youtube-cookies.txt you@your-server:/path/to/gam/secrets/ + + # on the server again + chmod 600 secrets/youtube-cookies.txt + ``` + +2. **Add the cookies overlay to `COMPOSE_FILE`** in `.env`: + + ```env + COMPOSE_FILE=docker-compose.yml:docker-compose.prod.yml:docker-compose.cookies.yml + ``` + + (Without the reverse-proxy overlay, `docker-compose.yml:docker-compose.cookies.yml`.) + `docker-compose.cookies.yml` mounts `secrets/` read-only at `/app/secrets` and points + `URL_IMPORT_COOKIES_FILE` at `/app/secrets/youtube-cookies.txt`. Set that variable in + `.env` only to use a different file name. + +3. **Apply it**, and check the container can see the file: + + ```bash + docker compose up -d + docker compose exec backend ls -l /app/secrets/ + ``` + + No rebuild is needed. Then retry the import that failed. + +**Not `docker-compose.override.yml`.** An earlier version of these docs said to put the +mount there. Compose reads that file only when `COMPOSE_FILE` is unset, and this +deployment sets `COMPOSE_FILE` for the reverse proxy — so the mount would be silently +ignored. The overlay is named in `COMPOSE_FILE` explicitly for that reason. + +Each import hands yt-dlp a *copy* of the file, because yt-dlp writes the cookie jar back +out when it finishes. The read-only mount is therefore fine, and your file is never +rewritten. + +#### When it stops working + +The sign-in message coming back means YouTube has expired or revoked that session. +Launch the `gam-youtube` profile, sign in again if asked, play a video, close the tab and +quit Firefox. Then repeat steps 4 and 5 on your computer and copy the new file over the +old one on the server. No restart: the directory is mounted, not the file, so the next +import reads the new one. + +If every import instead fails with *"URL_IMPORT_COOKIES_FILE is set to …, but there is no +file there"*, the overlay is on but `secrets/youtube-cookies.txt` is missing or named +differently. + +#### Other browsers + +`--cookies-from-browser` also accepts `chrome`, `chromium`, `brave` and `edge`. On Linux +those encrypt their cookie store with the desktop keyring, and yt-dlp may need the keyring +named — `chrome+gnomekeyring`, for instance; `yt-dlp --help` lists the choices. Firefox +needs none of that. ### Disk @@ -116,7 +240,7 @@ size free while that happens. |---|---|---| | `URL_IMPORT_MAX_HEIGHT` | `1080` | Tallest video fetched. | | `URL_IMPORT_MAX_PLAYLIST_ITEMS` | `200` | Most videos one playlist or channel link may queue. | -| `URL_IMPORT_COOKIES_FILE` | empty | See above. | +| `URL_IMPORT_COOKIES_FILE` | empty | Set for you by `docker-compose.cookies.yml` — see [YouTube cookies](#youtube-cookies). | ---