diff --git a/ops/backup/README.md b/ops/backup/README.md index d3eaa9f..15511e8 100644 --- a/ops/backup/README.md +++ b/ops/backup/README.md @@ -1,140 +1,407 @@ # Backup setup: gecko-notes → Synology NAS -This sets up the `backup` service from `docker-compose.yml`: a small sidecar -that takes a whole-database snapshot and pushes it, together with the entire -`data/media/` tree, to a Synology NAS over SFTP on a schedule. See -`docs/backup-strategy.md` for the design rationale — this file is just the -step-by-step setup. - -Nothing restic-specific needs to run on the NAS. It only needs its stock -SSH/SFTP service turned on; all the backup logic runs in the `backup` -container on the same host as the app. - -This iteration wires up exactly one destination (the NAS), reached over a -direct router port-forward since gecko-notes runs on a separate remote host. -That means SSH is genuinely exposed to the internet, so the steps below lean -on key-only auth, a non-default port, a locked-down account, and DSM's -brute-force protection rather than skipping them. - -## 1. On the Synology (DSM 7), as an admin - -- **Control Panel → Terminal & SNMP** → enable **SSH service**. -- **Control Panel → File Services → FTP tab** → enable **SFTP Service**. -- **Control Panel → Shared Folder** → create a dedicated shared folder, e.g. - `gecko-notes-backup`. Don't reuse an existing folder that holds anything else. -- **Control Panel → User & Group** → create a dedicated, non-admin local user, - e.g. `gecko-backup`. Under its **Advanced** tab, enable **User Home - service** — without it, `~/.ssh/authorized_keys` has nowhere to live. -- **Shared Folder permissions** → grant `gecko-backup` **Read/Write** on - `gecko-notes-backup` only, and **No access** on every other shared folder. - -## 2. Generate the backup keypair - -On any trusted machine (not the NAS itself): - -```sh -ssh-keygen -t ed25519 -N "" -f id_backup -``` - -Move both files into `ops/backup/secrets/` in this repo (`id_backup` and -`id_backup.pub`). That directory is gitignored — the private key must never -be committed. - -## 3. Install the public key on the NAS - -1. Log in once as `gecko-backup` via SSH using its password, just to make DSM - create the home directory. -2. Via File Station (or an admin SSH session), create - `homes/gecko-backup/.ssh/authorized_keys` containing the contents of - `id_backup.pub`. -3. Permissions matter here and DSM is stricter than you'd expect: - - the home directory itself: `755` (not `700` — DSM's sshd silently - rejects key auth if the home directory is group/world-inaccessible) - - `.ssh`: `700` - - `authorized_keys`: `600` - -## 4. Harden sshd and enable brute-force protection - -- Confirm `PubkeyAuthentication yes` is set in `/etc/ssh/sshd_config` (check - via an admin SSH session). Restart the SSH service from the DSM GUI - (toggle it off/on in Terminal & SNMP) rather than a raw service-reload - command — the GUI restart survives DSM updates more reliably. -- **Control Panel → Security** → enable **Auto Block** (naming/location - varies a bit by DSM version) so repeated failed logins get the source IP - banned automatically. - -## 5. Router: port-forward - -Forward an external, **non-default** port (e.g. `22022`, not `22`) to the -NAS's internal SSH port on its LAN IP. If the machine running gecko-notes has -a stable/static public IP, restrict the forward — or add a matching DSM -firewall rule — to allow only that one source IP. This single step removes -most of the risk of exposing SSH directly, so it's worth doing if at all -possible. - -## 6. Pre-seed the host key - -Run from the gecko-notes host (or anywhere that can already reach the NAS's -forwarded port): - -```sh +Sets up the `backup` service from `docker-compose.yml`: a sidecar that takes a +whole-database snapshot and pushes it, with the entire `data/media/` tree, to a +Synology NAS over SFTP on a schedule. See `docs/backup-strategy.md` for design +rationale. This file is the setup procedure. + +All backup logic runs in the `backup` container on the gecko-notes host. Nothing +restic-specific runs on the NAS — it only needs its stock SFTP service. + +gecko-notes runs on a separate remote host, so the NAS is reached over a router +port-forward. SSH is therefore genuinely exposed to the internet, and the steps +below lean on key-only auth, a non-default port, a locked-down account, a +source-IP firewall rule, and DSM's brute-force protection. + +--- + +## Read this first: two undocumented DSM requirements + +Two things are required that no Synology documentation mentions, and both fail +with an unhelpful `Permission denied (publickey,password)` and nothing useful in +the logs. + +**1. `PubkeyAuthentication` is OFF in Synology's sshd build.** Stock OpenSSH +defaults it on. Synology's does not, and ships `sshd_config` with the line +commented out — which looks like "default applies" but isn't. Without an +explicit `PubkeyAuthentication yes`, sshd rejects every key at the offer stage +without ever reading `authorized_keys`. + +**2. The backup account needs access to the `homes` share.** sshd reads +`~/.ssh/authorized_keys` as the target user. DSM puts an ACL on `/volume1/homes`, +and if the account is set to No Access on that share, sshd cannot traverse into +the account's own home. Symptom in a debug log: + +``` +debug1: Could not open authorized keys '.../authorized_keys': Permission denied +``` + +Setting `homes` to No Access looks like correct hardening. It silently breaks key +auth. + +**Also worth knowing before you start:** DSM does not permit interactive SSH +shell logins for non-admin accounts. The backup account will never give you a +shell, by design. That is fine — restic uses the SFTP subsystem, not a shell. +Verify with `sftp`, never `ssh`. + +Because of that, the way to run root commands on the NAS without enabling the SSH +terminal service is **Control Panel → Task Scheduler → Create → Scheduled Task → +User-defined script**, with User set to `root`. Untick "Enabled" so it never runs +on a schedule, then select it and click Run. Several steps below use this. + +--- + +## 1. NAS: enable SFTP only + +**Control Panel → File Services → FTP tab** + +- Leave "Enable FTP service" and FTPS **unticked**. FTP and FTPS are different + protocols from SFTP and you don't want them. +- Tick **Enable SFTP service**. +- Set a non-default port (e.g. `22022`). This is separate from the SSH port. +- Apply. + +**Control Panel → Terminal & SNMP** — leave "Enable SSH service" **off**. SFTP +and SSH are independent services on DSM; SFTP works without the terminal service. + +## 2. NAS: enable User Home service + +**Control Panel → User & Group → Advanced tab → User Home** → tick **Enable user +home service** → Apply. + +This is a global setting on the User & Group page, not a per-user one in the +account editor. It creates `/volume1/homes/` for every account +automatically. No login required. + +## 3. NAS: create the shared folder and account + +**Control Panel → Shared Folder** → create a dedicated share, e.g. `GeckoNotes`. +Don't reuse a folder holding anything else. + +**Control Panel → User & Group → User** → create a non-admin local user, e.g. +`gecko-notes-backup`. + +**Permissions tab:** + +| Share | Access | +| ---------- | ----------------------------------------------- | +| `GeckoNotes` | Read/Write | +| `homes` | Read Only (required — see "Read this first") | +| everything else | No Access | + +**Applications tab:** Allow **SFTP** explicitly. Set **DSM** to Deny so the +account cannot reach the web UI. Deny everything else. + +## 4. Generate the keypair (on the gecko-notes host) + +Generate on the host that will run the backups, not on the NAS and not on your +laptop. Only the public half ever leaves this machine. + +```bash +cd /opt/gecko-notes/ops/backup +mkdir -p secrets && chmod 700 secrets +ssh-keygen -t ed25519 -N "" -f secrets/id_backup +chmod 600 secrets/id_backup +cat secrets/id_backup.pub +``` + +No passphrase, because nothing can type one at 3am. `secrets/` is gitignored — +the private key must never be committed. + +The comment at the end of the public key (`user@hostname`) is a free-text label. +sshd ignores it. It does not need to match the NAS account name. + +## 5. NAS: install the public key + +Task Scheduler script, run as root. Paste your public key into `PUBKEY` as a +**single unbroken line** — DSM's task editor is a plain textarea and a wrapped +key produces a file that looks fine and fails silently. + +```bash +#!/bin/sh +USER_NAME=gecko-notes-backup +HOME_DIR=$(grep "^$USER_NAME:" /etc/passwd | cut -d: -f6) +PUBKEY="ssh-ed25519 AAAA...paste-yours-here... comment" + +mkdir -p "$HOME_DIR/.ssh" +echo "$PUBKEY" > "$HOME_DIR/.ssh/authorized_keys" +chown -R "$USER_NAME:users" "$HOME_DIR" "$HOME_DIR/.ssh" +chmod 755 "$HOME_DIR" +chmod 700 "$HOME_DIR/.ssh" +chmod 600 "$HOME_DIR/.ssh/authorized_keys" + +# verify +{ + ls -ld "$HOME_DIR" "$HOME_DIR/.ssh" + ls -l "$HOME_DIR/.ssh/authorized_keys" + wc -l "$HOME_DIR/.ssh/authorized_keys" +} > /volume1/GeckoNotes/keycheck.txt 2>&1 +``` + +Read `keycheck.txt` via File Station. Home should be `755` owned by the user, +`.ssh` `700`, `authorized_keys` `600`, and the line count exactly **1**. A count +of 2+ means the key wrapped on paste. + +The script derives `HOME_DIR` from `/etc/passwd` rather than hardcoding it. If +User Home service hasn't taken effect, a hardcoded `mkdir -p` will happily create +a directory tree sshd never consults. + +Re-running this script overwrites `authorized_keys` rather than appending, so +it's safe to run again with a new key. + +## 6. NAS: enable public key authentication + +The step Synology doesn't document. Task Scheduler, root: + +```bash +#!/bin/sh +CONF=/etc/ssh/sshd_config +cp "$CONF" "$CONF.bak" + +sed -i '/^[[:space:]]*#*[[:space:]]*PubkeyAuthentication/d' "$CONF" +sed -i '/^[[:space:]]*#*[[:space:]]*AuthorizedKeysFile/d' "$CONF" +sed -i 's|^PasswordAuthentication yes|PasswordAuthentication yes\nPubkeyAuthentication yes\nAuthorizedKeysFile .ssh/authorized_keys|' "$CONF" + +grep -v '^#' "$CONF" | grep -v '^$' > /volume1/GeckoNotes/sshdconf.txt 2>&1 +``` + +Check `sshdconf.txt` shows both new lines **above** any `Match User` blocks. +Directives after a `Match` block apply only to those users. + +Restart SFTP: Control Panel → File Services → FTP tab, untick Enable SFTP +service, Apply, re-tick, Apply. Then re-check `sshdconf.txt` — the toggle +sometimes rewrites `sshd_config`. + +**This edit does not survive DSM updates.** See "Maintenance" below. + +## 7. NAS: lock it down + +- **Control Panel → Security → Protection** → enable **Auto Block** (e.g. 5 + attempts in 5 minutes, block permanently). +- **Control Panel → Security → Firewall** → allow the SFTP port only from the + gecko-notes host's public IP; deny everything else on that port. +- **Router** → forward an external non-default port to the NAS's SFTP port. + Restrict the forward to the source IP if your provider allows it. + +Get the gecko-notes host's real outbound address from the host itself, not by +resolving its domain — a proxy or CDN in front of the app will give you the wrong +answer: + +```bash +curl -4 ifconfig.me +``` + +If that host's IP is dynamic, a source-IP rule will silently break your backups +when it rotates. + +## 8. gecko-notes host: pin the NAS host key + +```bash +cd /opt/gecko-notes ssh-keyscan -p > ops/backup/secrets/known_hosts ssh-keygen -lf ops/backup/secrets/known_hosts ``` -`ssh-keyscan` itself doesn't verify anything — it just fetches whatever key -is offered. Compare the fingerprint it prints against the one DSM displays -under **Control Panel → Terminal & SNMP**. That comparison is what actually -establishes trust; skipping it defeats the point of `StrictHostKeyChecking`. +`ssh-keyscan` verifies nothing — it accepts whatever answers. Compare the +fingerprint against the NAS's real key. **DSM does not display a host key +fingerprint anywhere in the UI**, so get it with a Task Scheduler script: + +```bash +#!/bin/sh +for f in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf "$f"; done \ + > /volume1/GeckoNotes/hostkeys.txt +``` -## 7. Fill in `.env` +Match the ED25519 line. That comparison is what establishes trust; skipping it +defeats the point of `StrictHostKeyChecking`. Delete `hostkeys.txt` afterwards. -In `.env` (not `.env.example` — that file stays a template), set: +**`known_hosts` must exist as a real file before any `docker compose` command.** +`docker-compose.yml` bind-mounts it, and Docker creates a **directory** at any +bind-mount source that doesn't exist. Nothing errors; you just get an empty +directory where a file should be, and confusing failures later. Same applies to +`id_backup`. If you find directories there: +```bash +rmdir ops/backup/secrets/id_backup ops/backup/secrets/known_hosts ``` -RESTIC_PASSWORD= -BACKUP_SFTP_HOST= -BACKUP_SFTP_PORT= -BACKUP_SFTP_USER=gecko-backup -BACKUP_SFTP_REMOTE_PATH=/gecko-notes + +## 9. Test SFTP with the key + +The gate. Nothing downstream works until this does. + +```bash +sftp -i ops/backup/secrets/id_backup \ + -o IdentitiesOnly=yes \ + -o PreferredAuthentications=publickey \ + -o UserKnownHostsFile=ops/backup/secrets/known_hosts \ + -P @ +``` + +`PreferredAuthentications=publickey` matters. Without it a broken key silently +falls back to the password and you won't discover the problem until the container +fails unattended. + +Once connected: + +``` +sftp> pwd +sftp> ls +sftp> cd GeckoNotes +sftp> put /etc/hostname writetest.txt +sftp> rm writetest.txt +``` + +Synology chroots the user, so `pwd` at login returns `/` — that is the chroot +root, not the filesystem root. The share appears inside it as `/GeckoNotes`. Use +that, never `/volume1/GeckoNotes`. + +## 10. Fill `.env` + +```bash +openssl rand -base64 32 ``` -Before trusting `BACKUP_SFTP_REMOTE_PATH`, test manually: +Save that string in a password manager **before** using it. restic encrypts +client-side; losing it makes every existing snapshot permanently unrecoverable. -```sh -sftp -P gecko-backup@ +``` +RESTIC_PASSWORD= +BACKUP_SFTP_HOST= +BACKUP_SFTP_PORT= +BACKUP_SFTP_USER=gecko-notes-backup +BACKUP_SFTP_REMOTE_PATH=/GeckoNotes/restic ``` -then `pwd` / `ls` once connected. Synology's SFTP quirk: what that user sees -as `/` is actually their home/shared-folder, not the NAS's real filesystem -root — so the path restic should use is whatever you see from inside that -session, not something like `/volume1/gecko-notes-backup`. +The `/restic` subdirectory keeps the repo's `data/`, `snapshots/` and `keys/` +from scattering across the share root. `restic init` creates it. -Keep `RESTIC_PASSWORD` somewhere durable outside this repo too (a password -manager, etc.) — losing it makes every existing backup unrecoverable, since -restic encrypts client-side. +Edit `.env`, not `.env.example` — that stays a template. -## 8. One-time init and first run +## 11. Init and first run -```sh +```bash docker compose build backup docker compose run --rm backup restic init -docker compose run --rm backup /usr/local/bin/backup.sh # optional: don't wait for the first cron tick +docker compose run --rm backup /usr/local/bin/backup.sh docker compose up -d backup ``` -## 9. Verify +`restic init` is one-time and creates the repository on the NAS. The third line +forces an immediate backup instead of waiting for the first cron tick. + +`docker compose run --rm` creates a throwaway container each time, so it always +picks up current files. `docker compose restart` does **not** — bind mounts and +environment variables are resolved at container *creation*. After changing `.env` +or anything in `secrets/`: -```sh +```bash +docker compose up -d --force-recreate backup +``` + +## 12. Verify + +```bash docker compose run --rm backup restic snapshots -docker logs backup # shows cron's output on later scheduled runs +docker logs backup +docker inspect --format '{{.HostConfig.RestartPolicy.Name}}' +docker compose exec backup crontab -l ``` -## Later: adding a second destination +One snapshot listed means the chain works. Confirm the restart policy is +`unless-stopped` or `always`, or the container won't return after a host reboot. +Check `docker logs backup` again after the next scheduled tick — an unattended +run is the only real proof. + +--- + +## Maintenance + +**The `sshd_config` edit from step 6 does not survive DSM updates**, and may not +survive toggling the SFTP service. When it's reverted, backups fail silently. + +Add a **Triggered Task → Boot-up**, User `root`, with the step 6 script body. +Re-run it manually after any DSM update, and re-check `sshdconf.txt`. + +Consider putting Tailscale on both machines and dropping the port-forward +entirely. Synology has a package; the host takes one command. The backup then +runs over a private tailnet address, nothing is exposed to the internet, and the +firewall rule, non-default port and Auto Block stop being load-bearing. + +--- + +## Troubleshooting + +The client-side error is always the same unhelpful `Permission denied +(publickey,password)`. Get the server's actual reason instead. + +### Read sshd's reasoning directly + +Task Scheduler, root. Runs a debug sshd on a spare port that logs why it refused: + +```bash +#!/bin/sh +timeout 180 /usr/bin/sshd -ddd -p 2299 -f /etc/ssh/sshd_config \ + > /volume1/GeckoNotes/sshd_debug.txt 2>&1 +``` + +Connect to port 2299 within three minutes from a machine on the LAN (that port +isn't forwarded). It accepts one connection and exits. + +Note: Synology's sshd only enables the sftp subsystem on the *configured* SFTP +port, so a debug daemon on another port will authenticate you and then report +`subsystem request failed`. That's expected. Authentication is what you're +testing. + +What to look for: + +| Log line | Cause | +| --- | --- | +| `Could not open authorized keys: Permission denied` | `homes` share access — step 3 | +| No `trying public key file` line at all | `PubkeyAuthentication` off — step 6 | +| `matching key found` then `Accepted publickey` | Auth is working | +| `Authentication refused: bad ownership or modes` | Permissions on home/`.ssh`/`authorized_keys` — step 5 | + +### Check the client is offering the key + +```bash +sftp -vvv -i -o IdentitiesOnly=yes -P @ +``` + +`Offering public key:` followed by no `Server accepts key:` means the server +rejected it before requesting a signature — server-side, not a client problem. +Compare the offered fingerprint against what's stored on the NAS: + +```bash +ssh-keygen -lf /id_backup.pub # on the host +ssh-keygen -lf /.ssh/authorized_keys # on the NAS, via Task Scheduler +``` + +### Things that look like the problem but aren't + +Each of these was tried during the original setup and none of them mattered: + +- **Login shell `/sbin/nologin`.** DSM assigns it to non-admin accounts and + resets it on reboot. Irrelevant here — `internal-sftp` needs no shell. +- **Membership of `administrators`.** Required for interactive SSH shells, not + for SFTP key auth. Don't grant it; it puts a privileged account behind an + internet-facing port. +- **`chmod 711` on `.ssh`.** Frequently recommended online. `700` is correct and + works. +- **`StrictModes` on parent directories.** Worth checking, but DSM's defaults + (`/volume1/homes` at `711` root-owned) are already correct. + +### Other notes + +- **`synoservicectl` doesn't exist on DSM 7.** It's `synosystemctl`. +- **File Station won't show `.ssh`** (hidden directory) and can't reliably set + Unix modes on ACL-managed shares. Use Task Scheduler scripts. +- **`/var/services/homes` is a symlink to `/volume1/homes`.** Either path works. + +--- + +## Later: second destination -`docs/backup-strategy.md` also describes fanning the same backup out to an -off-site S3-compatible bucket for real geographic redundancy. That's -deliberately deferred — this iteration only wires up the NAS. Adding it -later just means a second `restic backup ... -r ` (and its own -`restic forget`) in `backup.sh`, with its own credentials — no changes to -the NAS-side setup above. +`docs/backup-strategy.md` describes fanning the same backup out to an off-site +S3-compatible bucket for geographic redundancy. Deliberately deferred. Adding it +means a second `restic backup ... -r ` and its own `restic forget` +in `backup.sh`, with separate credentials. No changes to the NAS setup above.