Gotchas — server, deploy and CDN traps, as symptom → cause → fix

raw markdown reference/gotchas.md verified 2026-09-26

Each entry: symptom as you will see it, the cause, the fix. Search this file for the exact error text first. Placeholders: <domain>, <server>, <server-ip>, <app>, <db-user>, <web-root>.

Quick index

Symptom	Section
Key auth fails, no explanation	SSH
`PermitRootLogin no` has no effect	SSH
SSH to the domain times out	SSH
`pkill -f` killed your SSH session	Shell
`sudo echo x >> /etc/f` → Permission denied	Shell
`permission denied for schema public`	Postgres
`invalid URI query parameter: "schema"`	Postgres
`pg_restore: ... Permission denied` on your own dump	Postgres
Local Postgres won't start after a reboot	Postgres
`unknown directive "http2"`	Nginx
`unknown "connection_upgrade" variable`	Nginx
Security headers missing on some paths only	Nginx
A feature (location, camera) silently blocked	Nginx
5xx on everything right after turning on the proxy	TLS / Cloudflare
Browser cert error, `curl` fine	DNS caches
New file on the server, old one in the browser	Cloudflare cache
One user's data served to someone else	Cloudflare cache
Browser `max-age` differs from what nginx sends	Cloudflare cache
Cloudflare API calls from a script get 403	Cloudflare
PWA won't install / never updates / shows the old build	PWA
Login "does nothing"	Apps
`.env` values "missing"	Apps
Deploy dies at a step after `source .env`	Deploys
`rsync: --chown: unknown option`	Deploys
New asset returns 403	Deploys
Server `git pull` refuses to fast-forward	Deploys
Deploy succeeded, old code still runs	Deploys
502 on every app right after a reboot	systemd
One app uses several times the memory of its peers	Memory
Laptop backups stale although "scheduled"	Scheduling
Cron line with `%` does nothing	Scheduling
Mail never arrives from the server	Email
Adding an email provider broke all mail for the domain	Email
Bill higher than the server price	Costs

SSH

Key auth fails with "Permission denied (publickey)" and no hint

Cause: sshd ignores an authorized_keys file that others could modify. Wrong modes or owner on ~/.ssh or the file. Fix: chmod 700 ~/.ssh; chmod 600 ~/.ssh/authorized_keys; chown -R <user>:<user> ~/.ssh.

`PermitRootLogin no` (or `PasswordAuthentication no`) has no effect

Cause: /etc/ssh/sshd_config starts with Include /etc/ssh/sshd_config.d/*.conf and sshd uses the first value found. Cloud images put files there. Editing the main file changes nothing — and you believe the server is hardened. Fix: put settings in /etc/ssh/sshd_config.d/10-hardening.conf (low number wins). Verify the effective config: sudo sshd -T | grep -iE "permitrootlogin|passwordauth". Keep a second session open while changing login settings.

`ssh user@<domain>` times out

Cause: the domain is proxied by Cloudflare, which carries HTTP/HTTPS only. Fix: SSH to the IP. Put it in ~/.ssh/config as Host <server> / HostName <server-ip>.

Locked out right after `ufw enable`

Cause: port 22 was not allowed first. Fix: provider's web console → ufw allow 22/tcp. Next time: allow, ufw show added, then enable.

Shell

`sudo echo x >> /etc/file` → Permission denied

Cause: the redirect is done by your shell, which is not root; sudo only elevates echo. Fix: echo x | sudo tee -a /etc/file. Same for sudo cmd < file: pass the path as an argument.

`pkill -f pattern` over SSH killed the session

Cause: the pattern matches pkill's own command line (and the ssh command carrying it). Fix: pkill -f "[p]attern" — the bracket cannot match itself.

A deleted cloud resource still responds

Cause: teardown is asynchronous; connections drain. Fix: re-check after a minute before concluding either way.

Postgres

`permission denied for schema public` on the first migration

Cause: since Postgres 15, owning a database does not grant CREATE on public. Fix:

GRANT ALL ON SCHEMA public TO <db-user>;
ALTER SCHEMA public OWNER TO <db-user>;

Confusing errors like `invalid integer value "ON" for connection option "port"`

Cause: pasting multi-line SQL that contains \c — backslash commands consume the next line. Fix: psql -d <db-name> -c "..." instead of \c.

`invalid URI query parameter: "schema"`

Cause: Prisma appends ?schema=public to DATABASE_URL; libpq (psql, pg_dump) rejects it. Fix: "${DATABASE_URL%%\?*}".

`pg_restore: could not open input file ... Permission denied` on your own dump

Cause: dumps are 600 owned by the deploy user; sudo -u postgres pg_restore runs as postgres. Fix: stage a short-lived postgres-owned copy, removed by trap ... EXIT. Never chmod 644 dumps. You meet this the first time you really need a restore — run a drill on a schedule.

`unsupported version (1.x) in file header` / `SET transaction_timeout` fails

Cause: dump made by a newer pg_dump than the restoring server (PG17+ → PG16). Fix: matching client versions, or a plain --data-only dump with that line stripped: grep -v '^SET transaction_timeout' dump.sql | psql "$URL" -v ON_ERROR_STOP=1.

Row counts differ slightly between source and restored copy

Cause: pg_stat_user_tables is an estimate. Fix: compare select count(*).

Local (Homebrew) Postgres won't start after an unclean reboot

Symptom: brew services start → Bootstrap failed: 5: Input/output error, launchd retries forever. Cause: the log says lock file "postmaster.pid" already exists — the PID in it was recycled by another process. Fix: confirm that PID isn't postgres (ps -p <pid> -o comm=), then delete <brew-prefix>/var/postgresql@16/postmaster.pid. Never kill the PID in the file. Check this first when local integration tests fail with connection refused on 5432.

Nginx

`unknown directive "http2"`

Cause: Ubuntu 24.04 ships Nginx 1.24; http2 on; is 1.25+. Fix: listen 443 ssl http2;.

`unknown "connection_upgrade" variable`

Cause: $connection_upgrade is not built in. Fix: /etc/nginx/conf.d/upgrade-map.conf with the map $http_upgrade $connection_upgrade block.

Security headers present on `/` but missing on `/index.html` (or `/sw.js`, `/assets/...`)

Cause: a location with any add_header of its own discards every inherited add_header. Not merged — replaced. always does not help (it is about error responses). Fix: repeat include snippets/security-headers.conf; in every location that sets a header. Verify per path: curl -sI https://<domain>/index.html | grep -ciE 'x-content-type|x-frame|referrer|strict-transport' → 4.

HSTS set too long, now can't go back to HTTP

Cause: browsers cache Strict-Transport-Security for the full max-age; preload takes months to undo. Fix (prevention): start at max-age=86400; raise to a year once stable; think hard before preload.

A browser feature (geolocation, camera) silently does nothing on one site

Cause: a Permissions-Policy header such as geolocation=() blocks it for your own origin too. Fix: geolocation=(self) on the sites that need it.

`GET /login 404` in the log — but which site?

Cause: the default combined format has no host. Fix: a vhost log format with $host first.

A blocked scanner path shows 520 through Cloudflare

Cause: return 444 closes the connection; Cloudflare reports "origin sent nothing". Fix: none — expected.

fail2ban bans do nothing (or block everyone)

Cause: behind Cloudflare, the TCP peer is Cloudflare; the logged IP (after real-IP) never connects directly. Fix: rate-limit in the app keyed on X-Real-IP (set by nginx from CF-Connecting-IP). Never key on the first entry of X-Forwarded-For — the client controls it.

`.md` / `.webmanifest` files download instead of displaying (or are ignored)

Cause: not in nginx's mime.types → application/octet-stream. Fix: types { text/markdown md; } in that location; application/manifest+json webmanifest in conf.d/.

TLS and Cloudflare

5xx on every request right after switching the record to proxied

Cause: SSL mode Full (strict) while the origin has no valid certificate yet. Fix: grey cloud until HTTPS works on the origin (test with curl -k --resolve <domain>:443:<server-ip>), then proxy.

Certificate warning when opening the server by IP

Cause: Cloudflare origin certificates are trusted only by Cloudflare. Fix: none — expected. Visitors see Cloudflare's certificate.

Nginx won't start after installing the certificate

Cause: cert and key are not a pair (pasted the wrong one). Fix: compare openssl x509 -pubkey and openssl pkey -pubout hashes.

"Full" vs "Full (strict)"

Cause: both encrypt to the origin; only strict validates its certificate. Dashboards often default to Full or Automatic. Fix: read the setting; choose Full (strict).

Cloudflare API calls made from a script or from the dashboard's own session get 403

Cause: Cloudflare's WAF blocks unusual clients (e.g. Python's default user agent), and the dashboard's internal API accepts reads but blocks many writes from injected scripts. Fix: use a scoped API token with the official API for automation; send a normal User-Agent; do writes you can't script through the dashboard UI. Note: API cache purge may be unavailable this way — dashboard → Caching → Custom Purge (the free plan has Prefix purge there, e.g. <domain>/hero/v1/).

Cloudflare edge cache

New file on the server, old file in the browser

Diagnosis:

curl -sI https://<domain>/icon.png | grep -iE 'content-length|cf-cache-status|age'
# cf-cache-status: HIT + wrong content-length + an age → the edge holds the old one
curl -sI -k --resolve <domain>:443:<server-ip> https://<domain>/icon.png   # what the origin sends

Cause: Cloudflare caches static extensions (images, css, js, fonts, pdf, csv...) at the edge, by default for hours (images ~4 h). Fix: purge now; long-term, content-hash filenames (import assets through the bundler so a changed file is a new URL), long max-age only on hashed names, max-age=3600, must-revalidate on fixed names. For a folder of frames or media you must replace, version the folder (/v2/).

One user's private response served to other people

Cause: an authenticated URL with a cacheable extension (e.g. /api/export.csv) and no Cache-Control — Cloudflare caches it at the edge for everyone, logged in or not. Fix: Cache-Control: no-store on every API response (one global hook) plus a test. Check with no cookies: curl -sI https://<domain>/api/export.csv → no-store, cf-cache-status BYPASS/DYNAMIC, never HIT.

Browser receives a different `max-age` than nginx sends

Cause: the zone's Browser Cache TTL (default 4 h) rewrites it. Fix: set it to "Respect Existing Headers" if the origin should decide. Compare edge vs origin before concluding a config didn't apply.

Stale content debugging order: service worker → browser → Cloudflare edge → origin. cf-cache-status rules out two layers in one request.

DNS caches

`dig` shows the new address, the browser the old one (or a cert error)

Cause: OS resolver cache and the browser's own cache. Fix: macOS sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder; Chrome chrome://net-internals/#dns → Clear host cache. Confirm where a request went with curl -w '%{remote_ip}' before touching the server.

PWA / service workers

Site loads but won't install

Cause: .webmanifest served as application/octet-stream; or no HTTPS. Fix: types { application/manifest+json webmanifest; }; TLS first.

Installed app never gets updates

Cause: sw.js cached by the browser/CDN. Fix: Cache-Control: no-cache, no-store, must-revalidate on /sw.js.

Every deploy shows up one load late

Cause: service worker is cache-first for navigations; the page renders from cache before the new worker activates (skipWaiting/clients.claim don't help). Fix: network-first for request.mode === "navigate" (cache only as offline fallback); cache-first for static assets. Headless screenshots/checks must unregister the worker first.

iPhone "Add to Home Screen" shows a page screenshot as the icon

Cause: iOS ignores manifest icons. Fix: <link rel="apple-touch-icon" href="/apple-touch-icon.png"> (180×180).

Apps

Login "does nothing" — no error, back to the login page

Cause: Secure cookie discarded over plain HTTP. Fix: get TLS working. Don't set NODE_ENV=development to work around it.

App reports a missing env var the file clearly has

Cause: leading whitespace from pasting (" PORT"), or an unquoted value with spaces. Fix: sed -i 's/^[[:space:]]*//' .env; quote values with spaces.

App reachable on its port from the internet

Cause: it binds 0.0.0.0. Fix: bind 127.0.0.1; make HOST configurable.

Deploys

Deploy dies at "migrate" (or any step after `set -a; . ./.env`)

Cause: an unquoted value like EMAIL_FROM=Name <noreply@<domain>> — bash reads < as a redirect. Fix: quote every value containing <, >, spaces or quotes.

`rsync: --chown: unknown option` on a Mac

Cause: macOS ships openrsync. Fix: rsync to a staging dir the deploy user owns, then on the server sudo rsync -a --chown=root:root --chmod=D755,F644 staging/ <web-root>/.

A newly deployed file returns 403

Cause: some tool (e.g. an SVG optimizer) wrote it with mode 600; nginx can't read it. Fix: publish with --chmod=D755,F644; have the deploy request every asset and fail on non-200.

`git pull` / fast-forward on the server fails: "local changes present"

Cause: the server ran npm install, which rewrote package-lock.json in the checkout. Fix: git checkout -- package-lock.json then redeploy. Better: npm ci, or build on the laptop and rsync.

Deploy reported success, old behaviour remains

Cause: a build step failed but the script continued and restarted on the stale dist/. Fix: set -euo pipefail and npm run build || fail before any restart.

A lint step fails on files that aren't yours

Cause: linter/tsc globbing picks up agent worktrees (e.g. .claude/worktrees/*) or other checkouts inside the repo. Fix: remove stale worktrees before deploying, or exclude them in the lint config.

Old tabs break right after a deploy

Cause: the publish used --delete, removing hashed chunks an already-open tab still requests. Fix: no --delete on publish; prune chunks older than 30 days instead.

systemd

502 on every app right after `sudo reboot`, `systemctl status` says active

Cause: started, not yet listening — 15–20 s per Node app on a small box. Fix: wait 1–2 minutes before debugging.

Service keeps restarting

Cause: crash loop, or hitting MemoryMax. Fix: journalctl -u <app> -n 50 --no-pager; systemctl show <app> -p NRestarts -p MemoryPeak.

`ExecStart=npm start` — flags ignored, stop takes ages

Cause: npm wrapper: extra process, signals not forwarded. Fix: ExecStart=/usr/bin/node --max-old-space-size=128 dist/index.js.

Memory

One app uses 5–6× the memory of similar apps

Cause: running TypeScript through tsx/ts-node (compiler resident, no heap cap). Measured: ~325 MB vs 40–55 MB for built peers. Fix: build to dist/, run with node --max-old-space-size=128, MemoryMax=256M. See workflows/lean-production.md.

`npm install` / build on the server gets killed

Cause: OOM on a 1 GB box. Fix: 2 GB swap as a safety net; build on the laptop.

Scheduling

Laptop backup pull "set up in cron" but files are days old

Cause: macOS cron skips runs whose time passed while the Mac slept — no catch-up, no log. Fix: launchd StartCalendarInterval (runs on wake) + RunAtLoad. Prove schedules by the log's last line.

Crontab line containing `date +%F` never works

Cause: unescaped % is a newline in crontab. Fix: \%.

"Scheduled" CI job runs hours late

Cause: hosted CI cron (e.g. GitHub Actions) is best-effort; delays of 40 min to 12 h happen. Fix: time-sensitive jobs run in-process or from cron/systemd timers on a server that doesn't sleep.

Email

The server can't send mail; connections to mail servers time out

Cause: many VPS providers block outbound 25/465/587; and the IP's reverse DNS is a generic name that inboxes reject anyway. Fix: send through a transactional provider's HTTPS API (port 443). Don't install Postfix. See workflows/email-sending.md.

Adding an email provider broke mail for the whole domain

Cause: a second v=spf1 TXT record — a domain may have exactly one; two means SPF fails. Fix: one SPF record; merge includes into it. Many API providers need no SPF include at all (DKIM + DMARC suffice).

DKIM CNAMEs don't verify

Cause: records created as proxied (orange) in Cloudflare. Fix: DKIM/verification records must be DNS only (grey).

Costs

The bill is higher than the server price

Cause: snapshots bill per snapshot, per GB, per month, forever — separately from automated backups. Fix: snapshot only right before a risky change, delete it once the change is proven. Keep the flat-price automated backups.

Considering a bigger server

Cause: usually a dev-mode app, not real load. Fix: measure (free -m, per-service MemoryCurrent, load, disk) first. If you must resize, CPU/RAM only — disk resizes are permanent.