--- title: Gotchas — server, deploy and CDN traps, as symptom → cause → fix summary: Every infrastructure failure mode met while running several apps on one small Ubuntu server behind Cloudflare. Most are silent. when_to_use: Something is broken and the error points somewhere unhelpful — or before touching SSH, Postgres permissions, certificates, caching or scheduled jobs. related: [workflows/new-server.md, workflows/ship-a-site.md, workflows/lean-production.md, workflows/backups-and-restore.md] last_verified: 2026-09-26 --- # Gotchas Each entry: **symptom** as you will see it, the **cause**, the **fix**. Search this file for the exact error text first. Placeholders: ``, ``, ``, ``, ``, ``. ## Quick index | Symptom | Section | |---|---| | Key auth fails, no explanation | SSH | | `PermitRootLogin no` has no effect | SSH | | SSH to the domain times out | SSH | | `pkill -f` killed your SSH session | Shell | | `sudo echo x >> /etc/f` → Permission denied | Shell | | `permission denied for schema public` | Postgres | | `invalid URI query parameter: "schema"` | Postgres | | `pg_restore: ... Permission denied` on your own dump | Postgres | | Local Postgres won't start after a reboot | Postgres | | `unknown directive "http2"` | Nginx | | `unknown "connection_upgrade" variable` | Nginx | | Security headers missing on some paths only | Nginx | | A feature (location, camera) silently blocked | Nginx | | 5xx on everything right after turning on the proxy | TLS / Cloudflare | | Browser cert error, `curl` fine | DNS caches | | New file on the server, old one in the browser | Cloudflare cache | | One user's data served to someone else | Cloudflare cache | | Browser `max-age` differs from what nginx sends | Cloudflare cache | | Cloudflare API calls from a script get 403 | Cloudflare | | PWA won't install / never updates / shows the old build | PWA | | Login "does nothing" | Apps | | `.env` values "missing" | Apps | | Deploy dies at a step after `source .env` | Deploys | | `rsync: --chown: unknown option` | Deploys | | New asset returns 403 | Deploys | | Server `git pull` refuses to fast-forward | Deploys | | Deploy succeeded, old code still runs | Deploys | | 502 on every app right after a reboot | systemd | | One app uses several times the memory of its peers | Memory | | Laptop backups stale although "scheduled" | Scheduling | | Cron line with `%` does nothing | Scheduling | | Mail never arrives from the server | Email | | Adding an email provider broke all mail for the domain | Email | | Bill higher than the server price | Costs | --- ## SSH ### Key auth fails with "Permission denied (publickey)" and no hint **Cause:** sshd ignores an `authorized_keys` file that others could modify. Wrong modes or owner on `~/.ssh` or the file. **Fix:** `chmod 700 ~/.ssh; chmod 600 ~/.ssh/authorized_keys; chown -R : ~/.ssh`. ### `PermitRootLogin no` (or `PasswordAuthentication no`) has no effect **Cause:** `/etc/ssh/sshd_config` starts with `Include /etc/ssh/sshd_config.d/*.conf` and sshd uses the **first** value found. Cloud images put files there. Editing the main file changes nothing — and you believe the server is hardened. **Fix:** put settings in `/etc/ssh/sshd_config.d/10-hardening.conf` (low number wins). Verify the *effective* config: `sudo sshd -T | grep -iE "permitrootlogin|passwordauth"`. Keep a second session open while changing login settings. ### `ssh user@` times out **Cause:** the domain is proxied by Cloudflare, which carries HTTP/HTTPS only. **Fix:** SSH to the IP. Put it in `~/.ssh/config` as `Host / HostName `. ### Locked out right after `ufw enable` **Cause:** port 22 was not allowed first. **Fix:** provider's web console → `ufw allow 22/tcp`. Next time: allow, `ufw show added`, then enable. --- ## Shell ### `sudo echo x >> /etc/file` → Permission denied **Cause:** the redirect is done by *your* shell, which is not root; sudo only elevates `echo`. **Fix:** `echo x | sudo tee -a /etc/file`. Same for `sudo cmd < file`: pass the path as an argument. ### `pkill -f pattern` over SSH killed the session **Cause:** the pattern matches pkill's own command line (and the ssh command carrying it). **Fix:** `pkill -f "[p]attern"` — the bracket cannot match itself. ### A deleted cloud resource still responds **Cause:** teardown is asynchronous; connections drain. **Fix:** re-check after a minute before concluding either way. --- ## Postgres ### `permission denied for schema public` on the first migration **Cause:** since Postgres 15, owning a database does not grant CREATE on `public`. **Fix:** ```sql GRANT ALL ON SCHEMA public TO ; ALTER SCHEMA public OWNER TO ; ``` ### Confusing errors like `invalid integer value "ON" for connection option "port"` **Cause:** pasting multi-line SQL that contains `\c` — backslash commands consume the next line. **Fix:** `psql -d -c "..."` instead of `\c`. ### `invalid URI query parameter: "schema"` **Cause:** Prisma appends `?schema=public` to `DATABASE_URL`; libpq (psql, pg_dump) rejects it. **Fix:** `"${DATABASE_URL%%\?*}"`. ### `pg_restore: could not open input file ... Permission denied` on your own dump **Cause:** dumps are `600` owned by the deploy user; `sudo -u postgres pg_restore` runs as `postgres`. **Fix:** stage a short-lived postgres-owned copy, removed by `trap ... EXIT`. Never `chmod 644` dumps. You meet this the first time you really need a restore — run a drill on a schedule. ### `unsupported version (1.x) in file header` / `SET transaction_timeout` fails **Cause:** dump made by a newer `pg_dump` than the restoring server (PG17+ → PG16). **Fix:** matching client versions, or a plain `--data-only` dump with that line stripped: `grep -v '^SET transaction_timeout' dump.sql | psql "$URL" -v ON_ERROR_STOP=1`. ### Row counts differ slightly between source and restored copy **Cause:** `pg_stat_user_tables` is an estimate. **Fix:** compare `select count(*)`. ### Local (Homebrew) Postgres won't start after an unclean reboot Symptom: `brew services start` → `Bootstrap failed: 5: Input/output error`, launchd retries forever. **Cause:** the log says `lock file "postmaster.pid" already exists` — the PID in it was recycled by another process. **Fix:** confirm that PID isn't postgres (`ps -p -o comm=`), then delete `/var/postgresql@16/postmaster.pid`. Never kill the PID in the file. Check this first when local integration tests fail with connection refused on 5432. --- ## Nginx ### `unknown directive "http2"` **Cause:** Ubuntu 24.04 ships Nginx 1.24; `http2 on;` is 1.25+. **Fix:** `listen 443 ssl http2;`. ### `unknown "connection_upgrade" variable` **Cause:** `$connection_upgrade` is not built in. **Fix:** `/etc/nginx/conf.d/upgrade-map.conf` with the `map $http_upgrade $connection_upgrade` block. ### Security headers present on `/` but missing on `/index.html` (or `/sw.js`, `/assets/...`) **Cause:** a `location` with any `add_header` of its own **discards every inherited** `add_header`. Not merged — replaced. `always` does not help (it is about error responses). **Fix:** repeat `include snippets/security-headers.conf;` in every location that sets a header. Verify per path: `curl -sI https:///index.html | grep -ciE 'x-content-type|x-frame|referrer|strict-transport'` → 4. ### HSTS set too long, now can't go back to HTTP **Cause:** browsers cache `Strict-Transport-Security` for the full `max-age`; `preload` takes months to undo. **Fix (prevention):** start at `max-age=86400`; raise to a year once stable; think hard before `preload`. ### A browser feature (geolocation, camera) silently does nothing on one site **Cause:** a `Permissions-Policy` header such as `geolocation=()` blocks it for your own origin too. **Fix:** `geolocation=(self)` on the sites that need it. ### `GET /login 404` in the log — but which site? **Cause:** the default `combined` format has no host. **Fix:** a `vhost` log format with `$host` first. ### A blocked scanner path shows 520 through Cloudflare **Cause:** `return 444` closes the connection; Cloudflare reports "origin sent nothing". **Fix:** none — expected. ### fail2ban bans do nothing (or block everyone) **Cause:** behind Cloudflare, the TCP peer is Cloudflare; the logged IP (after real-IP) never connects directly. **Fix:** rate-limit in the app keyed on `X-Real-IP` (set by nginx from `CF-Connecting-IP`). Never key on the first entry of `X-Forwarded-For` — the client controls it. ### `.md` / `.webmanifest` files download instead of displaying (or are ignored) **Cause:** not in nginx's `mime.types` → `application/octet-stream`. **Fix:** `types { text/markdown md; }` in that location; `application/manifest+json webmanifest` in `conf.d/`. --- ## TLS and Cloudflare ### 5xx on every request right after switching the record to proxied **Cause:** SSL mode Full (strict) while the origin has no valid certificate yet. **Fix:** grey cloud until HTTPS works on the origin (test with `curl -k --resolve :443:`), then proxy. ### Certificate warning when opening the server by IP **Cause:** Cloudflare origin certificates are trusted only by Cloudflare. **Fix:** none — expected. Visitors see Cloudflare's certificate. ### Nginx won't start after installing the certificate **Cause:** cert and key are not a pair (pasted the wrong one). **Fix:** compare `openssl x509 -pubkey` and `openssl pkey -pubout` hashes. ### "Full" vs "Full (strict)" **Cause:** both encrypt to the origin; only strict validates its certificate. Dashboards often default to Full or Automatic. **Fix:** read the setting; choose Full (strict). ### Cloudflare API calls made from a script or from the dashboard's own session get 403 **Cause:** Cloudflare's WAF blocks unusual clients (e.g. Python's default user agent), and the dashboard's internal API accepts reads but blocks many writes from injected scripts. **Fix:** use a scoped API token with the official API for automation; send a normal User-Agent; do writes you can't script through the dashboard UI. Note: API cache purge may be unavailable this way — dashboard → Caching → Custom Purge (the free plan has Prefix purge there, e.g. `/hero/v1/`). --- ## Cloudflare edge cache ### New file on the server, old file in the browser Diagnosis: ```bash curl -sI https:///icon.png | grep -iE 'content-length|cf-cache-status|age' # cf-cache-status: HIT + wrong content-length + an age → the edge holds the old one curl -sI -k --resolve :443: https:///icon.png # what the origin sends ``` **Cause:** Cloudflare caches static extensions (images, css, js, fonts, pdf, csv...) at the edge, by default for hours (images ~4 h). **Fix:** purge now; long-term, **content-hash filenames** (import assets through the bundler so a changed file is a new URL), long `max-age` only on hashed names, `max-age=3600, must-revalidate` on fixed names. For a folder of frames or media you must replace, version the folder (`/v2/`). ### One user's private response served to other people **Cause:** an authenticated URL with a cacheable extension (e.g. `/api/export.csv`) and no `Cache-Control` — Cloudflare caches it at the edge for everyone, logged in or not. **Fix:** `Cache-Control: no-store` on **every** API response (one global hook) plus a test. Check with no cookies: `curl -sI https:///api/export.csv` → `no-store`, `cf-cache-status` BYPASS/DYNAMIC, never HIT. ### Browser receives a different `max-age` than nginx sends **Cause:** the zone's **Browser Cache TTL** (default 4 h) rewrites it. **Fix:** set it to "Respect Existing Headers" if the origin should decide. Compare edge vs origin before concluding a config didn't apply. Stale content debugging order: **service worker → browser → Cloudflare edge → origin.** `cf-cache-status` rules out two layers in one request. --- ## DNS caches ### `dig` shows the new address, the browser the old one (or a cert error) **Cause:** OS resolver cache and the browser's own cache. **Fix:** macOS `sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder`; Chrome `chrome://net-internals/#dns` → Clear host cache. Confirm where a request went with `curl -w '%{remote_ip}'` before touching the server. --- ## PWA / service workers ### Site loads but won't install **Cause:** `.webmanifest` served as `application/octet-stream`; or no HTTPS. **Fix:** `types { application/manifest+json webmanifest; }`; TLS first. ### Installed app never gets updates **Cause:** `sw.js` cached by the browser/CDN. **Fix:** `Cache-Control: no-cache, no-store, must-revalidate` on `/sw.js`. ### Every deploy shows up one load late **Cause:** service worker is cache-first for navigations; the page renders from cache before the new worker activates (`skipWaiting`/`clients.claim` don't help). **Fix:** network-first for `request.mode === "navigate"` (cache only as offline fallback); cache-first for static assets. Headless screenshots/checks must unregister the worker first. ### iPhone "Add to Home Screen" shows a page screenshot as the icon **Cause:** iOS ignores manifest icons. **Fix:** `` (180×180). --- ## Apps ### Login "does nothing" — no error, back to the login page **Cause:** `Secure` cookie discarded over plain HTTP. **Fix:** get TLS working. Don't set `NODE_ENV=development` to work around it. ### App reports a missing env var the file clearly has **Cause:** leading whitespace from pasting (`" PORT"`), or an unquoted value with spaces. **Fix:** `sed -i 's/^[[:space:]]*//' .env`; quote values with spaces. ### App reachable on its port from the internet **Cause:** it binds `0.0.0.0`. **Fix:** bind `127.0.0.1`; make HOST configurable. --- ## Deploys ### Deploy dies at "migrate" (or any step after `set -a; . ./.env`) **Cause:** an unquoted value like `EMAIL_FROM=Name >` — bash reads `<` as a redirect. **Fix:** quote every value containing `<`, `>`, spaces or quotes. ### `rsync: --chown: unknown option` on a Mac **Cause:** macOS ships `openrsync`. **Fix:** rsync to a staging dir the deploy user owns, then on the server `sudo rsync -a --chown=root:root --chmod=D755,F644 staging/ /`. ### A newly deployed file returns 403 **Cause:** some tool (e.g. an SVG optimizer) wrote it with mode 600; nginx can't read it. **Fix:** publish with `--chmod=D755,F644`; have the deploy request every asset and fail on non-200. ### `git pull` / fast-forward on the server fails: "local changes present" **Cause:** the server ran `npm install`, which rewrote `package-lock.json` in the checkout. **Fix:** `git checkout -- package-lock.json` then redeploy. Better: `npm ci`, or build on the laptop and rsync. ### Deploy reported success, old behaviour remains **Cause:** a build step failed but the script continued and restarted on the stale `dist/`. **Fix:** `set -euo pipefail` and `npm run build || fail` before any restart. ### A lint step fails on files that aren't yours **Cause:** linter/`tsc` globbing picks up agent worktrees (e.g. `.claude/worktrees/*`) or other checkouts inside the repo. **Fix:** remove stale worktrees before deploying, or exclude them in the lint config. ### Old tabs break right after a deploy **Cause:** the publish used `--delete`, removing hashed chunks an already-open tab still requests. **Fix:** no `--delete` on publish; prune chunks older than 30 days instead. --- ## systemd ### 502 on every app right after `sudo reboot`, `systemctl status` says active **Cause:** started, not yet listening — 15–20 s per Node app on a small box. **Fix:** wait 1–2 minutes before debugging. ### Service keeps restarting **Cause:** crash loop, or hitting `MemoryMax`. **Fix:** `journalctl -u -n 50 --no-pager`; `systemctl show -p NRestarts -p MemoryPeak`. ### `ExecStart=npm start` — flags ignored, stop takes ages **Cause:** npm wrapper: extra process, signals not forwarded. **Fix:** `ExecStart=/usr/bin/node --max-old-space-size=128 dist/index.js`. --- ## Memory ### One app uses 5–6× the memory of similar apps **Cause:** running TypeScript through `tsx`/`ts-node` (compiler resident, no heap cap). Measured: ~325 MB vs 40–55 MB for built peers. **Fix:** build to `dist/`, run with `node --max-old-space-size=128`, `MemoryMax=256M`. See `workflows/lean-production.md`. ### `npm install` / build on the server gets killed **Cause:** OOM on a 1 GB box. **Fix:** 2 GB swap as a safety net; build on the laptop. --- ## Scheduling ### Laptop backup pull "set up in cron" but files are days old **Cause:** macOS cron skips runs whose time passed while the Mac slept — no catch-up, no log. **Fix:** launchd `StartCalendarInterval` (runs on wake) + `RunAtLoad`. Prove schedules by the log's last line. ### Crontab line containing `date +%F` never works **Cause:** unescaped `%` is a newline in crontab. **Fix:** `\%`. ### "Scheduled" CI job runs hours late **Cause:** hosted CI cron (e.g. GitHub Actions) is best-effort; delays of 40 min to 12 h happen. **Fix:** time-sensitive jobs run in-process or from cron/systemd timers on a server that doesn't sleep. --- ## Email ### The server can't send mail; connections to mail servers time out **Cause:** many VPS providers block outbound 25/465/587; and the IP's reverse DNS is a generic name that inboxes reject anyway. **Fix:** send through a transactional provider's **HTTPS API** (port 443). Don't install Postfix. See `workflows/email-sending.md`. ### Adding an email provider broke mail for the whole domain **Cause:** a second `v=spf1` TXT record — a domain may have exactly one; two means SPF fails. **Fix:** one SPF record; merge includes into it. Many API providers need no SPF include at all (DKIM + DMARC suffice). ### DKIM CNAMEs don't verify **Cause:** records created as proxied (orange) in Cloudflare. **Fix:** DKIM/verification records must be **DNS only** (grey). --- ## Costs ### The bill is higher than the server price **Cause:** snapshots bill per snapshot, per GB, per month, forever — separately from automated backups. **Fix:** snapshot only right before a risky change, delete it once the change is proven. Keep the flat-price automated backups. ### Considering a bigger server **Cause:** usually a dev-mode app, not real load. **Fix:** measure (`free -m`, per-service `MemoryCurrent`, load, disk) first. If you must resize, CPU/RAM only — disk resizes are permanent.