The first week live: hardening a $6 server

Self-hosting a web app on a $6 droplet stops on launch day, 4 September. This picks up the morning after.

Nothing was on fire. Everything here came from looking — at the response headers, at the access log, at what Cloudflare was holding, at the backups — and finding that "it works" and "it is right" were two different things. Most of the problems were invisible from a browser. One of them could hand a user's data to strangers.

What changed in the first nine days

• Security headers on every site, and on every path of every site
• Scanner requests dropped with no response and no log line
• One access log that says which site each request was for
• An API that can never be cached by the CDN in front of it
• A restore drill that runs monthly, after the first real restore failed
• A second app on the same droplet, fenced so it cannot starve the first
• Measurements that answer "do I need a bigger server?" — no, and why

The shape of it now

        Cloudflare  ── DNS, TLS, proxy, edge cache
             │
             ▼
   ┌───────────────────────────────────────────────────┐
   │  Ubuntu 24.04, 1 GB RAM, 2 GB swap                 │
   │                                                    │
   │  Nginx :443 ─┬─▶ tracker.  → EpisodeTracker :3001  │
   │              ├─▶ gym.      → gym-api :3002 + files │
   │              └─▶ five static sites                 │
   │                                                    │
   │  PostgreSQL 16, localhost only                     │
   │     episode_tracker · gym                          │
   └───────────────────────────────────────────────────┘

Same three ideas as on launch day: two doors open, everything else on localhost, nothing running as root that does not need to.

Security headers, and the nginx rule that eats them

Four headers, in one snippet, included by every site:

# /etc/nginx/snippets/security-headers.conf
add_header X-Content-Type-Options "nosniff" always;
add_header X-Frame-Options "SAMEORIGIN" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
add_header Strict-Transport-Security "max-age=86400" always;

always sends them on error responses too. Without it, a 404 goes out bare.

add_header does not merge

This is the trap, and nothing warns you about it. A location block that sets even one add_header of its own discards every header inherited from the server block. Not merges — discards.

So this looks right and is wrong:

server {
    include snippets/security-headers.conf;

    location = /index.html {
        add_header Cache-Control "public, max-age=300";
        # every security header is now gone — on this path only
    }
}

/ has its headers. /index.html has none. Test the home page and you will never see it. The fix is to repeat the include inside every location that sets a header:

location = /index.html {
    include snippets/security-headers.conf;
    add_header Cache-Control "public, max-age=300";
}

The gym tracker's server block has seven locations that set Cache-Control. Every one of them repeats the include.

Verify per path, not per site:

for p in / /index.html /sw.js /assets/app.js; do
  printf "%-16s " "$p"
  curl -sI "https://app.example.com$p" | grep -ciE 'x-content-type|x-frame|referrer|strict-transport'
done
# every line should say 4

Why HSTS starts at one day

HSTS tells the browser to refuse plain HTTP for your domain, and the browser remembers that for the whole max-age. Set a year, then find you need plain HTTP on that hostname for any reason, and every browser that has seen the header refuses for a year — you cannot take it back, because those browsers will not ask you again.

So it ships at 86400, one day. Raise it to 31536000 once nothing has broken for a while. Adding preload is harder still to undo: removal from the browsers' built-in list takes months.

What the access log is actually made of

I opened /var/log/nginx/access.log expecting visitors. On 6 September, the day I looked, it held 5,404 requests, and 4,316 of them were 404s — 80%. Almost none were people. They were scanners asking every IP on the internet for the same files:

/.env   /.git/config   /wp-admin/install.php   /xmlrpc.php   /config.json

None of those exist on this server, so nothing was at risk. The cost was that the log was unreadable: anything real was buried under thousands of lines of someone else's shopping list.

Drop them with 444

# /etc/nginx/snippets/block-scanners.conf  (trimmed)

# Keep this reachable, and first: ^~ beats the regexes below.
location ^~ /.well-known/ {
    try_files $uri =404;
}

# .env, .env.backup, /app/.env, .git/config ...
location ~* (^|/)\.(env|git|svn|hg|aws|ssh|npmrc|htpasswd|bash_history)($|[./]) {
    access_log off;
    log_not_found off;
    return 444;
}

location ~* ^/(wp-admin|wp-login|wp-content|wp-includes|wordpress|wp/|xmlrpc\.php) {
    access_log off;
    log_not_found off;
    return 444;
}

• return 444 is nginx-specific: close the connection and send nothing. Cheaper than rendering a 404, and it tells the scanner nothing.
• access_log off is the real prize. The request no longer exists as far as the log is concerned.
• location ^~ /.well-known/ comes first. A ^~ prefix match stops nginx from checking regexes at all, so certificate challenges and similar files can never be caught by an over-eager pattern.

Through Cloudflare, a blocked path answers 520 — Cloudflare's catch-all for an origin that gave it nothing usable, here a closed connection. That is the 444 working, not a fault.

What the log looks like since. Between 7 and 10 September it held 2,499 to 4,334 lines a day. The 404s did not vanish — on 10 September there were still 2,802 of them. But 2,542 of those 2,802 were addressed to the bare IP address, not to any hostname: bots sweeping IP ranges, which never used the domain and never passed through Cloudflare. They ask for the long tail — service-account.json, settings.py, /proxy — that no pattern list will ever fully cover. A 404 from a static file costs nothing, so I leave them. The requests that are blocked do not appear at all, so those numbers understate what the snippet is doing.

One log, many sites

Nginx writes every site to the same file, and the default combined format does not record which site a request was for. GET /login 404 could be any of them. Put the host first:

# /etc/nginx/nginx.conf, in the http block
log_format vhost '$host $remote_addr - [$time_local] "$request" $status '
                 '$body_bytes_sent "$http_referer" "$http_user_agent"';
access_log /var/log/nginx/access.log vhost;

Then "which site is busiest today" is one line:

sudo awk "/$(date +%d/%b/%Y)/ {print \$1}" /var/log/nginx/access.log | sort | uniq -c | sort -rn

Ubuntu's logrotate already rotates it daily and keeps 14 days.

sudo wc -l < /var/log/nginx/access.log → Permission denied. Same trap as sudo echo >> in the first guide: your shell opens the file for the <, and your shell is not root. Give the path as an argument instead — sudo wc -l file.

Real visitor IPs, and why fail2ban would not help

Behind Cloudflare, every connection to nginx comes from a Cloudflare address. The first guide added cloudflare-realip.conf, which trusts Cloudflare's 21 published ranges and takes the visitor's address from CF-Connecting-IP. The log shows real clients.

That is also why fail2ban is not installed. fail2ban reads an IP from the log and bans it at the firewall. But the firewall never sees that IP — the TCP connection is from Cloudflare's edge:

Where a limit does matter — login attempts — it belongs in the app, keyed on the header nginx sets from the real address:

// Nginx sets X-Real-IP from Cloudflare's CF-Connecting-IP. X-Forwarded-For
// is not used: its left-most entry is whatever the client chose to send.
keyGenerator: (request) => request.headers["x-real-ip"] ?? request.ip,

The comment is the important part. X-Forwarded-For is a list that nginx appends to. Anyone can send one with a fake first entry, and a rate limiter keyed on that entry can be dodged by changing it on every request.

Cloudflare's cache is a layer you did not configure

Proxying through Cloudflare (the orange cloud) also puts a cache between your server and everyone else. You did not write its rules, and it has opinions.

A deploy that did not appear

A redeployed icon kept showing the old one. The server had the new file. The public URL did not:

ssh server 'ls -l /var/www/site/icon.png'        # 30082 bytes
curl -sI https://site/icon.png | grep -iE 'content-length|cf-cache-status|age'
#   content-length: 5926        <- the old file
#   cf-cache-status: HIT
#   age: 1080                   <- Cloudflare has held it 18 minutes

cf-cache-status: HIT with the wrong size is the whole diagnosis. To see what your server is sending with Cloudflare out of the way, send the request straight to the origin while keeping the hostname:

curl -sI -k --resolve site:443:ORIGIN_IP https://site/icon.png

(-k because the origin certificate is only trusted by Cloudflare.)

Two fixes, and you want both:

1. Purge — dashboard → Caching → Configuration → Purge. Immediate.
2. Stop causing it. A long max-age belongs only on filenames that change when their contents do. Fixed names get a short one:

# Hashed names: the bytes never change under the same name.
location /assets/ {
    add_header Cache-Control "public, max-age=31536000, immutable";
    include snippets/security-headers.conf;
}

# Icons keep their names across deploys.
location ~* \.(png|ico|svg)$ {
    add_header Cache-Control "public, max-age=3600, must-revalidate";
    include snippets/security-headers.conf;
}

One more wrinkle: Cloudflare's Browser Cache TTL setting (four hours by default) rewrites the max-age browsers receive. The public response and the origin response can disagree, and only the origin tells you what nginx is actually configured to send.

When something stale is showing, work outwards: service worker → browser → Cloudflare → origin. cf-cache-status rules out two of those in one request.

An API response it should never have kept

This one was serious.

Cloudflare decides what to cache largely by file extension, whatever the path. .css, .js, images, .pdf and .csv are on its default list. .html and .json are not — which is why an ordinary JSON API does not trip over this, and why it came as a surprise.

The gym tracker has a CSV export at /api/export.csv. It returns the signed-in user's workout history. The API sent no Cache-Control header at all. So Cloudflare saw a .csv, cached the first response at its edge, and would serve that user's export to anyone who requested the URL — signed in or not — until the cache entry expired. Nothing on the server was wrong. Every request that reached the origin was authenticated correctly. The leak was entirely in a layer the app knew nothing about.

It was found on 12 September, during a full test pass of the app, and fixed the same day.

The fix is one hook, because every API answer is for one user and none of them may be stored anywhere:

// Every API answer is for one signed-in user. Without this, Cloudflare caches
// any URL with a file extension at the edge.
app.addHook("onSend", async (_request, reply) => {
  reply.header("cache-control", "no-store");
});

— with a test that asserts the header, so a refactor cannot quietly remove it.

Check it from outside, with no cookies:

curl -sI https://app.example.com/api/export.csv | grep -iE 'cache-control|cf-cache-status'
#   cache-control: no-store
#   cf-cache-status: BYPASS       <- DYNAMIC is also fine. HIT never is.

The rule I took from it: a response that belongs to one user must say so. The default is not "private" — the default is whatever the proxy in front of you decides.

Restoring a backup on purpose

The first guide says to restore a backup before you need one. Two days after launch I finally did, and the first attempt failed:

pg_restore: error: could not open input file ".../episode-tracker-20260906-010001.dump": Permission denied

The permission trap

The nightly dumps are chmod 600, owned by omar. That is correct — each one contains every user's data. But restores run as the postgres user (sudo -u postgres pg_restore), and postgres is not omar.

This fails the first time you really need a restore, which is the worst moment to meet it.

The tempting fix is chmod 644 on the dumps. That makes every user's data readable by every account on the machine, forever, to make one command work once. Instead, stage a short-lived copy that postgres owns, and make sure it disappears even if the script dies halfway:

STAGED="/var/lib/postgresql/.restore-$$.dump"
trap 'sudo rm -f "$STAGED"' EXIT
sudo install -o postgres -g postgres -m 600 "$DUMP" "$STAGED"
sudo -u postgres pg_restore --no-owner --role=episode -d "$DB" "$STAGED"

install copies, sets the owner and sets the mode in one command, and the trap removes the copy on any exit — success, error or Ctrl-C.

The drill

~/bin/restore-db.sh does the whole exercise, and the safe thing is the default:

1. Take the newest dump, and refuse it if pg_restore --list cannot read it.
2. Create a scratch database with a timestamped name.
3. Restore into it.
4. For every table, compare count(*) in the scratch copy against live. (Not pg_stat_user_tables — that is an estimate.)
5. Count the indexes and constraints that came back. A dump can hold every row and still restore into something the app cannot use.
6. Drop the scratch database.

The first run, on 6 September:

TABLE                          RESTORED       LIVE
NotificationLog                      57         57   match
Session                              26         26   match
ShowCache                           113        113   match
TrackedShow                          89         89   match
WatchedEpisode                      876        876   match
...
indexes restored: 19
constraints:      15

RESULT: backup is good - restored cleanly and every table matches live.

Nine tables, all matching. It now runs itself at 03:00 UTC on the 1st of every month and appends to ~/backups/restore-drill.log:

0 3 1 * * { echo "===== $(date -u '+\%Y-\%m-\%d \%H:\%M UTC') ====="; /home/omar/bin/restore-db.sh; } >> /home/omar/backups/restore-drill.log 2>&1

(% has to be escaped in a crontab, or cron treats it as a newline.)

The dump is taken at 01:00 and the drill runs at 03:00, so if anyone uses the app in between, the live counts will be higher. The script reports that as "restored cleanly, but some counts differ" rather than as a failure.

When it is real

restore-db.sh --live replaces the live database, so it is built to be hard to run by accident:

• you must type RESTORE — not y
• it takes a safety dump of the current live data first, and refuses to continue if that fails
• it stops the app, recreates the database, restores, starts the app, prints table counts and the site's HTTP status
• if pg_restore reported errors, it prints the exact command to roll back to the safety dump

There is deliberately no automatic rollback. A person looking at the output decides.

What writing this turned up

Checking every claim in this guide against the server found two gaps, which is the argument for checking:

• The drill only covers EpisodeTracker. The gym database is dumped nightly and each dump is verified as readable, but none has been restored on a schedule. Readable is not restorable.
• The copy off the machine had gone quiet. The laptop pulls the dumps every morning at 05:00 — according to its crontab. Its log's last entry was 8 September, and no gym dump had ever arrived. The script was fine, and so was the crontab line. The laptop was asleep at 05:00, and macOS cron does not run a job whose time passed while the machine slept. It skips that day, and logs nothing.

The fix was to move the job to a launchd agent, which runs a missed calendar job as soon as the Mac wakes, plus once at login in case it was shut down:

<key>StartCalendarInterval</key>
<dict><key>Hour</key><integer>5</integer><key>Minute</key><integer>0</integer></dict>
<key>RunAtLoad</key><true/>

Nothing was lost — the server keeps seven days, and the first run brought every one of them down — but only because it was noticed inside that week.

A scheduled job is proven by the last line of its log, not by the line in the crontab.

A second app on the same droplet

On 11 September the gym tracker moved in. Nothing about the first app changed; the second one got its own copy of everything:

Shared: nginx, Postgres, the certificate, the swap. Separate users and databases mean a bug in one app's queries cannot read the other's tables.

Fence it off

On a 1 GB machine, the new app's worst day should not become the old app's outage. Its systemd unit caps it:

# Compiled JS run by node directly: no tsx in memory. The heap cap and MemoryMax
# keep this app from starving EpisodeTracker on the shared 1 GB droplet.
ExecStart=/usr/bin/node --max-old-space-size=128 dist/index.js
Restart=always
MemoryMax=256M
NoNewPrivileges=true

• MemoryMax=256M — if it leaks, systemd kills this service and Restart=always brings it back. Without a cap, the kernel's out-of-memory killer chooses the victim from everything on the box, and it does not choose by blame.
• --max-old-space-size=128 — Node's own heap limit, set well under that cap, so a runaway heap ends the process cleanly before systemd ever has to.
• NoNewPrivileges=true — nothing the process starts can gain privileges it did not have.

The difference between the two apps shows up in memory. Measured with systemctl show -p MemoryCurrent -p MemoryPeak on 13 September:

Build somewhere else

The gym tracker is built on the laptop, and only the finished output is copied up. The droplet runs npm ci and the migration, then restarts. Installing and building are the most memory-hungry things this box is ever asked to do — the reason the 2 GB of swap exists — and the build no longer happens there.

After a reboot, wait before debugging

The reboot test on 11 September: the kernel started at 20:08:50, gym-api was listening at 20:09:06 and EpisodeTracker at 20:09:09. In between, nginx is up and answers 502 for every app, and systemctl status says active — the process has started, it just is not listening yet. Give it a minute or two before treating a post-reboot 502 as a failure.

Is 1 GB still enough?

I keep asking myself this, so it gets answered with numbers. uptime, free -m, df -h and the kernel journal:

The two databases are about 9 MB each.

No. The server is idle. The number that moved is available memory, down by a third with the second app — still more than twice the line. And the table above says where most of it went: the cheapest RAM on this server is running EpisodeTracker as compiled JavaScript, the way gym-api already runs, not a bigger droplet.

Upgrading from 1 GB to 2 GB doubles the bill to $12/month, every month, to solve a problem the numbers do not show. If it ever becomes real:

• Resize CPU and RAM only. That is reversible — you can step back down.
• Do not resize the disk. A disk resize is permanent; the droplet can never shrink below it again.

Gotchas, collected

The habits worth keeping

• Check per path, not per site. Headers, caching and blocking are all decided per location.
• Look at the edge and the origin separately. cf-cache-status and --resolve answer "is it my server or is it Cloudflare" in two requests.
• Anything that belongs to one user says no-store. Do not rely on a default you did not set.
• Restore on a schedule, and read the result. A backup is a hope until it has been restored; a schedule is a hope until its log says it ran.
• Measure before you pay. The answer to "do I need a bigger server" is in free -m, not in a feeling.