Production builds only: three apps on 1 GB

The first week live ended with a question: is 1 GB still enough now that a second app lives on the droplet? The answer then was yes, with numbers. Two weeks later there were three Node APIs on it — EpisodeTracker, the gym tracker and Triple Hit — plus five static sites, and the numbers had moved.

On 19 September free memory had fallen to about 250 MB available. Nothing had crashed. Nobody had noticed anything slow. But when I looked at which process was holding the memory, the answer was embarrassing: the oldest app, the one with real users, was running its TypeScript source through a development tool.

What changed

• EpisodeTracker runs a compiled bundle, not tsx src/index.ts — ~325 MB → ~64 MB
• Every Node app starts with --max-old-space-size=128 and sits under a systemd MemoryMax=256M
• Builds happen on the Mac, not on the server (with one exception, below)
• Every deploy ends with a memory reading, and a second one a day later
• The Mac-side backup pull moved from cron to launchd, after cron silently skipped a week

The shape of it now

   ┌────────────────────────────────────────────────────────────┐
   │  Ubuntu 24.04 · 961 MB RAM · 2 GB swap                      │
   │                                                             │
   │  episode-tracker   node dist/index.js       ~81 MB / 256 MB │
   │  gym-api           node dist/index.js       ~72 MB / 256 MB │
   │  triplehit-api     node dist/src/index.js   ~59 MB / 256 MB │
   │  postgresql@16     three databases          ~53 MB          │
   │  nginx             eight hostnames           ~9 MB          │
   │                                                             │
   │  available: ~510–540 MB        swap used: 0                 │
   └────────────────────────────────────────────────────────────┘
          ▲  compiled on the Mac, shipped as dist/ (gym, Triple Hit)
          │  dumps pulled back nightly by launchd
        Mac

Measured on 25 September, six hours after a reboot, with systemctl show -p MemoryCurrent. The second number on each app line is its hard ceiling.

Finding the one that did not belong

free -m tells you the server is getting full. It does not tell you who is filling it. For that, ask each service's cgroup — systemd already counts every byte a unit and its children hold:

for s in episode-tracker gym-api triplehit-api postgresql@16-main nginx; do
  printf "%-22s %4s MB\n" "$s" $(( $(systemctl show "$s" -p MemoryCurrent --value) / 1048576 ))
done

On 19 September that list looked roughly like this:

Three APIs of similar size — Fastify, Prisma, a few dozen routes each. One of them cost six times what the other two did.

Why tsx costs so much

tsx is a development convenience: it compiles TypeScript as the process runs. That means a TypeScript compiler (esbuild under the hood, plus its loader hooks) lives inside the production process for its whole life, and every source file it touched stays in memory in two forms. On top of that there was an npm process wrapping it, doing nothing but waiting.

And with no heap limit, V8 makes its own choice about when to collect garbage. On a machine with 1 GB it assumes it may grow a long way before bothering. Memory that could be freed simply is not, because nothing asked.

The gym tracker and Triple Hit were built later, from a plan that said "compile on deploy, cap the heap" from day one. EpisodeTracker predates that rule. It was never wrong enough to break, which is exactly why it lasted.

The fix: bundle it, run it with plain node

The API is a workspace in a monorepo, and it imports a sibling package, @episode-tracker/shared, that ships TypeScript source. Plain tsc output would still try to import that .ts file at runtime and fail. So the build is a bundle: esbuild pulls our own code into one file and leaves npm packages as ordinary imports.

// apps/api/build.mjs
import { build } from "esbuild";

await build({
  entryPoints: ["src/index.ts"],
  outfile: "dist/index.js",
  bundle: true,
  platform: "node",
  format: "esm",
  target: "node22",
  plugins: [
    {
      name: "external-npm-packages",
      setup(b) {
        // Any bare import (not relative, not absolute) except our own workspace.
        b.onResolve({ filter: /^[^./]/ }, (args) =>
          args.path.startsWith("@episode-tracker/") ? undefined : { external: true },
        );
      },
    },
  ],
});

Why keep npm packages external rather than bundle everything: Prisma's client loads a native query engine from node_modules by path, and several packages read their own files at runtime. Bundling those is a fight for no gain — they are already JavaScript.

dist/index.js sits at the same depth as src/index.ts on purpose. The API serves the React build with a path relative to import.meta.url (../../web/dist). Put the output one folder deeper and the site 404s while the health check, which never touches those files, stays green.

The package scripts after the change:

"dev":   "tsx watch --env-file-if-exists=../../.env src/index.ts",
"build": "node build.mjs",
"start": "node --max-old-space-size=128 dist/index.js",

tsx moved to devDependencies. It still runs the dev server, the tests and the seed script — on the Mac, where it belongs.

The unit file

# /etc/systemd/system/episode-tracker.service  (the parts that matter)
[Service]
WorkingDirectory=/home/omar/apps/episode-tracker/apps/api
EnvironmentFile=/home/omar/apps/episode-tracker/.env
# Set after EnvironmentFile so it wins over anything stale in that file.
Environment=NODE_ENV=production
# Compiled JS run by node directly: no tsx and no npm wrapper in memory. The heap cap
# makes V8 collect garbage early; MemoryMax is the hard ceiling for the whole process.
ExecStart=/usr/bin/node --max-old-space-size=128 dist/index.js
MemoryMax=256M
Restart=always
RestartSec=5

Two limits, doing two different jobs:

The heap cap is the one that changes day-to-day behaviour: it tells V8 "you are on a small machine, clean up early". MemoryMax is the seatbelt. It covers what the heap flag does not — native memory, buffers, the Prisma engine — and it guarantees that one misbehaving app cannot take the other two down with it. The gap between them (128 → 256) is room for everything outside the JavaScript heap.

Put ExecStart= straight on node, never on npm start. Through npm you get two processes instead of one, signals that may not reach the real app on restart, and a --max-old-space-size that has to survive being passed through a script. The flag belongs where systemd can see it.

The result

The deploy script got one extra line, because the service now runs a file that only exists if someone builds it:

npm run build -w @episode-tracker/web || { fail "web build failed"; exit 1; }
# The service runs apps/api/dist/index.js, so a missing API build means stale code.
npm run build -w @episode-tracker/api || { fail "api build failed"; exit 1; }

Without it the deploy would "succeed", restart the service, pass its health check — and keep running last week's code from the old dist/. A deploy that silently ships nothing is worse than one that fails.

Build on the laptop, not on the server

Running compiled code fixes the resting memory. The other pressure point is the build itself: tsc and Vite can each take several hundred megabytes for a minute. On a 1 GB machine that already holds three apps and a database, that minute is the moment something gets pushed into swap — or killed.

So the gym tracker and Triple Hit never compile on the server. Their deploy scripts run on the Mac:

   Mac                                          droplet
   ───                                          ───────
   tests  →  build api/dist + web/dist
                        │
                        ├── rsync api/dist, prisma, package*.json ──▶  npm ci --omit=dev
                        │                                              prisma migrate deploy
                        └── rsync web/dist ─────────────────────────▶  staging dir → /var/www
                                                                       systemctl restart
                                                                       health check

The server installs production dependencies only (npm ci --omit=dev) — no TypeScript, no Vite, no test runner ever lands in node_modules there.

The openrsync trap

The web files must end up owned by root, readable by Nginx. The obvious command:

rsync -az --chown=root:root web/dist/ oyware:/var/www/gym/

fails on a current Mac, because macOS no longer ships GNU rsync. It ships openrsync, which has no --chown at all. And even with it, the SSH user is not root.

The fix is two hops: ship to a folder the SSH user owns, then let the server's rsync — the real one, run with sudo — publish it:

# macOS ships openrsync, which cannot set remote ownership. Ship to a staging
# folder owned by omar, then let the server's rsync publish it as root:644.
rsync -az --delete web/dist/ "oyware:$APP_DIR/web-dist/"
ssh oyware "sudo rsync -a --chown=root:root --chmod=D755,F644 $APP_DIR/web-dist/ $WEB_DIR/ \
  && sudo find $WEB_DIR/assets -type f -mtime +30 -delete"

Note the publish step has no --delete. Vite gives every chunk a hashed name, and a phone that opened the app before this deploy still holds the old index.html, which asks for the old chunks the moment you tap into a lazy-loaded screen. Deleting them immediately breaks that tab. They go after 30 days instead.

The exception

EpisodeTracker still builds on the droplet — its deploy script pulls from git, runs npm install and both builds there. It works because the 2 GB swap file absorbs that one minute, and it is the next thing to move to the Mac-side pattern. I note it here rather than pretend all three are the same.

Measure after every deploy

A memory cap hides a leak instead of fixing it: the app hits the ceiling, restarts, and starts climbing again. From outside that looks healthy. So every deploy now ends with a reading, and another the next day:

systemctl show gym-api -p MemoryCurrent -p MemoryPeak -p NRestarts

The rule: a number that keeps climbing across days is a leak. Fix it; don't raise the cap. Raise the cap only with a measured reason, such as a known import job that genuinely needs the room.

Today's readings, for the record (25 September, after a reboot):

These are a little above the numbers from the 19th. MemoryCurrent counts the cgroup, which includes file cache the kernel charges to the service, not only the Node heap, so a service that has read its files settles a bit higher than one that just started. The thing to watch is the trend, not the first digit.

Machine-wide, free -m said:

               total        used        free      shared  buff/cache   available
Mem:             961         453         194          20         479         508
Swap:           2047           0        2047

Load 0.00 0.00 0.00, disk 28%, zero out-of-memory kills. Read available, not free: buff/cache is the kernel using spare RAM for disk cache, and it gives it back the moment a process asks.

Don't write leaks in the first place

Caps and measurements catch leaks; these rules stop most of them being written:

• Every timer has an owner that clears it. A setInterval in a module that is imported twice runs twice, forever.
• No module-level array or map without a size limit or an expiry. "Cache the last results" with no eviction is a leak with a nicer name.
• One database client for the whole process. A new PrismaClient() per request opens a new connection pool per request. Postgres shows the right shape — one API, a few idle connections, always the same ones.
• Listeners are removed when their thing goes away — sockets, event emitters, anything with .on(.

The backups that stopped arriving

The droplet dumps each database every night at 01:00, 01:05 and 01:10 UTC and keeps seven days. A second copy on the Mac is what makes that a backup rather than a file on the same disk as the thing it protects. The Mac pulls them with a small script:

rsync -az --ignore-existing \
  -e "ssh -o BatchMode=yes -o ConnectTimeout=15" \
  "oyware:~/backups/$APP-*.dump" "$HOME/backups/$APP/"

It started life as a cron job at 05:00. The script's log, read on 13 September:

2026-09-04 18:48  OK: 1 dump(s) held locally
2026-09-05 02:00  OK: 5 dump(s) held locally
2026-09-08 02:00  OK: 8 dump(s) held locally
2026-09-13 11:18  OK: 21 dump(s) held locally     ← run by hand

Two scheduled runs in nine days — and no gym dump had ever reached the Mac, even though the gym tracker had been live for two days.

Why cron skipped it

On macOS, cron does not catch up. If the machine is asleep at 05:00, that run simply does not happen: no error, no log line, nothing in the morning. A laptop is asleep at 05:00 most nights. The job worked every time it ran; it just almost never ran.

launchd, the macOS service manager, behaves differently: a StartCalendarInterval job whose time passed during sleep runs when the Mac wakes. So the job moved:

<!-- ~/Library/LaunchAgents/com.omaryousef.pull-oyware-backups.plist -->
<key>Label</key><string>com.omaryousef.pull-oyware-backups</string>
<key>ProgramArguments</key>
<array><string>/Users/omaryousef/bin/pull-oyware-backups.sh</string></array>
<key>StartCalendarInterval</key>
<dict><key>Hour</key><integer>5</integer><key>Minute</key><integer>0</integer></dict>
<key>RunAtLoad</key><true/>

RunAtLoad covers the other gap — a Mac that was fully shut down at 05:00 runs it at login. Running twice is harmless because --ignore-existing skips dumps already held.

launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.omaryousef.pull-oyware-backups.plist
crontab -e   # and delete the old line, or both run

It caught up the same day, and it has run every day since. The Mac now holds 31 EpisodeTracker dumps, 66 gym dumps and 15 Triple Hit dumps — more than one per day, because every deploy takes a dump before it migrates, and those are pulled too. Anything older than 14 days is pruned, so the Mac copy outlives the droplet's seven.

One failure in the log is not the same as a stopped job. On 18 September the pull logged FAIL: rsync from droplet failed (exit 255) — SSH could not connect in the moment. The next run picked everything up. What matters is that a failure is logged; the cron version failed by not existing, which no log can show.

The same lesson twice

Both halves of this guide are the same mistake: a thing that works whenever you look at it. EpisodeTracker answered every request; the backup script succeeded every time it ran. Neither was checked for what it cost or how often it actually happened. The fix in both cases was a number written down on a schedule — a memory reading after each deploy, an OK: line per night — so that silence becomes visible.

Checklist for the next app

• [ ] npm run build produces dist/; production never sees tsx, ts-node, nodemon or a dev server
• [ ] systemd ExecStart=/usr/bin/node --max-old-space-size=128 dist/… — node directly, not npm start
• [ ] MemoryMax=256M and Restart=always in the unit
• [ ] Environment=NODE_ENV=production after EnvironmentFile=
• [ ] Built on the Mac, shipped with rsync to a staging dir, published by the server's rsync
• [ ] npm ci --omit=dev on the server
• [ ] Deploy script fails if the build fails, and never restarts on a stale dist/
• [ ] Right after the deploy: systemctl show <svc> -p MemoryCurrent. A day later: again
• [ ] Nightly dump on the server, pulled to the Mac by launchd, not cron