Skip to content

Database backups & restore

The production Postgres (postgres service in docker-compose.yml, data in the pgdata named volume on SERVER) holds identity + billing (users, subscriptions, oauth accounts, tokens). It is not stored in /opt/kimtos, so make deploy (which wipes and re-extracts /opt/kimtos) never touches it. Backups exist for the things a deploy can't save you from: host loss, disk failure, corruption, or a bad migration / accidental delete.

The rule that actually matters

A backup kept only on the same server does not survive that server dying. The off-server copy is the disaster recovery.

So this is two moving parts:

SERVER (12:00 Cairo cron)            THIS machine (12:30 Cairo cron)
  pg_dump -Fc                          rsync pull
  → ~/kimtos/data/backup-<date>.dump   → ~/kimtos/data/backup-<date>.dump
  keep newest BACKUP_KEEP (7)          keep newest BACKUP_KEEP (14)
  (later: rclone → S3/B2)              ← the durable, off-server copy
  • Format: pg_dump -Fc (custom, compressed) → restore with pg_restore.
  • Rotation: keeps the newest N and prunes the rest. Default 7 on the server, 14 on your machine. Keeping only 1 is intentionally not the default — a single corrupt dump would then be your only copy. Override with BACKUP_KEEP.
  • Atomic: writes *.partial then mvs into place, and refuses to publish an empty dump — a broken backup never overwrites a good one.

Scripts: scripts/backup-db.sh (runs on the server, shipped to /opt/kimtos/scripts by make deploy) and scripts/backup-pull.sh (runs here).

Setup (one time)

bash
make deploy            # ships scripts/backup-db.sh to the server
make backup-install    # server: install the noon-Egypt dump + rotation cron (CRON_TZ=Africa/Cairo, DST-safe)
make backup            # server: run one now to verify it works
make backup-pull-install  # this machine (Egypt time): install the 12:30 pull cron
make backup-pull       # this machine: pull now to ~/kimtos/data

Tune retention / paths with env or on the CLI, e.g. make backup-install BACKUP_KEEP=14, make backup-pull BACKUP_DIR=/mnt/backups.

Restore

Copy a dump to the server (or use one already there), then restore into the running container. --clean --if-exists drops existing objects first, so this is an in-place overwrite of the current DB:

bash
scp ~/kimtos/data/backup-2026-07-11.dump root@SERVER:/tmp/
ssh root@SERVER 'cd /opt/kimtos && \
  docker compose exec -T postgres sh -c \
  "pg_restore -U \$POSTGRES_USER -d \$POSTGRES_DB --clean --if-exists" \
  < /tmp/backup-2026-07-11.dump'

Full DR onto a fresh server: make deploy (brings up an empty stack + runs migrations), then run the pg_restore above with your latest dump. Data back.

Test a restore periodically (e.g. into a throwaway docker run postgres or a scratch DB) — an untested backup is a hope, not a backup.

Later: off-site to object storage (S3 / B2 / R2)

Pulling to your machine is the interim off-server copy; object storage is the production target (off-region, 11-nines durable, always-on). It's a drop-in — no script change:

  1. Install rclone on the server and configure a remote (e.g. b2).
  2. On the server, create ~/kimtos/backup.env (never committed):
    sh
    OFFSITE_RCLONE=b2:kimtos-backups
    HEALTHCHECK_URL=https://hc-ping.com/<uuid>   # optional dead-man's-switch
    BACKUP_KEEP=7
    backup-db.sh sources it and uploads each dump after the local write.
  3. Encrypt — the dump contains PII + billing links. Use bucket-side SSE (or gpg/rclone crypt) so the off-site copy is encrypted at rest.

Monitoring

Set HEALTHCHECK_URL in ~/kimtos/backup.env to a healthchecks.io/cronitor ping URL. A backup that silently stops is the real danger; the dead-man's switch alerts you when the daily ping doesn't arrive.