Keeping It Running

Since my homelab was expanding in services and containers, my home directory and folder structures started to become more clustered and unorganised, a lot of my containers were on my home directory without a proper nested structure, this made navigating through my configuration files harder, to ensure a smooth transition, I got my Codex agent to do a read audit check before migrating. Twelve service directories that had been sitting at the top level went into ~/homelab/, split by what they are: ...

Two halves apt patches Debian. My homelab is not Debian — it’s about a dozen containers sitting on top of Debian. AdGuard, Authentik, Vaultwarden, Caddy, n8n, Greenbone, Homepage. apt will never once mention that Vaultwarden cut a release. So “keep the lab patched” is really two jobs: Debian packages — unattended-upgrades, already a solved problem, just needs scheduling and a report. Everything I actually run — nobody’s solved that for me. Containers only update when I decide to pull. This post is both halves. The first one taught me something uncomfortable about monitoring, so I’ll start there. ...

The problem I set out to solve My homelab takes a backup every night at 20:30. A cron job runs a script, the script dumps my Authentik database, snapshots the Vaultwarden vault, tars up the configs, encrypts a copy with age and drops it into OneDrive. It writes everything it does to /var/log/lab-backup.log. And nobody reads /var/log/lab-backup.log. That was the thing bothering me. If the backup silently stopped, I wouldn’t find out when it stopped — I’d find out the day I needed a restore, which is the worst possible day to learn anything. So the plan was simple: get n8n to check the backup actually happened, and buzz my phone if it didn’t. ...

The identity provider quietly became load-bearing and had no backup. Building one taught me more about database internals than building the identity provider did. Stack: Bash · Postgres · SQLite · age · cron · OneDrive 1. Purpose The identity provider started as a nice-to-have. Single sign-on for a dashboard — pleasant, not critical. Then it took over the firewall’s login. That’s the moment the risk profile changed, and it changed without any decision being made about it. Nobody sat down and said “let’s make the firewall depend on this container.” It happened one integration at a time, and the dependency graph grew quietly underneath. ...