The 3am Problem: Building Systems That Don't Need You Awake
Monitoring, alerting, and the automation mindset that changed how I work. Sleep is the metric.
3:17am. My phone buzzes. I unlock it, squinting at the brightness, and read: "ZFS pool degraded — disk 3 checksum errors increasing, scrub recommended." I look at it for four seconds, put the phone down, and go back to sleep.
That's the whole story. I ran a scrub after coffee. Pool was fine. The disk went on my weekly watchlist. The only reason that story is boring — and the only reason I got to stay asleep — is because the system was built to make it boring. A year earlier I wouldn't have known there was a problem until the pool entered a critical DEGRADED state — a very different notification to receive at 3am.
The Alert Hierarchy That Saved My Sleep
Most monitoring failures aren't monitoring failures — they're calibration failures. The system watches everything, alerts on everything, and because everything pages you, nothing registers. You start ignoring alerts. Then you miss the one that mattered.
The question that fixed my calibration: if I ignore this for 8 hours, what's the worst case? That single question sorts almost every alert into one of three buckets.
L1 — Page me now. Data loss risk: ZFS pool entering DEGRADED from multiple disk failure. Security breach indicators: unexpected outbound connections, failed auth from unusual IPs. Services with real-world exposure: my scheduled automation pipelines going dark, anything with an external dependency that compounds overnight. These get ntfy at high priority. They wake me up. There are maybe five of them.
L2 — Tell me when I'm awake. Degraded but stable: one disk with elevated checksum errors, a single container crash that auto-restarted. Error rate increases that aren't yet service-affecting. Disk health trending in the wrong direction. These get ntfy at normal priority. They don't wake me up. I check them with coffee.
L3 — Log it, don't alert. Everything else. Backup completions. Cron job outcomes. Health check successes. Routine scrub results. These go to logs. Never to a push notification. When I first applied this framework, I deleted roughly 60% of my existing alerts. Response quality went up. Alert fatigue went down. Sleep quality followed.
Uptime Kuma for service availability — checks every service on a schedule, alerts on downtime with enough context to evaluate severity without logging in. Netdata for system metrics — CPU, memory, ZFS pool health, network throughput in real time. Scrutiny for disk health — SMART data aggregated and trended so I see degradation before failure. ntfy for push alerts — mobile notifications that include the alert level, the affected service, and a one-line recommended action. The whole stack runs on self-hosted infrastructure. No cloud dependencies, no SaaS fees.
Most homelab monitoring failures aren't monitoring failures. They're calibration failures. An uncalibrated alerting system is worse than no alerts at all.
Self-Healing Scripts: The 80% Solution
The automation mindset shift that changed everything: stop asking "how do I fix this when it breaks?" and start asking "how do I make it fix itself when it breaks?" The answer to the second question is almost always a 20-line bash script that checks one condition and takes one action. Written once. Running on cron forever.
Here's what's actually running on my NAS:
- ZFS health check — every 6 hours. Reads checksum error counts from
zpool status. If errors exceed threshold, triggers a scrub immediately and sends an L2 notification. On scrub completion, logs the result and sends a follow-up — cleared errors get logged, persistent errors escalate to L1. I haven't had to manually initiate a scrub in months. - Container health check — every 5 minutes. Checks every Docker container for
unhealthyorexitedstate. If found, restarts with exponential backoff: 1 minute wait, then 5 minutes, then 15. After the third consecutive failure it escalates to L1 and stops retrying. This prevents transient crash-loops from waking me up while still catching genuine failures. - Ollama inference check — every 10 minutes. Sends a minimal test prompt to the inference endpoint. Three consecutive failures trigger a container restart, followed by an endpoint health verification before marking it recovered. Ollama is load-bearing infrastructure now — it powers half my automation stack — so silent failures are not acceptable.
- n8n workflow queue check — every 10 minutes. If the workflow queue depth stays above threshold for more than 10 minutes, it restarts the n8n worker and logs the trigger event. Workflow backlogs that nobody notices are how automation pipelines silently rot.
Every script follows the same structure: check the condition, take exactly one action if the condition is met, log what was done with timestamp and result, notify if the action failed or if this is the Nth recurrence in a window. The key constraint is "exactly one action." Scripts that try to be smart and take multiple recovery paths become unpredictable. One check, one action, escalate if it doesn't resolve. That's it.
The script I'm most glad exists is a small watchdog, running every 5 minutes. It checks whether the dashboard's node process is alive and serving requests on its health endpoint. If not, it runs a dependency check, restarts the process, and logs the recovery. Since I deployed it, this script has silently recovered the dashboard from crashes 11 times. I noticed zero of those crashes in real time. The service just stayed up. That's the only stat that matters.
The core management daemon watchdog is a different kind of script. It runs every minute and checks the NAS's core management process. The key difference: it can't be auto-recovered — restarting it causes worse problems than the original failure (I learned this the hard way). So the script doesn't try to fix it. Instead, it detects the failure and sends an L1 alert immediately. For a process you can't safely auto-recover, fast detection is the entire self-healing strategy. Two minutes from failure to alert is the goal. It consistently hits under 90 seconds.
They tell you how to install Prometheus and Grafana. They don't tell you that the hardest part isn't instrumentation — it's calibration. The calibration work is a week of paying attention to which alerts you actually act on versus which ones you dismiss without reading. That work is worth more than better dashboards. An uncalibrated alerting system sends too many alerts, you learn to ignore them, and you end up worse off than having no alerts at all. Do the calibration first. Install more dashboards second.
Sleep Is the Metric
I measure the quality of my homelab ops by one number: how many times I got out of bed for it this month. Right now that number is zero — and it's been zero for four months, including the ZFS checksum event that opened this post. Every non-zero is a data point about where the system broke down: either an L2 event got miscategorized as L1, or a self-healing script failed to fire, or it was a genuine L1 event that I should have caught and automated away during the day.
Build the automation first. Tune the calibration second. Automation without alerting is blind. Alerting without automation is just a pager that wakes you up to do things a bash script could do. Both together, calibrated correctly, is a system that runs itself and tells you exactly what you need to know. Sleep is the metric. Four months and counting.
Stay with us · dig next
How Do You Handle Alerts at Night?
What is your approach to handling system alerts during off-hours? Share your tips!
No account needed — pick a take, then keep reading. We rotate these prompts so each piece feels like a conversation, not a clone.
