Submitted 23 hours ago by user224@lemmy.sdf.org to selfhosted@lemmy.world
Running services, network usage, memory usage, bandwidth, disk I/O, successful logins, whether the thing is even alive, etc…
So far my only method has been “hope and pray”.
Uptime Kuma, only really care if it’s up or not. Got about 30 services monitored, sending notifications to Signal.
Prometheus + grafana. It’s overkill tbh, most of the services restart automatically and I mostly ignore it. (it’s an artifact from life where I cared about it)
I sync my podcasts hourly with Nextcloud and poddersync, so when my servers go down, my phone beeps at me.
Cockpit on each server (microos) and bezel for a lite centralized monitoring/alert stack. I have a bespoke gitops on each server running ansible hourly, and ntfy pings for failure on these. Also healthchecks.Io for heartbeat/backups, other critical infra checks
When logged in locally, I use btop to see an overview of what’s happening.
Other than that, I have relevant Prometheus exporters in every machine (node exporter in all machines, specific exporters by the workload), hooked up over Wireguard to my monitoring solution offsite.
The phone I actually carry around has a ntfy client talking to ntfy server on the aforementioned monitoring solution, so I get buzzes when something goes down.
Btw, does anybody happen to know where I could get a comprehensive
I vibe coded a monitor that shows me all the logs and other stuff. And im using netbird cloud (free version) to setup dns so all my services have url “servicexyz.home.internal”.
So i just connect to netbird vpn on my phone and go to that url.
You can pretty much use any of the other tools mentioned in this thread with this kind of setup, heck you can write a simple fastapi server that just runs “top” and return it (it would be like 20-30 lines of python) over a url that you can from anywhere as long as you are connected to your vpn
SNMP.
The old engine was nagios. The new engine will be telegraf -> mqtt -> brokerfest -> timescale -> Prometheus.
I guess. Not sure yet. The brokerfest is a series of broker pubsub between sites to both split out data from the stream for off-site CC, or pull it’s own subscriptions in. Data could go host <- telegraf -> broker -> remote broker -> remote timescale -> remote Prometheus
Limeade3425@lemmy.zip 22 hours ago
I started with uptime kuma since it was simple and light on resources. I’ve switched to checkmk community (formerly raw) and really enjoying it. It’s telling me so many things that are wrong that I missed. Like that one time I made a snapshot of a VM then forgot about it for 113 days…