I use some batch scripts in my proxmox installation. They are in cron.hourly and daily checking for virus and ram/CPU load of my LXC containers. An email is send on condition.
What are your tipps or solution without unnecessary load on disc io or CPU time. Lets keep it simple.
Prometheus, Loki and Grafana.
And so so many Prometheus metric exporters.
Observability is such an endless rabbit hole, it’s so easy for me to spend huge amounts of time accomplishing not that much lol. But very enjoyable and cool to see it all come together.
My pro tips: using Kubernetes actually makes this stuff a heck of a lot easier to set up thanks to the common patterns that k8s has - lots of turnkey helm charts out there that make it all so easy and are powerful. Another tip would be to use Prometheus service discovery if you can. Also, Loki/Promtail is actually quite easy to set up - but using LogQL queries can be very tricky. Just be warned, observability is a full time hobby in itself lol