System Monitoring
Prometheus + Grafana, scraping Node Exporter on all nine cluster members, with ten alert rules that page the team over Telegram.
What was asked
Deploy a monitoring solution that observes the health status of the hardware — OS parameters, services, and network functionality — using a tool like Prometheus, Grafana, or CheckMK.
Architecture
Runs standalone via Docker Compose — deliberately not a workload on the Kubernetes cluster it's monitoring, so it stays observable even if the cluster itself is unhealthy.
| Component | Runs where | Role |
|---|---|---|
| Node Exporter | directly on every Pi, as a systemd service | exposes hardware metrics (CPU, RAM, disk, network, temperature) on :9100 |
| Prometheus | Docker container | scrapes all 9 Node Exporter targets every 5 seconds, stores the time series |
| Grafana | Docker container | visualizes Prometheus data; datasource auto-provisioned on startup, no manual UI step |
Nine Node Exporter targets are scraped: the Pi 5 (10.0.0.1, "ServerNode") plus
all eight Pi 3 workers by static IP.
Metrics tracked
| Panel | Metric |
|---|---|
| CPU utilization | node_cpu_seconds_total |
| System load (1m) | node_load1 |
| RAM usage | node_memory_MemAvailable_bytes |
| Disk space (SD card) | node_filesystem_avail_bytes |
| Network received / transmitted | node_network_receive_bytes_total / _transmit_bytes_total |
| CPU temperature | node_thermal_zone_temp |
| System uptime | node_boot_time_seconds |
Alert rules → Telegram
Ten Grafana alert rules, all wired to the same Telegram chat as a contact point — this monitoring stack is the other consumer of the notification channel built for Task 9.
| # | Rule | Condition | Pending |
|---|---|---|---|
| 1 | CPU Temperature | above 80°C | 1m |
| 2 | Node Down (any) | up below 1 | 1m |
| 3 | ServerNode Down | up below 1 for 10.0.0.1:9100 | fires immediately |
| 4 | Low Disk Space (ServerNode SSD) | below 10% free | 5m |
| 5 | High CPU Usage | above 85% (5m avg) | 5m |
| 6 | Low Available RAM | below 100 MiB | 5m |
| 7 | High System Load | node_load1 above 4 | 5m |
| 8 | Network In Spike | receive rate above 10 MiB/s | 1m |
| 9 | Cluster Size Changed | fewer than 9 node exporters up | 1m |
| 10 | NFS Root Read-Only | NFS root filesystem flips read-only | fires immediately |
Rules 3 and 10 are worth noting: the ServerNode (Pi 5) is both the control plane and the NFS root for every diskless worker (see Architecture and the risk register), so its health — and specifically whether its NFS exports flip read-only — is treated as a first-class, zero-delay alert rather than a generic node-down case.