Skip to content

Monitoring & Dashboards

This page is the map of the observability stack: what collects what, which Grafana dashboard answers which question, where coverage is strong, and the ranked roadmap of known gaps. For the collection pipeline itself (Alloy installers, Loki/Prometheus configuration, LogQL/PromQL recipes) see Logging & Metrics.

Where everything runs

Grafana (:3300), Prometheus (:9090) and Loki (:3100) are Portainer git stacks under docker-pve1/monitoring/ on the Docker LXC (192.168.1.241, CT 101 on pve1). Dashboards are file-provisioned from docker-pve1/monitoring/grafana/provisioning/dashboards/ — a merge to main redeploys them via GitOps; never import by hand. All dashboards use templated datasource variables (${prometheus} / ${loki}), never hardcoded datasource UIDs. Grafana is reachable at https://grafana.mdhmedia.uk (behind tinyauth SSO) — a dashboard lives at /d/<uid>.

Stack overview

Five hosts run the Grafana Alloy agent and push telemetry to the collector on .241: metrics via Prometheus remote_write, logs into Loki. Prometheus additionally pull-scrapes a handful of targets: the CrowdSec agent on the Caddy LXC (192.168.1.245:6060), the blackbox exporter (HTTPS probes of the public *.mdhmedia.uk endpoints, including TLS cert expiry), and its own stack. Grafana sits on top of both datasources and owns alert evaluation and delivery.

flowchart LR
    subgraph Hosts[Alloy host agents]
        PVE1[pve1<br/>.199]
        PVE2[pve2<br/>.151]
        CADDY[caddy CT 112<br/>.245]
        D1[docker-pve1 CT 101<br/>.241]
        HOLLY[holly<br/>.200]
    end

    subgraph Collector[Docker LXC - 192.168.1.241]
        PROM[Prometheus<br/>:9090]
        LOKI[Loki<br/>:3100]
        BB[Blackbox<br/>exporter]
        GRAF[Grafana<br/>:3300]
    end

    CS[CrowdSec<br/>CT 112 :6060]
    WEB[Public endpoints<br/>*.mdhmedia.uk]
    TG[Telegram contact point<br/>tokens still placeholders]

    Hosts -->|metrics remote_write| PROM
    Hosts -->|logs push| LOKI
    PROM -->|scrape| CS
    PROM -->|scrape| BB
    BB -->|HTTPS probes| WEB
    PROM --> GRAF
    LOKI --> GRAF
    GRAF -.->|alerts| TG

Alloy hosts: pve1, pve2, caddy (CT 112), docker-pve1 (CT 101), holly (Unraid). Each ships node metrics (CPU/memory/disk/network) plus systemd-journal and docker-container logs; the Caddy host additionally parses JSON access logs with pre-indexed request_host / status / method labels.

Dashboard catalog

Nine provisioned dashboards, layered from single-pane-of-glass down to raw logs. Open them at https://grafana.mdhmedia.uk/d/<uid>.

Dashboard UID What it answers When to open it
Homelab Command Center mdh-command-center New top-level landing board — the single pane of glass combining fleet health, alert state and endpoint status with drill-down links to everything below. Start here. Bookmark it; every other dashboard is a drill-down from this one.
MDH Media - Homelab Overview mdh-homelab-overview "Is anything wrong right now?" — hosts reporting, endpoints down, firing alerts, min cert days, CrowdSec active bans, fleet CPU/mem/disk/network (recording rules host:*), recent Loki errors. First stop when something feels off and you don't yet know where.
MDH Media - Infrastructure mdh-infrastructure Everything about one host via the $host variable — uptime, CPU by mode, load, memory, filesystem %, disk I/O, network errors, plus that host's logs ($log_job). A specific host is slow, full, or misbehaving.
MDH Media - Security mdh-security CrowdSec decisions/alerts/scenario overflows, parser throughput, Caddy edge 4xx/5xx by hostname, top client IPs, SSH failed/successful logins, sudo usage. Suspicious traffic, an IP ban question, or an auth audit.
MDH Media - Ops Jobs mdh-ops-jobs "Did last night's jobs run?" — vzdump backups, pve-offsite-sync to Hetzner, caddy-deploy + DDNS on CT 112, container log-error health on CT 101. Morning check, or after touching backups / the Caddyfile deploy.
MDH Media - Caddy Access Logs mdh-caddy-logs Web traffic analytics: requests by status/service/method, error rates by service, 4xx/5xx log tail, Caddy service errors, full access-log browser. A *.mdhmedia.uk route errors or you're investigating who hit what.
MDH Media - Homelab Logs mdh-homelab-logs Cross-host log exploration: volume by host/job, errors and warnings over time, SSH/auth panels, full log browser. Log-first investigations when you don't know which host to blame.
MDH Media - Jarvis Ops mdh-jarvis-ops Jarvis (VM 121) agent liveness, monitoring self-health (Loki/Prometheus up), currently-firing alerts, neteng incident/action timeline, recent homelab errors. Jarvis misbehaves, or you want the monitoring-of-monitoring view.
Holly NAS (Unraid) mdh-holly-nas Array/cache capacity and per-disk fill, disk temperature/SMART health, Holly CPU/mem/network, per-container log volume and error rate. NAS capacity planning or disk-health checks (only useful while Holly is powered on).
Drive Health (SMART) mdh-drive-health New — per-drive SMART across every host with disks (pve1/pve2/holly): overall health, temperature, reallocated/pending sectors, CRC errors, SSD wear, power-on hours, one row per physical drive. "How are my drives doing?" — the at-a-glance disk-failure early-warning board.

What's on each dashboard

Homelab Overview — rows & panels

Rows: At a Glance, Fleet (recording rules: host:*), Endpoints & TLS (Blackbox), Recent Errors (Loki). Panels: Hosts Reporting, Endpoints Down, Firing Alerts, Min Cert Days, CrowdSec Active Bans, Probe Latency p95, Firing Alerts (detail), CPU / Memory / Root Disk / Network by Host, Endpoint Status, TLS Certificate Days Remaining, Probe Duration, Recent Errors, plus a markdown panel with drill-down links to /d/mdh-infrastructure, /d/mdh-caddy-logs, /d/mdh-security, /d/mdh-ops-jobs, /d/mdh-homelab-logs.

Infrastructure — rows & panels

Rows (all filtered by $host): Overview, CPU, Memory, Disk, Network, Logs. Panels: Uptime, CPU Usage, Memory Usage, Root Disk Usage, Load 1m / 5m / 15m, CPU Usage by Mode, Load Averages, Filesystem Used %, Disk I/O Bytes, Disk IOPS, Network Traffic, Network Errors / Drops, Errors / Warnings ($log_job), All Logs, Log Volume by Job.

Security — rows & panels

Rows: Security at a Glance, CrowdSec (CT 112), Caddy Edge Traffic (JSON access logs, since 2026-07-07), SSH & Auth Audit. Panels: CrowdSec Active Decisions, CrowdSec Alerts, Scenario Overflows, HTTP 4xx / 5xx (range), Failed SSH Logins, Scenario Overflows by Scenario, Parser Throughput, Whitelist Hits by Rule, LAPI Bouncer Requests, Requests by Hostname, Status Class Breakdown, 4xx/5xx by Hostname, Top Client IPs, Failed SSH Logins by Host, Failed / Successful Logins, Sudo Usage, plus a markdown panel (CrowdSec mode note + drill-down links). Note: the row title still says "simulation mode" but CrowdSec has been enforcing since 2026-07-08 — see gap 11.

Ops Jobs — rows & panels

Rows: Backups - vzdump (pve1), Offsite Sync - pve-offsite-sync.service (pve1 → Hetzner Storage Box), Caddy Deploy & DDNS (CT 112), Container Health - docker-pve1 (CT 101). Panels: Backup Runs / Successes / Errors (7d), vzdump Activity (pve1 journal), Offsite Sync Runs / Completed / Failures (7d), Offsite Sync Log, Deploy Timer Runs, Caddyfile Deploys, Deploy Failures, DDNS Cron Runs (7d), Recent Deploy Activity (CT 112 journal), Log Volume by Compose Project, Top 10 Containers by Error Lines, plus a markdown "Job schedules" panel with links to the Overview and Security dashboards.

Caddy Access Logs — rows & panels

Rows: Traffic Overview, Services & Hosts, Error Analysis, Caddy Service, Access Log Browser. Panels: Total Requests, 2xx / 3xx / 4xx / 5xx counts, Requests/sec, Requests by Status Code, Requests by Service/Host, Top Services, Request Methods, Status Code Distribution, Errors by Service, Error Rate Over Time, Error Logs (4xx & 5xx), Caddy Service Errors/Warnings, All Access Logs.

Homelab Logs — rows & panels

Rows: Overview, Errors & Warnings, Security & Authentication, Systemd Services, Log Browser. Panels: Total Log Lines, Active Hosts, Error / Warning Count, Auth Failures, Successful Logins, Log Volume by Host / Job, Errors & Warnings Over Time, Errors by Host / Job, SSH Login Attempts, Failed Logins by Host, Logs by Systemd Unit, All Logs, Error Logs Only. Note: the Security & Authentication and Systemd Services rows are currently dead panels (they query a job="auth" that doesn't exist in Loki) — see gap 11.

Jarvis Ops — rows & panels

Single row: Jarvis / neteng Ops. Panels: Jarvis agent, Firing alerts, Loki, Prometheus, Currently firing, Jarvis liveness, neteng incident / action timeline, Recent homelab errors. Note: the two headline Jarvis panels show No Data because the blackbox-jarvis scrape job in prometheus.yml was never loaded by the running Prometheus — see gap 5 (a Prometheus reload fixes it).

Holly NAS — rows & panels

Rows: Array & Cache Capacity, Per-Disk Fill, Disk Temperatures & SMART, System (CPU / Memory / Network), Docker Containers (Loki). Panels: Parity / Redundancy Status, Array Used % / Free (/mnt/user), Cache SSD Used % (/mnt/cache), Disks SMART-Healthy, Per-Disk Used %, Current Fill by Disk, Disk Temperatures, SMART Health by Disk, CPU / Memory / Network, Container Log Volume, Container Error Rate, Recent Container Errors. Note: the SMART panels are now repointed at real smartctl_device_smart_status / smartctl_device_temperature metrics (gap 9 closed 2026-07-10) and light up once Holly's smartctl_exporter is deployed; the parity banner is still a hardcoded vector(1) (no Unraid parity metric — separate follow-up). For cross-host drive health use the dedicated Drive Health (SMART) dashboard.

Coverage matrix

Honest assessment of what the stack sees today (2026-07-10):

Area Status Detail
Host metrics (node/Alloy) Partial 5 hosts ship Alloy remote_write node metrics (pve1, pve2, caddy/CT112, docker-pve1/CT101, holly) with recording rules and two dashboards — but CT113/114/116/120, VM121 (jarvis), CT122 and the HA appliance ship nothing, no host produces up{} (so death = silent staleness), and node_hwmon_temp_celsius + node_systemd_unit_state are absent everywhere.
NAS / storage (Holly) Partial mdh-holly-nas covers array/cache/per-disk fill from node_filesystem, and SMART temp/health now flow via smartctl_exporter into the new mdh-drive-health dashboard (deployed on pve1; pending on Holly/pve2 until they're back up). Remaining: the parity banner is still hardcoded vector(1), no disk-I/O panels, and Holly is excluded from vzdump.
Docker containers Partial Container logs flow well (job=docker-logs per container/compose_project into Loki), but container metrics exist only from Holly (currently off) — docker-pve1/CT101, the host running every Portainer stack, exports zero cAdvisor/docker-state metrics, and no container-down/restart-loop alert exists anywhere.
Reverse proxy / web (Caddy) Good Best-covered area: JSON access logs with pre-indexed labels (status, request_host, method) feed the Caddy Access Logs + Security dashboards, the caddy-service job catches ACME/reload errors, and all blackbox probes transit it — the only real holes are no latency (duration) panels and no client-IP/user-agent or tinyauth-denial breakdown.
Security / CrowdSec / auth Partial CrowdSec's 34 cs_* metric families are scraped from .245:6060 with a solid Security dashboard, and SSH audit panels exist — but CrowdSec has zero alert rules, tinyauth (the SSO gate for ~12 routes) has no probe/panel, WireGuard is invisible, the SSH panels double-count auth.log+journal, and the dashboard's simulation-mode text is stale (enforcing since 2026-07-08).
Uptime probes / certs Partial 5 blackbox targets all healthy with probe_ssl_earliest_cert_expiry + a 14-day expiry alert — but grafana, mdhdocs, kicad, wgadmin, auth.mdhmedia.uk, grampsweb and all Holly apps are unprobed, the icmp module is defined yet unused (no host-liveness probes), and the blackbox-jarvis job in prometheus.yml is not loaded by the running Prometheus (Jarvis dashboard headline panels show No Data).
Logs (Loki pipeline) Good 4 jobs (caddy, caddy-service, docker-logs, systemd-journal) from all Alloy hosts, ~216k lines/24h, browsable via 3 log-centric dashboards — weaknesses are consumption-side: naive keyword regexes ignoring the detected_level label, 6 dead panels in mdh-homelab-logs, and zero Loki-based alert rules.
Backups (vzdump / offsite / HA) Partial Ops Jobs dashboard counts vzdump + pve-offsite-sync runs via brittle journal string-matches only — no success/age metric, no "time since last good backup" panel, no alert on failure (a dead timer renders all-green zeros), Home Assistant's Google Drive backups are wholly unmonitored, and CT113/CT120/Holly aren't in any vzdump job.
Proxmox guest level Missing No pve-exporter on either node: guest up/down status, per-guest CPU/mem, and storage-pool usage on pve1/pve2 are invisible to Prometheus — a stopped CT is only noticed if it individually ships telemetry or is probed (most don't and aren't).
Home automation (HA) Partial Only a blackbox https_2xx probe of ha.mdhmedia.uk; HA's native Prometheus integration is not wired up, so no entity/automation/zigbee metrics, no host metrics from the appliance, and its Google Drive backup add-on outcome is invisible.
Network / DNS Missing AdGuard (CT120) — the split-horizon DNS every LAN blackbox probe depends on — has zero telemetry (no Alloy, no dns-module probe, no exporter); WireGuard (CT116) likewise; no ICMP probes; an AdGuard outage would fail all domain probes at once with nothing pointing at DNS.
Alerting / notification delivery Missing 6 Prometheus rules + the Grafana mirror rule (7 total) evaluate correctly, but the ONLY contact point (telegram-jarvis) has placeholder bot token/chat id and SMTP is suppressed — every alert silently fails to deliver; no Alertmanager, no deadman/heartbeat check, so the whole pipeline is observe-only.
Media stack (*arr / Plex / downloads) Missing Zero per-app telemetry for Plex, Radarr/Sonarr/Prowlarr/Readarr, qBittorrent, NZBGet, Audiobookshelf, Ombi — only Holly host metrics + docker log lines when up, and none of the *.mdhmedia.uk media routes are probed; the MCP servers give ad-hoc queryability but nothing continuous (apps currently off with the heatwave, which is expected).

Gap roadmap

Ranked by leverage. Effort: S = small (config-only), M = medium (new component).

# Gap Effort Value
1 Notification delivery is broken (placeholder Telegram creds, no deadman) S high
2 Host-down blind spot (no up{} from remote_write hosts, no staleness alert) S high
3 Backup success is invisible (no metrics, age tracking, or alerts) M high
4 No container metrics on docker-pve1 M high
5 Narrow probe coverage + blackbox-jarvis never loaded S high
6 No Proxmox guest-level metrics (pve-exporter) M high
7 AdGuard DNS: zero-telemetry single point of failure S high
8 Log-based alert rules missing (dashboards detect, nothing pages) M high
9 No SMART / temperature telemetry anywhere M med
10 HA, MariaDB and WireGuard essentially unobserved M med
11 Dashboard hygiene: dead/duplicated panels, inconsistent $host M med
12 Monitoring-of-monitoring: Grafana unscraped, single delivery channel S med
1. Notification delivery is broken: placeholder Telegram credentials + no deadman check — effort S, value high

Why. Every alert in the system — HostDown, DiskSpaceCritical, cert expiry, the Grafana mirror — routes to the single telegram-jarvis contact point whose TELEGRAM_BOT_TOKEN/TELEGRAM_CHAT_ID are placeholders. All detection built so far notifies nobody; every other gap fix is worthless until this lands, and nothing would ever tell you delivery broke again later.

How. Create the bot via @BotFather, set real TELEGRAM_BOT_TOKEN/ TELEGRAM_CHAT_ID in the grafana stack's Portainer env vars and redeploy (contactpoints.yaml already env-substitutes them; PR #50 fixed the quoting). Then add a deadman: an always-firing Prometheus rule (expr: vector(1), alertname Watchdog) mirrored by the existing neteng-prometheus-mirror rule so a Telegram ping arrives every repeat_interval — silence means the pipe broke. Optionally split critical vs warning into two routes in policies.yaml with a shorter repeat_interval for critical.

2. Host-down blind spot: Alloy remote_write hosts produce no up{} and no staleness alert exists — effort S, value high

Why. HostDown (up==0) only covers the 5 pull-scraped targets on .241. The 5 real hosts (pve1, pve2, caddy, docker-pve1, holly) arrive via remote_write, so if pve1 itself dies — taking the hypervisor, backups, and most guests with it — metrics just stop and NOTHING fires. Dashboards show stale green lastNotNull stats; a dead host is indistinguishable from a healthy one today.

How. Add to docker-pve1/monitoring/prometheus/alerts.yml a staleness rule: time() - max by (host) (timestamp(node_cpu_seconds_total)) > 600 (label severity=critical, one series per host, auto-covers new hosts), plus per-critical-host absent() rules (absent(node_load1{host="pve1"})). Complement with blackbox ICMP: the icmp module already exists in blackbox.yml unused — add a blackbox-icmp scrape job in prometheus.yml probing .199/.151/.245/.241/.200/.244/.252. Add a per-host "last seen" table panel to mdh-homelab-overview using the same timestamp() expr so the dashboard names WHICH host vanished.

3. Backup success is invisible: no metrics, no age tracking, no alerts for vzdump / pve-offsite-sync / HA Google Drive — effort M, value high

Why. The repo treats backups as critical (encrypted offsite, non-negotiable), yet the only visibility is brittle exact-string journal matching on the Ops Jobs dashboard — a stopped timer or dead journal shipper renders as all-green zeros, a hung rclone that never logs "failed" is invisible, and HA's Google Drive backups have zero monitoring. "Did last night's backup run?" currently requires mentally diffing two stat panels.

How. Use the node_exporter textfile pattern the Alloy unix integration already supports: have pve-offsite-sync.sh (and a vzdump-hookscript on pve1) write backup_last_success_timestamp_seconds{job_name=...} to the textfile dir on success. Alert: time() - backup_last_success_timestamp_seconds > 90000 (26h). Add "Time since last success" stat panels to mdh-ops-jobs. For HA, expose the Google Drive Backup add-on's sensor via the HA Prometheus integration (see gap 10) or probe its /backups endpoint. Interim (no new plumbing): Grafana-managed Loki alert rules on the existing "Backup job finished successfully" / "offsite sync complete" journal lines using absent-over-26h logic.

4. No container metrics on docker-pve1, the host running every Portainer stack — effort M, value high

Why. cAdvisor metrics exist only from Holly (powered off); CT101 — running Grafana, Loki, Prometheus, tinyauth, docs, Mealie, Portainer itself — exports zero per-container metrics. A crash-looping or OOM-killed stack is completely silent: no restart-count, no container-down alert, and the docker-logs Loki stream only helps if someone reads it.

How. Add a cadvisor service (gcr.io/cadvisor/cadvisor, version-pinned for Dependabot) to docker-pve1/monitoring/prometheus/docker-compose.yml on monitoring-net with the standard /var/run/docker.sock + /sys + /var/lib/docker mounts, add a cadvisor scrape job to prometheus.yml, then alert on time() - container_last_seen > 300 for expected containers and increase(container_restart_count[15m]) > 3. Add a Containers row (CPU/mem/restarts per compose_project) to mdh-homelab-overview — the Loki-based container panels already exist, metrics complete the picture.

5. Probe coverage is narrow and blackbox-jarvis never loaded (Jarvis dashboard headline panels are No Data) — effort S, value high

Why. Only 5 of ~15 routable services are probed. tinyauth — the single point of failure for every SSO route (grafana, portainer, pve1/2, jarvis, wgadmin, holly, homepage root) — has no probe at all; grafana, mdhdocs, kicad, wgadmin and auth.mdhmedia.uk are unmonitored. Meanwhile the blackbox-jarvis job defined at prometheus.yml line 88 is not loaded by the running Prometheus, so the Jarvis Ops dashboard's two headline panels have shown No Data since creation — a verified live defect.

How. Reload Prometheus to pick up blackbox-jarvis (curl -X POST http://192.168.1.241:9090/-/reload or redeploy the prometheus stack in Portainer) — the config is already merged. Then extend the target lists in prometheus.yml: add https://grafana.mdhmedia.uk, https://mdhdocs.mdhmedia.uk, https://wgadmin.mdhmedia.uk, https://kicad.mdhmedia.uk to a module accepting 2xx/3xx/401 (SSO routes 302), tcp_connect for tinyauth 192.168.1.241:9092 and http://192.168.1.241:8081 (grampsweb). Add or vector(0) / absent() guards to the Jarvis liveness panels so a vanished scrape job renders red, not blank.

6. No Proxmox guest-level metrics (pve-exporter) — effort M, value high

Why. Neither pve1 nor pve2 exports PVE API data: guest up/down, per-guest CPU/mem/disk, and storage-pool usage are invisible. This is the cheapest way to cover the 6+ guests that have no Alloy agent (CT113/114/116/120, VM121, CT122) — one exporter watches all of them, and a stopped MariaDB CT or full local-lvm pool would finally be detectable.

How. Add prometheus-pve-exporter (prompve/prometheus-pve-exporter, pinned) to the monitoring compose on monitoring-net, with a read-only PVE API token (PVEAuditor role) created on pve1; add a prometheus.yml job with the standard /pve multi-target relabel config for both nodes. Alert on pve_up{id=~"lxc/(101|112|113|114|116|120)|qemu/121"} == 0 and pve storage usage

85%. Add a "Proxmox Guests" row (status table + per-guest CPU/mem) to mdh-infrastructure or a new dashboard. Note pve2 guests will show down during heatwave shutdowns — group them so they can be muted.

7. AdGuard DNS: a zero-telemetry single point of failure that all LAN probes depend on — effort S, value high

Why. CT120 carries the split-horizon rewrites (*.mdhmedia.uk → .245) that every LAN blackbox probe relies on. If AdGuard dies, all domain probes fail simultaneously while the internet path stays fine — a confusing all-red storm with nothing pointing at DNS, and LAN clients lose name resolution entirely. It currently has no Alloy, no probe, no exporter.

How. Three layers, cheapest first: (1) add a dns module to blackbox.yml (query_name grafana.mdhmedia.uk, expect A 192.168.1.245) probing 192.168.1.244:53 — detects both process-down and broken rewrites; (2) run install-alloy.sh on CT120 for host metrics + journal; (3) optionally add adguard-exporter (ebrianne/adguard-exporter) to the monitoring stack for query/block-rate stats. Alert on the dns probe_success with severity=critical and mention DNS in the annotation so probe-storm triage starts at the right place.

8. Log-based alert rules: dashboards detect conditions that never page — effort M, value high

Why. Loki is a provisioned datasource but zero Loki alert rules exist. The dashboards already color-code backup failures, Caddy 5xx spikes, caddy-deploy rollback failures, failed-SSH bursts and CrowdSec overflows — but every one depends on a human happening to look. This is the systemic pattern: detection logic lives in panel thresholds instead of alert rules.

How. Add Grafana-managed alert rules (provisioning/alerting/rules.yaml, datasource uid loki) reusing the exact panel queries: (1) offsite-sync/vzdump failure strings from mdh-ops-jobs, (2) sum(count_over_time({job="caddy", status=~"5.."}[15m])) > 50, (3) the caddy-deploy "failed validation|rollback" regex, (4) failed-SSH burst > threshold/10m. Also add one Prometheus rule on CrowdSec: increase(cs_bucket_overflowed_total[1h]) > N now that it is enforcing. They all route to the (fixed) Telegram contact point via the existing root policy — zero new plumbing.

9. Drive SMART telemetry — DONE (2026-07-10): smartctl_exporter + Drive Health dashboard

Was. smartctl_device_* metrics were absent everywhere, so the Holly SMART panels never rendered and pve1/pve2's disks had no failure early-warning.

Now. host-agents/install-smartctl-exporter.sh deploys the prometheus-community smartctl_exporter (v0.14.0) on :9633 — native systemd on PVE/Debian hosts, a privileged container on Unraid — firewalled to the Prometheus host like the CrowdSec endpoint. A smartctl scrape job in prometheus.yml (per-target host label) pulls it; the new Drive Health (SMART) dashboard (mdh-drive-health) shows one row per physical drive with health, temperature, reallocated/pending sectors, CRC, SSD wear and power-on hours, and the Holly dashboard's SMART panels are repointed at the real metrics (smartctl_device_smart_status, not the guessed _smart_healthy). Alerts added: SmartHealthFailed (critical), SmartReallocatedSectors / SmartPendingSectors / DriveTemperatureHigh > 60C (warning).

Live now: pve1 (1 Kingston SSD). Pending: run the installer on pve2 and Holly once they're back from the heatwave shutdown — their smartctl targets show down until then (expected; a powered-off host is not a drive fault). node_hwmon_temp_celsius (host CPU/board thermals) is still absent — a separate, smaller follow-up (enable the hwmon collector in install-alloy.sh).

10. Home Assistant, MariaDB and WireGuard have essentially zero observability — effort M, value med

Why. HA (one HTTPS probe, nothing else) runs home automation; MariaDB (CT114) serves databases with no telemetry, no exporter, not even a TCP probe; WireGuard (CT116) is the remote-access door with an admin UI that is SSO-gated but unmonitored. All three fail silently today; MariaDB and WireGuard are also invisible to any dashboard.

How. HA: enable the native Prometheus integration (configuration.yaml prometheus: block + long-lived token), add a scrape job to prometheus.yml — also surfaces the Google Drive backup sensor for gap 3. MariaDB: add mysqld_exporter (prom/mysqld-exporter, pinned) to the monitoring stack pointed at 192.168.1.251:3306 with a monitoring-only grant, or minimally a tcp_connect blackbox probe of :3306. WireGuard: run install-alloy.sh on CT116 and add a textfile metric from wg show latest-handshakes (or prometheus_wireguard_exporter); probe wgadmin per gap 5. pve-exporter (gap 6) provides the guest-level backstop for all three CTs meanwhile.

11. Dashboard hygiene: dead panels, duplicated panels, inconsistent $host filtering — effort M, value med

Why. Six permanently-empty panels in mdh-homelab-logs (job="auth" does not exist; no unit label), the Jarvis headline panels (gap 5), and the Holly SMART panels (gap 9) all erode trust — a dashboard where some blanks are "known dead" teaches you to ignore blanks. Meanwhile status-code data is rendered ~8 ways across 3 dashboards, and half-applied $host variables make filtered views silently disagree with their own headline stats.

How. One PR touching the provisioned JSON: (1) mdh-homelab-logs — point the 5 auth panels at {job="systemd-journal"} |~ "Failed password|Accepted" and delete "Logs by Systemd Unit"; (2) apply $host to the overview/caddy-logs headline stats or remove the variable from the exprs' scope note; (3) collapse the 3 duplicate blackbox tiles on mdh-homelab-overview and one of each duplicate pair on mdh-caddy-logs (pie vs timeseries, Top Services vs Requests-by-Host); (4) replace keyword-regex error queries with detected_level=~"error|warn" where streams carry it; (5) fix "(7d)" hardcoded stat titles to use $__range semantics; (6) normalise schemaVersion 38 dashboards and ${datasource}→${loki} naming to repo convention. Add the missing Caddy latency panel (| json | unwrap duration, p95 by request_host) while in there.

Dashboard overlap & dead-panel inventory (verified)
  • mdh-homelab-overview duplicates itself: 3 of 6 at-a-glance tiles (Endpoints Down, Min Cert Days, Probe Latency p95) restate the Endpoints & TLS row's panels — 6 of 16 panels are blackbox-derived on a whole-homelab dashboard.
  • mdh-security's "Caddy Edge Traffic" row substantially restates mdh-caddy-logs: Requests by Hostname, Status Class Breakdown and 4xx/5xx by Hostname mirror the dedicated Caddy dashboard; HTTP status-class data is rendered ~8 different ways across the two dashboards plus the overview.
  • mdh-caddy-logs duplicates internally: "Requests by Status Code" vs "Status Code Distribution" pie, "Requests by Service/Host" vs "Top Services" bargauge, and the 4xx/5xx stat tiles vs "Error Rate Over Time" all restate the same series.
  • Near-identical keyword-regex error-log tails appear on four dashboards (overview "Recent Errors", infrastructure "Errors/Warnings", homelab-logs "Error Logs Only", jarvis-ops "Recent homelab errors") — the jarvis-ops one scope-creeps to every host, diluting a Jarvis-focused board; the firing-alerts stat+table pair is also duplicated on overview and jarvis-ops.
  • Dead panels, mdh-homelab-logs: 5 panels query a job="auth" that does not exist in Loki, and "Logs by Systemd Unit" filters on a unit label journal streams never ship (verified 0 series) — the entire Security & Authentication and Systemd Services rows are permanently blank.
  • Dead panels, mdh-jarvis-ops: "Jarvis agent" stat and "Jarvis liveness" timeseries query probe_success{job="blackbox-jarvis"}, which returns zero series on the live Prometheus (job defined in prometheus.yml but never loaded) — the dashboard's two most important panels show No Data.
  • Dead panels, mdh-holly-nas: both SMART panels (temperature timeseries, health table) have never had data (smartctl metrics absent from all hosts even while Holly was up), and the Parity banner is a hardcoded vector(1) rather than metric-driven.
  • mdh-infrastructure triple-reports load average ("Load 1m" stat + "Load 5m/15m" stat + "Load Averages" timeseries, none with core-count thresholds) and overlaps mdh-holly-nas's System row (CPU/mem/network for host=holly) — the latter overlap is mild and arguably fine.
  • Inconsistent $host/$job variable application makes several dashboards disagree with themselves: overview stats/alerts-table/blackbox row, caddy-logs headline stats + Top Services, and all six homelab-logs overview stats ignore the filter their sibling panels honour — selecting a host yields half-filtered views.
12. Monitoring-of-monitoring: Grafana unscraped, single-channel delivery, no scrape-health panel — effort S, value med

Why. Grafana exposes no metrics to Prometheus (no grafana_* series) and is also the sole alert-delivery engine — if Grafana dies, alerting dies with it and nothing notices (the up{job="prometheus"} self-scrape stat on jarvis-ops is similarly near-dead). No dashboard shows "up by job" scrape health, so a dead crowdsec or blackbox exporter surfaces only as frozen tiles.

How. Add a grafana scrape job (grafana:3000/metrics on monitoring-net) to prometheus.yml; the existing HostDown (up==0) rule then covers it — though delivery still transits Grafana, so pair it with the gap-1 deadman (missing Watchdog pings expose a dead Grafana externally) or probe https://grafana.mdhmedia.uk per gap 5. Add a small "Scrape Targets" stat/table panel (up by job/instance) to mdh-homelab-overview's At a Glance row.

Quick wins

Cheapest changes first — most are config-only edits in this repo:

  1. Set real TELEGRAM_BOT_TOKEN/TELEGRAM_CHAT_ID in the grafana stack's Portainer env and redeploy — the contact point, mirror rule and root policy are already wired; this single change turns all 7 existing alert rules from observe-only into functional (~15 min incl. @BotFather).
  2. Reload/redeploy Prometheus so the already-merged blackbox-jarvis job loads — instantly fixes the two No-Data headline panels on mdh-jarvis-ops (curl -X POST http://192.168.1.241:9090/-/reload).
  3. Add a staleness alert to alerts.yml using existing data: time() - max by (host) (timestamp(node_cpu_seconds_total)) > 600 — closes the biggest alert blind spot (remote_write host death) with a one-rule config change, no new exporters.
  4. Add a per-host "Last Seen" table panel to mdh-homelab-overview with the same timestamp() expr, so "Hosts Reporting: 3" becomes "holly last seen 26h ago" — names which host vanished.
  5. Fix the 6 dead panels in mdh-homelab-logs: repoint the job="auth" queries (Auth Failures, Successful Logins, SSH Login Attempts, Failed Logins by Host) at {job="systemd-journal"} with the same line filters, and delete the unit-label panel — SSH lines already arrive via systemd-journal.
  6. Extend blackbox target lists in prometheus.yml with existing modules: tcp_connect for tinyauth 192.168.1.241:9092 (the SSO SPOF), plus https targets for grafana/mdhdocs/wgadmin — config-only, exporter already running.
  7. Add an "up by job" scrape-health stat panel to the overview's At a Glance row (sum by (job) (up)) — a dead crowdsec/blackbox/loki scrape currently only shows as frozen tiles.
  8. Fix the CrowdSec Alerts stat on mdh-security from sum(cs_alerts) (counter-since-restart) to sum(increase(cs_alerts[$__range])), and update the dashboard's stale "simulation mode" text — CrowdSec has been enforcing since 2026-07-08.
  9. Add "Time Since Last Successful Backup" and "Time Since Last Offsite Sync" stat panels to mdh-ops-jobs using the existing journal lines (time() minus last log timestamp via a Loki instant query) — answers the #1 backup question at a glance with data already in Loki.
  10. Add p95 latency-by-request_host panels to mdh-caddy-logs via {job="caddy",type="access"} | json | unwrap duration | quantile_over_time(0.95, ...) — the duration field is already in every JSON access log line.

Known quirks

Read before panicking at a blank panel

  • pve2, Holly and CT 122 show No Data whenever they're powered off — heatwave shutdowns are normal ops here, not an outage. Their panels (and the whole mdh-holly-nas dashboard) go dark until the hosts come back; last data before the current shutdown is ~2026-07-09.
  • No alert will reach you yet: the telegram-jarvis contact point still has placeholder bot token/chat id (gap 1). Alert rules evaluate and show as Firing in Grafana, but delivery silently fails until real credentials are set in the grafana stack's Portainer env vars.
  • SMTP is suppressed by choice — email is intentionally not a notification channel; don't "fix" it by wiring SMTP back up. Telegram is the designated channel once its tokens are real.

References