Skip to content

Monitoring & Dashboards

This page is the map of the observability stack: what collects what, which Grafana dashboard answers which question, where coverage is strong, and the ranked roadmap of known gaps. For the collection pipeline itself (Alloy installers, Loki/Prometheus configuration, LogQL/PromQL recipes) see Logging & Metrics.

Where everything runs

Grafana (:3300), Prometheus (:9090) and Loki (:3100) are Arcane git-sync stacks under stacks/monitoring/ on the Docker LXC (192.168.1.241, CT 101 on pve1). Dashboards are file-provisioned from stacks/monitoring/grafana/provisioning/dashboards/ — a merge to main redeploys them via GitOps; never import by hand. All dashboards use templated datasource variables (${prometheus} / ${loki}), never hardcoded datasource UIDs. Grafana is reachable at https://grafana.mdhmedia.uk (behind tinyauth SSO) — a dashboard lives at /d/<uid>.

Stack overview

Six hosts have the Grafana Alloy agent installed and push telemetry to the collector on .241: metrics via Prometheus remote_write, logs into Loki. Five report around the clock (pve1, pve2, caddy, docker-pve1, holly); pve3 (onboarded 2026-07-25) became a Wake-on-LAN backup node on 2026-07-30 — normally powered off, it only reports during its evening backup window. Prometheus additionally pull-scrapes a handful of targets: the CrowdSec agent on the Caddy LXC (192.168.1.245:6060), the blackbox exporter (HTTPS/TCP probes of public endpoints, Jarvis and PBS, including TLS cert expiry), smartctl_exporter on each bare-metal host, and its own stack. Grafana sits on top of both datasources and owns alert evaluation and delivery.

flowchart LR
    subgraph Hosts[Alloy host agents]
        PVE1[pve1<br/>.199]
        PVE2[pve2<br/>.151]
        PVE3[pve3<br/>.28 - WoL, evenings only]
        CADDY[caddy CT 112<br/>.245]
        D1[docker-pve1 CT 101<br/>.241]
        HOLLY[holly .200]
    end

    subgraph Collector[Docker LXC - 192.168.1.241]
        PROM[Prometheus<br/>:9090]
        LOKI[Loki<br/>:3100]
        BB[Blackbox<br/>exporter]
        GRAF[Grafana<br/>:3300]
    end

    CS[CrowdSec<br/>CT 112 :6060]
    WEB[Public endpoints<br/>*.mdhmedia.uk]
    TG["Telegram contact point<br/>@MDHJarvis_bot - live"]

    Hosts -->|metrics remote_write| PROM
    Hosts -->|logs push| LOKI
    PROM -->|scrape| CS
    PROM -->|scrape| BB
    BB -->|HTTPS probes| WEB
    PROM --> GRAF
    LOKI --> GRAF
    GRAF -->|alerts| TG

Alloy hosts: pve1, pve2, pve3 (while awake), caddy (CT 112), docker-pve1 (CT 101), holly (Unraid; recovered 2026-07-26, Docker daemon restored 2026-07-29). Each ships node metrics (CPU/memory/disk/network) plus systemd-journal and docker-container logs; the Caddy host additionally parses JSON access logs with pre-indexed request_host / status / method labels. Since 2026-07-30 every Alloy config also enables the netclass collector — node_network_speed_bytes reports each interface's live negotiated link speed, feeding the NetworkLinkDegraded alert — and the Holly + Caddy configs gained the time and stat collectors, bringing them under the HostMetricsStale staleness alert (which keys off node_time_seconds; Holly was invisible to it before).

Dashboard catalog

Twelve provisioned dashboards, layered from single-pane-of-glass down to raw logs and grouped into Grafana folders via foldersFromFilesStructure in dashboards.yml (Start Here / Infrastructure / Hardware & Storage / Edge & Security / Logs & Diagnostics / Services). Homelab Command Center is the default home dashboard (GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH), and allowUiUpdates:false keeps the git JSON authoritative. There is one Prometheus and one Loki instance, so the datasource template selectors are hidden (hide:2) on every board. Every data panel carries a description (100% coverage as of the 2026-07-30 polish pass, PR #172), and two boards were renamed the same day: Fleet Trends (was "MDH Media - Homelab Overview") and Log Explorer (was "MDH Media - Homelab Logs") — UIDs unchanged. Open them at https://grafana.mdhmedia.uk/d/<uid>.

Dashboard UID What it answers When to open it
Homelab Command Center mdh-command-center New top-level landing board — the single pane of glass combining fleet health, alert state and endpoint status with drill-down links to everything below. Start here. Bookmark it; every other dashboard is a drill-down from this one.
Fleet Trends mdh-homelab-overview The fleet-trends view (renamed from "MDH Media - Homelab Overview" 2026-07-30) — per-host CPU / memory / root-disk / network from the host:* recording rules. (Alerts, endpoints/TLS, CrowdSec and the error-tail moved to the Command Center.) Watching fleet utilisation trends over time.
MDH Media - Infrastructure mdh-infrastructure Everything about one host via the $host variable — uptime, CPU by mode, load, memory, filesystem %, disk I/O, network errors, plus that host's logs ($log_job). A specific host is slow, full, or misbehaving.
System Health mdh-system-health Per-host CPU temperature (hwmon/coretemp), CPU busy / memory / load, and live CPU clock — bare-metal hosts (pve1/pve2/pve3) expose thermal sensors; LXCs do not. "Is a host running hot?" under load or in a heatwave.
MDH Media - Security mdh-security CrowdSec decisions (local vs the CAPI community list)/alerts/scenario overflows, parser throughput, and the SSH/auth audit (failed/successful logins, sudo). Edge status is summarised in the at-a-glance tiles; full traffic analytics live on Caddy Access Logs. Suspicious traffic, an IP ban question, or an auth audit.
MDH Media - Ops Jobs mdh-ops-jobs "Did last night's jobs run?" — vzdump backups, pve-offsite-sync to Hetzner, caddy-deploy + DDNS on CT 112, container log-error health on CT 101. Morning check, or after touching backups / the Caddyfile deploy.
MDH Media - Caddy Access Logs mdh-caddy-logs Web traffic analytics: requests by status/service/method, error rates by service, 4xx/5xx log tail, Caddy service errors, full access-log browser. A *.mdhmedia.uk route errors or you're investigating who hit what.
Log Explorer mdh-homelab-logs A clean cross-host log explorer (renamed from "MDH Media - Homelab Logs" 2026-07-30): volume by host/job, errors & warnings over time, and a Host/Job/Search-filtered log browser. Log-first investigations when you don't know which host to blame.
MDH Media - Jarvis Ops mdh-jarvis-ops Jarvis (VM 121) agent liveness (blackbox-jarvis, now loaded + UP), monitoring self-health (Loki/Prometheus up), actionable firing alerts, and the neteng incident/action timeline. Jarvis misbehaves, or you want the monitoring-of-monitoring view.
Holly NAS (Unraid) mdh-holly-nas Array/cache capacity and per-disk fill, a SMART-healthy rollup, Holly CPU/mem/network, per-container log volume and error rate. Per-drive SMART detail lives on Drive Health. NAS capacity planning (only useful while Holly is powered on).
Drive Health (SMART) mdh-drive-health New — per-drive SMART across every host with disks (pve1/pve2/pve3/holly): overall health, temperature, reallocated/pending sectors, CRC errors, SSD wear, power-on hours, one row per physical drive. "How are my drives doing?" — the at-a-glance disk-failure early-warning board.
Power & Energy mdh-power Homelab power draw from a pluggable per-host source model — whole-system watts via IPMI (pve2 R710) and PSU hwmon (NAS rebuild), a CPU-package RAPL proxy (pve1, understates wall draw), and HA smart plugs; plus a measured whole-system total (excludes RAPL) and a rough kWh/day. "How much is the homelab drawing, and costing?" — cost uses the editable Home Assistant input_number.electricity_unit_rate (24.0952 p/kWh), scraped into Prometheus with a constant fallback.

What's on each dashboard

Fleet Trends (was Homelab Overview) — rows & panels

Refocused to fleet trends (2026-07-26), renamed Fleet Trends (2026-07-30): the alert, endpoint/TLS, CrowdSec and error-tail tiles moved to the Command Center, which now owns them. Rows: At a Glance (Hosts Reporting), Fleet (recording rules: host:*). Panels: Hosts Reporting, CPU / Memory / Root Disk / Network by Host (all from the host:* recording rules), plus a markdown panel linking to the Command Center and the deep-dives.

System Health — rows & panels

Rows: CPU Temperature, CPU / Memory / Load, CPU Clock Speed. Panels (all by $host): CPU Temperature now / over time / per-core (hwmon coretemp), CPU Busy %, Memory Used %, Load average, CPU clock avg + peak-vs-idle. Only bare-metal hosts (pve1/pve2/pve3) expose thermal + cpufreq sensors; LXCs appear only in CPU/Mem/Load.

Infrastructure — rows & panels

Rows (all filtered by $host): Overview, CPU, Memory, Disk, Network, Logs. Panels: Uptime, CPU Usage, Memory Usage, Root Disk Usage, Load 1m / 5m / 15m, CPU Usage by Mode, Load Averages, Filesystem Used %, Disk I/O Bytes, Disk IOPS, Network Traffic, Network Errors / Drops, Errors / Warnings ($log_job), All Logs, Log Volume by Job.

Security — rows & panels

Rows: Security at a Glance, CrowdSec (CT 112 — enforcing), SSH & Auth Audit. The duplicative Caddy Edge Traffic row was removed 2026-07-26 (Caddy Access Logs owns edge traffic analytics). Panels: CrowdSec Local Decisions (origin != CAPI — not the ~29.6k CAPI community-list total), CrowdSec Alerts, Scenario Overflows, HTTP 4xx / 5xx (range), Failed SSH Logins, Scenario Overflows by Scenario, Parser Throughput, Whitelist Hits by Rule, LAPI Bouncer Requests, Failed SSH Logins by Host, Failed / Successful Logins, Sudo Usage, plus a markdown drill-down panel.

Ops Jobs — rows & panels

Rows: Backups - vzdump (pve1), Offsite Sync - pve-offsite-sync.service (pve1 → Hetzner Storage Box), Caddy Deploy & DDNS (CT 112), Container Health - docker-pve1 (CT 101). Panels: Backup Runs / Successes / Errors (range), vzdump Activity (pve1 journal), Offsite Sync Runs / Completed / Failures (range), Offsite Sync Log, Deploy Timer Runs, Caddyfile Deploys, Deploy Failures, DDNS Cron Runs (range), Recent Deploy Activity (CT 112 journal), Log Volume by Compose Project, Top 10 Containers by Error Lines, plus a markdown "Job schedules" panel with links to the Overview and Security dashboards.

Caddy Access Logs — rows & panels

Rows: Traffic Overview, Services & Hosts, Error Analysis, Caddy Service, Access Log Browser. Panels: Total Requests, 2xx / 3xx / 4xx / 5xx counts, Requests/sec, Requests by Status Code, Requests by Service/Host, Top Services, Request Methods, Status Code Distribution, Errors by Service, Error Rate Over Time, Error Logs (4xx & 5xx), Caddy Service Errors/Warnings, All Access Logs.

Log Explorer (was Homelab Logs) — rows & panels

Rebuilt 2026-07-26 into a clean log explorer; renamed Log Explorer 2026-07-30. Rows: Overview, Errors & Warnings, Log Browser. Panels: Total Log Lines, Active Hosts, Error Count, Warning Count, Log Volume by Host / Job, Errors & Warnings Over Time, Errors by Host / Job, All Logs, Error Logs Only — all honouring the Host / Job / Search variables. The dead job="auth" stat/timeseries panels and the unit-label "Logs by Systemd Unit" panel were removed; SSH & auth audit lives on the Security dashboard.

Jarvis Ops — rows & panels

Single row: Jarvis / neteng Ops. Panels: Jarvis agent, Firing alerts (Watchdog excluded), Loki, Prometheus, Currently firing, Jarvis liveness, neteng incident / action timeline. The blackbox-jarvis job is now loaded + UP and the two Jarvis liveness panels carry or vector(0) guards, so a vanished scrape renders red not blank; the fleet-wide error-tail panel was removed 2026-07-26 (the Command Center owns it) to keep the board Jarvis-focused, and the empty neteng annotation layer is hidden.

Holly NAS — rows & panels

Rows: Array & Cache Capacity, Per-Disk Fill, System (CPU / Memory / Network), Docker Containers (Loki). Panels: Parity / Redundancy Status, Array Used % / Free (/mnt/user), Cache SSD Used % (/mnt/cache), Disks SMART-Healthy (rollup count), Per-Disk Used %, Current Fill by Disk, CPU / Memory / Network, Container Log Volume, Container Error Rate, Recent Container Errors. The per-drive Disk Temperatures & SMART detail row was removed 2026-07-26 — the dedicated Drive Health (SMART) dashboard owns cross-host per-drive SMART (label device, not the old buggy disk). The parity banner's vector(1) is deliberate, not a stub (reviewed 2026-08-09): it is a constant driving a value-mapping that paints a permanent red "NO PARITY DISK — ARRAY IS UNPROTECTED" background. There is no Unraid parity metric to drive it from, and the array is zero-parity by choice, so the banner is accurate. The one real caveat: it is static, so if a parity disk is ever assigned, this panel must be changed by hand or it will keep claiming the array is unprotected.

Power & Energy — sources & honesty

Power is read from whatever each host has, tagged by a source label of differing accuracy, all landing on one gauge node_power_watts{host,source}. ipmi — pve2's R710, in-band /dev/ipmi0 "System Level", whole-system (~160-200 W). rapl — pve1's i3-4005U CPU-package energy (~5 W), a proxy that misses disks/fans/PSU loss; it is read by the root-run power-metric.sh, not the node_exporter rapl collector, because the Platypus-CVE mitigation makes /sys/.../energy_uj root-only and Alloy runs non-root. hwmon / plug are reserved for the NAS-rebuild Corsair HX1500i PSU (native hwmon collector) and Home-Assistant smart plugs (Matter = local). Panels: a whole-system total that sums only ipmi|hwmon|plug (RAPL deliberately excluded — a CPU proxy in a wall total would mislead), a rough kWh/day, the RAPL proxy on its own, per-host now, and a time series. Cost is sourced from an editable Home Assistant helper (input_number.electricity_unit_rate, 24.0952 p/kWh) exposed via HA's prometheus integration and scraped as the homeassistant job; the electricity_unit_rate_gbp_per_kwh recording rule reads it with a constant fallback. Edit the rate in the HA app - the dashboard follows on the next scrape, time-versioned (a tariff change is a step, never a retro re-cost). Wired on the production .240 HA; repoints to the new VM at the HAOS cutover. See proxmox/power-monitoring/.

Coverage matrix

Honest assessment of what the stack sees today (updated 2026-07-30):

Area Status Detail
Host metrics (node/Alloy) Partial 6 hosts ship Alloy remote_write node metrics (pve1, pve2, caddy/CT112, docker-pve1/CT101, holly, plus pve3 while awake — a Wake-on-LAN backup node since 2026-07-30, normally powered off) with recording rules and the Fleet Trends / Infrastructure / System Health dashboards. Holly recovered 2026-07-26 after a boot-USB failure (its Docker daemon came back after a VM reboot 2026-07-29); since PR #140 the Alloy filesystem collector excludes network fstypes (cifs\|nfs\|smb3\|fuse.*), so a dead Holly CIFS mount can no longer hang the node scrape and blank a host's node_* metrics (which had caused a pve1 + CT101 blackout while Holly was down). node_hwmon_temp_celsius + node_cpu_scaling_frequency_hertz flow from the bare-metal hosts (System Health dashboard); the netclass collector (added to every Alloy config 2026-07-30) exposes node_network_speed_bytes — live link speed, feeding the NetworkLinkDegraded alert — and remote_write silence now alerts via HostMetricsStale (time() - node_time_seconds > 300) on all five always-on hosts, the time+stat collectors having been added to Holly + Caddy to make that true. Remaining holes: CT114/116/120, VM121 (jarvis), CT122 and the HA appliance ship nothing, and node_systemd_unit_state is absent everywhere.
NAS / storage (Holly) Partial mdh-holly-nas covers array/cache/per-disk fill from node_filesystem, and SMART temp/health now flow via smartctl_exporter into the mdh-drive-health dashboard (deployed on pve1, pve2, pve3 and Holly — pve3 added 2026-07-26). Remaining: no disk-I/O panels, and Holly is excluded from vzdump (it is backed up at the Unraid level, not by Proxmox).
Docker containers Good Container logs flow per container/compose_project into Loki (job=docker-logs), and since 2026-07-27 (PR #153) cAdvisor runs on CT 101 — per-container CPU/mem/restart metrics for the whole Arcane estate (job=cadvisor, --docker_only; it works fine in the unprivileged LXC thanks to nesting=1), with ContainerRestartLooping + ContainerDownTinyauth alerting on it since PR #156. Remaining: no generic container-down rule for the other expected containers (the 2026-08-09 blackbox-lan-services tcp probes now cover the user-facing ones from the outside).
Reverse proxy / web (Caddy) Good Best-covered area: JSON access logs with pre-indexed labels (status, request_host, method) feed the Caddy Access Logs + Security dashboards, the caddy-service job catches ACME/reload errors, and all blackbox probes transit it — the only real holes are no latency (duration) panels and no client-IP/user-agent or tinyauth-denial breakdown.
Security / CrowdSec / auth Partial CrowdSec's 34 cs_* metric families are scraped from .245:6060 with a solid Security dashboard, and SSH audit panels exist. Since 2026-08-09 it finally has alert rules — CrowdSecParsingStalled and CrowdSecBouncerStalled, both alerting on work done rather than process-up (a ban-spike rule was tried and dropped — bans are normal operation, and its baseline had been measured while CrowdSec was blind), written after the gap hid a 9-day parsing outage (see the changelog). Remaining: tinyauth is probed (https via auth.mdhmedia.uk + a direct :9092 tcp probe since 2026-08-09) yet still has no panel, WireGuard's tunnel is invisible (wgadmin :10086 now tcp-probed), the SSH panels double-count auth.log+journal, and the dashboard's simulation-mode text is stale (enforcing since 2026-07-08).
Uptime probes / certs Partial 5 blackbox-http targets healthy with probe_ssl_earliest_cert_expiry + a 14-day expiry alert; the blackbox-jarvis tcp probe (VM 121 :9119) is now loaded + UP, and a blackbox-pbs tcp probe watches PBS (192.168.1.30:8007) liveness — excluded from the EndpointDown alert since 2026-07-30, because PBS is a Wake-on-LAN box that is expected to be offline outside its evening window. Since 2026-08-09 the blackbox-lan-services job tcp-probes the always-on SSO-gated/LAN services directly on their ports (tinyauth :9092, grafana :3300, audiobookrequest :8000, homepage :3080, wgadmin :10086) — probing their public hostnames is meaningless because Caddy's forward_auth 302s to auth before touching the upstream, so a parked service and a failed one look identical from outside (demonstrated 2026-08-09 by kicad + grampsweb, which are intentionally stopped and are deliberately NOT probe targets — re-add them when they return to service). Holly's apps gained a guarded job the same day (blackbox-holly-apps: audiobookshelf, qbittorrent, nzbget, radarr, sonarr, prowlarr, ombi, plex) — excluded from EndpointDown and routed to HollyAppDown, which fires only when Holly itself is demonstrably alive, so a deliberate NAS shutdown stays silent instead of emitting 8 criticals. Remaining: the icmp module is defined yet unused (no host-liveness probes).
Logs (Loki pipeline) Good 4 jobs (caddy, caddy-service, docker-logs, systemd-journal) from all Alloy hosts, ~216k lines/24h, browsable via 3 log-centric dashboards — weaknesses are consumption-side: naive keyword regexes ignoring the detected_level label, and few Loki-based alert rules (the first three — BackupJobFailed, WolPbsCycleProblem and the fleet-wide OomKillDetected — landed 2026-07-31 as Grafana-managed rules; most panel-detected conditions still don't page).
Backups (PBS / vzdump / offsite / HA) Partial PBS is the primary target, since 2026-07-30 on an evening Wake-on-LAN window: the wol-pbs container on CT 101 wakes pve3 at 19:00, the PBS job covers all 9 pve1 guests at 19:15 (prune 19:45, GC 19:50, verify 20:00, Holly's host/holly upload 20:35, then offsite sync and shutdown once PBS is idle). Since 2026-07-31 the window is alert-covered without Jarvis: PbsBackupWindowMissed (Prometheus deadman — PBS never reachable in 26h), BackupJobFailed (Loki rule on the pve1 journal's vzdump failure lines) and WolPbsCycleProblem (ERROR/WARN in the wol-pbs container log; both summaries rewritten 2026-08-04, PR #194, to be unmistakable). The Jarvis neteng-backup-watchdog cron (daily 20:05) additionally alerts if any guest's newest PBS snapshot is >30h old or missing, or the datastore exceeds 90%, and tracks the host/holly age (~23.5h at check time is healthy) — an unreachable PBS is expected (WoL box) and stays silent. The Ops Jobs dashboard still counts the local vzdump runs via brittle journal string-matches — no success/age metric, no "time since last good backup" panel — and its offsite row watches pve1's pve-offsite-sync, which was disabled 2026-07-31 (superseded by the PBS-side sync). Home Assistant's Google Drive backups are wholly unmonitored.
Proxmox guest level Good pve-exporter live since 2026-07-27 (PR #152): prompve/prometheus-pve-exporter in the prometheus stack, multi-target /pve job over all three nodes (~84 pve_up series), with ProxmoxGuestDown + ProxmoxStoragePoolFull alerting on it since PR #156 (stopped-by-design guests + the sleeping pve3 excluded).
Home automation (HA) Partial Only a blackbox https_2xx probe of ha.mdhmedia.uk. The rebuilt HAOS VM (haos, VMID 123 on pve1) is built but parked at 192.168.2.120 (stopped, onboot=0 since 2026-08-27), while the old .240 instance still owns the route. At cutover, VM 123 starts, returns to onboot=1, and takes .240 + the route; its /api/prometheus endpoint is already enabled and the monitoring-stack scrape job lands then. Until cutover no entity/automation/zigbee metrics flow, the appliance ships no host metrics, and the Google Drive backup add-on outcome is invisible.
Network / DNS Missing AdGuard (CT120) — the split-horizon DNS every LAN blackbox probe depends on — has zero telemetry (no Alloy, no dns-module probe, no exporter); WireGuard (CT116) likewise; no ICMP probes; an AdGuard outage would fail all domain probes at once with nothing pointing at DNS.
Alerting / notification delivery Good The rule set grew 2026-07-30 (PRs #169/#170): HostMetricsStale (remote_write silence), NetworkLinkDegraded (a physical link renegotiated down to <=100 Mb/s) and BlackboxExporterDown (names the root cause when every probe goes blind). HostDown now excludes the blackbox-* probe jobs (a probe target's up reflects the exporter, not the service) and the deliberately-sleeping pve3/PBS (.28/.30); EndpointDown excludes job=blackbox-pbs. The backup window gained its own rules on 2026-07-31: PbsBackupWindowMissed (Prometheus deadman) plus the first Grafana-managed Loki rules — BackupJobFailed, WolPbsCycleProblem (their summaries rewritten 2026-08-04, PR #194, so the two are unmistakable) and a fleet-wide OomKillDetected. Everything delivers to Telegram via the telegram-jarvis contact point (bot @MDHJarvis_bot; chat id hardcoded in contactpoints.yaml to dodge grafana#69950), now under a noise policy (PR #171): group_by is [alertname] only, repeats are tiered — critical 6h, warning 24h, default 12h — and the always-firing Watchdog deadman has its own 24h route (a silent Telegram for >24h is itself the signal the pipe broke). Motivating incident: the blackbox exporter died in the 2026-07-28 pve1 reboot and stayed down 43h, producing 11 false HostDown criticals (~120 Telegram messages). SMTP stays suppressed by choice; the remaining weakness is the single channel (no Alertmanager).
Media stack (*arr / Plex / downloads) Missing Zero per-app telemetry for Plex, Radarr/Sonarr/Prowlarr, qBittorrent, NZBGet, Audiobookshelf, Ombi, or AudioBookRequest (on CT 101 since 2026-07-21) — only host metrics + docker log lines, and none of the *.mdhmedia.uk media routes are probed; the MCP servers give ad-hoc queryability but nothing continuous.

Gap roadmap

Ranked by leverage. Effort: S = small (config-only), M = medium (new component).

# Gap Effort Value
1 Notification delivery (placeholder creds, no deadman) DONE 2026-07-13 — Telegram live + Watchdog deadman S high
2 Host-down blind spot (no up{} from remote_write hosts, no staleness alert) DONE 2026-07-30 — HostMetricsStale covers every always-on host S high
3 Backup success monitoring — partly done 2026-07-31 (window deadman + failure-log alerts live; no success-age metric yet) M high
4 No container metrics on docker-pve1 DONE 2026-07-27 (PR #153 — cAdvisor on CT 101; restart-loop + tinyauth-down alerts PR #156) M high
5 Narrow probe coverage — largely DONE 2026-08-09 (blackbox-lan-services tcp-probes every SSO-gated/LAN service directly; Holly apps + icmp remain) S high
6 No Proxmox guest-level metrics DONE 2026-07-27 (PR #152 — pve-exporter, 3 nodes; guest-down + pool-full alerts PR #156) M high
7 AdGuard DNS: partly done 2026-07-27 (dns_udp probe + AdGuardDnsDown live; still no host telemetry from CT 120) S high
8 Log-based alert rules — partly done 2026-07-31 (first three Loki rules live; most panel-detected conditions still don't page) M high
8b CrowdSec has zero alert rules DONE 2026-08-09 — parsing-stalled + bouncer-stalled (the gap had hidden a 9-day WAF outage) S high
9 No SMART / temperature telemetry anywhere M med
10 HA, MariaDB and WireGuard essentially unobserved M med
11 Dashboard hygiene: dead/duplicated panels, inconsistent $host largely DONE 2026-07-26 (see below) M med
12 Monitoring-of-monitoring: Grafana unscraped, single delivery channel S med
1. Notification delivery — DONE (2026-07-13): Telegram live + Watchdog deadman — effort S, value high

What landed. The telegram-jarvis contact point now delivers to Telegram (bot @MDHJarvis_bot). The bot token is set in the Grafana stack's Arcane project .env (TELEGRAM_BOT_TOKEN); the chat id is hardcoded in contactpoints.yaml because grafana 13.1.2 coerces an all-numeric $TELEGRAM_CHAT_ID back to a JSON int and crash-loops provisioning — plain quotes and the !!str tag were both verified to still fail (grafana#69950). The chat id is a Telegram user id, not a secret (useless without the bot token).

Deadman. An always-firing Watchdog rule (expr: vector(1)) in alerts.yml surfaces as ALERTS{alertstate="firing"}, which the existing Grafana mirror rule turns into a Telegram ping — since 2026-07-30 on the Watchdog's own 24h route. No Watchdog ping in >24h means the Prometheus→Grafana→Telegram pipe is broken — the only signal that delivery itself failed.

Noise policy (2026-07-30, PR #171). The severity-split routes landed: policies.yaml groups by alertname only (one message covers the same alert on N hosts) and tiers the repeats — critical 6h, warning 24h, default 12h. Still single-channel (no Alertmanager).

2. Host-down blind spot — DONE (2026-07-30, PR #169): HostMetricsStale covers remote_write silence — effort S, value high

Was. HostDown (up==0) only covers the pull-scraped targets on .241. The real hosts arrive via remote_write and produce no up{}, so if pve1 itself died — taking the hypervisor, backups, and most guests with it — metrics just stopped and NOTHING fired; dashboards showed stale green lastNotNull stats.

Now. HostMetricsStale in alerts.yml fires (warning, one series per host) when time() - node_time_seconds > 300 — it keys off the node time collector, so all five always-on hosts (pve1, pve2, caddy, docker-pve1, holly) are covered. The time + stat collectors had to be added to the Holly and Caddy Alloy configs first (2026-07-30) — Holly was invisible to the rule before. pve3 is deliberately not covered: it is a Wake-on-LAN backup box that is expected to be asleep. The Command Center Last-Seen table still names which host vanished. Blackbox ICMP host-liveness probes remain unimplemented (optional belt-and-braces).

3. Backup success monitoring: PARTIALLY CLOSED 2026-07-31 (window deadman + failure-log alerts live); HA Google Drive still unmonitored — effort M residual

What landed (2026-07-31). Three alerts now cover the nightly PBS cycle end-to-end without Jarvis: PbsBackupWindowMissed (Prometheus — the blackbox-pbs probe never succeeded in 26 h, i.e. the WoL window never happened), BackupJobFailed (Grafana-managed Loki rule on the pve1 journal's vzdump failure lines, incl. could not activate storage 'pve3-pbs' — the exact line the 2026-07-30 missed night produced; pages within minutes of a 19:15 failure), and WolPbsCycleProblem (ERROR/WARN in the wol-pbs container log). Together with Jarvis's 20:05 watchdog (snapshot ages via the PBS API — since 2026-08-03 it also tracks Holly's host/holly backup age, where ~23.5h at check time is healthy) a silent no-backup night is no longer possible without a page. On 2026-08-04 (PR #194) the BackupJobFailed and WolPbsCycleProblem summaries were rewritten so the two alerts are unmistakable at a glance.

Proactive daily surface (2026-08-22). The new backup-digest Arcane stack (stacks/management/backup-digest/) posts a once-a-day 08:00 Telegram summary that puts the backup state in front of a human every morning - turning silence into a signal (a firing critical had gone unread ~10 days). It reads the PBS window + verify state time-windowed (max_over_time(probe_success{job="blackbox-pbs"}[26h]) and the blackbox-pbs-verify equivalent) so a WoL-asleep PBS is never misread as a failure. The PbsVerificationFailed alert itself was reworked to the same 26h deadman on 2026-08-25 - it previously used a bare probe_success == 0, which fired critical all day the WoL box slept and could not be told apart from a real verify failure; the digest still mutes it from its raw alert list only to avoid double-reporting the authoritative windowed line above.

Built and live (PR #318, 2026-08-27). The wol-pbs orchestrator writes pbs_last_backup_timestamp_seconds{guest=...} to a node_exporter textfile on CT 101, and Alloy's textfile collector serves it. GuestBackupStale (48h) alerts on per-guest backup age, while PbsBackupMetricsStale is the writer deadman. HA's Google Drive backups remain unmonitored (expose the add-on sensor via the HA Prometheus integration, gap 10).

Partly mitigated (2026-07-30). The Jarvis neteng-backup-watchdog cron (daily 20:05, moved from 06:00 — PBS is asleep at 06:00 now) alerts if a guest's newest PBS snapshot is >30h old or missing, or the datastore exceeds 90%; an unreachable PBS is expected (Wake-on-LAN box) and stays silent. vzdump / offsite / HA remain journal-string-matched only.

4. Container metrics on docker-pve1 — DONE (2026-07-27, PR #153) — effort M, value high

What landed. cAdvisor (gcr.io/cadvisor/cadvisor, version-pinned, --docker_only) runs in the prometheus stack on CT 101 — per-container CPU/mem/restart metrics for the entire Arcane estate under job=cadvisor. It works in the unprivileged LXC (nesting=1 gives cgroup delegation; the pre-deploy worry was unfounded). ContainerRestartLooping and ContainerDownTinyauth (PR #156) alert on it. Remaining nice-to-have: a generic container-down rule for the other expected containers — partly mitigated 2026-08-09 by the blackbox-lan-services tcp probes, which watch the user-facing services from outside the docker engine.

5. Probe coverage is narrow (blackbox-jarvis + PBS now loaded — partly done) — effort S, value high

Done. blackbox-jarvis is now loaded + UP (the Jarvis Ops headline panels render, and carry or vector(0) guards so a vanished scrape shows red not blank), and a new blackbox-pbs tcp probe watches PBS (192.168.1.30:8007) — both surface on the Command Center endpoint grid automatically.

Also done (2026-08-09). The blackbox-lan-services job tcp-probes the always-on SSO-gated/LAN services directly on their ports: tinyauth :9092, grafana :3300, audiobookrequest :8000, homepage :3080, wgadmin :10086 — all feeding the existing EndpointDown alert. Direct ports, not the public hostnames, because Caddy's forward_auth answers 302 → auth before the request reaches the upstream, so an SSO-gated https probe passes whether the backend is dead or intentionally parked — from outside the two are indistinguishable (kicad + grampsweb demonstrated this on 2026-08-09; both are parked on purpose, so they are deliberately not targets — re-add each probe target when its service returns). auth.mdhmedia.uk (https, added earlier) covers the Caddy→tinyauth path.

Still to do. The Holly apps (Plex, *arr, downloads) are unprobed, and the icmp module is defined but unused (no host-liveness probes).

6. Proxmox guest-level metrics — DONE (2026-07-27, PR #152) — effort M, value high

What landed. prometheus-pve-exporter (prompve/prometheus-pve-exporter, pinned) in the prometheus stack, reusing the homepage PVEAuditor token (note: the real token is homepage@pam!home); a multi-target /pve scrape job covers all three nodes (~84 pve_up series). ProxmoxGuestDown + ProxmoxStoragePoolFull (PR #156) alert on it, with stopped-by-design guests (100/113/115) and the WoL-sleeping pve3/PBS excluded. The guests with no Alloy agent (CT114/116/120, VM121, CT122) are now visible at the up/down + CPU/mem level.

7. AdGuard DNS: a zero-telemetry single point of failure that all LAN probes depend on — effort S, value high

Why. CT120 carries the split-horizon rewrites (*.mdhmedia.uk → .245) that every LAN blackbox probe relies on. If AdGuard dies, all domain probes fail simultaneously while the internet path stays fine — a confusing all-red storm with nothing pointing at DNS, and LAN clients lose name resolution entirely. It currently has no Alloy, no probe, no exporter.

How. Three layers, cheapest first: (1) done 2026-07-27 — the dns_udp module in blackbox.yml probes 192.168.1.244:53 and AdGuardDnsDown (PR #156) names the root cause ahead of the probe cascade; (2) run install-alloy.sh on CT120 for host metrics + journal; (3) optionally add adguard-exporter (ebrianne/adguard-exporter) to the monitoring stack for query/block-rate stats. Alert on the dns probe_success with severity=critical and mention DNS in the annotation so probe-storm triage starts at the right place.

8. Log-based alert rules: partly done 2026-07-31 — the first three Loki rules are live — effort M residual

What landed (2026-07-31). The first Grafana-managed Loki alert rules: BackupJobFailed (vzdump failure lines in the pve1 journal), WolPbsCycleProblem (ERROR/WARN in the wol-pbs container log — the two summaries were rewritten 2026-08-04, PR #194, to be unmistakable) and a fleet-wide OomKillDetected.

Why the rest still matters. The dashboards also color-code Caddy 5xx spikes, caddy-deploy rollback failures, failed-SSH bursts and CrowdSec overflows — but those still depend on a human happening to look. The systemic pattern remains: detection logic living in panel thresholds instead of alert rules.

How. Add further Grafana-managed rules (provisioning/alerting/rules.yaml, datasource uid loki) reusing the exact panel queries: (1) offsite-sync/vzdump failure strings from mdh-ops-jobs (done — BackupJobFailed), (2) sum(count_over_time({job="caddy", status=~"5.."}[15m])) > 50, (3) the caddy-deploy "failed validation|rollback" regex, (4) failed-SSH burst > threshold/10m. Also add one Prometheus rule on CrowdSec: increase(cs_bucket_overflowed_total[1h]) > N now that it is enforcing. They all route to the (fixed) Telegram contact point via the existing root policy — zero new plumbing.

9. Drive SMART telemetry — DONE (2026-07-10): smartctl_exporter + Drive Health dashboard

Was. smartctl_device_* metrics were absent everywhere, so the Holly SMART panels never rendered and pve1/pve2's disks had no failure early-warning.

Now. host-agents/install-smartctl-exporter.sh deploys the prometheus-community smartctl_exporter (v0.14.0) on :9633 — native systemd on PVE/Debian hosts, a privileged container on Unraid — firewalled to the Prometheus host like the CrowdSec endpoint. A smartctl scrape job in prometheus.yml (per-target host label) pulls it; the new Drive Health (SMART) dashboard (mdh-drive-health) shows one row per physical drive with health, temperature, reallocated/pending sectors, CRC, SSD wear and power-on hours, and the Holly dashboard's SMART panels are repointed at the real metrics (smartctl_device_smart_status, not the guessed _smart_healthy). Alerts added: SmartHealthFailed (critical), SmartReallocatedSectors / SmartPendingSectors / DriveTemperatureHigh > 60C (warning).

Live now: pve1, pve2, pve3 and Holly (pve2 + Holly added 2026-07-20 after the heatwave shutdown; pve3 added 2026-07-26) — every host with disks now reports SMART to Prometheus/Grafana. node_hwmon_temp_celsius (host CPU/board thermals) is still absent — a separate, smaller follow-up (enable the hwmon collector in install-alloy.sh).

10. Home Assistant, MariaDB and WireGuard have essentially zero observability — effort M, value med

Why. HA (one HTTPS probe, nothing else) runs home automation; MariaDB (CT114) serves databases with no telemetry, no exporter, not even a TCP probe; WireGuard (CT116) is the remote-access door with an admin UI that is SSO-gated but unmonitored. All three fail silently today; MariaDB and WireGuard are also invisible to any dashboard.

How. HA: the rebuilt HAOS VM is built but parked (stopped, onboot=0) since 2026-08-27 while the old .240 instance owns the route. At cutover, start VM 123, set onboot=1, and move .240 + the route; the VM already exposes /api/prometheus - only the prometheus.yml scrape job remains, planned to land at cutover - which also surfaces backup sensors for gap 3. MariaDB: add mysqld_exporter (prom/mysqld-exporter, pinned) to the monitoring stack pointed at 192.168.1.251:3306 with a monitoring-only grant, or minimally a tcp_connect blackbox probe of :3306. WireGuard: run install-alloy.sh on CT116 and add a textfile metric from wg show latest-handshakes (or prometheus_wireguard_exporter); probe wgadmin per gap 5. pve-exporter (gap 6) provides the guest-level backstop for all three CTs meanwhile.

11. Dashboard hygiene — largely DONE (2026-07-26) — effort M, value med

What landed. One JSON PR: (1) mdh-homelab-logs rebuilt as a clean log explorer — the dead job="auth" panels and the unit-label "Logs by Systemd Unit" panel removed, ${datasource}→${loki} renamed, schemaVersion 39; (2) mdh-homelab-overview refocused to fleet-trends (the duplicated alert/endpoint/CrowdSec/error tiles dropped — the Command Center owns them); (3) mdh-security's duplicative Caddy Edge Traffic row removed (Caddy Access Logs owns edge traffic); (4) mdh-holly-nas per-drive SMART detail removed (Drive Health owns it) and the buggy disk→device label retired with it; (5) mdh-jarvis-ops narrowed to Jarvis-only (fleet error-tail dropped, blackbox-jarvis guarded, empty neteng annotation hidden); (6) mdh-ops-jobs "(7d)" stat titles → "(range)"; (7) the Watchdog deadman is excluded from every Firing Alerts count and given its own green heartbeat tile on the Command Center; (8) the Prometheus/Loki datasource selectors are hidden fleet-wide and dashboards grouped into folders.

Still open (smaller). Replace keyword-regex error queries with detected_level=~"error|warn" where streams carry it; add the Caddy p95 latency panel (| json | unwrap duration, quick-win #10); dedupe mdh-caddy-logs' internal pie-vs-timeseries / Top-Services-vs-Requests-by-Host pairs.

Dashboard overlap & dead-panel inventory (pre-2026-07-26 audit — most now fixed)
  • mdh-homelab-overview duplicates itself: 3 of 6 at-a-glance tiles (Endpoints Down, Min Cert Days, Probe Latency p95) restate the Endpoints & TLS row's panels — 6 of 16 panels are blackbox-derived on a whole-homelab dashboard.
  • mdh-security's "Caddy Edge Traffic" row substantially restates mdh-caddy-logs: Requests by Hostname, Status Class Breakdown and 4xx/5xx by Hostname mirror the dedicated Caddy dashboard; HTTP status-class data is rendered ~8 different ways across the two dashboards plus the overview.
  • mdh-caddy-logs duplicates internally: "Requests by Status Code" vs "Status Code Distribution" pie, "Requests by Service/Host" vs "Top Services" bargauge, and the 4xx/5xx stat tiles vs "Error Rate Over Time" all restate the same series.
  • Near-identical keyword-regex error-log tails appear on four dashboards (overview "Recent Errors", infrastructure "Errors/Warnings", homelab-logs "Error Logs Only", jarvis-ops "Recent homelab errors") — the jarvis-ops one scope-creeps to every host, diluting a Jarvis-focused board; the firing-alerts stat+table pair is also duplicated on overview and jarvis-ops.
  • Dead panels, mdh-homelab-logs: 5 panels query a job="auth" that does not exist in Loki, and "Logs by Systemd Unit" filters on a unit label journal streams never ship (verified 0 series) — the entire Security & Authentication and Systemd Services rows are permanently blank.
  • Dead panels, mdh-jarvis-ops: "Jarvis agent" stat and "Jarvis liveness" timeseries query probe_success{job="blackbox-jarvis"}, which returns zero series on the live Prometheus (job defined in prometheus.yml but never loaded) — the dashboard's two most important panels show No Data.
  • Dead panels, mdh-holly-nas: both SMART panels (temperature timeseries, health table) have never had data (smartctl metrics absent from all hosts even while Holly was up). (The Parity banner's vector(1) was also listed here as a defect; on review 2026-08-09 it is intentional — see the note above.)
  • mdh-infrastructure triple-reports load average ("Load 1m" stat + "Load 5m/15m" stat + "Load Averages" timeseries, none with core-count thresholds) and overlaps mdh-holly-nas's System row (CPU/mem/network for host=holly) — the latter overlap is mild and arguably fine.
  • Inconsistent $host/$job variable application makes several dashboards disagree with themselves: overview stats/alerts-table/blackbox row, caddy-logs headline stats + Top Services, and all six homelab-logs overview stats ignore the filter their sibling panels honour — selecting a host yields half-filtered views.
12. Monitoring-of-monitoring: Grafana unscraped, single-channel delivery, no scrape-health panel — effort S, value med

Why. Grafana exposes no metrics to Prometheus (no grafana_* series) and is also the sole alert-delivery engine — if Grafana dies, alerting dies with it and nothing notices (the up{job="prometheus"} self-scrape stat on jarvis-ops is similarly near-dead). No dashboard shows "up by job" scrape health, so a dead crowdsec or blackbox exporter surfaces only as frozen tiles.

How. Add a grafana scrape job (grafana:3000/metrics on monitoring-net) to prometheus.yml; the existing HostDown (up==0) rule then covers it — though delivery still transits Grafana, so pair it with the gap-1 deadman (missing Watchdog pings expose a dead Grafana externally) or probe https://grafana.mdhmedia.uk per gap 5. Add a small "Scrape Targets" stat/table panel (up by job/instance) to mdh-homelab-overview's At a Glance row.

Quick wins

Cheapest changes first — most are config-only edits in this repo:

  1. Set real Telegram creds DONE 2026-07-13 — the pipe is live (bot @MDHJarvis_bot; chat id hardcoded in contactpoints.yaml per grafana#69950) and a Watchdog deadman guards it. All 7 alert rules now deliver.
  2. Reload/redeploy Prometheus so blackbox-jarvis loads DONE — the job is loaded
  3. UP; the mdh-jarvis-ops liveness panels render and carry or vector(0) guards. A blackbox-pbs tcp probe (PBS 192.168.1.30:8007) was added alongside.
  4. Add a staleness alert DONE 2026-07-30 — HostMetricsStale (time() - node_time_seconds > 300) in alerts.yml covers every always-on remote_write host; the time+stat collectors were added to the Holly + Caddy Alloy configs so Holly is no longer invisible to it (pve3 is deliberately excluded — a Wake-on-LAN box that is expected to sleep).
  5. Add a per-host "Last Seen" table panel DONE — the Command Center's Host Last Seen (metrics) table (with a reporting-timeline state-timeline) names which remote_write host vanished, so "Hosts Reporting: 3" becomes "holly last seen 26h ago".
  6. Fix the 6 dead panels in mdh-homelab-logs DONE — the board was rebuilt as a clean log explorer: the dead job="auth" stat/timeseries panels and the unit-label panel were removed, and SSH/auth audit now lives solely on mdh-security.
  7. Extend blackbox target lists in prometheus.yml with existing modules: tcp_connect for tinyauth 192.168.1.241:9092 (the SSO SPOF), plus https targets for grafana/mdhdocs/wgadmin — config-only, exporter already running.
  8. Add an "up by job" scrape-health stat panel to the overview's At a Glance row (sum by (job) (up)) — a dead crowdsec/blackbox/loki scrape currently only shows as frozen tiles.
  9. Update the dashboard's stale "simulation mode" text DONE — mdh-security's footer panel now reads "CrowdSec is enforcing (since 2026-07-08)". Still open: fix the CrowdSec Alerts stat on mdh-security from sum(cs_alerts) (counter-since-restart) to sum(increase(cs_alerts[$__range])).
  10. Add "Time Since Last Successful Backup" and "Time Since Last Offsite Sync" stat panels to mdh-ops-jobs using the existing journal lines (time() minus last log timestamp via a Loki instant query) — answers the #1 backup question at a glance with data already in Loki.
  11. Add p95 latency-by-request_host panels to mdh-caddy-logs via {job="caddy",type="access"} | json | unwrap duration | quantile_over_time(0.95, ...) — the duration field is already in every JSON access log line.

Known quirks

Read before panicking at a blank panel

  • Holly (Unraid NAS, VM 110) recovered 2026-07-26 after a 2026-07-25 boot-USB failure (the USB physically dropped off the bus; reseated); the same failure had corrupted docker.img, and the Docker daemon came back after a VM reboot on 2026-07-29. If Holly is ever down again, its node metrics, SMART rows on Drive Health, and the Holly NAS dashboard go expected-blank.
  • pve3 (192.168.1.28) is a Wake-on-LAN backup node (since 2026-07-30) — normally powered off, woken daily at 19:00 for the PBS backup window and shut down once PBS is idle. Its Alloy node metrics + smartctl_exporter report only while it is awake; expected-blank the rest of the day. HostDown and HostMetricsStale deliberately exclude it (and the PBS VM at .30).
  • blackbox-jarvis is loaded + UP and a blackbox-pbs tcp probe watches PBS (192.168.1.30:8007) — both show on the Command Center endpoint grid. blackbox-pbs is excluded from the EndpointDown alert: PBS being unreachable is normal outside the evening backup window.
  • A dead blackbox exporter no longer masquerades as dead sites: after the exporter died in the 2026-07-28 pve1 reboot and sat unnoticed for 43h (11 false HostDown criticals, ~120 Telegram messages), HostDown ignores the blackbox-* probe jobs and a dedicated BlackboxExporterDown alert names the root cause.
  • Alerts now deliver to Telegram (bot @MDHJarvis_bot): the telegram-jarvis contact point is live under a tiered noise policy (grouped by alert name only; repeats critical 6h / warning 24h / default 12h) and the Watchdog deadman pings on its own ~24h route, so a silent Telegram for over a day is itself the signal that the delivery pipe broke.
  • SMTP is suppressed by choice — email is intentionally not a notification channel; don't "fix" it by wiring SMTP back up. Telegram is the designated channel (now live).

References