Skip to content

Open items

Things that are genuinely outstanding, and who they are waiting on. Kept short on purpose — if something here is done, delete the entry rather than annotating it. Last reviewed 2026-08-24.

Investigating

Why did a firing critical go ~5 days unacted? (2026-08-19)

PbsBackupWindowMissed (critical) fired continuously 2026-08-14 23:00 -> 2026-08-19 15:00 UTC (~19 Telegram messages at the 6 h critical repeat) during the pve3 WoL outage, and was not acted on. Single delivery channel, alert fatigue, or notifications not landing - check the Telegram thread for those dates before changing the rule. The 2026-08-22 backup-digest (daily 08:00) is the first mitigation.

Waiting on you

Nothing here is blocked on work — only on credentials, physical access, or a decision.

Send to Kindle (Audiobookshelf) — the live one

Everything mechanical is done: the /mnt/user/Media/Books → /books mount is on the ABS container, the Books library (067eda4d-1114-4270-87ce-27189d7a6623) is scanned with 10 items, and the book-router pushes new arrivals in automatically, and as of 2026-08-27 the SMTP sender is configured too. Two steps remain, both needing your Amazon account:

  1. ABS → Settings → Email — done 2026-08-27. A dedicated mailbox abs@mdhdev.uk was created on eu1 (cPanel user mdhdev, 250 MB) and wired into ABS: host mail.mdhdev.uk, port 465, implicit TLS, rejectUnauthorized left true — the cert validates for that hostname, checked rather than assumed. SMTP AUTH verified on both 465 and 587 before wiring, and the saved settings were read back rather than trusted from the API response. A dedicated account rather than reusing ombi@mdhdev.uk, which ABS had been pointed at: the From address has to go on Amazon's approved-sender list in step 3, and that should not be a mailbox another service depends on.

Deliverability was checked first, because Amazon silently discards mail that fails authentication: mdhdev.uk publishes SPF explicitly listing eu1's sending IP 167.235.23.68, DKIM (default._domainkey), and DMARC p=none.

The password is not in git — it is in the operator's password manager. 2. Same page → e-reader devices — add the @kindle.com address (find it in Amazon → Manage Your Content and Devices → Preferences → Personal Document Settings). 3. Amazon → Personal Document Settings → Approved Personal Document E-mail List — add abs@mdhdev.uk (the From address configured in step 1).

Step 3 is the one that bites

Skip it and Amazon silently discards the email while ABS still reports success. If a send "works" but nothing arrives, check this first.

Prefer EPUB — Amazon dropped MOBI for Send-to-Kindle in 2022 and converts EPUB on receipt. PDFs go through as-is. See Books & Comics.

Physical / hardware

  • pve1 boot SSD — wear alert at ~15% life left (Aug 2026). Mirror the rpool or replace; the alert only warns, it does not act.

Decisions

  • HAOS cutover night — VM 123 parallel-ran 2026-08-01 → 08-27 and is now powered off with onboot=0, to return 2.5 GB to pve1 while this waits for a free evening. The old .240 instance still owns the IP and the route, so nothing is degraded. You pick the evening; the checklist is in haos-vm/README.md — start with its step 0 (qm start 123 + qm set 123 -onboot 1), or HA will not survive a power cut post-cutover.

Queued work (no human input needed)

  • mcp 2.0 fleet port done 2026-08-09 (PR #234, superseding #190) — all nine servers moved to MCPServer; 9/9 verified over a real stdio handshake. If you pull this repo on another machine, reinstall each server's .venv — the code now requires mcp 2.0 and an old venv will fail at import.
  • Backup Phase 2 — retire pve1's two local vzdump jobs and extend the PBS job to pve2's guests. The corruption root-cause gate is now cleared (see Recently closed: the failures were a one-off, resolved by the 2026-08-04 whole-disk migration + the RAM removal, and verifies have passed since 2026-08-19). But the local vzdump floor saved the estate twice in nine days — the verify corruption and the five-night WoL gap — so retiring it is now a risk-appetite decision for you, not a blocked task. If you'd rather keep the floor, this item can simply close.
  • Monitoring leftovers done 2026-08-09 — CrowdSec now has alert rules (and writing them uncovered a 9-day parsing outage, since fixed), Holly's apps are probed behind a guarded HollyAppDown, and the parity banner turned out to be intentional rather than a stub. Grafana itself is unscraped fixed 2026-08-27 — Prometheus now scrapes grafana:3000, so the component that evaluates every alert rule is no longer the one thing nothing watches. Single delivery channel decided 2026-08-27, see Parked on purpose.

Recently closed

  • PBS verify-job corruption — root cause found (2026-08-11 → 08-24) — the failed-verify backups (5 remain) are all dated 2026-07-26 → 08-10, i.e. entirely the pre-migration / faulty-RAM era: a one-off from the 2026-08-03 zil_commit oops on the old ext4-in-zvol datastore layout plus the bad RAM removed after the 2026-08-22 memtest. Confirmed resolved — the bak pool scrubbed with 0 errors on 2026-08-04 (repairing 12K during the migration), SMART + kernel are clean, and every verify job since 2026-08-19 has passed; no post-migration backup has ever failed. The old failed snapshots prune out on retention. This clears the corruption root-cause gate on Backup Phase 2.
  • pve3 stopped waking to WoL (2026-08-19 -> 24) — after the 2026-08-19 RAM swap pve3 missed five consecutive backup nights (14th-18th): the magic packet was sent each night but the box never booted. Root cause was the CMOS reset from the RAM swap clearing the Gigabyte Z68XP-UD3 BIOS power settings. Re-set in Power Management Setup (ErP/EuP Ready = Disabled, Wake on LAN = Enabled) on 2026-08-24, and a full sleep -> magic-packet -> wake round-trip verified (woke in ~5 s). The five-night gap was covered by pve1's local vzdump floor and caught up with a manual vzdump of all 9 pve1 guests on 2026-08-24 — the second time in nine days the local floor saved the estate, so Backup Phase 2 stays held.
  • Docs site silently stale after content-only merges — found 2026-08-11 (stale 19+ hours across 4 merged docs PRs; zensical build runs once at container start and nothing recreated the container for content-only changes). Arcane's own pre-deploy hook was researched and ruled out (sandboxed, network=none, no Docker socket access). Fixed with docs/deploy/docs-deploy.timer on CT 101 (#267) — a content-hash check, not mtime (verified live that Arcane rewrites every file's mtime on every poll regardless of content change, which would have restart-looped an mtime-based version forever). Verified end-to-end: a real merge (#268) was auto-detected and triggered a restart with the live site updating; then the enabled timer held steady through 3 real scheduled poll cycles (20:46, 20:52, 20:57) with zero unexpected restarts.
  • pve1 RAM pressure — root cause was an uncapped ZFS ARC (~14.5 GiB ceiling on a 16 GiB host), not the VMs/CTs. Capped to 2 GiB 2026-08-11; available memory 1.6 GiB → 4.3 GiB, HighMemoryUsage cleared.
  • Old PBS zvol — the ext4-in-a-zvol rollback kept after the 2026-08-04 datastore migration. Destroyed 2026-08-11 after a clean week; ~55G reclaimed on pve3's bak pool.
  • PBS verify-job monitoring — PBS had zero telemetry of its own. A new blackbox probe (blackbox-pbs-verify / PbsVerificationFailed) now watches the PBS API directly for the nightly verify job's status. Found and confirmed a real, ongoing failure the same day it was deployed (root cause since found and closed — see above). Reworked to a 26h max_over_time deadman on 2026-08-25 — the original bare probe_success == 0 fired critical all day the WoL box slept and could not be told apart from a real verify failure.

  • Libation GUI container (Libation-mdh) — retired 2026-08-10. Its nginx crash-looped on a missing /var/log/nginx after the 06:00 appdata backup cycled it, serving 502 while docker still called the container "running". Nothing depended on it. libate.mdhmedia.uk still works — it is now served by the headless stack's libation-login helper, alongside libation.mdhmedia.uk.

  • CT 112 (Caddy) root filesystem — was 88% full (740 MB free of 6 GB) and falling ~300 MB/week. Cleared 2026-08-09: 88% → 32% (4.2 GB free). It was all regenerable cache — 2.0 GB of apt .debs plus ~1.8 GB of Go module/build cache left behind by successive xcaddy builds. build-caddy.sh now reclaims both at the end of a build, so it cannot silently regrow.

Parked on purpose

Not forgotten, not open — listed so nobody "fixes" them.

  • kicad and grampsweb are stopped deliberately (low use). They are excluded from the blackbox-lan-services probes for that reason; re-add each target when its service comes back.
  • NAS rebuild — on hold while component prices are inflated; board and chassis already bought.

  • Single alert delivery channel — all four Grafana notification policies route to the one telegram-jarvis contact point. Considered 2026-08-27 and deliberately left as-is: BetterStack already alerts independently by email and push from outside the network, and it survives the homelab going down entirely — which Grafana, running on CT 101, cannot. The 08:00 digest is a third signal. Neither a second Grafana contact point nor a full Alertmanager stack adds a channel that survives the failure that matters most. Do not "fix" this.

  • Authentik — staged, trigger-gated. Move only on a real need (groups, enforced MFA, native OIDC roles, ~10+ users), not for its own sake.