Changelog¶

This page tracks significant changes to the homelab infrastructure.

2026¶

2026-09-08 - Jarvis (VM 121) migrated pve1 → pve2 and resized¶

Category: Infrastructure

Summary: The Hermes agent VM was moved off the primary node onto pve2, the heavy node, and resized in the same pass — 3GB/32GB on pve1 becomes 8GB RAM, 8 cores, 48GB on local-zfs-pve2. pve1 is a 16GB i3-4005U with a 2GB-capped ARC; pve2 has 72GB and two X5670s, so an agent that runs builds and sandboxed container tasks belongs there.

Details:

• Offline migration, and it took two attempts. The first (15:55) aborted; the VM was restarted on pve1 at 15:56, a fresh vzdump taken at 16:08 and a stale snapshot removed at 16:21 before the retry. The second migration at 16:23 completed OK and the VM started on pve2 at 16:35. Taking the backup between the failed and successful attempts is the right instinct — the retry then had a known-good restore point behind it.
• Backups never lapsed. The vzdump jobs select by vmid with no node restriction, so pve2 picked up the guest it now owns: VM 121 backed up to pve3-pbs at 19:15 the same evening, status OK, and again the following night.
• The repo had recorded pve1 in six places (AGENTS.md ×2, README.md ×2, PROJECT.md, hermes-vm/README.md) plus this docs site. The stale node also broke the homepage Jarvis tile, which asked pve1 for VM 121, got nothing back and rendered a permanent UNKNOWN status badge — fixed in the same pass.
• onboot=1 is preserved on pve2, so the agent's always-on data-protection crons still come back by themselves after a host reboot. qm stop 121 remains the kill switch, but it is now a pve2 command — a habit worth updating.

2026-08-26 - PBS verify probe was watching the wrong task type¶

Category: Monitoring

Summary: The blackbox-pbs-verify probe filtered on typefilter=verify (manual re-verify tasks) and had been pinned to a stale FAILED manual verify from 2026-08-11 for months, never seeing the nightly scheduled verify jobs (which pass). So PbsVerificationFailed was a permanent digest-muted false-alarm — and the 2026-08-25 deadman couldn't clear it because the probe never read OK. Corrected the scrape filter to typefilter=verificationjob and the module regex to worker_type:"verificationjob", verified against the live API.

Details:

• Found while confirming the 2026-08-25 deadman fix end-to-end. Tonight's cycle ran clean — WoL woke pve3 unattended in 75s (first real hands-off wake since the BIOS fix), backup done, verify OK at 20:38, GC OK, offsite reconcile ran, pve3 slept 21:06 — yet the probe still read 0 with an HTTP 200, i.e. the content check was failing on a genuinely-OK verify.
• Root cause: typefilter=verify matches worker_type:"verify" (manual) tasks, not the nightly worker_type:"verificationjob" scheduled job. The latest manual verify was the 2026-08-11 corruption re-check (status: verification failed), so the probe was stuck at 0 permanently.
• The 26h max_over_time deadman is kept (it is correct); this points it at the right task. The alert clears when PBS next wakes and the probe reads the latest OK verificationjob. The scrape URL and the module regex must stay in lockstep on the task type.

2026-08-25 - Corrupt PBS snapshots pruned; PbsVerificationFailed reworked to a deadman¶

Category: Operations / Monitoring

Summary: Pruned the last 5 bad-RAM-era corrupt PBS snapshots that were failing the nightly verify, and fixed the PbsVerificationFailed alert so it stops crying wolf while the WoL-box PBS is asleep.

Details:

• Corrupt snapshots pruned: the 5 remaining failed-verify snapshots (ct/100, ct/113, ct/114 x2, vm/121, all dated 2026-07-26 to 08-10 — the pre-migration / bad-RAM era) were forgotten from the PBS datastore and garbage-collected; 0 failed snapshots remain. The offsite reconcile stamp was aged so that evening's rclone sync sweeps the now-orphaned chunks off the Hetzner copy.
• PbsVerificationFailed reworked to a deadman: the expr changed from a bare probe_success{job="blackbox-pbs-verify"} == 0 to max_over_time(probe_success{job="blackbox-pbs-verify"}[26h]) == 0, mirroring PbsBackupWindowMissed. The old form fired critical ~22 h/day whenever PBS was WoL-asleep — probe_success is 0 both for a real verify failure and for an unreachable box, so a genuine failure was indistinguishable from the daily noise. The deadman fires only if no verify has passed at all in 26 h, and the prior night's pass staying in-window also covers the ~35 min nightly run.
• Grafana dashboards: the Command Center's Endpoints Down / Endpoint Status / Availability Timeline panels now exclude the WoL-box PBS probes (blackbox-pbs*), so a sleeping PBS no longer paints them red; the Ops Jobs job-schedule table was corrected to the real weekly Sun 03:30 / keep-last 1 vzdump policy.
• CT backup schedule doc (#313): corrected from "daily 02:30" to the actual "weekly Sun 03:30" across AGENTS.md and the docs.

2026-08-24 - pve3 WoL fixed, and Jarvis Phase 11 beyond-VM hardening¶

Category: Operations / Security

Summary: Fixed pve3's five-night WoL failure at the BIOS, and applied the Phase 11 "beyond-VM" hardening pass to the Jarvis VM (VM 121).

Details:

• pve3 WoL fixed: the 2026-08-19 RAM-swap CMOS reset had cleared the Gigabyte Z68XP-UD3 power settings. Re-armed in Power Management Setup (ErP/EuP Ready = Disabled, Wake on LAN = Enabled); a full sleep -> magic-packet -> wake round-trip verified (~5 s). The multi-night backup gap was caught up with a manual vzdump of all 9 pve1 guests. Closes the corresponding Open Items investigation.
• DOCKER-USER firewall guard (#302): the Arcane agent's :3553 on VM 121 was reachable from any 192.168.0.0/22 host because Docker's NAT bypasses ufw. A boot-persistent arcane-agent-fw guard (stacks/management/arcane-agent/deploy/) now restricts :3553 to the CT 101 manager (192.168.1.241) only.
• Credential hardening: the PBS API token api@pbs!jarvis was scoped from Audit-on-/ down to DatastoreAudit-on-/datastore/main; the over-scoped classic GitHub PAT was removed from the agent .env (gh now uses stored OAuth); the ARCANE_AGENT_TOKEN was rotated; an off-VM encrypted credential backup was made.
• Containment decision: the VM boundary plus qm stop 121 is the accepted containment, so the jarvis user deliberately keeps passwordless sudo and docker-group membership — the audit's two "CRITICAL" findings are accepted trade-offs, not open items. Intra-VM hardening was dropped; HA-token scoping is deferred to the HAOS cutover; the Grafana SA token stays Editor (Grafana OSS cannot scope below Editor).

2026-08-23 - Deployment batch and pve3 memtest completed¶

Category: Operations / Hardware

Summary: Reconciled the 2026-08-22/23 deployment batch: 11 PRs were merged, the daily backup digest and pve2 R710 fan controller are live, the local iDRAC6 KVM helper is repo-managed, and the Arcane manager is running v2.8.1. pve3's memtest passed after faulty RAM was identified and removed.

Details:

• Daily digest live: the backup-digest Arcane git-sync is registered and running on CT 101, posting its backup + host-health summary to Telegram at 08:00 Europe/London.
• pve2 host tooling: proxmox/fan-control/ is installed as a systemd service using in-band IPMI and a fail-safe thermostatic curve. The separate proxmox/idrac-kvm/ helper runs on the operator laptop on demand; it is not a homelab-deployed stack.
• Arcane: the hand-deployed CT 101 manager was upgraded from v2.7.0 to v2.8.1. The deliberately parked Gramps Web and KiCad git-syncs now use autoSync=false, so Dependabot merges cannot un-park them.
• pve3 memtest passed: the retained RAM sticks passed individually and together on 2026-08-22; old sticks that threw errors were removed. Bad RAM is the likely cause of the 2026-08-03 zil_commit oops and the prime suspect for the PBS verify-job corruption, though that link is not yet proven.

2026-08-22 - Daily backup + host-health digest to Telegram¶

Category: Monitoring

Summary: Added a new Arcane git-sync stack, backup-digest (the 15th), that posts one compact plain-text summary a day (08:00 Europe/London) to Telegram. It adds no new detection - the alerts already exist - but makes silence unambiguous: a firing critical that nobody read went unacted for ~10 days because the alert stream is easy to tune out, so a proactive morning digest turns its own absence into a signal and puts the backup state in front of a human every day.

Details:

• What it reports (all from the read-only Prometheus HTTP API, a single always-on source): always-on host node-metric freshness (time() - node_time_seconds, with pve3 asleep treated as normal), firing ALERTS (minus the muted set), the PBS backup window + verify state, and pve1 boot-SSD life (SSD_Life_Left). If Prometheus itself is unreachable the digest still sends - saying so, because that is itself a signal.
• WoL-asleep nuance: at 08:00 pve3 (the PBS box) is powered off, so probe_success{job="blackbox-pbs*"} reads 0 normally and is indistinguishable from a real failure by the live value. The backup section is therefore time-windowed - "did last night's window run (max_over_time[26h]) and verify pass (max_over_time[14h])" - and PbsVerificationFailed is muted from the raw alert list (it false-fires ~22 h/day while PBS sleeps; the windowed verify line carries the real state).
• Not built (documented TODO): a per-guest "time since last good backup" line. It needs a node_exporter textfile collector, which is not enabled on CT 101 today (neither Alloy config lists textfile) - so it was left as a clear TODO in the stack README rather than standing up new collector infra in this MVP. See gap 3 in Monitoring & Dashboards.
• Deployed like every other git-sync (bridge network - it reaches Prometheus on the LAN IP and Telegram over the internet, so unlike wol-pbs it needs no host networking); the bot token lives in the Arcane project .env, never in git. See stacks/management/backup-digest/README.md.

2026-08-19 - pve3 RAM 8GB -> 16GB, and a five-night WoL backup gap¶

Category: Hardware / Incident

Summary: pve3's memory was doubled to 16 GB (all four DIMM slots now filled). Investigating why the box was still powered on that evening surfaced a separate, larger problem: pve3 had failed to wake for five consecutive nights, so no PBS backup ran between 2026-08-13 and 2026-08-19.

Details:

• RAM upgrade: 8 GB -> 16 GB, 4x 4 GB DDR3-1600 at 1600 MT/s, banks A0-A3 - every slot on the board is now populated, so the next step up would mean replacing modules rather than adding them (board max 32 GB).
• The host's ZFS ARC is still capped at the 8 GB-era 779 MiB (zfs_arc_max=816840704 in /etc/modprobe.d/zfs.conf). Left alone deliberately - raising it is a live-system change on load-marginal hardware and should follow a memtest, not precede one.
• memtest is now overdue, not merely standing. The 2026-08-03 kernel oops already put old non-ECC DDR3 under suspicion; there are now four untested modules in the same box.
• The five-night gap: the orchestrator shut pve3 down cleanly on 08-13 at 21:20 and sent its magic packet at 19:00 on the 14th, 15th, 16th, 17th and 18th - each one logged ERROR: PBS did not answer within 420s. last -x shows no boot in that window at all; the box came back only because it was powered on by hand at 16:54 on the 19th for the RAM. Newest snapshot for every guest before then was 2026-08-13T18:15Z; host/holly had not uploaded since 2026-08-11T19:35Z.
• What covered it: pve1's local vzdump jobs - the same keep-last-2 floor that covered the 2026-08-11 verify-job corruption. Two saves in nine days; Backup Phase 2 remains held.

• Ruled out on the pve3 side: the WoL flag is armed (Wake-on: g), pve3-wol-enable.service is enabled with both ExecStart and ExecStop re-applying it, and the link is up at 1 Gb. See Open Items for the remaining candidates and the next test.

• Monitoring did its job; the loop did not close. PbsBackupWindowMissed fired critical continuously from 08-14 23:00 UTC to 08-19 15:00 UTC (4 d 16 h) - at the tiered 6 h critical repeat, roughly 19 Telegram messages. The alert was correct, timely and ignored.

2026-08-11 - PBS backup integrity incident, RAM fix, zvol cleanup¶

Category: Incident / Infrastructure

Summary: A corrected PBS re-verify found 13 backups had genuinely failed verification, including the two most operationally critical guests (ct/101, vm/121) - PBS had no monitoring of its own, so this was invisible until checked by hand. Fixed the same session: fresh backups taken, a new alert deployed so it can't happen silently again, pve1's real RAM-pressure root cause found and fixed, and the old PBS rollback zvol destroyed.

Details:

• RAM pressure on pve1 traced to an uncapped ZFS ARC using up to ~14.5 GiB of the 16 GiB host (the VMs/CTs were all reasonably sized for their actual usage - not the cause). Capped to 2 GiB via /etc/modprobe.d/zfs.conf (zfs_arc_max=2147483648), applied live then persisted with update-initramfs -u -k all. Available memory 1.6 GiB -> 4.3 GiB; HighMemoryUsage cleared. Low-risk LXC memory-ceiling trims alongside it: CT 114 (mariadb) 1024->384M, CT 120 (adguard) 1024->384M, CT 122 (netboot-pxe) 2048->512M - hygiene, not the fix.
• Old PBS zvol destroyed - the ext4-in-a-zvol rollback kept after the 2026-08-04 datastore migration, confirmed unmounted/unreferenced first. ~55G reclaimed on pve3's bak pool.
• PBS verify-job corruption, found the hard way: a first verify pass used --ignore-verified true (the default) while trying to force a full recheck - that flag actually skips backups already marked verified, so it silently missed the two most critical snapshots. Re-ran with --ignore-verified false, which genuinely rechecks everything: 13 failures, confirmed via the raw PBS log (a background monitor's summary had truncated the count to 2 - caught before reporting it). Took fresh, fully-verified backups of ct/101 and vm/121 immediately to close the risk window.
• New monitoring: PBS shipped no telemetry of its own (no Alloy agent; it sleeps ~22 h/day). Built a blackbox HTTP probe against the PBS API directly (pbs_verify_ok module, blackbox-pbs-verify scrape job) backing a new PbsVerificationFailed alert (PR #263) - a dedicated least-privilege PBS token (api@pbs!blackbox), tested end-to-end against both a real success and a real failure response before deploying. Confirmed live: it correctly reported the still-bad verify state (probe_success=0), not a false positive.
• Backup Phase 2 (retiring pve1's local vzdump floor) was under consideration this session - this incident is exactly why it's held, not just deferred: the local floor is what closed the gap while the PBS-only leg was silently bad. See Open Items for the still-open root-cause question.

See Proxmox -> Backups for the full incident writeup and current Phase 2 status.

2026-08-11 - Holly rebalance: ~460 GB moved, a self-caught misdiagnosis¶

Category: Maintenance

Summary: Moved four TV-show folders off the two fullest disks (disk1, disk4) onto disk3. Fully verified: array healthy throughout, every byte accounted for, zero data loss - but a mv pitfall required a manual mid-operation fix, and an early conclusion about a "pre-existing junk folder" was wrong and had to be corrected before writing anything back.

Details:

• Unbalance was installed for this but its scatter/gather engine turned out to be an undocumented WebSocket protocol, not a REST API - reverse-engineering it live against production data, hours after a disk-drop incident, was not a reasonable risk. Did the equivalent operation directly instead: rsync to a temp name -> independent verify -> delete source only after verification -> rename into place. Array health re-checked before and after every item.
• Result: disk1 219 GB -> 371 GB free, disk3 1.3 TB -> 852 GB free, disk4 222 GB -> 503 GB free. Union file count for the affected show unchanged (26) throughout - nothing added, nothing lost, only relocated.
• The bug: mv SRC DST where DST already exists as a directory nests SRC inside it instead of merging, silently, with no error, if the nested name doesn't collide with anything. This happened on the SECOND House-of-the-Dragon move without any error being printed - it only surfaced when the fourth item hit the same nested name, now occupied, and errored loudly (File exists).
• The self-correction: first read of the nested folder concluded it was old pre-existing library clutter, based on file mtimes from 2022-2024. That was wrong. A mv preserves the original mtime - old dates on episode files mean the episodes are old, not that they arrived at that path long ago. Reconciling by name+size against what each rsync job actually reported showed the nested folder's 6 files summed to exactly one job's reported total, byte-for-byte. It was verified, moved content sitting in the wrong spot, not junk. Caught before anything was written back, by insisting on an exact byte reconciliation rather than accepting a first plausible explanation.

See the Unraid docs for the full write-up and the merge-not-nest fix for any future scripted move.

2026-08-10 (late) - Holly: two array disks vanished, nothing alerted¶

Category: Incident

Summary: disk1 and disk2 disappeared from Holly entirely. The array kept serving for 1h45m with no alert and no UI indication. Recovered in full by a VM restart - the drives were never faulty.

Details:

• Found by accident. While auditing share settings for a rebalance, ls /mnt/user/onedrive returned Input/output error. That led to XFS metadata I/O errors (error 5) on md1p1/md2p1, and then to the real finding: /dev/sdc and /dev/sdd did not exist. smartctl could not open them.
• Unraid showed DISK_OK for both throughout. With a zero-parity array there is nothing to disable a disk against, so the UI cannot report it. Error counters had reached 8,209 (disk1) and 7,037 (disk2).
• Cause: a vfio-pci controller reset. pve2's dmesg shows resets on the passed-through HBAs (03:00.0, 05:00.0, 06:00.0/.1). The drives dropped inside the VM; the host never sees them (passthrough by design). Two drives failing at once would be extraordinary - a controller reset dropping two is ordinary.
• Recovery: stop Docker -> qm shutdown 110 (graceful ACPI, so Unraid stops the array cleanly) -> qm start 110. Both disks returned with matching serials; all six DISK_OK errors=0, /mnt/user/onedrive lists its 23 entries again, 0 XFS errors since boot, all 12 containers back.
• All six disks SMART PASSED - 0 reallocated, 0 pending, 0 uncorrectable, 0 CRC. The two that vanished are at 29,024 and 54,803 power-on hours.
• A reboot backgrounded over SSH silently did nothing - it died with the session and uptime proved the box never restarted. Verify a reboot happened rather than assuming the command took.

2026-08-10 - docker-pve1/ dissolved: stacks/, edge/, host-agents/¶

Category: Repository

Summary: The host-named top-level folder is gone. Everything under it moved to a folder named for what it is, not where it runs.

Why: the prefix had stopped carrying information - all 14 Arcane stacks run on the same host (CT 101), so docker-pve1/ distinguished nothing - and it was actively wrong for caddy and crowdsec, which run on CT 112.

Details:

• stacks/ = docker compose stacks. Category folders kept deliberately: they say what an app is for at a glance.
• edge/ = CT 112, native systemd, no compose. Caddy and CrowdSec are one unit - CrowdSec parses Caddy's access logs and its bouncer is compiled into the Caddy binary, so they cannot be upgraded independently.
• host-agents/ = curl'd installers, no compose at all. It had been sitting inside the grafana stack folder, so syncDirectory: true was shipping it onto CT 101 as dead weight.
• docker-pve1 the string survives untouched as a metric label. It is CT 101's actual hostname, so it is the Prometheus instance and Loki host label. The rewrite used an anchored mapping (docker-pve1/<category>) that can never match the bare token, and a test now guards it - renaming it would split every historical series and blank the fleet dashboards, silently.
• 108 files moved as git renames, so history follows them.
• The out-of-repo couplings (13 Arcane composePaths and CT 112's installed update-caddyfile.sh) were repointed as a deliberate cutover, not left to chance - see the deployment notes.

2026-08-10 - docs site lifted to a top-level docs/, dead folders archived¶

Category: Repository

Summary: The documentation site moved from docker-pve1/documentation/zensical/ to a top-level docs/, and four retired or never-deployed folders moved into a new archive/.

Why: the repo was organised by deployment host at the top level, so things got filed by where they run rather than what they are. That prefix had stopped carrying information - all 14 Arcane stacks run on the same host (CT 101), so docker-pve1/ distinguished nothing, and it was actively wrong for caddy and crowdsec, which run on CT 112. The docs site was the clearest case: it is content that happens to be served by a container, and nothing ties it to pve1.

Details:

• docs/content/, not docs/docs/. zensical.toml now sets docs_dir = "content". Verified against zensical 0.0.53 before committing - build exit 0, 20 pages, nav intact. site_dir stays at its default ./site; it must not live inside docs_dir.
• Public URLs are unchanged - the built paths are still /services/logging/ etc. Nothing external breaks.
• Archived (nothing here deploys): arcane-trial/, authentik/ (staged, never deployed), arcane-agent-ct101/ (retired 2026-07-21; it points at the decommissioned CT 100 manager, so it is a record and not a usable template), and the retired neteng-profile/. See archive/README.md.
• The neteng profile directory is still live on VM 121 and was deliberately left alone - only the repo copy moved. Removing VM state is a separate, backed-up change.
• Deleted mcp-servers/readarr/: an abandoned scaffold with a venv and a populated .env but no server.py, unregistered in .mcp.json. 78 MB and a live credential file for a server that does not exist.
• Fixed a dead docs anchor the build surfaced (network.md pointed at proxmox.md#pve3-servicesutility, which does not exist).
• 15 broken cross-references were found by the regression tests added earlier the same day, not by reading - which is exactly what they were for.

2026-08-10 - uptime.mdhmedia.uk served directly by BetterStack¶

Category: Monitoring

Summary: The homelab status page moved off the interim Caddy reverse-proxy onto a dedicated DNS record, so its traffic no longer touches the homelab at all.

Details:

• Before: uptime.mdhmedia.uk was a reverse_proxy block on CT 112 that rewrote the Host header to mdhmedia.betteruptime.com. That worked, but it routed the status page through the very infrastructure it reports on - if CT 112 or the home connection went down, so did the page saying so.
• Now: CNAME uptime.mdhmedia.uk -> statuspage.betteruptime.com, DNS-only (not Cloudflare-proxied). A specific record beats the *.mdhmedia.uk wildcard, so no wildcard surgery was needed - which was the original reason for the proxy workaround. BetterStack issued its own Let's Encrypt certificate (CN=uptime.mdhmedia.uk) automatically once the record resolved.
• Verified: resolves to statuspage.betteruptime.com (167.235.220.62), HTTPS 200 with a valid cert. The uptime block was removed from the Caddyfile only after that check passed - removing it first would have taken the page down.
• Do not re-add a Caddyfile block for this host; it would be dead config that never sees a request. A comment in the Caddyfile records why.
• The page stays public on purpose: a status page behind the SSO gate cannot report that the SSO gate is down.
• All 9 DNS records for the zone were backed up to /root/cf-dns-mdhmedia.uk.bak-2026-08-10-140150.json on CT 112 before the change.

2026-08-10 - libate 502: Libation GUI retired, its URL kept¶

Category: Services

Summary: libate.mdhmedia.uk started returning 502. The GUI container behind it was already redundant, so it was retired rather than repaired - and the hostname repointed at the headless stack's own UI so muscle memory still works.

Details:

• Cause: nginx inside Holly's Libation-mdh container was crash-looping once a second on open() "/var/log/nginx/error.log" failed (2: No such file or directory) - the directory did not exist in the container. Docker's proxy still held :3001, so Caddy connected, got nothing, and returned 502.
• Why nothing alerted: docker reported the container running with RestartCount=0. Only the web server inside was dead - the same "process-up != working" shape as the CrowdSec outage the day before.
• It broke at 06:09, when Holly's appdata backup stops and restarts containers - but intermittently, not every backup: Caddy logged 502s for this host only on 2026-08-06 and 2026-08-10, serving 200s in between. A startup race in an unpinned mdhmatt/libate:latest image.
• Retired instead of fixed. The headless libation-sync stack does the work (cycles completed 03:39 and 09:40 that morning) and its libation-login helper already covers the GUI's one unique function, adding Audible accounts. Nothing depended on the container; it was already a listed retirement candidate.
• The URL was kept, deliberately: libate.mdhmedia.uk is the name typed out of habit, so it is served from the same block as libation.mdhmedia.uk (192.168.1.241:8099, SSO-gated) rather than removed or redirected. Same page, nothing to relearn.
• The Unraid dockerMan template stays in place, so the GUI is a few clicks away if it is ever wanted.

2026-08-09 - MCP server fleet ported to SDK 2.0¶

Category: Tooling

Summary: mcp 2.0.0 removed mcp.server.fastmcp, which all nine Python MCP servers imported. Ported them to the new MCPServer API and merged the dependency bump with the code, since either alone would have broken the fleet.

Details:

• Checked the real package, not the release notes: MCPServer keeps the same annotations= kwarg on .tool() and .run() still defaults to stdio, so the change is the import plus the constructor per file - far smaller than the pyarr-6 rewrite it resembled on paper. mcp.types.ToolAnnotations still resolves and was left alone.
• Why #190 could not be merged on its own: it bumped only requirements.in/constraints.txt. Merging the lock without the code port would have broken all nine servers at import - the reason it sat open.
• Verified with a real stdio handshake (initialize -> tools/list) against every server under a clean mcp-2.0.0 venv: 9/9 healthy with unchanged tool counts. The repo's own AST contract test needed updating too - it keyed on a FastMCP() call, so post-rename it matched nothing and reported an empty name list for every server.
• Operational note: the servers run from per-directory .venvs, so the venvs must be reinstalled after pulling this change or they fail at import.

2026-08-09 - CrowdSec had been BLIND for 9 days; found while adding its first alert rules¶

Category: Security / Monitoring

Summary: Writing the missing CrowdSec alerts turned up the outage those alerts were meant to catch: CrowdSec stopped parsing Caddy's access log around 2026-07-31 and had read zero lines since — while the service was active, the bouncer kept polling, and every dashboard stayed green.

Details:

• Symptom that gave it away: sum(increase(cs_parser_hits_total[1h])) was 0 while cs_lapi_bouncer_requests_total showed a healthy 240/hour. Old decisions were still being enforced; nothing new could ever be detected.
• Root cause: the process held file descriptors on two deleted rotated logs (newest access-2026-07-31T11-41-29.950-size.log) and had no descriptor on the live access.log. Caddy rotates its own log (roll_size), and CrowdSec never followed the rename - a known upstream issue (crowdsec#2449). Contributing factor: cscli setup had generated a second, overlapping acquisition using the glob /var/log/caddy/*.log, which also matches the rotated files - the file's own header warns about double acquisition.
• Fix: removed the generated glob acquisition (kept aside, not deleted), added force_inotify: true to the repo's acquis-caddy.yaml so the directory is watched rather than a single inode, validated with crowdsec -t, restarted. Verified: the fd now points at the live file, and 284 lines flowed through the full parser chain within minutes. install-crowdsec.sh now removes the duplicate so a rebuild cannot reintroduce it.
• Alerts so it cannot hide again (the first CrowdSec rules ever): CrowdSecParsingStalled (0 parsed lines in an hour = the WAF is blind), CrowdSecBouncerStalled (detection fine but enforcement silently off) and both alerting on work done, never on process-up - up{job="crowdsec"} was green throughout the outage.
• A third rule was written, shipped and removed the same hour - worth recording. CrowdSecBanSpike was calibrated against a measured baseline of "0 overflows in 24h"... which was measured while CrowdSec was blind. The moment parsing was restored the real rate appeared - ~55 overflows/hour of ordinary internet scanning - and the rule fired permanently. Removed rather than left as noise. The lesson: measure first is not enough if the thing you measure is broken; and bans are the system working, not an incident.
• Holly app probes, guarded: new blackbox-holly-apps job (audiobookshelf, qbittorrent, nzbget, radarr, sonarr, prowlarr, ombi, plex) feeding a new HollyAppDown that fires only when Holly itself is alive - a deliberate NAS shutdown stays silent instead of producing 8 criticals.
• Parity banner reviewed, not changed: the roadmap listed the Holly parity stat's vector(1) as a defect. It is intentional - a constant driving a red "NO PARITY DISK" banner, accurate for a zero-parity array. Noted that it must be edited by hand if a parity disk is ever assigned.

2026-08-09 - Kavita retired: one reader (Audiobookshelf) for audiobooks, ebooks and comics¶

Category: Services

Summary: The Kavita stack, deployed 2026-08-07, was retired two days later. Audiobookshelf already does everything it was brought in for - and the feature that actually matters here is Send-to-Kindle, which ABS has.

Details:

• The premise was wrong. Kavita's README claimed there was "no way to read anything" beforehand. ABS reads EPUB, PDF, CBZ and CBR, and probing the live server settled the one open question: /api/opds returns 401, not 404 - ABS serves OPDS natively too. The real problem in early August was that books sat loose in the audiobook share and comics were unextracted RAR archives; the book-router and mount chain fixed that, and both are reader-agnostic.
• What was actually lost by keeping two apps: a second container to patch, a second backup surface, and a books.mdhmedia.uk route carrying a bespoke split-auth exception (UI behind SSO, /api/opds* open) that existed only because a fresh Kavita hands admin to whoever registers first.
• Cost of retiring: nil. Kavita had 0 libraries and 0 series - it was never configured. The decision got cheaper by being made now rather than after reading progress accumulated.
• Changes: book-router's scan trigger repointed from Kavita to the ABS books library (ABS_BOOKS_LIBRARY_ID); stack folder, Caddy route, Dependabot entry and the blackbox-lan-services probe target removed; docs rewritten around ABS. Stack count 15 -> 14.
• ABS side wired the same day, no browser needed. The /mnt/user/Media/Books -> /books mount was added to the Audiobookshelf container by editing its dockerMan XML template and running Unraid's own rebuild_container - the CLI equivalent of Edit -> Apply, validated by diffing the generated docker create command before/after (exactly one -v line changed). The Books library was then created over the ABS API and scanned: 10 items indexed. Method captured as the unraid-docker skill. Gotchas found: Unraid has no python at all (PHP only), and rebuild_container leaves a container stopped unless it is listed in /var/lib/docker/unraid-autostart.
• Still needs a human: SMTP credentials + the Kindle address in ABS, and the sender approved in Amazon's Personal Document list.
• Kept as a real trade-off, not a dismissal: Kavita genuinely models comic series/volume/issue structure better than ABS, which treats each CBZ as a standalone book. With 6 comic files that bought nothing; the compose is in git history if a comic-native reader is ever wanted.

2026-08-09 - Backlog day: deps batch, LAN probes, Jarvis single-profile¶

Category: Maintenance / Monitoring / Automation

Summary: Cleared the open-work backlog: 11 Dependabot PRs merged + verified, direct LAN tcp probes added for the always-on SSO-gated services, and Jarvis consolidated back to a single Hermes profile. (kicad + grampsweb were briefly revived in error - they are parked on purpose - and re-stopped same day.)

Details:

• Dependency batch: zensical 0.0.52, mealie v3.22.0, grampsweb 26.7.1 + redis 8.10.0, cadvisor v0.55.1, @hono/node-server 2.0.12 (grafana-stack tests pass), MCP SDK 1.30.0 npm, and four pip lock bumps - all deployed via autoSync and verified on their endpoints. mcp 2.0.0 stays open: its CI failure was just the stale pre-#217 server-count test, rebase requested, local compat test still owed before merge.
• kicad + grampsweb: revived in error, re-stopped. Both had been Exited since 2026-07-28; reading that as reboot fallout, they were redeployed (grampsweb straight onto the new images) - the user then clarified both are intentionally parked (low use right now), so they were stopped again, keeping the updated images for whenever they return. The useful lesson stands: through Caddy's forward_auth (302 before the upstream), a parked service and a silently-failed one look identical from outside - which is exactly why the probe job below exists, and why it only lists always-on services. Restart either via Arcane when wanted (re-add its probe target in the same change).
• New blackbox-lan-services probe job: tcp_connect directly to the always-on gated services - tinyauth :9092, grafana :3300, kavita :5000, audiobookrequest :8000, homepage :3080 and wgadmin :10086 - feeding the existing EndpointDown alert, so a genuinely failed always-on service now pages in 3 minutes. Monitoring doc brought back in line with reality (pve-exporter + cAdvisor marked done, gap table refreshed).
• Jarvis back to one profile: the neteng experiment's crons (pins intact), credentials, SOUL.md/HOMELAB.md, memories and workspace all ported to the base profile; neteng's own cron copies paused; stray user-level gateway units disabled (one had fought the system unit through 12,526 failed restarts since 2026-08-03); gateway healthy under systemd, watchdog test-run green. Claude + Codex auth verified on base; Nous needs a portal re-login (refresh token rejected 2026-08-08, predating the reset).
• Kavita follow-up found: the container is healthy but no libraries were ever created (0 series) and no user API key exists yet - creating Comics -> /books/Comics + Books -> /books/Ebooks and generating an API key are admin-UI actions; the key then gets wired into the libation-sync .env so the book-router's scan trigger stops no-opping.

2026-08-07 - Books & comics pipeline: Kavita, auto-filing, Readarr retired¶

Category: Services / Storage / Automation

Summary: Ebooks and comics went from "loose files in the audiobook share, comics stuck inside unextracted RAR archives, nothing to read them with" to a proper library with a reader and automatic filing. Also added the second Audible account and retired the Readarr MCP.

Details:

• New library: Holly Media/Books/{Ebooks,Comics}, reached from CT 101 by the same CIFS->bind chain as the audiobooks (hotplugged with pct set, no container restart). Unraid share dirs must be 777 or the SMB user cannot write.
• Kavita (15th Arcane stack, books.mdhmedia.uk) reads both comics and ebooks: iPad via the web reader or OPDS, Kindle via Send to Device. Proxy auth is split - the UI is SSO-gated, /api/opds* is open - because a fresh Kavita gives admin to whoever registers first, and bots probed the new hostname within seconds of its certificate appearing in the CT logs.
• book-router (in the libation-sync stack) watches new books/comics download-client categories, extracts archives, packs loose images as .cbz, copies into the library (never moves - torrents keep seeding) and triggers a Kavita scan. Categories were created through the qBittorrent MCP, which gained a write-gated create_category tool; NZBGet's went into nzbget.conf.
• Libation file-type routing: it cannot split output by type (one Books location, and FileTemplate covers "the saved pdf and audio files"), so the post-sync hook now moves .pdf/.epub/.mobi into Books/Ebooks - 30 such files were rescued from the audiobook tree. ASIN naming was deliberately left alone: Libation needs the id in the file or folder name to detect already-downloaded books.
• Migration: all 6 Sin City volumes extracted and filed (they were RAR5 nested inside ZIPs - Alpine's p7zip and official 7zip lack the RAR codec, so the router uses unrar-free).
• Second Audible account added and scanned: 45 titles, no overlap with the existing 65, ~37 GB liberated. The login helper now shows per-account Books / Downloaded / Pending counts.
• Readarr retired - archived upstream June 2025; the MCP server and all references removed (MCP fleet: eleven -> ten).

2026-08-04 - PBS datastore rebuilt on native ZFS, offsite reconcile, Holly stagger¶

Category: Infrastructure / Storage / Backups / Monitoring

Summary: A 2026-08-03 kernel oops on pve3 (a ZFS zil_commit memcpy page fault under sustained zvol writes) wedged the bak pool mid-window and forced the follow-through on the load-marginal-hardware caveat: the PBS datastore was re-architected from ext4-in-a-zvol to a native ZFS dataset owned by the PBS VM, the offsite gained a guarded ~monthly reconcile, Holly's upload was staggered to 20:35, and the two backup-window Loki alerts were made unmistakable. (PRs #194/#195 + the datastore migration.)

Details:

• Datastore on native ZFS (2026-08-04): datastore main now lives on the ZFS dataset bak/pbs-datastore. The 500G disk is passed whole into PBS VM 102, which imports the bak pool itself (ARC capped at 1G); the old ext4-in-a-zvol layout is retired (the zvol kept as a short-term rollback) and the pve3 host storages pve3-bak/pve3-backup were deleted. A final verify of the whole datastore came back OK, and the VM's pool auto-import is reboot-proven.
• Cause: the 2026-08-03 kernel oops (ZFS zil_commit memcpy page fault) under sustained zvol writes wedged the pool — more evidence pve3's old non-ECC hardware is load-marginal (a memtest on pve3 remains the standing hardware follow-up).
• Dual-import scare, fully recovered: during the migration pvestatd re-imported the freshly-exported pool on the host because the storage entry was still enabled — a transient dual-import, recovered without loss. Lesson: disable the storage entry first, then export.
• Offsite reconcile: the nightly rclone copy never deletes (a one-way valve), so pruned chunks used to accumulate on Hetzner forever. A new ~monthly guarded reconcile runs rclone sync with a --max-delete tripwire and a dated --backup-dir archive; reconcile archives are purged after 60 days.
• Holly upload staggered 19:25 → 20:35 (PR #195): pve3 threw transient chunk-digest errors under the concurrent evening load, so Holly now uploads after the guest-backup storm and the 20:00 verify. The first successful host/holly appdata backups exist; ~10 born-broken guest snapshots from the corrupt Jul-30/31 + Aug-1/2 evenings were removed (clean pre-corruption + current history retained). Every corrupt chunk traced to the old stack under load — nothing written by the new stack was bad.
• Alert clarity (PR #194): the BackupJobFailed and WolPbsCycleProblem summaries were rewritten so the two are unmistakable at a glance.

2026-08-01 - Home Assistant rebuilt as a HAOS VM + Jarvis wired in¶

Category: Infrastructure / Services / Automation

Summary: Rebuilt Home Assistant from scratch as a HAOS VM on pve1 — parallel-running beside the old laptop install until cutover — transplanted the 19 live integrations, moved Zigbee fully to Zigbee2MQTT over a network coordinator, and gave Jarvis scoped Home Assistant API access. (PRs #183–#185; plan + scaffold in haos-vm/.)

Details:

• New VM haos (VMID 123 on pve1, UEFI/q35) running HAOS 18.2 / core 2026.7.4, parallel-running at 192.168.2.120. The old laptop install stays at 192.168.1.240 and keeps the ha.mdhmedia.uk route until cutover.
• Clean rebuild, not a restore: years of dead tweaks (AWS dev-key remote access, the Cloudflare tunnel) were deliberately dropped — remote access stays Caddy + WireGuard.
• 19 integrations transplanted with their auth tokens intact (config-entry transplant over SSH).
• Zigbee is Zigbee2MQTT only, via the SMLIGHT SLZB-MR-series network coordinator at 192.168.1.253 — no USB passthrough. ZHA + the Sonoff USB dongle were dead leftovers and were dropped.
• Alexa and Bluetooth/iBeacon dropped entirely; future voice is HA Assist with an AI conversation agent and/or the Jarvis Hermes HA skill.
• Jarvis has scoped HA API access (PR #185): read any entity, act on lights/switches/scenes only (HA_URL/HA_TOKEN in its profile env).
• HA exposes /api/prometheus — the monitoring-stack scrape job lands at cutover.
• Cutover pending: the Z2M 4-file data move, taking over .240, joining the nightly PBS backup job, and a blackbox probe.
• pve1 RAM trims (2026-08-03): jarvis 4 → 3 GB and haos 3 → 2.5 GB — pve1 has ~5.5 GiB available again.

2026-07-31 - Backup-failure alerting, legacy offsite retirement, log-noise fix¶

Category: Monitoring / Backups

Summary: Closed the backup-visibility gap exposed by the 2026-07-30 missed backup night (a UTC/BST timezone bug, fixed same-day in PRs #174/#175): the nightly WoL window is now alert-covered by Prometheus and Loki rules, the long-superseded pve1 offsite mirror timer was disabled, and the pvestatd log-noise from the sleeping PBS box is dropped at the collector.

Details:

• New alerts: PbsBackupWindowMissed (Prometheus deadman - PBS never reachable in 26 h = the WoL window never happened), BackupJobFailed (Grafana-managed Loki rule on the pve1 journal's vzdump failure lines - it would have paged at 19:15 on the missed night, 50 minutes before the Jarvis watchdog) and WolPbsCycleProblem (ERROR/WARN in the wol-pbs container log), plus a fleet-wide OomKillDetected Loki rule.
• Caddy latency: the Access Logs dashboard gained a per-site p95 latency panel and a "Slow Requests (>2s)" browser.
• Legacy offsite retired: pve1's pve-offsite-sync.timer (superseded by the PBS-side incremental offsite since 2026-07-26) was disabled; units stay installed and the final hetzner-crypt:pve1 mirror is kept as a frozen copy.
• Log noise: pvestatd retries the pve3-pbs storage every ~10 s while pve3 sleeps (~8.6 k identical error lines/day); pve1's Alloy now drops the bare poll-noise line (anchored so the real could not activate storage failure line still ships - the BackupJobFailed rule depends on it).
• Cleanups: stray tender_lewin helper container removed from CT 101; Holly's five empty share directories cleared; Unraid Connect cloud flash backup verified current (appdata backup identified as a genuine gap).
• Holly appdata → PBS backup built (holly/backup/): the CA Appdata Backup plugin had silently vanished in the 2026-07-25 boot-USB failure (last sets July 2025) — appdata had no backup at all. New scheme: 06:00 container-quiesced rsync of appdata to a cache-SSD staging copy, then a 19:25 static proxmox-backup-client upload of staging + /boot to host/holly on PBS inside the WoL wake window; the PBS→Hetzner offsite carries it automatically. (vzdump of VM 110 was a dead end — USB-boot + passthrough disks.)
• Holly cleanups: ~215 GB reclaimed (failed + superseded caab backup sets 88 G, a never-rotated 25 G nzbget.log — NZBGet now set to rotate — and ~39 G of import-verified stale NZB leftovers); 17 missing-data torrent entries removed from qBittorrent (debris of the 2026-07-28 duplicate purge); the 122.8 GB retired-Holly-VM archive purged from the Hetzner Storage Box.

2026-07-30 - Wake-on-LAN PBS, static docs serving, alert overhaul, Jarvis data protection¶

Category: Infrastructure / Services / Monitoring / Automation

Summary: Turned pve3 into a wake-on-LAN backup box woken nightly by a new wol-pbs orchestrator, switched the docs site from a live-reload dev server to a one-shot static build, overhauled alert semantics and Telegram noise after a 43-hour false-alert storm, polished the Grafana dashboards, and gave Jarvis three pinned data-protection cron jobs. (PRs #164-#172.)

Details:

• pve3 is now a wake-on-LAN PBS box — normally powered off. A new wol-pbs container on CT 101 (the 13th Arcane git-sync, docker-pve1/management/wol-pbs/) wakes it daily for an evening window (all times moved from the old 01:30/04:00-05:00): 19:00 WoL wake → 19:15 PBS backup (all 9 pve1 guests: 100, 101, 112, 113, 114, 116, 120, 121, 122) → 19:45 prune → 19:50 GC → 20:00 verify → offsite sync → shutdown once PBS is idle (minimum wake 75 min). (PRs #166-#168.)
• Cluster quorum: pve3 stays in the cluster; pve1's quorum_votes went 2 → 3 (total 5, quorum 3), so pve1 remains quorate alone while pve3 sleeps.
• WoL specifics: magic packet to enp6s0 (r8169); the driver clears the WoL flag at shutdown, so a pve3-wol-enable.service re-arms it; BIOS needs WoL on and ErP/EuP off. PBS VM 102 has onboot=1 (autostarts on wake, ~75 s to API). The orchestrator's SSH key is forced-command restricted on pve3 (only poweroff / start pbs-offsite-sync); PBS access is a read-only token (api@pbs!wolpbs). pve1's separate local vzdump jobs are unchanged (Sun 02:00 VM 121, daily 02:30 CTs, keep-last 2).
• Docs site serves statically (PRs #164/#165). The mdhmedia-docs stack no longer runs the Zensical live-reload dev server: the container runs zensical build once at start, then serves ./site with a threaded python3 -m http.server (entrypoint overridden to [/sbin/tini, --]). The autoheal sidecar was removed — the old "live-reload wedge + autoheal restart" behaviour is obsolete. A failed build serves the last good ./site; a docs merge redeploys via Arcane autoSync → container recreate → rebuild.
• Alert semantics (PR #170): HostDown now excludes blackbox-* probe jobs (a probe target's up reflects the exporter, not the service) and excludes pve3/.28/.30 (expected asleep); a new BlackboxExporterDown alert names the root cause; EndpointDown excludes job=blackbox-pbs.
• Telegram noise policy (PR #171): Grafana groups by [alertname] only, with tiered repeat intervals — critical 6 h / warning 24 h / default 12 h — and the always-firing Watchdog deadman on its own 24 h route. The motivating incident: the blackbox exporter died during the 2026-07-28 pve1 reboot and stayed down 43 h → 11 false HostDown criticals (~120 Telegram messages).
• Grafana G3 (PR #172): renamed "MDH Media - Homelab Overview" → "Fleet Trends" and "MDH Media - Homelab Logs" → "Log Explorer"; 100% of data panels now carry descriptions. Still 11 dashboards in the six provisioned folders, command-center as home, allowUiUpdates: false.
• Jarvis data protection: three cron jobs, all pinned to openai-codex/gpt-5.6-sol (Hermes fail-closes unpinned crons whose inference config drifted — "#44585"), under a new SOUL.md "Reclaim and malware protocol" (the only sanctioned auto-deletes): neteng-malware-sweep (daily 07:30) auto-removes fake video torrents (video category + executable extension + no real video file; cap 5/run), preferring the Radarr/Sonarr queue delete with blocklist=true (blocklist + re-search) over a direct qBittorrent delete (orphans only) — its first run removed three fakes (an Odyssey .exe, two House of the Dragon .scr); neteng-reclaim-scan (Mon 09:00) cross-refs arr imported history vs qBittorrent into workspace/reclaim-pending.tsv, actioned only by a Telegram reply (reclaim delete / list / cancel); neteng-backup-watchdog (daily 20:05, moved from 06:00 — PBS is asleep at 06:00 now) alerts on a guest snapshot >30 h old/missing or datastore >90%, and stays silent on an unreachable* PBS (expected for a WoL box).

2026-07-29 - Holly fully recovered + host-staleness telemetry coverage¶

Category: Services / Storage / Monitoring

Summary: Holly's Docker daemon came back after a VM reboot, a full health audit cleared the array (the real problem is capacity, not disk health), and the Alloy fleet gained the collectors that make staleness and link-speed alerting work.

Details:

• Holly recovered: the Docker daemon is back after a VM reboot — the 2026-07-25 boot-USB failure had corrupted docker.img.
• Health audit: all disks SMART-healthy (0 reallocated / pending / CRC). The array is deliberately zero-parity — the "2 disabled" slots are the empty parity slots, not failures. Two data drives are at ~54,500 power-on hours (~6.2 years).
• The real issue is capacity: disk4 96%, disk1 95%, disk6 93%, disk2 90% used (263.9 GB of duplicates already cleared on 2026-07-28 — see below).
• Telemetry coverage (2026-07-29..30, PR #169): the netclass collector was added to every Alloy config (node_network_speed_bytes = live link speed), and the time + stat collectors were added on Holly and the caddy installer — HostMetricsStale keys off node_time_seconds and Holly was invisible to it before. Five hosts now covered: caddy, docker-pve1, holly, pve1, pve2 (+pve3 only while awake).

2026-07-28 - pve1 gigabit uplink fixed, kernel update, Holly dedup¶

Category: Infrastructure / Network / Storage

Summary: Root-caused and fixed pve1's long-standing 100 Mb/s uplink (a wedged USB negotiation, not a bad cable), added boot-time NIC remediation and a link-degradation alert, updated the pve1 kernel, set Jarvis to autostart, and reclaimed 263.9 GB of duplicates on Holly.

Details:

• pve1 uplink at 1 Gb/s. The old 100 Mb problem was a wedged USB negotiation on the USB3 RTL8153 gigabit adapter (enx3c18a0d4eddd), fixed by an xHCI reset. The adapter is now the sole uplink; the onboard eno1 — an unplugged 10/100 RTL8106E (slow, not dead) — was split out of vmbr0.
• Boot remediation: restart-usb.service → usbup.sh → /usr/local/sbin/nic-link-up.sh (NIC_EXPECTED_SPEED=1000 + xHCI reset + force-gigabit retry). A new repo folder proxmox/network/ holds nic-link-up.sh + nic-link-up@.service, with boot oneshots enabled on all three nodes (pve2 uplink enp4s0f0 = 10 GbE, pve3 enp6s0 = 1 GbE).
• New Prometheus alert NetworkLinkDegraded fires when a physical link drops to <=100 Mb.
• pve1 kernel updated to 7.0.14-6-pve and rebooted. (During that reboot the blackbox exporter died and stayed down 43 h — the incident behind the 2026-07-30 alert-semantics overhaul.)
• Jarvis VM 121 set onboot=1 so the agent autostarts with pve1.
• Holly dedup: deleted 263.9 GB of byte-verified duplicates — movie downloads 146.5 GB + audiobook dupes 95.6 GB + redundant caab backups 21.8 GB (the GoPro and OneDrive trees were kept).

2026-07-26 - pve3 storage build, Proxmox Backup Server, backup realignment, Grafana overhaul¶

Category: Infrastructure / Storage / Monitoring

Summary: Turned pve3 into a real services/backup node — built its ZFS pools, stood up a Proxmox Backup Server VM as the new primary guest-backup target, realigned the whole backup topology around it, overhauled the Grafana dashboards, fixed a telemetry blackout, and confirmed Holly's recovery.

Details:

• pve3 ZFS build. Wiped pve3's two former NTFS disks and built tank (1.81 TB → storages pve3-vm for images/rootdir + pve3-iso dir) and bak (464 GB → pve3-backup dir + the PBS datastore). pve3 boots off its small OCZ SSD; both pools ashift=12, lz4, atime=off, xattr=sa.
• Proxmox Backup Server deployed as VM 102 on pve3 (192.168.1.30:8007, PBS 4.2.3), installed unattended via the netboot auto-install kit (extended for PBS in PR #141 — a static [network] needs a NIC filter that validate-answer does not enforce). Datastore main (/mnt/datastore/main on the bak pool), registered as cluster storage pve3-pbs (active). PBS is deliberately its own VM, not co-located with a hypervisor.
• PBS is now the primary guest-backup target. A daily 01:30 PBS job backs up all 9 pve1 guests (100, 101, 112, 113, 114, 116, 120, 121, 122; keep-all — retention delegated to PBS). Datastore lifecycle: prune 04:00 (keep-last 3 / daily 7 / weekly 4 / monthly 6), GC 04:30, verify 05:00. A restore test proved recoverability.
• Backup realignment. The local vzdump jobs were trimmed to keep-last 2 as a belt-and-braces floor during the soak (VM 121 Sun 02:00; CTs 100/101/112/114/116 daily 02:30 → local). The offsite leg is now incremental + deduplicated: a new pbs-offsite-sync.timer (05:30) runs on the PBS VM and rclone copys the datastore to hetzner-crypt:pbs (client-side rclone crypt, content + filenames; gated against active PBS backup/GC/verify tasks). Phase 2 (deferred until a ~1–2 week green soak): retire the local vzdump jobs + pve1's old pve-offsite-sync, reclaim pve1 local, and extend PBS to the pve2 + pve3 guests (excluding Holly VM 110 and the PBS VM itself).
• Hetzner offsite cleanup. Pruned ~249 GiB of stale full-dump uploads left by the old (non-incremental) pve1 offsite mirror.
• Grafana overhaul (PR #142). The 11 dashboards were regrouped into six Grafana folders (Start Here / Infrastructure / Hardware & Storage / Edge & Security / Logs & Diagnostics / Services) via foldersFromFilesStructure; Homelab Command Center became the default home dashboard; allowUiUpdates: false locks the git JSON as authoritative; a blackbox-pbs tcp probe watches PBS liveness; the "Firing Alerts" tile now excludes the always-on Watchdog deadman; CrowdSec tiles standardised on local decisions (not the CAPI community list); and SMART panels use the device label. pve3's SMART now shows on Drive Health.
• Telemetry fix (PR #140). The Alloy filesystem collector now excludes network fstypes (cifs|nfs|smb3|fuse.*) in both install-alloy.sh and monitoring/alloy/config.alloy. A dead Holly CIFS mount used to make statfs() hang past the scrape timeout and blank a host's node_* metrics — which caused a pve1 + CT 101 node-metric blackout when Holly went down. All metric hosts report again, resiliently.
• Holly recovery. Unraid VM 110 recovered after its boot flash physically dropped off pve2's USB bus (2026-07-25 → reseated 2026-07-26). Root cause was dead/dropped hardware, not wrong-device passthrough — the usb0: host=1-3.1 config was already correct.

2026-07-24 - PVE3 joined and PXE moved to PVE1¶

Category: Infrastructure

Summary: Added a third Proxmox node, normalised its static address and storage, and moved the PXE container from pve2 to pve1.

Details:

• pve3 joined the homelab cluster at 192.168.1.28/22 with PVE 9.2.5, current Intel microcode and the existing personal + automation SSH keys.
• Registered the installer-created 10GB LVM-thin pool as local-lvm-pve3, restricted to pve3. Its two additional NTFS disks remain untouched pending a storage decision.
• Corrected the persistent console argument to nomodeset video=1280x720@60 and verified it active after reboot.
• Migrated CT 122 (netboot-pxe, 192.168.1.231) from pve2 to pve1 and verified it running there.

2026-07-21 - Arcane consolidated to a single environment + ABS ASIN backfill¶

Category: Infrastructure / Services

Summary: Removed the redundant Arcane agent so CT 101 is managed as one environment, and backfilled Audible ASINs across the Audiobookshelf library.

Details:

• Arcane is now a single environment (Local Docker, env 0). The manager (co-located on CT 101 since 2026-07-20) was realigned to PROJECTS_DIRECTORY=/opt/docker with a /opt/docker:/opt/docker bind, so it manages the host's docker directly over the local socket. The arcane-headless agent (:3553) and its docker-ct101 environment were removed — redundant once the manager shared the host, and the cause of every container/image being enumerated twice.
• All 11 git-syncs migrated to env 0 and re-bound to env-0 projects (verified: 11/11 success, a live sync + deploy succeeded with the agent down). Root cause of the earlier split-brain: the agent held its own sync definitions and projects, which the manager only proxied — so /environments/0/gitops-syncs appeared empty.
• Manager staging was also fixed: /etc/arcane/projects was root-owned and unwritable by the manager's runtime uid (65532), which blocked any new stack from deploying. A chown unblocked it (this is what had made the audiobookrequest sync fail to create).
• Rollback kept on CT 101 at /root/arcane-phase3-backup/ (manager + agent compose, manager.env, and pre/post arcane-data volume tarballs). docker-pve1/management/arcane-agent/ is now a historical reference only and was dropped from Dependabot.
• Audiobookshelf ASIN backfill: ASIN coverage went 90 → 470 of 540 items. Audible (audible.uk) was matched per item and only the asin field was written (verified partial update — title/authors/narrators/genres/description untouched). 52 low-confidence and 18 no-match items were deliberately skipped: the author+title guardrail correctly rejected bad matches such as Emma, Pride and Prejudice and Jane Eyre mapping to omnibus "Complete Novels"/"British Classics" collections.

2026-07-21 - LazyLibrarian retired → AudioBookRequest + beets¶

Category: Services

Summary: Replaced LazyLibrarian (brought live only the day before) with a lean AudioBookRequest + beets stack — LazyLibrarian was the wrong weight class: too heavy, and its existing-library import was unreliable (it ballooned to 3,215 books following series → related authors and leaned on the long-dead GoodReads API).

Details:

• AudioBookRequest (ABR) on CT 101 port 8000, SSO-gated at abr.mdhmedia.uk (tinyauth; ABR runs open internally with FORCE_LOGIN_TYPE=none). A lightweight request UI that searches Audible for metadata and routes grabs through the existing Prowlarr → qBittorrent/NZBGet, then triggers an Audiobookshelf scan. It needs no media mounts of its own.
• beets sidecar (lscr.io/linuxserver/beets + the beets-audible plugin) organises finished downloads into /audiobooks/$Author/$Series/$Title (ABS-compatible) — the tidy-filing piece ABR deliberately leaves out, done with Audible metadata. It reuses the exact CT 101 CIFS binds LazyLibrarian used (/mnt/nas/downloads, /mnt/nas/audiobooks), kept when LazyLibrarian was torn down.
• New MCP server audiobookrequest (Bearer-token /api) replaces the lazylibrarian MCP; the MCP count is unchanged.
• LazyLibrarian fully retired: stack folder, MCP, Arcane git-sync, and the lazylibrarian_ll-config volume all removed; only the shared CIFS binds stay. See Moving an app off Holly.

2026-07-20 - Consolidation onto CT 101 + LazyLibrarian live + SMART everywhere¶

Category: Infrastructure / Services / Storage

Summary: Consolidated the docker estate to a single host (CT 101, 192.168.1.241), retiring CT 100 and CT 113 and deleting the last Portainer remnant; brought LazyLibrarian live over SMB; extended SMART drive monitoring to pve2 and Holly.

Details:

• Arcane manager moved off its dedicated LXC CT 100 (192.168.1.243:3552) and is now co-located on CT 101 (192.168.1.241:3552, still arcane.mdhmedia.uk). Data volume + encryption key preserved; image pinned from :latest to v2.4.0. CT 100 retired — stopped with onboot=0 and kept as a rollback, not destroyed.
• Homepage moved off its dedicated LXC CT 113 (192.168.1.242:3000) onto CT 101 at host port 3080 (kicad already owns 3000 there). Caddy repoints mdhmedia.uk at 192.168.1.241:3080 and the root is now behind the tinyauth SSO gate. CT 113 retired the same way (stopped, onboot=0, rollback); its Arcane agent is gone — CT 101 is now the sole agent host and also runs the manager.
• Portainer fully gone: the portainer_data cold-rollback volume on CT 101 — the last remnant of the July 2026 decommission — was deleted. Arcane is the sole Docker manager with no Portainer rollback remaining.
• LazyLibrarian live on CT 101, port 5299 (LAN-only; an SSO route is still backlog). Storage is a pve1 host SMB/CIFS mount of Holly's Media/Music/Audiobooks + downloads shares (dedicated Holly SMB user pveshare, uid=101000 → container uid 1000), bind-mounted into CT 101 — NFS was unusable (CT 101 is an unprivileged LXC that can't mount network filesystems, and Holly's nfsd doesn't serve external clients). Feeds Audiobookshelf. See Moving an app off Holly.
• smartctl-exporter deployed on pve2 and Holly (previously pve1 only) — all three hosts now report drive SMART health to Prometheus and the Drive Health dashboard.

2026-07-15 - Portainer → Arcane migration (Docker manager replaced)¶

Category: Infrastructure / Services

Summary: Replaced Portainer with Arcane as the homelab's Docker manager and decommissioned Portainer entirely.

Details:

• New Arcane manager on CT 100 (192.168.1.243:3552, arcane.mdhmedia.uk) with native Google OIDC login plus a claude break-glass admin.
• Arcane headless agents (ghcr.io/getarcaneapp/arcane-headless) on CT 101 (192.168.1.241) and CT 113 (192.168.1.242), each with PROJECTS_DIRECTORY=/opt/docker and a matching /opt/docker:/opt/docker host bind.
• Every stack re-created as an Arcane git-sync (repo + main + compose path, ~5-min autoSync — a merge to main redeploys) with projectName set to the old Portainer stack name (preserves the <stack>_<volume> volume names) and syncDirectory: true, so the whole compose dir materialises and ./config / ./provisioning binds keep working.
• Secrets moved from Portainer stack env vars to each stack's Arcane project .env at /opt/docker/<stack>/.env.
• Portainer server and all agents removed; only the portainer_data volume on CT 101 was kept as a cold rollback (since deleted 2026-07-20 — see the entry above).

2026-07-08 - Jarvis neteng network-engineer profile¶

Category: Automation / Monitoring

Summary: Turned the Jarvis Hermes agent into an always-on homelab network engineer and wired a Grafana-native Telegram alert pipe.

Details:

• New version-controlled Jarvis profile neteng (hermes-vm/agent/profiles/neteng/): a network-engineer persona (SOUL.md) with a hard autonomy contract, a homelab knowledge pack (HOMELAB.md) seeded into the agent's memory, and an idempotent setup-neteng.sh deploy script.
• Wired the grafana-stack MCP (Loki + Prometheus on .241) into the profile so the agent can query telemetry directly; registered three cron jobs (30-min health sweep, daily digest, weekly review → Telegram), created paused pending the Nous Portal login.
• Scoped auto-fix: the agent may only restart existing containers via a revocable jarvis-neteng Portainer token; everything structural is proposed via Telegram and routed through GitOps. It annotates Grafana for every action.
• Grafana pane of glass: added a "Jarvis Ops" dashboard (agent heartbeat, firing alerts, annotation-driven incident timeline) and provisioned Grafana-managed alerting that mirrors the Prometheus ALERTS metric to a Telegram contact point — the dumb-pipe layer that fires even if the agent is down. Added a Prometheus blackbox heartbeat probe for the Jarvis dashboard on :9119.

2026-01-16 - Documentation Overhaul¶

Category: Documentation

Summary: Major documentation update with infrastructure scanning and Home Assistant API integration.

Details:

• Added Mermaid diagram support and converted all ASCII diagrams
• Added MkDocs Material QoL features (sticky nav, instant navigation, code annotations, etc.)
• Scanned PVE1 and PVE2 via SSH - documented hardware, storage, VMs, LXCs
• Connected to Home Assistant API and documented:
  • 9 add-ons with versions
  • 12 integrations
  • 164 sensors, 35 binary sensors, 8 automations
  • Lights, media players, cameras
• Created system architecture diagrams for network and Home Assistant
• Simplified docker-compose.yml (removed nginx/build services)

2026-01-16 - SSH Key Configuration¶

Category: Infrastructure

Summary: Configured SSH key access for automated Proxmox management.

Details:

• Generated ed25519 SSH key
• Deployed to PVE1 (192.168.1.199) and PVE2 (192.168.1.151)
• Enables automated infrastructure scanning

2025¶

January¶

• Initial MkDocs documentation setup

Template Entry¶

### YYYY-MM-DD - Brief Title

**Category:** Infrastructure / Services / Network / Storage

**Summary:** What changed and why.

**Details:**
- Bullet points with specifics
- Links to relevant docs if needed

**Issues encountered:**
- Any problems and their solutions