Skip to content

Proxmox

Nodes

PVE1 (Primary)

Hardware

Component Details
CPU Intel Core i3-4005U @ 1.70GHz (2 cores, 4 threads)
RAM 16GB
Boot Disk Kingston SA400S37 240GB SSD
PVE 9.2.5 (kernel 7.0.14-6-pve — updated + rebooted 2026-07-28)

ZFS ARC capped 2026-08-11

The ARC on rpool/local-zfs was uncapped, growing to ~14.5 GiB on this 16 GiB host and firing HighMemoryUsage (available memory down to 1.6 GiB). The VMs/CTs themselves were correctly sized for their actual usage — this was the real cause. Capped to 2 GiB via options zfs zfs_arc_max=2147483648 in /etc/modprobe.d/zfs.conf (applied live, then persisted with update-initramfs -u -k all). Available memory recovered to 4.3 GiB.

Storage

Pool Type Size Purpose
rpool ZFS 222GB Main storage pool
local Directory 183GB ISO images, backups
local-zfs ZFS Pool 174GB VM/CT disks

Network

Since 2026-07-28 the sole uplink is the USB3 RTL8153 gigabit adapter (enx3c18a0d4eddd), running at 1 Gb/s. The onboard eno1 is an unplugged 10/100 RTL8106E (working, just slow and not connected) and was split out of vmbr0 the same day. A boot remediation chain — restart-usb.serviceusbup.sh/usr/local/sbin/nic-link-up.sh — guards against the USB NIC renegotiating at 100 Mb; see Network.

flowchart LR
    subgraph Bridge[vmbr0 - 192.168.1.199/22]
        usb[enx3c18a0d4eddd<br/>USB3 RTL8153 - 1 Gbit]
    end
    eno1[eno1<br/>RTL8106E 10/100 - unplugged, out of vmbr0]
    Bridge --> GW[Gateway<br/>192.168.1.254]

PVE2 (Secondary)

  • IP: 192.168.1.151
  • URL: pve2.mdhmedia.uk
  • Role: Secondary hypervisor, heavy workloads
  • BMC: Dell R710-era iDRAC6 at 192.168.1.150 — remote power/console independent of the host OS (confirmed 2026-08).
  • Fans: Managed by the live proxmox/fan-control/ systemd service using in-band IPMI (/dev/ipmi0). Its thermostatic curve sets 12/18/25/35/50% at 0/45/55/63/70 C; any error, service stop, or 75 C ceiling restores iDRAC automatic control.
  • Power: the proxmox/power-monitoring/ reader publishes node_power_watts{source="ipmi"} from the in-band /dev/ipmi0 "System Level" sensor (whole-system, ~160-200 W). pve1 instead reports source="rapl" (a CPU-package proxy from RAPL, read as root because the Platypus-CVE mitigation makes energy_uj root-only and Alloy runs non-root). Both feed the Power & Energy Grafana dashboard (mdh-power), costed by the editable Home Assistant input_number.electricity_unit_rate (24.0952 p/kWh) - HA's prometheus integration on .240 exposes it, the homeassistant scrape job pulls it, and the electricity_unit_rate_gbp_per_kwh recording rule reads it (constant fallback). Repoints to the new HA VM at the HAOS cutover.

Hardware

Component Details
CPU 2x Intel Xeon X5670 @ 2.93GHz (12 cores, 24 threads)
RAM 72GB
Boot Disk PNY CS900 120GB SSD
PVE 9.1.14 (kernel 7.0.2-4-pve)

Storage

Pool Type Size Purpose
rpool ZFS 110GB Main storage pool
local Directory 93GB ISO images, backups
local-zfs-pve2 ZFS Pool 107GB VM/CT disks

Network

flowchart LR
    subgraph vmbr0[vmbr0 - 192.168.1.151/22]
        eno1[eno1]
    end
    subgraph vmbr1[vmbr1 - 192.168.1.152/22]
        enp4s0f0[enp4s0f0]
    end
    subgraph vmbr2[vmbr2 - 192.168.1.153/22]
        enp4s0f1[enp4s0f1]
    end
    vmbr0 --> GW[Gateway<br/>192.168.1.254]

PVE3 (Wake-on-LAN Backup Node)

  • IP: 192.168.1.28
  • URL: 192.168.1.28:8006
  • Role: Wake-on-LAN backup node (ZFS storage + Proxmox Backup Server) — normally powered off since 2026-07-30

Joined to the homelab cluster on 2026-07-24. On 2026-07-26 its two former NTFS disks were built into ZFS pools — tank (1.81 TB, VM/CT + ISO storage) and bak (464 GB, the PBS datastore) — and the node now hosts the Proxmox Backup Server VM (see Virtual Machines and Backups). Since 2026-08-04 the 500G bak disk is passed whole into the PBS VM, which imports the bak pool itself — the host no longer mounts it, and the pve3-bak/pve3-backup host storages were deleted. It still boots off its small OCZ SSD.

Since 2026-07-30 pve3 is a wake-on-LAN backup box: normally powered off, it is woken daily at 19:00 by the wol-pbs orchestrator container on CT 101 (the 13th Arcane git-sync stack, stacks/management/wol-pbs/), which runs the evening backup window and shuts the node down again once PBS is idle (minimum wake 110 minutes). If PBS is already awake at wake-time the orchestrator now continues the cycle and still shuts pve3 down (via an atomic mkdir lock); STANDBY_IF_UP=true restores the old standby behaviour. (Fixed 2026-08-12, #276.) pve3 stays in the cluster: pve1's quorum_votes was raised from 2 to 3 (total 5, quorum 3), so pve1 remains quorate on its own while pve3 sleeps.

Wake-on-LAN specifics: the magic packet targets enp6s0 (r8169). The driver clears the WoL flag at shutdown, so a pve3-wol-enable.service oneshot re-arms it, and the BIOS needs WoL enabled with ErP/EuP off. The PBS VM 102 has onboot=1 and autostarts on wake (~75 s to a responsive API). The orchestrator's SSH key is forced-command restricted on pve3 — it can only power the node off or start pbs-offsite-sync — and its PBS API access uses the read-only token api@pbs!wolpbs.

Bad RAM found and removed (2026-08-22)

pve3 produced transient chunk-digest errors and a kernel oops on 2026-08-03 — a ZFS zil_commit memcpy page fault under sustained zvol writes — which wedged the bak pool. The response was to stagger the window (Holly's upload moved to 20:35) and re-architect the datastore off zvols onto a native ZFS dataset on 2026-08-04 (see Backups). The 2026-08-22 memtest tested the retained RAM sticks individually and together; all passed. Old sticks that threw errors were removed. That bad RAM is the likely cause of the August 3 oops and the prime suspect for the PBS verify-job corruption, although the latter link is not proven.

Hardware

Component Details
CPU Intel Core i5-2500K (4 cores)
RAM 16GB (4× 4GB DDR3-1600, all four DIMM slots filled 2026-08-19) - up from 8GB; board max 32GB, so a future step would mean replacing DIMMs, not adding them. Memtest passed 2026-08-22: the retained sticks passed individually and together; old sticks that threw errors were removed
Boot Disk OCZ-NOCTI 30GB SSD
Storage Disks 500GB → passed whole into PBS VM 102 (ZFS bak, 464 GB, imported in-VM since 2026-08-04); 2TB → ZFS tank (1.81 TB)
PVE 9.2.5 (kernel 7.0.14-6-pve)

Storage

Pool Type Size Purpose
local Directory (ext4 root) 12GB ISO images, templates; avoid large backups
local-lvm-pve3 LVM-thin 10GB VM/CT disks, restricted to pve3
tank → pve3-vm ZFS zpool 1.81 TB VM/CT disks (images/rootdir), restricted to pve3
tank/iso → pve3-iso ZFS (dir) on tank ISO images + CT templates
bak (inside PBS VM 102) ZFS zpool 464 GB The whole 500G disk is passed into the PBS VM since 2026-08-04, which imports the pool itself (ARC capped at 1G). Datastore main = dataset bak/pbs-datastore (cluster storage pve3-pbs). The pve3-bak/pve3-backup host storages were deleted; the old ext4 zvol was kept as a short-term rollback and destroyed 2026-08-11 (~55G reclaimed) once the new layout had soaked clean

Network

flowchart LR
    subgraph vmbr0[vmbr0 - 192.168.1.28/22]
        enp6s0[enp6s0<br/>1 Gbit]
    end
    vmbr0 --> GW[Gateway<br/>192.168.1.254]

LXC Containers

Statuses as of 2026-07-24. The cluster was rationalised 2026-07-07 — 13 stopped test/legacy guests (10 CTs + 3 VMs) were removed (see the note below). On 2026-07-20 the docker estate was consolidated onto CT 101: the Arcane manager (CT 100) and Homepage (CT 113) moved there and both LXCs were retired — stopped with onboot=0 and kept as rollbacks, not destroyed.

VMID Name Node Status IP Purpose
100 arcane pve1 stopped (retired) 192.168.1.243 Retired 2026-07-20 — Arcane manager moved to CT 101; kept stopped (onboot=0) as a rollback
101 portainer pve1 running 192.168.1.241 Docker host — Arcane manager (2 environments), all git-sync stacks, Homepage (:3080), AudioBookRequest (:8000), wol-pbs (wakes pve3 for the 19:00 backup window)
112 caddy pve1 running 192.168.1.245 Reverse proxy
113 homepage pve1 stopped (retired) 192.168.1.242 Retired 2026-07-20 — Homepage moved to CT 101 :3080; kept stopped (onboot=0) as a rollback
114 mariadb pve1 running 192.168.1.251 Database server
116 alpine-wireguard pve1 running 192.168.1.236 VPN server (WG Dashboard on :10086)
120 adguard pve1 running 192.168.1.244 DNS filtering (active)
122 netboot-pxe pve1 running 192.168.1.231 netboot.xyz + dnsmasq proxyDHCP + unattended Proxmox installer; migrated from pve2 on 2026-07-24

Cluster rationalisation — 2026-07-07

Removed these stopped guests with pct/qm destroy --purge. Those with a pre-destroy vzdump on their node's local are marked ✓; the rest were throwaway/superseded and taken without a backup:

Removed Node Was Backup
CT 102 pve1 teleport (WireGuard + tinyauth SSO supersede it)
CT 103 pve1 docker0X template (stale)
CT 104 pve1 pihole (never integrated)
CT 105 pve1 ubuntu template (stale)
CT 106 pve1 adguard (old, superseded by CT 120)
CT 108 pve2 plex (dup; Plex runs on Holly .210)
CT 111 pve1 librenms
CT 115 pve1 vaultwarden (test)
CT 117 pve2 scanopy (test)
CT 119 pve2 n8n (LAN automations move to Jarvis)
VM 100 pve1 zabbix
VM 107 pve1 apiscp-dev
VM 109 pve2 PMS (dup Plex)

CT 118 (grafana) was likewise destroyed 2026-07-06. Network mapping (Zabbix/LibreNMS) is deferred to Jarvis rather than replaced. The vzdumps live on each node's local and can be pruned or offsite-archived later.

Virtual Machines

VMID Name Node Status RAM Disk Purpose
102 pbs pve3 up during pve3's wake window (onboot=1) 4GB 32GB (ZFS on tank) + the whole 500G bak disk Proxmox Backup Server 4.2.3 at 192.168.1.30primary guest-backup target (datastore main on ZFS dataset bak/pbs-datastore, pool imported in-VM since 2026-08-04; cluster storage pve3-pbs)
121 hermes-jarvis pve2 running 8GB, 8 cores 48GB (local-zfs-pve2, discard=on,iothread=1,ssd=1) Jarvis host — Hermes agent at 192.168.1.252 (see Jarvis). Migrated pve1 → pve2 on 2026-09-08 and resized onto the heavy node (was 3GB/32GB on pve1)
123 haos pve1 stopped 2.5GB (trimmed from 3GB 2026-08-03) 32GB (ZFS on local-zfs) Home Assistant OS 18.2 (core 2026.7.4) at DHCP 192.168.2.120 - clean rebuild, built and parked (stopped, onboot=0) since 2026-08-27; the old 192.168.1.240 instance still owns the ha.mdhmedia.uk route. At cutover, start VM 123, set onboot=1, and move .240 + the route to it (see Home Automation)
110 unraid pve2 running 32GB USB boot Unraid NAS "Holly" — 16 cores, 3× PCI passthrough, OVMF + TPM 2.0

Backups

Since 2026-07-26 the backup topology is PBS-primary with a local vzdump floor and an incremental encrypted offsite. This realigns the earlier vzdump-only setup around the new Proxmox Backup Server (VM 102 on pve3). Since 2026-07-30 the whole PBS leg runs in an evening wake window: the wol-pbs orchestrator on CT 101 wakes pve3 at 19:00, the PBS jobs run, the offsite sync fires, and the node shuts down again once PBS is idle (minimum wake 110 minutes). The times below replaced the original 01:30 / 04:00–05:00 schedule. Full detail + tooling: proxmox/backup/ in the repo.

Primary — Proxmox Backup Server (dedup / incremental)

PBS 4.2.3 on VM 102 (pve3, 192.168.1.30), datastore main — since 2026-08-04 a native ZFS dataset (bak/pbs-datastore) on the bak pool, which the VM imports itself (the whole 500G disk is passed through; ARC capped at 1G; the old ext4-in-a-zvol layout is retired, the zvol destroyed 2026-08-11 after a clean week) — registered cluster-wide as storage pve3-pbs. A daily 19:15 job backs up all 9 pve1 guests (100, 101, 112, 113, 114, 116, 120, 121, 122; keep-all — retention is delegated to the PBS prune job). The full evening window:

Time Step
19:00 wol-pbs sends the magic packet; pve3 boots, the PBS VM autostarts (onboot=1, ~75 s to API)
19:15 PBS backup job — all 9 pve1 guests
19:45 Prune (pj-main) — keep-last 3 / daily 7 / weekly 4 / monthly 6
19:50 Garbage collection — reclaims unreferenced chunks
20:00 Verify (vj-main) — re-checksums, outdated-after 7
20:35 Holly's host/holly upload (appdata staging + boot flash from inside the Unraid VM — staggered here from 19:25 on 2026-08-03 to dodge pve3's concurrent-load chunk-digest errors; see Unraid)
after verify Offsite sync (carries host/holly along), then shutdown once PBS is idle (minimum wake 110 minutes)

A restore test on 2026-07-26 proved recoverability, and a full-datastore verify after the 2026-08-04 migration came back OK.

Verify-job incident — 2026-08-11

PBS shipped no telemetry of its own (no Alloy agent; asleep ~22 h/day), so a genuine vj-main TASK ERROR was invisible until checked by hand. A corrected re-verify (--ignore-verified false — the default true skips exactly the backups you're trying to recheck) found 13 backups had actually failed verification, including the latest ct/101 (the whole docker host) and vm/121 (Jarvis). Fresh, fully-verified backups of both were taken immediately as a stopgap. Root cause found 2026-08-24: a one-off — every failed backup predates the 2026-08-04 datastore migration (the 2026-08-03 zil_commit oops on the old ext4-in-zvol layout, plus the faulty RAM removed after the 2026-08-22 memtest). The bak pool has scrubbed with 0 errors and every verify has passed since 2026-08-19; no post-migration backup has ever failed (see Open Items). Prometheus now probes the PBS API directly for the latest verify status (blackbox-pbs-verify job, PbsVerificationFailed alert) so this can't go unnoticed again.

Local floor — vzdump to local (belt-and-braces during the soak)

Two local vzdump jobs remain, trimmed to keep-last 1 as a floor (cut from keep-last 2 on 2026-08-24 to reduce pve1 SSD churn — one local copy under PBS):

Job Guests Schedule Retention
jarvis VM 121 Sun 02:00 keep-last 1
priority CTs 100, 101, 112, 114, 116 Sun 03:30 keep-last 1

Snapshot mode / ZSTD → local. VM 110 (Unraid) is intentionally excluded (PCI passthrough / USB boot) — Holly backs itself up instead: a proxmox-backup-client inside the VM ships appdata + the boot flash to PBS as host/holly (since 2026-07-31; upload at 20:35 — see Unraid), with Unraid Connect's cloud flash backup as the second flash copy.

Offsite — incremental, encrypted, from the PBS datastore

The offsite leg runs on the PBS VM: pbs-offsite-sync rclone copys the datastore to hetzner-crypt:pbs on a jailed Hetzner Storage Box sub-account (u195506-sub5), wrapped in rclone crypt (client-side encryption of content and filenames) and gated against active PBS backup/GC/verify tasks. Since 2026-07-30 it is started by the wol-pbs orchestrator at the end of the evening window (after verify, before shutdown) — starting it is one of only two commands the orchestrator's restricted SSH key permits — rather than by the old 05:30 timer, when pve3 is now asleep. Because PBS chunks are content-addressed, only changed chunks transfer — a strict improvement over the old full-dump mirror. The crypt password/salt live only in the gitignored proxmox/backup/.env.

The nightly copy never deletes (a one-way valve), so since 2026-08-04 a ~monthly guarded reconcile complements it: rclone sync with a --max-delete tripwire and a dated --backup-dir archive — pruned chunks eventually leave Hetzner, a mass delete trips the guard instead of running, and reconcile archives are purged after 60 days.

Phase 2 — HELD, not just deferred (2026-08-11)

Done so far: pve1's legacy pve-offsite-sync.timer was disabled 2026-07-31 (units kept installed; the final hetzner-crypt:pve1 mirror stays as a frozen extra copy), and Holly's appdata + boot-flash backup is live (host/holly, since 2026-07-31). Still deferred: retire the two local vzdump jobs, reclaim pve1 local, and extend the PBS job to the pve2 + pve3 guests (excluding Holly VM 110 and the PBS VM 102 itself).

This was tracked as "resume after two clean weeks" until the 2026-08-11 verify-job incident above proved the PBS-only leg can go silently bad — the local vzdump floor is exactly what closed the gap while that was found. The corruption root cause is now understood (a one-off, resolved — see the Recently-closed entry in Open Items), so this is no longer blocked on that. But the floor saved the estate twice in nine days (the corruption and the five-night WoL gap), so retiring it is now a risk-appetite decision, not a calendar one.

This gives a live 3-2-1: PBS (primary, dedup) → local vzdump (keep-last-1 floor) → encrypted Hetzner offsite.