Proxmox¶
Nodes¶
PVE1 (Primary)¶
- IP: 192.168.1.199
- URL: pve1.mdhmedia.uk
- Role: Primary hypervisor
Hardware¶
| Component | Details |
|---|---|
| CPU | Intel Core i3-4005U @ 1.70GHz (2 cores, 4 threads) |
| RAM | 16GB |
| Boot Disk | Kingston SA400S37 240GB SSD |
| PVE | 9.2.5 (kernel 7.0.14-6-pve — updated + rebooted 2026-07-28) |
ZFS ARC capped 2026-08-11
The ARC on rpool/local-zfs was uncapped, growing to ~14.5 GiB on this
16 GiB host and firing HighMemoryUsage (available memory down to 1.6 GiB).
The VMs/CTs themselves were correctly sized for their actual usage — this was
the real cause. Capped to 2 GiB via options zfs zfs_arc_max=2147483648
in /etc/modprobe.d/zfs.conf (applied live, then persisted with
update-initramfs -u -k all). Available memory recovered to 4.3 GiB.
Storage¶
| Pool | Type | Size | Purpose |
|---|---|---|---|
| rpool | ZFS | 222GB | Main storage pool |
| local | Directory | 183GB | ISO images, backups |
| local-zfs | ZFS Pool | 174GB | VM/CT disks |
Network¶
Since 2026-07-28 the sole uplink is the USB3 RTL8153 gigabit adapter
(enx3c18a0d4eddd), running at 1 Gb/s. The onboard eno1 is an unplugged
10/100 RTL8106E (working, just slow and not connected) and was split out of
vmbr0 the same day. A boot remediation chain — restart-usb.service →
usbup.sh → /usr/local/sbin/nic-link-up.sh — guards against the USB NIC
renegotiating at 100 Mb; see Network.
flowchart LR
subgraph Bridge[vmbr0 - 192.168.1.199/22]
usb[enx3c18a0d4eddd<br/>USB3 RTL8153 - 1 Gbit]
end
eno1[eno1<br/>RTL8106E 10/100 - unplugged, out of vmbr0]
Bridge --> GW[Gateway<br/>192.168.1.254]
PVE2 (Secondary)¶
- IP: 192.168.1.151
- URL: pve2.mdhmedia.uk
- Role: Secondary hypervisor, heavy workloads
- BMC: Dell R710-era iDRAC6 at
192.168.1.150— remote power/console independent of the host OS (confirmed 2026-08). - Fans: Managed by the live
proxmox/fan-control/systemd service using in-band IPMI (/dev/ipmi0). Its thermostatic curve sets 12/18/25/35/50% at 0/45/55/63/70 C; any error, service stop, or 75 C ceiling restores iDRAC automatic control. - Power: the
proxmox/power-monitoring/reader publishesnode_power_watts{source="ipmi"}from the in-band/dev/ipmi0"System Level" sensor (whole-system, ~160-200 W). pve1 instead reportssource="rapl"(a CPU-package proxy from RAPL, read as root because the Platypus-CVE mitigation makesenergy_ujroot-only and Alloy runs non-root). Both feed the Power & Energy Grafana dashboard (mdh-power), costed by the editable Home Assistantinput_number.electricity_unit_rate(24.0952 p/kWh) - HA'sprometheusintegration on.240exposes it, thehomeassistantscrape job pulls it, and theelectricity_unit_rate_gbp_per_kwhrecording rule reads it (constant fallback). Repoints to the new HA VM at the HAOS cutover.
Hardware¶
| Component | Details |
|---|---|
| CPU | 2x Intel Xeon X5670 @ 2.93GHz (12 cores, 24 threads) |
| RAM | 72GB |
| Boot Disk | PNY CS900 120GB SSD |
| PVE | 9.1.14 (kernel 7.0.2-4-pve) |
Storage¶
| Pool | Type | Size | Purpose |
|---|---|---|---|
| rpool | ZFS | 110GB | Main storage pool |
| local | Directory | 93GB | ISO images, backups |
| local-zfs-pve2 | ZFS Pool | 107GB | VM/CT disks |
Network¶
flowchart LR
subgraph vmbr0[vmbr0 - 192.168.1.151/22]
eno1[eno1]
end
subgraph vmbr1[vmbr1 - 192.168.1.152/22]
enp4s0f0[enp4s0f0]
end
subgraph vmbr2[vmbr2 - 192.168.1.153/22]
enp4s0f1[enp4s0f1]
end
vmbr0 --> GW[Gateway<br/>192.168.1.254]
PVE3 (Wake-on-LAN Backup Node)¶
- IP: 192.168.1.28
- URL: 192.168.1.28:8006
- Role: Wake-on-LAN backup node (ZFS storage + Proxmox Backup Server) — normally powered off since 2026-07-30
Joined to the homelab cluster on 2026-07-24. On 2026-07-26 its two former
NTFS disks were built into ZFS pools — tank (1.81 TB, VM/CT + ISO storage) and
bak (464 GB, the PBS datastore) — and the node now hosts the
Proxmox Backup Server VM (see Virtual Machines and
Backups). Since 2026-08-04 the 500G bak disk is passed
whole into the PBS VM, which imports the bak pool itself — the host no
longer mounts it, and the pve3-bak/pve3-backup host storages were deleted.
It still boots off its small OCZ SSD.
Since 2026-07-30 pve3 is a wake-on-LAN backup box: normally powered
off, it is woken daily at 19:00 by the wol-pbs orchestrator container on
CT 101 (the 13th Arcane git-sync stack, stacks/management/wol-pbs/),
which runs the evening backup window and shuts the node down again once PBS is
idle (minimum wake 110 minutes). If PBS is already awake at wake-time the
orchestrator now continues the cycle and still shuts pve3 down (via an atomic
mkdir lock); STANDBY_IF_UP=true restores the old standby behaviour. (Fixed
2026-08-12, #276.) pve3 stays in the cluster: pve1's
quorum_votes was raised from 2 to 3 (total 5, quorum 3), so pve1 remains
quorate on its own while pve3 sleeps.
Wake-on-LAN specifics: the magic packet targets enp6s0 (r8169). The driver
clears the WoL flag at shutdown, so a pve3-wol-enable.service oneshot
re-arms it, and the BIOS needs WoL enabled with ErP/EuP off. The PBS VM
102 has onboot=1 and autostarts on wake (~75 s to a responsive API). The
orchestrator's SSH key is forced-command restricted on pve3 — it can only
power the node off or start pbs-offsite-sync — and its PBS API access uses
the read-only token api@pbs!wolpbs.
Bad RAM found and removed (2026-08-22)
pve3 produced transient chunk-digest errors and a kernel oops on
2026-08-03 — a ZFS zil_commit memcpy page fault under sustained zvol
writes — which wedged the bak pool. The response was to stagger the
window (Holly's upload moved to 20:35) and re-architect the datastore
off zvols onto a native ZFS dataset on 2026-08-04 (see
Backups). The 2026-08-22 memtest tested the retained RAM sticks
individually and together; all passed. Old sticks that threw errors were
removed. That bad RAM is the likely cause of the August 3 oops and the
prime suspect for the PBS verify-job corruption, although the latter link
is not proven.
Hardware¶
| Component | Details |
|---|---|
| CPU | Intel Core i5-2500K (4 cores) |
| RAM | 16GB (4× 4GB DDR3-1600, all four DIMM slots filled 2026-08-19) - up from 8GB; board max 32GB, so a future step would mean replacing DIMMs, not adding them. Memtest passed 2026-08-22: the retained sticks passed individually and together; old sticks that threw errors were removed |
| Boot Disk | OCZ-NOCTI 30GB SSD |
| Storage Disks | 500GB → passed whole into PBS VM 102 (ZFS bak, 464 GB, imported in-VM since 2026-08-04); 2TB → ZFS tank (1.81 TB) |
| PVE | 9.2.5 (kernel 7.0.14-6-pve) |
Storage¶
| Pool | Type | Size | Purpose |
|---|---|---|---|
| local | Directory (ext4 root) | 12GB | ISO images, templates; avoid large backups |
| local-lvm-pve3 | LVM-thin | 10GB | VM/CT disks, restricted to pve3 |
tank → pve3-vm |
ZFS zpool | 1.81 TB | VM/CT disks (images/rootdir), restricted to pve3 |
tank/iso → pve3-iso |
ZFS (dir) | on tank |
ISO images + CT templates |
bak (inside PBS VM 102) |
ZFS zpool | 464 GB | The whole 500G disk is passed into the PBS VM since 2026-08-04, which imports the pool itself (ARC capped at 1G). Datastore main = dataset bak/pbs-datastore (cluster storage pve3-pbs). The pve3-bak/pve3-backup host storages were deleted; the old ext4 zvol was kept as a short-term rollback and destroyed 2026-08-11 (~55G reclaimed) once the new layout had soaked clean |
Network¶
flowchart LR
subgraph vmbr0[vmbr0 - 192.168.1.28/22]
enp6s0[enp6s0<br/>1 Gbit]
end
vmbr0 --> GW[Gateway<br/>192.168.1.254]
LXC Containers¶
Statuses as of 2026-07-24. The cluster was rationalised 2026-07-07 — 13 stopped
test/legacy guests (10 CTs + 3 VMs) were removed (see the note below). On
2026-07-20 the docker estate was consolidated onto CT 101: the Arcane manager
(CT 100) and Homepage (CT 113) moved there and both LXCs were retired — stopped
with onboot=0 and kept as rollbacks, not destroyed.
| VMID | Name | Node | Status | IP | Purpose |
|---|---|---|---|---|---|
| 100 | arcane | pve1 | stopped (retired) | 192.168.1.243 | Retired 2026-07-20 — Arcane manager moved to CT 101; kept stopped (onboot=0) as a rollback |
| 101 | portainer | pve1 | running | 192.168.1.241 | Docker host — Arcane manager (2 environments), all git-sync stacks, Homepage (:3080), AudioBookRequest (:8000), wol-pbs (wakes pve3 for the 19:00 backup window) |
| 112 | caddy | pve1 | running | 192.168.1.245 | Reverse proxy |
| 113 | homepage | pve1 | stopped (retired) | 192.168.1.242 | Retired 2026-07-20 — Homepage moved to CT 101 :3080; kept stopped (onboot=0) as a rollback |
| 114 | mariadb | pve1 | running | 192.168.1.251 | Database server |
| 116 | alpine-wireguard | pve1 | running | 192.168.1.236 | VPN server (WG Dashboard on :10086) |
| 120 | adguard | pve1 | running | 192.168.1.244 | DNS filtering (active) |
| 122 | netboot-pxe | pve1 | running | 192.168.1.231 | netboot.xyz + dnsmasq proxyDHCP + unattended Proxmox installer; migrated from pve2 on 2026-07-24 |
Cluster rationalisation — 2026-07-07
Removed these stopped guests with pct/qm destroy --purge. Those with a
pre-destroy vzdump on their node's local are marked ✓; the rest were
throwaway/superseded and taken without a backup:
| Removed | Node | Was | Backup |
|---|---|---|---|
| CT 102 | pve1 | teleport (WireGuard + tinyauth SSO supersede it) | ✓ |
| CT 103 | pve1 | docker0X template (stale) | — |
| CT 104 | pve1 | pihole (never integrated) | — |
| CT 105 | pve1 | ubuntu template (stale) | — |
| CT 106 | pve1 | adguard (old, superseded by CT 120) | — |
| CT 108 | pve2 | plex (dup; Plex runs on Holly .210) |
— |
| CT 111 | pve1 | librenms | ✓ |
| CT 115 | pve1 | vaultwarden (test) | ✓ |
| CT 117 | pve2 | scanopy (test) | ✓ |
| CT 119 | pve2 | n8n (LAN automations move to Jarvis) | ✓ |
| VM 100 | pve1 | zabbix | ✓ |
| VM 107 | pve1 | apiscp-dev | ✓ |
| VM 109 | pve2 | PMS (dup Plex) | — |
CT 118 (grafana) was likewise destroyed 2026-07-06. Network mapping
(Zabbix/LibreNMS) is deferred to Jarvis rather than replaced. The vzdumps live
on each node's local and can be pruned or offsite-archived later.
Virtual Machines¶
| VMID | Name | Node | Status | RAM | Disk | Purpose |
|---|---|---|---|---|---|---|
| 102 | pbs | pve3 | up during pve3's wake window (onboot=1) |
4GB | 32GB (ZFS on tank) + the whole 500G bak disk |
Proxmox Backup Server 4.2.3 at 192.168.1.30 — primary guest-backup target (datastore main on ZFS dataset bak/pbs-datastore, pool imported in-VM since 2026-08-04; cluster storage pve3-pbs) |
| 121 | hermes-jarvis | pve2 | running | 8GB, 8 cores | 48GB (local-zfs-pve2, discard=on,iothread=1,ssd=1) |
Jarvis host — Hermes agent at 192.168.1.252 (see Jarvis). Migrated pve1 → pve2 on 2026-09-08 and resized onto the heavy node (was 3GB/32GB on pve1) |
| 123 | haos | pve1 | stopped | 2.5GB (trimmed from 3GB 2026-08-03) | 32GB (ZFS on local-zfs) | Home Assistant OS 18.2 (core 2026.7.4) at DHCP 192.168.2.120 - clean rebuild, built and parked (stopped, onboot=0) since 2026-08-27; the old 192.168.1.240 instance still owns the ha.mdhmedia.uk route. At cutover, start VM 123, set onboot=1, and move .240 + the route to it (see Home Automation) |
| 110 | unraid | pve2 | running | 32GB | USB boot | Unraid NAS "Holly" — 16 cores, 3× PCI passthrough, OVMF + TPM 2.0 |
Backups¶
Since 2026-07-26 the backup topology is PBS-primary with a local vzdump
floor and an incremental encrypted offsite. This realigns the earlier vzdump-only
setup around the new Proxmox Backup Server (VM 102 on pve3). Since 2026-07-30
the whole PBS leg runs in an evening wake window: the wol-pbs orchestrator
on CT 101 wakes pve3 at 19:00, the PBS jobs run, the offsite sync fires, and
the node shuts down again once PBS is idle (minimum wake 110 minutes). The times
below replaced the original 01:30 / 04:00–05:00 schedule. Full detail +
tooling: proxmox/backup/
in the repo.
Primary — Proxmox Backup Server (dedup / incremental)
PBS 4.2.3 on VM 102 (pve3, 192.168.1.30), datastore main —
since 2026-08-04 a native ZFS dataset (bak/pbs-datastore) on the
bak pool, which the VM imports itself (the whole 500G disk is passed
through; ARC capped at 1G; the old ext4-in-a-zvol layout is retired, the
zvol destroyed 2026-08-11 after a clean week) — registered cluster-wide
as storage pve3-pbs. A daily 19:15 job backs up all 9 pve1 guests
(100, 101, 112, 113, 114, 116, 120, 121, 122; keep-all — retention is
delegated to the PBS prune job). The full evening window:
| Time | Step |
|---|---|
| 19:00 | wol-pbs sends the magic packet; pve3 boots, the PBS VM autostarts (onboot=1, ~75 s to API) |
| 19:15 | PBS backup job — all 9 pve1 guests |
| 19:45 | Prune (pj-main) — keep-last 3 / daily 7 / weekly 4 / monthly 6 |
| 19:50 | Garbage collection — reclaims unreferenced chunks |
| 20:00 | Verify (vj-main) — re-checksums, outdated-after 7 |
| 20:35 | Holly's host/holly upload (appdata staging + boot flash from inside the Unraid VM — staggered here from 19:25 on 2026-08-03 to dodge pve3's concurrent-load chunk-digest errors; see Unraid) |
| after verify | Offsite sync (carries host/holly along), then shutdown once PBS is idle (minimum wake 110 minutes) |
A restore test on 2026-07-26 proved recoverability, and a full-datastore verify after the 2026-08-04 migration came back OK.
Verify-job incident — 2026-08-11
PBS shipped no telemetry of its own (no Alloy agent; asleep ~22 h/day),
so a genuine vj-main TASK ERROR was invisible until checked by hand. A
corrected re-verify (--ignore-verified false — the default true skips
exactly the backups you're trying to recheck) found 13 backups had
actually failed verification, including the latest ct/101 (the whole
docker host) and vm/121 (Jarvis). Fresh, fully-verified backups of both
were taken immediately as a stopgap. Root cause found 2026-08-24: a
one-off — every failed backup predates the 2026-08-04 datastore
migration (the 2026-08-03 zil_commit oops on the old ext4-in-zvol layout,
plus the faulty RAM removed after the 2026-08-22 memtest). The bak pool
has scrubbed with 0 errors and every verify has passed since 2026-08-19;
no post-migration backup has ever failed (see Open Items).
Prometheus now probes the PBS API directly for the latest verify
status (blackbox-pbs-verify job, PbsVerificationFailed alert) so this
can't go unnoticed again.
Local floor — vzdump to local (belt-and-braces during the soak)
Two local vzdump jobs remain, trimmed to keep-last 1 as a floor (cut from
keep-last 2 on 2026-08-24 to reduce pve1 SSD churn — one local copy under PBS):
| Job | Guests | Schedule | Retention |
|---|---|---|---|
| jarvis | VM 121 | Sun 02:00 | keep-last 1 |
| priority CTs | 100, 101, 112, 114, 116 | Sun 03:30 | keep-last 1 |
Snapshot mode / ZSTD → local. VM 110 (Unraid) is intentionally excluded (PCI
passthrough / USB boot) — Holly backs itself up instead: a
proxmox-backup-client inside the VM ships appdata + the boot flash to PBS
as host/holly (since 2026-07-31; upload at 20:35 — see
Unraid), with Unraid Connect's cloud flash backup as
the second flash copy.
Offsite — incremental, encrypted, from the PBS datastore
The offsite leg runs on the PBS VM: pbs-offsite-sync rclone copys the
datastore to hetzner-crypt:pbs on a jailed Hetzner Storage Box sub-account
(u195506-sub5), wrapped in rclone crypt (client-side encryption of
content and filenames) and gated against active PBS backup/GC/verify
tasks. Since 2026-07-30 it is started by the wol-pbs orchestrator at the
end of the evening window (after verify, before shutdown) — starting it is one
of only two commands the orchestrator's restricted SSH key permits — rather
than by the old 05:30 timer, when pve3 is now asleep. Because PBS chunks are
content-addressed, only changed chunks transfer — a strict improvement over
the old full-dump mirror. The crypt password/salt live only in the gitignored
proxmox/backup/.env.
The nightly copy never deletes (a one-way valve), so since 2026-08-04
a ~monthly guarded reconcile complements it: rclone sync with a
--max-delete tripwire and a dated --backup-dir archive — pruned chunks
eventually leave Hetzner, a mass delete trips the guard instead of running,
and reconcile archives are purged after 60 days.
Phase 2 — HELD, not just deferred (2026-08-11)
Done so far: pve1's legacy pve-offsite-sync.timer was disabled 2026-07-31
(units kept installed; the final hetzner-crypt:pve1 mirror stays as a frozen
extra copy), and Holly's appdata + boot-flash backup is live (host/holly,
since 2026-07-31). Still deferred: retire the two local vzdump jobs, reclaim
pve1 local, and extend the PBS job to the pve2 + pve3 guests (excluding
Holly VM 110 and the PBS VM 102 itself).
This was tracked as "resume after two clean weeks" until the 2026-08-11 verify-job incident above proved the PBS-only leg can go silently bad — the local vzdump floor is exactly what closed the gap while that was found. The corruption root cause is now understood (a one-off, resolved — see the Recently-closed entry in Open Items), so this is no longer blocked on that. But the floor saved the estate twice in nine days (the corruption and the five-night WoL gap), so retiring it is now a risk-appetite decision, not a calendar one.
This gives a live 3-2-1: PBS (primary, dedup) → local vzdump (keep-last-1 floor) → encrypted Hetzner offsite.