Logging & Metrics Stack¶

MDH Media homelab uses Grafana Loki for centralized log aggregation, Prometheus for metrics storage, and Grafana Alloy as the unified collection agent for both logs and metrics.

Architecture¶

flowchart TD
    subgraph Hosts[Monitored non-docker Hosts]
        A1[VM/LXC 1<br/>Alloy]
        A2[VM/LXC 2<br/>Alloy]
        A3[Caddy<br/>Alloy]
    end

    subgraph Collector[Docker LXC - 192.168.1.241]
        Loki[Loki<br/>:3100]
        Prom[Prometheus<br/>:9090]
        Graf[Grafana<br/>:3300]
    end

    A1 -->|logs| Loki
    A2 -->|logs| Loki
    A3 -->|logs| Loki
    A1 -->|metrics| Prom
    A2 -->|metrics| Prom
    A3 -->|metrics| Prom
    Loki --> Graf
    Prom --> Graf

Services¶

Grafana Loki¶

• Host: Docker LXC (CT 101, pve1)
• IP: 192.168.1.241
• Port: 3100
• Role: Log aggregation and storage
• Stack: stacks/monitoring/loki/ (Arcane git-sync stack)
• Config: stacks/monitoring/loki/loki-config.yml

Prometheus¶

• Host: Docker LXC (CT 101, pve1)
• IP: 192.168.1.241
• Port: 9090
• Role: Metrics storage (via remote_write API)
• Stack: stacks/monitoring/prometheus/ (Arcane git-sync stack)
• Config: stacks/monitoring/prometheus/prometheus.yml

Grafana Alloy¶

• Role: Unified log and metrics collection agent
• Config: /etc/alloy/config.alloy
• Service: systemctl status alloy
• UI: http://localhost:12345 (when enabled)

Collects:

• System metrics: CPU, memory, disk, filesystem, load, network
• Systemd journal logs
• Syslog and auth logs
• Application-specific logs (Caddy, Docker, etc.)

Installation¶

Alloy (Generic Host)¶

Run on any non-docker VM/LXC to collect system logs and metrics and forward them to the collector on 192.168.1.241:

curl -O https://raw.githubusercontent.com/MDHMatt/mdhmedia/main/host-agents/install-alloy.sh
chmod +x install-alloy.sh
sudo ./install-alloy.sh                 # ships to 192.168.1.241 by default

# Or pass the collector IP explicitly
sudo ./install-alloy.sh 192.168.1.241

Features:

• Collects system metrics (CPU, memory, disk, filesystem, load, network)
• Forwards metrics to Prometheus via remote_write
• Collects systemd journal, syslog, and auth logs
• Forwards logs to Loki

Alloy (Caddy Reverse Proxy)¶

Specialized script for the Caddy VM (CT 112, pve1) with JSON access log parsing:

curl -O https://raw.githubusercontent.com/MDHMatt/mdhmedia/main/host-agents/install-alloy-caddy.sh
chmod +x install-alloy-caddy.sh
sudo ./install-alloy-caddy.sh 192.168.1.241

Features:

• Collects system metrics (CPU, memory, disk, filesystem, load, network)
• Parses Caddy JSON access logs at /var/lib/caddy/logs/access.log
• Extracts labels for filtering: request_host, request_method, status
• Collects Caddy systemd service logs
• Creates human-readable log lines from JSON

Manual Installation¶

# Add Grafana repository (Debian/Ubuntu - DEB822 format)
sudo mkdir -p /etc/apt/keyrings
sudo wget -q -O /etc/apt/keyrings/grafana.asc https://apt.grafana.com/gpg-full.key
sudo chmod 644 /etc/apt/keyrings/grafana.asc

cat << 'EOF' | sudo tee /etc/apt/sources.list.d/grafana.sources
Types: deb
URIs: https://apt.grafana.com
Suites: stable
Components: main
Signed-By: /etc/apt/keyrings/grafana.asc
EOF

# Install
sudo apt-get update
sudo apt-get install alloy

# Configure and start
sudo systemctl enable --now alloy

Configuration¶

Default Log Sources¶

Caddy Log Labels¶

Adding Custom Log Sources¶

Edit /etc/alloy/config.alloy:

local.file_match "myapp" {
  path_targets = [
    {
      __path__ = "/var/log/myapp/*.log",
      job      = "myapp",
    },
  ]
}

loki.source.file "myapp" {
  targets    = local.file_match.myapp.targets
  forward_to = [loki.write.default.receiver]
  tail_from_end = true
}

Reload configuration:

sudo systemctl reload alloy

Node Metrics Configuration¶

The scripts configure system metrics collection using prometheus.exporter.unix:

prometheus.exporter.unix "node" {
  set_collectors = ["cpu", "meminfo", "diskstats", "filesystem", "loadavg", "netdev", "netclass", "uname", "stat", "time", "vmstat", "hwmon", "thermal_zone", "cpufreq"]
  filesystem {
    fs_types_exclude = "^(autofs|binfmt_misc|bpf|cgroup2?|cifs|configfs|debugfs|devpts|devtmpfs|fuse.*|fusectl|hugetlbfs|iso9660|mqueue|nfs4?|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|smb3|squashfs|sysfs|tracefs)$"
  }
}

prometheus.scrape "node" {
  targets    = prometheus.exporter.unix.node.targets
  forward_to = [prometheus.remote_write.default.receiver]
  scrape_interval = "15s"
  job_name   = "node"
}

Example Configurations¶

Docker Container Logs¶

local.file_match "docker" {
  path_targets = [
    {
      __path__         = "/var/lib/docker/containers/*/*.log",
      job              = "docker",
      __path_exclude__ = "/var/lib/docker/containers/*/*-json.log.*.gz",
    },
  ]
}

loki.source.file "docker" {
  targets    = local.file_match.docker.targets
  forward_to = [loki.write.default.receiver]
  tail_from_end = true
}

Nginx Logs¶

local.file_match "nginx" {
  path_targets = [
    {__path__ = "/var/log/nginx/access.log", job = "nginx", type = "access"},
    {__path__ = "/var/log/nginx/error.log", job = "nginx", type = "error"},
  ]
}

loki.source.file "nginx" {
  targets    = local.file_match.nginx.targets
  forward_to = [loki.write.default.receiver]
  tail_from_end = true
}

Querying Logs¶

LogQL Examples¶

# All logs from a specific host
{host="docker-pve1"}

# Systemd journal logs for a specific unit
{job="systemd-journal", unit="nginx.service"}

# Search for errors across all hosts
{env="homelab"} |= "error"

# Auth failures
{job="auth"} |= "Failed password"

# Top error-producing hosts (last hour)
sum by (host) (count_over_time({env="homelab"} |= "error" [1h]))

# SSH login attempts
{job="auth"} |~ "sshd.*Accepted|sshd.*Failed"

# Systemd service failures
{job="systemd-journal"} |= "Failed to start"

# Caddy requests by service
{job="caddy", request_host="grafana.mdhmedia.uk"}

# Caddy 5xx errors
{job="caddy", status=~"5.."}

Querying Metrics¶

PromQL Examples¶

# CPU usage by host (percentage)
100 - (avg by(host) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory usage by host (percentage)
100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

# Disk usage by host and mountpoint (percentage)
100 - (node_filesystem_avail_bytes / node_filesystem_size_bytes * 100)

# Network traffic (bytes/sec)
rate(node_network_receive_bytes_total[5m])
rate(node_network_transmit_bytes_total[5m])

# System load average (1 minute)
node_load1

# Top 5 hosts by CPU usage
topk(5, 100 - (avg by(host) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100))

# Hosts with disk usage > 80%
node_filesystem_avail_bytes / node_filesystem_size_bytes * 100 < 20

# All metrics from a specific host
{host="caddy"}

Dashboards¶

Log Explorer (formerly Homelab Logs)¶

• File: stacks/monitoring/grafana/provisioning/dashboards/Logs & Diagnostics/loki-homelab-logs.json

Variables: host, job, search

Caddy Access Logs¶

• File: stacks/monitoring/grafana/provisioning/dashboards/Edge & Security/caddy-access-logs.json

Variables: host (service), status, search

Fleet Trends (formerly Homelab Overview)¶

• File: stacks/monitoring/grafana/provisioning/dashboards/Infrastructure/homelab-overview.json

Variables: host, job, timeRange

Dashboard provisioning¶

These JSON files are file-provisioned from the repo — the Grafana stack mounts stacks/monitoring/grafana/provisioning/dashboards/ and loads them on startup (see provisioning/dashboards/dashboards.yml), so a merge to main redeploys them via GitOps. No manual import. After editing a dashboard in the Grafana UI, export the JSON back into that folder to keep it version-controlled.

Troubleshooting¶

Alloy Not Starting¶

sudo systemctl status alloy
sudo journalctl -u alloy -f
alloy fmt /etc/alloy/config.alloy  # Check syntax and format

Common config errors: - Unrecognized attribute: Options like fs_types_exclude must be inside nested blocks (e.g., filesystem { }) - Missing quotes: String values must be quoted - Invalid regex: Check escape characters in patterns

No Logs Appearing in Loki¶

1. Check Alloy is running: systemctl status alloy
2. Test Loki connectivity: curl http://192.168.1.241:3100/ready
3. Check firewall: Ensure port 3100 is open
4. Verify permissions:

   sudo usermod -aG systemd-journal alloy
   sudo usermod -aG adm alloy
   sudo systemctl restart alloy
   

High Memory Usage (Alloy)¶

Reduce batch size in /etc/alloy/config.alloy:

loki.write "default" {
  endpoint {
    url = "http://192.168.1.241:3100/loki/api/v1/push"
    batch_size = "512KiB"
    batch_wait = "2s"
  }
}

Loki Not Starting¶

Loki runs as a docker container on the Docker LXC (CT 101, 192.168.1.241). Check it via pve1:

ssh pve1 "pct exec 101 -- docker ps --filter name=loki"
ssh pve1 'pct exec 101 -- docker logs loki --tail 50'

Config errors show up in the container logs. The config lives in the repo at stacks/monitoring/loki/loki-config.yml — edit it there and merge to main; Arcane GitOps redeploys the stack.

Loki High Disk Usage¶

Adjust retention in stacks/monitoring/loki/loki-config.yml (deploys via merge to main / Arcane GitOps):

limits_config:
  retention_period: 720h  # 30 days

compactor:
  retention_enabled: true
  retention_delete_delay: 2h

Loki Out of Memory¶

Reduce ingester settings in stacks/monitoring/loki/loki-config.yml:

ingester:
  chunk_idle_period: 30m
  max_chunk_age: 1h
  chunk_target_size: 1048576

Prometheus Not Starting¶

Prometheus runs as a docker container on the Docker LXC (CT 101, 192.168.1.241). Check it via pve1:

ssh pve1 "pct exec 101 -- docker ps --filter name=prometheus"
ssh pve1 'pct exec 101 -- docker logs prometheus --tail 50'

The config lives in the repo at stacks/monitoring/prometheus/prometheus.yml. Note that the Prometheus git stack does not auto-reload config changes — after a merge to main, the container needs a redeploy/restart to pick them up.

No Metrics in Prometheus¶

1. Check Alloy is running: systemctl status alloy
2. Test Prometheus connectivity: curl http://192.168.1.241:9090/-/ready
3. Check firewall: Ensure port 9090 is open
4. Verify remote_write is enabled in Prometheus (check for --web.enable-remote-write-receiver flag)
5. Check Alloy config has correct Prometheus URL

Prometheus High Disk Usage¶

Adjust retention via the --storage.tsdb.retention.time flag in the Prometheus compose file (stacks/monitoring/prometheus/docker-compose-prometheus.yml):

--storage.tsdb.retention.time=15d

Then merge to main — changing the compose file triggers an Arcane GitOps redeploy of the stack (a plain prometheus.yml config change would additionally need a manual restart, since Prometheus does not auto-reload its config).

References¶

• Grafana Alloy Documentation
• Loki Documentation
• LogQL Reference
• Prometheus Documentation
• PromQL Reference