Sign in Subscribe

Six Homelab Exporter Gotchas Nobody Writes Down

Hardening Prometheus exporters for Proxmox and PBS sounds like a copy-paste job. Then smartctl can't open a disk, pbs-exporter 404s on download, and pve-exporter 401s with a token that works everywhere else. Six small papercuts and their one-line fixes.

Six Homelab Exporter Gotchas Nobody Writes Down

Cozystack Gotchas — a series on the sharp edges of running this stack in a homelab. This one is a roundup: six small exporter papercuts, each with the fix. (Index →)

Standing up host-level metrics for a Proxmox + Proxmox Backup Server homelab is supposed to be a solved problem — grab the exporter, drop in a systemd unit, scrape it. In practice each of the three exporters I deployed had at least one undocumented (or under-documented) gotcha that cost real time, because every one of them fails silently: zero rows in a dashboard, a 404 on download, a 401 with a token that works everywhere else. Here are six, smallest blast radius to largest.

1. smartctl_exporter needs CAP_DAC_OVERRIDE — not just CAP_SYS_RAWIO + CAP_SYS_ADMIN

The READMEs and the few "hardened systemd unit" write-ups out there tell you that if you drop root, SMART scraping needs CAP_SYS_RAWIO (for the SATA ATA PASS-THROUGH ioctl) and CAP_SYS_ADMIN (for NVMe admin passthrough). On Debian 13 (Trixie) / Proxmox VE 9.x, those two are not enough — every scrape logs Permission denied on /dev/nvme0, /dev/sda, everything.

The catch: capabilities gate syscalls, but the failure is the open(2) of the device node, which is gated by ordinary DAC file permissions first. On PVE 9.x, NVMe character devices are crw------- root root (mode 0600, not even in the disk group). CAP_SYS_ADMIN doesn't bypass that — only CAP_DAC_OVERRIDE does.

Fix: add CAP_DAC_OVERRIDE to the unit's AmbientCapabilities (and CapabilityBoundingSet). Adding the user to the disk group helps for SATA/SCSI only — NVMe char devices aren't in disk on modern kernels.

2. smartctl_exporter 0.13.0 renamed the per-device metric

You import a community Grafana dashboard, point it at a 0.13.x exporter, and the "drives" panel is empty. Nothing is broken — the metric got renamed.

Older dashboards and tutorials query smartctl_device_info as the "one row per device" metric. As of 0.13.0 it's emitted as smartctl_device{...} — no _info suffix. Queries for the old name silently return zero rows.

Fix: query smartctl_device (and the smartctl_device_* family). If you're importing a dashboard, check it was updated for 0.13.x — Grafana dashboard ID 20204 already uses the new names.

3. pbs-exporter's release asset has a v prefix and a suffixed binary

The natural assumption (the Prometheus ecosystem convention) is an asset named pbs-exporter_X.Y.Z_linux_amd64.tar.gz with a pbs-exporter binary inside. Both halves are wrong for natrontech/pbs-exporter, so a naive installer 404s on download, and if you hard-code the binary name your install step fails with "file not found."

The actual asset is pbs-exporter_vX.Y.Z_linux_amd64.tar.gz (note the v in front of the version), and the binary inside is pbs-exporter-linux-amd64 (the platform suffix is kept, not stripped).

Fix: download the v-prefixed asset; reference the binary as pbs-exporter-linux-amd64 (rename it on install if you like).

4. pbs-exporter uses PBS_INSECURE, not PBS_INSECURE_SKIP_VERIFY

Most exporters that talk to a self-signed endpoint use a --tls.insecure-skip-verify flag or a *_SKIP_VERIFY env var, so that's what you reach for to scrape a PBS over its self-signed HTTPS. pbs-exporter uses PBS_INSECURE (flag pbs.insecure), taking "true"/"false". Set the longer name and the exporter silently keeps verification on and fails to scrape.

Fix: set PBS_INSECURE=true (or --pbs.insecure). Watch for the silent failure mode — wrong var name = no error, just no data.

5. prometheus-pve-exporter 401s until a full restart after a token edit

A common install flow writes /etc/prometheus/pve.yml with a placeholder token, starts the service, then sed-replaces the placeholder with the real token. The first scrape after that edit returns 401 Unauthorized: Authentication failed! — even though the exact same token works when you test it through proxmoxer.ProxmoxAPI() directly.

The exporter runs under gunicorn, which caches the parsed config at worker fork time. There's no reload-on-change watcher, so an in-place edit to pve.yml doesn't reach the running workers.

Fix: systemctl restart pve-exporter after any edit to pve.yml — a reload isn't enough. Make your installer always restart after touching the config.

6. PBS proxmox-backup-manager user generate-token rejects --output-format

PVE guides for monitoring tokens show pveum user token add ... --output-format json, and it's natural to assume the sibling PBS command takes the same flag. It doesn't — proxmox-backup-manager user generate-token rejects --output-format with schema does not allow additional properties. The PVE and PBS CLIs share a shape but diverge on small flags, and this is one of them.

The command always emits pretty-printed JSON (Result: {...}) on success, so you parse it ad hoc.

Fix: drop --output-format; strip the Result: prefix and pipe to jq (or grep the value field). On PBS, --output-format exists on some subcommands like list, but not generate-token.

The pattern

Five of these six fail silently — no crash, no error log, just an empty panel or an unauthenticated scrape that looks like a network problem. That's the real lesson for homelab monitoring: after you wire up an exporter, prove a metric actually lands in Prometheus before you move on. A green systemd unit tells you the process started, not that it can read your disks or that your dashboard speaks the same metric names it does.

References