We were compromised through CVE-2023-54391 on 1–2 September and rebuilt 22 nodes from backup. A few of the post-exploitation findings cost us real time, so probably worth sharing here.
Getting the obvious out of the way first: we were on 7.4, well past EOL, with 8006 reachable from the internet because our customers log into it. That was our decision and our consequence.
**What was left behind**
An eBPF rootkit on every node, loaded by a systemd unit with a random 8-character name and `Description=System Management Daemon`. It hides its own files and processes from `ls`, `find` and `ps`, blocks `kill` against its own PIDs, blocks `ptrace`, and rewrites Monero addresses in flight. `dpkg -V` came back clean on `top`, `procps` and `libc6` — because the binaries genuinely are clean. The interception is in the kernel.
Detection is one command, and it's the only check the rootkit can't filter:
bpftool map show | grep -E 'protected_pids|exempt_comm|hidden_filename|ghost_files|hidden_names'
Any output means infected. If `bpftool` is missing, copy it from another node — on a host where dpkg keeps getting killed, apt won't finish.
`kill` doesn't work on it either, since it hooks that too. Use the cgroup — `echo 1 > /sys/fs/cgroup/<path>/cgroup.kill` — and disable, mask and remove the unit before killing.
**The part that cost us the most time**
We restored a host in place with `rsync -aHAX --delete` from backup. It completed without error and the rootkit was still resident.
The rootkit hides its files from `getdents`, and `rsync --delete` enumerates the destination with `getdents`. Files it can't see are files it doesn't delete, so the unit file and binaries survived a restore that reported success.
The order that worked: remove the rootkit, confirm with `bpftool map show`, **reboot**, then rsync. Hosts we rebuilt from a live USB never had this problem, because the hooked kernel isn't running.
**Cluster config**
`/etc/pve` has to be preserved for quorum, so it doesn't get restored — and that's where the API tokens and any added accounts live. We rebuilt 22 clean hosts and every one came back into a config that still held both, and that was our omission.
It was our fortune that before we open port 8006 again, we build additional checks on authenticity of any login, in addition to Proxmox's own authentication. This guards are based on a few vectors including IP address, past login records and patterns, and immediately blocks suspicious logins. It was this mechanism that detected the use and existance of the planted tokens. May be we should contribute our code to Proxmox for this part!
Therefore, worth checking `/etc/pve/priv/token.cfg` and your user list against your own provisioning records. Tokens don't appear in `pveum user list`, so an account audit won't surface them.
Honestly Proxmox is good piece of software. Software has bugs and sysadmin should never trust the authentication logic of Proxmox as well as those of other hypervisors. Always install guards that fit your need and that's a lesson we learned.
Here full write-up with IOCs, timeline, how actors exploited qemu-ga, plus many malicious or hacked IP addresses:
https://hosting.netfront.net/announ...-cluster-compromised-and-how-we-restored.html
