Proxmox VE 7 is vulnerable to some type of 0day/RCE non auth

[proxuser77]

Yes this is true for most cloudd-init based images of major distributions but in my book this isn't an issue per se

Yep, I wouldn't exactly call myself an expert, but I know how it works, and I even use cloud-init images on my Proxmox hosts to set up new VMs. My comments were just meant as examples to show that it’s normal with basic VPSs and dedicated servers to typically receive the VM or server with just the OS in a very rudimentary configuration, and then you have to figure things out from there.

But yes, you can do a lot more these days. With cloud-init and the APIs offered by many providers, you can actually deploy fully automated, securely configured VPSs. But in the end, you still have to take a proactive approach to most of it. The providers deliberately set very few parameters to give users maximum flexibility, but as you say, that also means that more effort is needed to operate such a server securely.

 

[netfront]

We were compromised through CVE-2023-54391 on 1–2 September and rebuilt 22 nodes from backup. A few of the post-exploitation findings cost us real time, so probably worth sharing here.

Getting the obvious out of the way first: we were on 7.4, well past EOL, with 8006 reachable from the internet because our customers log into it. That was our decision and our consequence.

**What was left behind**

An eBPF rootkit on every node, loaded by a systemd unit with a random 8-character name and `Description=System Management Daemon`. It hides its own files and processes from `ls`, `find` and `ps`, blocks `kill` against its own PIDs, blocks `ptrace`, and rewrites Monero addresses in flight. `dpkg -V` came back clean on `top`, `procps` and `libc6` — because the binaries genuinely are clean. The interception is in the kernel.

Detection is one command, and it's the only check the rootkit can't filter:

bpftool map show | grep -E 'protected_pids|exempt_comm|hidden_filename|ghost_files|hidden_names'

Any output means infected. If `bpftool` is missing, copy it from another node — on a host where dpkg keeps getting killed, apt won't finish.

`kill` doesn't work on it either, since it hooks that too. Use the cgroup — `echo 1 > /sys/fs/cgroup/<path>/cgroup.kill` — and disable, mask and remove the unit before killing.

**The part that cost us the most time**

We restored a host in place with `rsync -aHAX --delete` from backup. It completed without error and the rootkit was still resident.

The rootkit hides its files from `getdents`, and `rsync --delete` enumerates the destination with `getdents`. Files it can't see are files it doesn't delete, so the unit file and binaries survived a restore that reported success.

The order that worked: remove the rootkit, confirm with `bpftool map show`, **reboot**, then rsync. Hosts we rebuilt from a live USB never had this problem, because the hooked kernel isn't running.

**Cluster config**

`/etc/pve` has to be preserved for quorum, so it doesn't get restored — and that's where the API tokens and any added accounts live. We rebuilt 22 clean hosts and every one came back into a config that still held both, and that was our omission.

It was our fortune that before we open port 8006 again, we build additional checks on authenticity of any login, in addition to Proxmox's own authentication. This guards are based on a few vectors including IP address, past login records and patterns, and immediately blocks suspicious logins. It was this mechanism that detected the use and existance of the planted tokens. May be we should contribute our code to Proxmox for this part!

Therefore, worth checking `/etc/pve/priv/token.cfg` and your user list against your own provisioning records. Tokens don't appear in `pveum user list`, so an account audit won't surface them.

Honestly Proxmox is good piece of software. Software has bugs and sysadmin should never trust the authentication logic of Proxmox as well as those of other hypervisors. Always install guards that fit your need and that's a lesson we learned.

Here full write-up with IOCs, timeline, how actors exploited qemu-ga, plus many malicious or hacked IP addresses:

https://hosting.netfront.net/announ...-cluster-compromised-and-how-we-restored.html

 

[Johannes S]

Thanks for the writeup. One thing I'm missing: Did you you recreate the cluster with a non-EOL version (PVE9) or did you resetup them up with an EOL version? If you did the latter I would reconsider since although you might have closed this particular attack vector the same version and kernel has also a lof of other known and (potential unknown) issues. If there is something which really only runs in that old versions (like some lxcs) consider hosting that stuff behind a VPN on a non-EOL PVE9 (yes you can put PVE in PVE ). For VMs this comes with an obvious performance penalty but for lxc this shouldn't hurt to much.

Kudos for your backup strategy, everybody can get hurt by a bad choice, what matters is, that you can recover

I would consider to add the ProxmoxBackupServer since it allows to implement a pull-scheme for offsite backups too and you can configure the credentials that the PVE hosts can't alter backups, but only add new: https://pbs.proxmox.com/docs/storage.html#ransomware-protection-recovery

I know that the German webhoster uberspace uses a combined rsync/PBS strategy: They backup the userdirs of their users with rsync and present read-only network shares to them (so the users can do restores on their own in case they messed something up). Additionally they use PBS so they can restore a VM quickly if that need should arise. If you are doing it right (by seperating the backups so the rsync target and PBS live on different physical hardware) even if one backup target get compromised you could still recover from the other one.

 

[netfront]

Thanks for the writeup. One thing I'm missing. Did you you recreate the cluster with a non-EOL version (PVE9) or did you resetup them up with an EOL version? If you did the latter I would reconsider since although you might have closed this particular attack vector the same version and kernel has also a lof of other known and (potential unknown) issues. If there is something which really only runs in that old versions (like some lxcs) consider hosting that stuff behind a VPN on a non-EOL PVE9 (yes you can put PVE in PVE . For VMs this comes with an obvious performance penalty but for lxc this shouldn't hurt to much.

Kudos for your backup strategy, everybody can get hurt by a bad choice, what matters is, that you can recover

I would consider to add the ProxmoxBackupServer since it allows to implement a pull-scheme for offsite backups too and you can configure the credentials that the PVE hosts can't alter backups, but only add new: https://pbs.proxmox.com/docs/storage.html#ransomware-protection-recovery

I know that the German webhoster uberspace uses a combined rsync/PBS strategy: They backup the userdirs of their users with rsync and present read-only network shares to them (so the users can do restores on their own in case they messed something up). Additionally they use PBS so they can restore a VM quickly if that need should arise. If you are doing it right (by seperating the backups so the rsync target and PBS live on different physical hardware) even if one backup target get compromised you could still recover from the other one.

We chose the quickest route of restoring to the EOL version from backup first, because the cluster was barely holding quorum and the ceph osds hanging on a thread. And it was in fact unrealistic for us to do in between upgrades without a fully healthy cluster. But we will be upgrading soon it is in the pipeline.

Thank you for reading the write up. The fact that we can fully recover from all 22 nodes rooted is really amazing and a lot of luck. And that shows Proxmox's design is really robust, in my opinion.

 

[Johannes S]

We chose the quickest route of restoring to the EOL version from backup first, because the cluster was barely holding quorum and the ceph osds hanging on a thread. And it was in fact unrealistic for us to do in between upgrades without a fully healthy cluster. But we will be upgrading soon it is in the pipeline.

Then I would definitively put a VPN before the port 8006. If your customers are annoyed remind them, that you don't want to risk another incident and that it's in their best interest to make life harder for ransomware gangs.