Proxmox VE 7 is vulnerable to some type of 0day/RCE non auth

[proxuser77]

Yes this is true for most cloudd-init based images of major distributions but in my book this isn't an issue per se

Yep, I wouldn't exactly call myself an expert, but I know how it works, and I even use cloud-init images on my Proxmox hosts to set up new VMs. My comments were just meant as examples to show that it’s normal with basic VPSs and dedicated servers to typically receive the VM or server with just the OS in a very rudimentary configuration, and then you have to figure things out from there.

But yes, you can do a lot more these days. With cloud-init and the APIs offered by many providers, you can actually deploy fully automated, securely configured VPSs. But in the end, you still have to take a proactive approach to most of it. The providers deliberately set very few parameters to give users maximum flexibility, but as you say, that also means that more effort is needed to operate such a server securely.

 

[netfront]

We were compromised through CVE-2023-54391 on 1–2 September and rebuilt 22 nodes from backup. A few of the post-exploitation findings cost us real time, so probably worth sharing here.

Getting the obvious out of the way first: we were on 7.4, well past EOL, with 8006 reachable from the internet because our customers log into it. That was our decision and our consequence.

**What was left behind**

An eBPF rootkit on every node, loaded by a systemd unit with a random 8-character name and `Description=System Management Daemon`. It hides its own files and processes from `ls`, `find` and `ps`, blocks `kill` against its own PIDs, blocks `ptrace`, and rewrites Monero addresses in flight. `dpkg -V` came back clean on `top`, `procps` and `libc6` — because the binaries genuinely are clean. The interception is in the kernel.

Detection is one command, and it's the only check the rootkit can't filter:

bpftool map show | grep -E 'protected_pids|exempt_comm|hidden_filename|ghost_files|hidden_names'

Any output means infected. If `bpftool` is missing, copy it from another node — on a host where dpkg keeps getting killed, apt won't finish.

`kill` doesn't work on it either, since it hooks that too. Use the cgroup — `echo 1 > /sys/fs/cgroup/<path>/cgroup.kill` — and disable, mask and remove the unit before killing.

**The part that cost us the most time**

We restored a host in place with `rsync -aHAX --delete` from backup. It completed without error and the rootkit was still resident.

The rootkit hides its files from `getdents`, and `rsync --delete` enumerates the destination with `getdents`. Files it can't see are files it doesn't delete, so the unit file and binaries survived a restore that reported success.

The order that worked: remove the rootkit, confirm with `bpftool map show`, **reboot**, then rsync. Hosts we rebuilt from a live USB never had this problem, because the hooked kernel isn't running.

**Cluster config**

`/etc/pve` has to be preserved for quorum, so it doesn't get restored — and that's where the API tokens and any added accounts live. We rebuilt 22 clean hosts and every one came back into a config that still held both, and that was our omission.

It was our fortune that before we open port 8006 again, we build additional checks on authenticity of any login, in addition to Proxmox's own authentication. This guards are based on a few vectors including IP address, past login records and patterns, and immediately blocks suspicious logins. It was this mechanism that detected the use and existance of the planted tokens. May be we should contribute our code to Proxmox for this part!

Therefore, worth checking `/etc/pve/priv/token.cfg` and your user list against your own provisioning records. Tokens don't appear in `pveum user list`, so an account audit won't surface them.

Honestly Proxmox is good piece of software. Software has bugs and sysadmin should never trust the authentication logic of Proxmox as well as those of other hypervisors. Always install guards that fit your need and that's a lesson we learned.

Here full write-up with IOCs, timeline, how actors exploited qemu-ga, plus many malicious or hacked IP addresses:

https://hosting.netfront.net/announ...-cluster-compromised-and-how-we-restored.html

 

[Johannes S]

Thanks for the writeup. One thing I'm missing: Did you you recreate the cluster with a non-EOL version (PVE9) or did you resetup them up with an EOL version? If you did the latter I would reconsider since although you might have closed this particular attack vector the same version and kernel has also a lof of other known and (potential unknown) issues. If there is something which really only runs in that old versions (like some lxcs) consider hosting that stuff behind a VPN on a non-EOL PVE9 (yes you can put PVE in PVE ). For VMs this comes with an obvious performance penalty but for lxc this shouldn't hurt to much.

Kudos for your backup strategy, everybody can get hurt by a bad choice, what matters is, that you can recover

I would consider to add the ProxmoxBackupServer since it allows to implement a pull-scheme for offsite backups too and you can configure the credentials that the PVE hosts can't alter backups, but only add new: https://pbs.proxmox.com/docs/storage.html#ransomware-protection-recovery

I know that the German webhoster uberspace uses a combined rsync/PBS strategy: They backup the userdirs of their users with rsync and present read-only network shares to them (so the users can do restores on their own in case they messed something up). Additionally they use PBS so they can restore a VM quickly if that need should arise. If you are doing it right (by seperating the backups so the rsync target and PBS live on different physical hardware) even if one backup target get compromised you could still recover from the other one.

 

[netfront]

Thanks for the writeup. One thing I'm missing. Did you you recreate the cluster with a non-EOL version (PVE9) or did you resetup them up with an EOL version? If you did the latter I would reconsider since although you might have closed this particular attack vector the same version and kernel has also a lof of other known and (potential unknown) issues. If there is something which really only runs in that old versions (like some lxcs) consider hosting that stuff behind a VPN on a non-EOL PVE9 (yes you can put PVE in PVE . For VMs this comes with an obvious performance penalty but for lxc this shouldn't hurt to much.

Kudos for your backup strategy, everybody can get hurt by a bad choice, what matters is, that you can recover

I would consider to add the ProxmoxBackupServer since it allows to implement a pull-scheme for offsite backups too and you can configure the credentials that the PVE hosts can't alter backups, but only add new: https://pbs.proxmox.com/docs/storage.html#ransomware-protection-recovery

I know that the German webhoster uberspace uses a combined rsync/PBS strategy: They backup the userdirs of their users with rsync and present read-only network shares to them (so the users can do restores on their own in case they messed something up). Additionally they use PBS so they can restore a VM quickly if that need should arise. If you are doing it right (by seperating the backups so the rsync target and PBS live on different physical hardware) even if one backup target get compromised you could still recover from the other one.

We chose the quickest route of restoring to the EOL version from backup first, because the cluster was barely holding quorum and the ceph osds hanging on a thread. And it was in fact unrealistic for us to do in between upgrades without a fully healthy cluster. But we will be upgrading soon it is in the pipeline.

Thank you for reading the write up. The fact that we can fully recover from all 22 nodes rooted is really amazing and a lot of luck. And that shows Proxmox's design is really robust, in my opinion.

 

[Johannes S]

We chose the quickest route of restoring to the EOL version from backup first, because the cluster was barely holding quorum and the ceph osds hanging on a thread. And it was in fact unrealistic for us to do in between upgrades without a fully healthy cluster. But we will be upgrading soon it is in the pipeline.

Then I would definitively put a VPN before the port 8006. If your customers are annoyed remind them, that you don't want to risk another incident and that it's in their best interest to make life harder for ransomware gangs.

 

[justin6.33]

We were compromised through CVE-2023-54391 on 1–2 September and rebuilt 22 nodes from backup. A few of the post-exploitation findings cost us real time, so probably worth sharing here.

Getting the obvious out of the way first: we were on 7.4, well past EOL, with 8006 reachable from the internet because our customers log into it. That was our decision and our consequence.

**What was left behind**

An eBPF rootkit on every node, loaded by a systemd unit with a random 8-character name and `Description=System Management Daemon`. It hides its own files and processes from `ls`, `find` and `ps`, blocks `kill` against its own PIDs, blocks `ptrace`, and rewrites Monero addresses in flight. `dpkg -V` came back clean on `top`, `procps` and `libc6` — because the binaries genuinely are clean. The interception is in the kernel.

Detection is one command, and it's the only check the rootkit can't filter:

bpftool map show | grep -E 'protected_pids|exempt_comm|hidden_filename|ghost_files|hidden_names'

Any output means infected. If `bpftool` is missing, copy it from another node — on a host where dpkg keeps getting killed, apt won't finish.

`kill` doesn't work on it either, since it hooks that too. Use the cgroup — `echo 1 > /sys/fs/cgroup/<path>/cgroup.kill` — and disable, mask and remove the unit before killing.

**The part that cost us the most time**

We restored a host in place with `rsync -aHAX --delete` from backup. It completed without error and the rootkit was still resident.

The rootkit hides its files from `getdents`, and `rsync --delete` enumerates the destination with `getdents`. Files it can't see are files it doesn't delete, so the unit file and binaries survived a restore that reported success.

The order that worked: remove the rootkit, confirm with `bpftool map show`, **reboot**, then rsync. Hosts we rebuilt from a live USB never had this problem, because the hooked kernel isn't running.

**Cluster config**

`/etc/pve` has to be preserved for quorum, so it doesn't get restored — and that's where the API tokens and any added accounts live. We rebuilt 22 clean hosts and every one came back into a config that still held both, and that was our omission.

It was our fortune that before we open port 8006 again, we build additional checks on authenticity of any login, in addition to Proxmox's own authentication. This guards are based on a few vectors including IP address, past login records and patterns, and immediately blocks suspicious logins. It was this mechanism that detected the use and existance of the planted tokens. May be we should contribute our code to Proxmox for this part!

Therefore, worth checking `/etc/pve/priv/token.cfg` and your user list against your own provisioning records. Tokens don't appear in `pveum user list`, so an account audit won't surface them.

Honestly Proxmox is good piece of software. Software has bugs and sysadmin should never trust the authentication logic of Proxmox as well as those of other hypervisors. Always install guards that fit your need and that's a lesson we learned.

Here full write-up with IOCs, timeline, how actors exploited qemu-ga, plus many malicious or hacked IP addresses:

https://hosting.netfront.net/announ...-cluster-compromised-and-how-we-restored.html

Why not deploy access to these host from internet with something like Cloudflare Tunnels/Cloudflare Access? I know a lot of people will say Tailscale, but if you are opening this up to clients and don't want to have to deal with them installing a client app, the Cloudflare option will give you a simple option of TOTP email codes, or you could integrate your own SSO platform that customers authenticate to, so no client apps are required. It's also free up to the first 50 users.

If it were me, I would go straight to setting up an IdP/SSO and integrating it with both Cloudflare and PVE so customers only need one account and you can set the authentication requirements with MFA options, password strength, geo-blocking, etc.

 

[proxuser77]

because our customers log into it

Sorry, but I'd say using Proxmox VE as a kind of VPS or RDP service over the internet is, is kind of risky by itself. I mean, have they ever had an audit, and if so, do they recommend or even sell it for that use case, i.e., using it for multi-tenant, public self-service offerings?

And even if all of that were in place, offering an EOL version to customers is unacceptable. Sorry, I don’t even need to hear the whole story. If you offer EOL software to customers, you’re being grossly negligent, and any story you tell afterward is simply one big excuse.

Maybe starting a different business would be a good idea if the profits of your current business model depend directly on the IT infrastructure you’re offering. Otherwise, using SaaS services might be better suited to your needs.

 

[proxuser77]

If it were me, I would go straight to setting up an IdP/SSO and integrating it with both Cloudflare and PVE so customers only need one account and you can set the authentication requirements with MFA options, password strength, geo-blocking, etc.

Those are certainly good ideas, but would they really have prevented this specific bypass? I don’t know. But even if they would, it doesn’t matter. Software has to be kept up to date, especially when you expose it to the public internet, regarless of SSO or no SSO, Cloudflare Tunnels or no Cloudflare Tunnels.

So, if anything, all of those measures should be used together. Or you could simply pay for support and ask the people who develop the software what you should do if you’re not able to assess the situation yourself.

But again: water is wet, the sky is blue, and if you expose EOL software to the internet, you’re practically begging to get pwned. You don’t need a support contract to know that. That’s kindergarten-level common knowledge in 2026. ;-)

 

[LongQT-sea]

For people stuck with 7.4 who can't upgrade right now but whose clients still require it to be accessible from the internet, I recommend doing this NOW: remove the default route (gateway) in the Proxmox network settings. This will kill most fully-automated commodity exploitation at the download/beacon stage.

 

[Johannes S]

For people stuck with 7.4 who can't upgrade right now but whose clients still require it to be accessible from the internet

What circumstances require this but don't allow to setup at least a VPN? People who are not able to use VPN shouldn't use Virtualization in the first place.

, I recommend doing this NOW: remove the default route (gateway) in the Proxmox network settings. This will kill most fully-automated commodity exploitation at the download/beacon stage.

This won't help if somebody uses an exploit to break out of a vm or lxc and afterwards put the default route again in place on the host.

Sorry folks but what's that hard in simply not running EOL software?

 

[LongQT-sea]

Just a temporary speed bump that could stop non-targeted or lazy hackers from exploiting that CVE to establish a connection back to their C2. Again, this is only for someone stuck on 7.4 who can't upgrade right now. It's not a solution, unless you still don't get it?

 

[shanreich]

For people stuck with 7.4 who can't upgrade right now but whose clients still require it to be accessible from the internet, I recommend doing this NOW: remove the default route (gateway) in the Proxmox network settings. This will kill most fully-automated commodity exploitation at the download/beacon stage.

In the related advisory there are instructions for a mitigation available [1]. As noted, the only long-term fix is of course upgrading from PVE 7 asap.

[1] https://forum.proxmox.com/threads/p...security-advisories.149331/page-4#post-867929

 

[LongQT-sea]

In the related advisory there are instructions for a mitigation available [1]. As noted, the only long-term fix is of course upgrading from PVE 7 asap.

[1] https://forum.proxmox.com/threads/p...security-advisories.149331/page-4#post-867929

I see, but what I mean is not just specific to this CVE. It could be any exploit. Egress filtering is still always recommended.

 

[netfront]

Hello all! Thank you for all the replies and suggestions. Yes, our solution is holding up really well so far. We have put in a lot of customizations and measures to all our infrastructure. Upgrades are on their way, without service interruption to clients. We just finished the ceph upgrades, and proxmox version updates in the pipe. But our additional security measures and codes will stay in place whatever new versions we gonna employ.

For the suggestions of using cloudflare tunnels and VPN, sure we understand one more level of security will reduce the risk surface by another magnitude. But we weighted the pros and cons and decided to keep our own security measures and upgrade the software. For instance, cloudflare tunnel would end-point the client TLS at cloudflare servers, and there might be some problem with the websocket applications of upload and download which some of our client utilizes. And there is another assumption, however slightly, that proxmox is not secure enough that's why we have to add another layer of protection, which we hope such assumption will eventually turn out to be invalid. Like proxmox does offer hooks for 2FA and other security improvements. And we trust these kinds of security problems will not happen again.

Thank you for all the good suggestions! Happy proxmoxing!

 

[toomanylogins]

I was quite alarmed to read this this morning. I have a V 7 and a load of old guests. And another running the latest version 9 that I'm migrating to. I'm just a single developer So it's quite time consuming. What constitutes connected to the internet for version 7? I have PVE running an IP fire guest as the firewall. This does not open port 22 or port 8006. And a separate network card goes straight to IP fire. The PVE Web Interfaces is on the local network. The only ports open on ipfire are for the email server no ssh access ti ipfire. Don't have any rules in the PVE firewall section. So for now, can I assume this is isolated from the internet as I assumed it was? Any advice much appreciated.
Thank you.

 

[Johannes S]

I was quite alarmed to read this this morning. I have a V 7 and a load of old guests. And another running the latest version 9 that I'm migrating to. I'm just a single developer So it's quite time consuming. What constitutes connected to the internet for version 7? I have PVE running an IP fire guest as the firewall. This does not open port 22 or port 8006. And a separate network card goes straight to IP fire. The PVE Web Interfaces is on the local network. The only ports open on ipfire are for the email server no ssh access ti ipfire. Don't have any rules in the PVE firewall section. So for now, can I assume this is isolated from the internet as I assumed it was? Any advice much appreciated.

No it's not. Since you run a PVE host on an EOL system there might be some (known or unknown) exploit which would enable an attacker to break out from an container or vm to shutdown your firewall guest or access it from the host.

Your best option would be to seperate your version V7 completely from the Internet. A potential mitigation might be to put something like cloudfare tunnels/pangolin/wireguard/tailscale/netbird before it, so that anything running on your old system is only accessible over that channel. It's not bullet-proof but would at least mitigate some issues of running EOL software.

Adding rules in PVEs firewall would be another useful mitigation but due to PVE7 being EOL this obviouvsly won't help if the attackaker manages to explot an issue with Linux kernel firewall.

 

[Johannes S]

Another option would be to run a PVE7 VM inside your PVE9 host. Obviovusly this would be a bad idea for VMs performancewise. For VMs it's best to migrate them to your new host and be done with it. But if you happen to have some old containers which also doesn't work on newer PVE versions migrating them to an tightly isolated PVE7 VM would at least reduce the blast radius. If you happen to have such old containers they will also propably be EOL so I would migrate their workloads (depending on whether they need an EOL distribution (some propietary software is "fun" in that regard) or not) to a VM of the same EOL distribution or a newer non-EOL version. I would also consider migrating from lxcs to vms in general since containers will always be more vulnerable to host issues than VMs.

Obviouvlsy if you happen to have any VMs with EOL systems you should still upgrade them to the last supported version (e.G. Debian11 to Debian13, Ubuntu18-Ubuntu26) or migrate the workloads to a newer, similiar distribution (e.G. from Centos7 to AlmaLinux10 or Fedora).

 

[proxuser77]

I'm just a single developer So it's quite time consuming.

PVE 7 is the second-to-last version. We’re now on version 9, and I think version 8 is also nearing end of life, or has it already reached it? I’m not 100% sure. Either way, the longer you wait, the more complicated and time-consuming it’ll be to update. Right now, there are two versions, but next year, when Forky is released, and with it PVE 10, it'l bee three versions.

And don't get me wrong: all the additional security measures presented here in this thread for mitigating such exploits are all well and good, and depending on the actual use case of the respective instances, at least some of them should probably be used anyway, even if everything is up to date. Keyword: "defense in depth" and so on.

But that shouldn't stop anyone from keeping their software up to date, especially the host and any VMs and containers that are somehow exposed to the internet. And I can only repeat myself: anything else is grossly negligent, even with additional mitigations.

 

[UdoB]

and I think version 8 is also nearing end of life, or has it already reached it?

Thread 'Proxmox VE - Support Lifecycle'

Jul 17, 2017

How long will my Proxmox VE version be supported?

Proxmox VE versions are supported at least as long as the corresponding Debian Version is oldstable. Proxmox VE uses a rolling release model and using the latest stable version is always recommended.

Proxmox VE 9
First Release: 2025-08, Debian Version 13 (Trixie), Debian EOL: TBA, Proxmox VE EOL: TBA

Proxmox VE 8
First Release: 2023-06, Debian Version 12 (Bookworm), Debian EOL: 2026-07, Proxmox VE EOL: 2026-08

Proxmox VE 7
First Release: 2021-07, Debian Version 11 (Bullseye), Debian...

• martin
• Replies: 41
• Forum: Proxmox VE: Installation and configuration

• 

 

[tschmidt]

This seems outdated. PVE 8's corresponding Debian version is Bookworm. Which is oldstable. Thus according to the policy PVE 8 should still be supported. Yet it isn't.