Proxmox VE 7 is vulnerable to some type of 0day/RCE non auth
- Thread starter pve7bug
- Start date
Obviously 7 is EOL, but the big main issue here is, why is something like this not firewalled off with restricted IP access. Also have you done any backups at all in those 4 years?
Even my oldest bodged MBR GRUB install, has been updated over time and behind IP ACL.
Never sit relying on vendor to keep things secure for you (which is why people have given tough love in the replies), assume it can be compromised, assume your server might go on fire one day, take precautions against these events.
Even my oldest bodged MBR GRUB install, has been updated over time and behind IP ACL.
Never sit relying on vendor to keep things secure for you (which is why people have given tough love in the replies), assume it can be compromised, assume your server might go on fire one day, take precautions against these events.
Last edited:
Sadly and most likely compilance theater. If they are in an environment where it's mandatory to have a cyber security insurance and said insurance state in their conditions that they only pay if you patched all of your systems then change managers might finally approve of doing patches. Even then they also might pass the ball to the legal department wheter the "patch or we don't pay"-clause might be taken to court.
Another potential way to enforce this via compilance theater/checkboxing would be a legal obligation like the moronic "you are not allowed to update without recertifying everything for a high amount of cash"-rules in parts of critical infrastructure. It was quite funny to read following report on an talk kernel developer Greg Kroah-Hartmann did on Linux kernels cve issueing process:
So if the same rules state that any updates needs an recertification would also state that any CVE or security issue with a known fix needs to be fixed asap otherwise you loose your clearance for such environments, more people will update.
To be clear: I hate the CVE process and compilance-instead-actual-security debacle of "enterprise it" with passion, but if managment only cares about ticking checkboxes, then that would help. I wouldn't hold my breath though, that lawmakers or insurance companies will actually do this.
Proxmox VE 7.1 - cryptojacking rootkit + root SSH backdoor: full teardown
TL;DR
Two PVE 7 hosts were compromised and running an XMRig cryptominer hidden by a preload rootkit. The same "install_and_mine.sh / libhide" toolchain planted:
The binaries (sshd, ps, ls, systemd, libc) were not tampered with — the attack is config/file based. Key lessons: don't trust ls/ps/find/ss even as root, and don't trust mtime.
Environment
Timeline (reconstructed)
What was planted
1) Miner (identical structure on both hosts)
2) Stealth library — why nothing showed up
3) Persistence — survives reboot and kill
4) SSH root backdoor (separate from miner) — the finding that matters most
/etc/ssh/sshd_config end of file:
5) PVE auth module replacement
6) Remote tools deleted
Anti-forensics observed
How we confirmed / what to check on your own hosts
Set LD_PRELOAD unset or the rootkit hides everything:
Use birth time, not mtime (crtime can't be faked this way):
Check the SSH authorized-keys command backdoor:
Package integrity:
Biggest blind spots we hit: find -mtime/-newermt won't see backdated files, and dpkg -V shows config diffs that look benign. Combine birth-time scans + dpkg -V + unsetting LD_PRELOAD.
What we recommend in cleanup/rebuild
Unknowns remaining
Sample IOCs (normalized)
TL;DR
Two PVE 7 hosts were compromised and running an XMRig cryptominer hidden by a preload rootkit. The same "install_and_mine.sh / libhide" toolchain planted:
- A miner plus stealth library hiding its process, port and files from every standard tool.
- A separate root SSH key backdoor that survives any miner cleanup (this was the unexpected one).
- A replaced AccessControl.pm (PVE auth module).
- Deleted who / top / w.
- Aggressive log/forensics destruction.
The binaries (sshd, ps, ls, systemd, libc) were not tampered with — the attack is config/file based. Key lessons: don't trust ls/ps/find/ss even as root, and don't trust mtime.
Environment
- Proxmox VE 7.1-7 (pve-manager/7.1-7/df5740ad), kernel 5.13.19-2-pve, Debian 11 Bullseye.
- Two independently-running hosts; both infected identically, ~2 min apart.
- Production VMs (several QEMU guests incl. web/DB) unaffected at application layer.
Timeline (reconstructed)
- T0 00:00-14:12: /var/log/pveproxy/access.log wiped for this whole window — attack entry vector lost.
- T0 12:52: Rogue PVE accounts created: [REDACTED]@pve (Administrator) + root@pam![REDACTED] API token.
- T0 13:14:4x: Rootkit + SSH backdoor installed on both nodes (birth times identical on both).
- T0 13:14-14:05: Root shell profile/history rewritten; logs symlinked to /dev/null; journald.conf changed; auth.log / wtmp / btmp / lastlog / secure -> symlinks.
- T0 14:05: /etc/shadow + /etc/passwd touched on both nodes — root password almost certainly rotated by attacker.
- T0 16:15+: Two VMs found stopped — consistent with host CPU starvation from the miner.
What was planted
1) Miner (identical structure on both hosts)
- Binary /var/lib/systemd/PVE-1 (XMRig), pool [REDACTED]:20004, C2 [REDACTED], wallet [REDACTED], TLS, --donate-level=0.
- Launcher /var/lib/systemd/ld-svc; mtime backdated to 2018-01-15 on both hosts.
- chattr +a / +i not visible as immutable at scan time (only 'e' flag) — but the build clearly intended stealth attrs (state file .stealth-attrs=1).
2) Stealth library — why nothing showed up
- /var/lib/systemd/.hide/libhide.so, injected via LD_PRELOAD in two places: /etc/environment and /etc/profile.d/sysenv.sh.
- Hides: its process name from ps, its port (20004) from ss/netstat, and its files from ls/find — even from a root shell. Only visible after env -u LD_PRELOAD.
3) Persistence — survives reboot and kill
- PVE-1.service (miner, Restart=always)
- PVE-1.timer + PVE-1-maintain.service — restart guard every ~3 min
- PVE-1-update.timer + PVE-1-update.service — daily auto-updater pulling official XMRig, SHA-256-verified, atomic replace
4) SSH root backdoor (separate from miner) — the finding that matters most
/etc/ssh/sshd_config end of file:
Code:
Match User root
AuthorizedKeysCommand /etc/ssh/.sshd-keys-cache
AuthorizedKeysCommandUser root
PubkeyAuthentication yes- /etc/ssh/.sshd-keys-cache -> outputs /etc/ssh/.cache/root.keys
- /etc/ssh/.cache/root.keys = attacker ed25519 key ([REDACTED])
- sshd -T -C user=root confirms it's live: authorizedkeyscommand /etc/ssh/.sshd-keys-cache
- /etc/ssh/.hist-arm + /root/.config/fish/config.fish = self-healing 2h shell-history suppression ("svc-history-guard") to hide their commands
5) PVE auth module replacement
- /usr/share/perl5/PVE/AccessControl.pm replaced, birth T0 13:14:42, mtime faked to 2021-11-25, identical SHA256 on both hosts. Almost certainly the enabler for the rogue-account/console actions, or a GUI re-access backdoor.
6) Remote tools deleted
- who, top, w deleted (not replaced) — so the admin's own checks silently fail. dpkg -V flags them.
Anti-forensics observed
- /var/log/auth.log, wtmp, btmp, lastlog, secure -> symlinks to /dev/null (re-pointed 14:05)
- journald.conf -> Storage=none
- pveproxy access log morning window deleted
- Root .bash_history / .history wiped or never logged; fish + bash history suppressed
- /etc/resolv.conf forced to 8.8.8.8 / 1.1.1.1 (x3) — DNS redirected to attacker-chosen resolvers
- /etc/hosts "donate-block" region added (blocks xmrig donation domains) — generated from data file /var/lib/systemd/.cache/.donate-block
- mtimes backdated everywhere (2018/2021/2022); birth times (crtime) give the truth
How we confirmed / what to check on your own hosts
Set LD_PRELOAD unset or the rootkit hides everything:
Code:
env -u LD_PRELOAD ps aux | grep -i -E 'xmrig|PVE-1|ld-svc'
env -u LD_PRELOAD ss -tulpn | grep ':20004'
env -u LD_PRELOAD find /var/lib/systemd -type f
env -u LD_PRELOAD ls -la /etc/environment /etc/profile.d/ /var/lib/systemd/.hide/Use birth time, not mtime (crtime can't be faked this way):
Code:
stat -c '%w %n' /path/to/suspectCheck the SSH authorized-keys command backdoor:
Code:
grep -A3 'Match User' /etc/ssh/sshd_config
sshd -T -C user=root | grep -i authorizedkeyscommand
find /etc/ssh -name '*.keys*' -o -name '.cache*'Package integrity:
Code:
dpkg -V openssh-server pve-manager libpve-access-control coreutils procpsBiggest blind spots we hit: find -mtime/-newermt won't see backdated files, and dpkg -V shows config diffs that look benign. Combine birth-time scans + dpkg -V + unsetting LD_PRELOAD.
What we recommend in cleanup/rebuild
- Remove the SSH backdoor first (it's the real access, separate from the miner): delete the Match User root block, .sshd-keys-cache, /etc/ssh/.cache, /etc/ssh/.hist-arm, set PermitRootLogin no, rotate all host keys + root authorized_keys.
- apt-get install --reinstall libpve-access-control coreutils procps (restore AccessControl.pm + who/top/w).
- Kill/remove miner + 4 systemd units + chattr -i/-a where present + delete /var/lib/systemd/{PVE-1,ld-svc,.hide,.cache}, /etc/profile.d/sysenv.sh, restore /etc/environment, /etc/hosts, /etc/resolv.conf.
- Re-point logs, restore journald.conf.
- Rotate ALL credentials: node root/passwords, PVE users/tokens, SSH host+user keys, IPMI, backup system, application secrets.
- Consider migrating VMs off the compromised hosts entirely (full images exist) — cleanest given the forensic unknowns (wiped log window).
Unknowns remaining
- The original entry vector (wipe of the pre-14:12 pveproxy window wiped it)
- Exact payload of the replaced AccessControl.pm
Sample IOCs (normalized)
Code:
/var/lib/systemd/PVE-1 (XMRig binary)
/var/lib/systemd/ld-svc (launcher)
/var/lib/systemd/.hide/libhide.so (preload rootkit)
/var/lib/systemd/.cache/ (state/donate-block/hugepages state)
/usr/libexec/PVE-1-maintain
/usr/libexec/PVE-1-update
/etc/systemd/system/PVE-1.service|timer
/etc/systemd/system/PVE-1-maintain.service
/etc/systemd/system/PVE-1-update.service|timer
/etc/profile.d/sysenv.sh
/etc/environment (LD_PRELOAD export)
/etc/ssh/.sshd-keys-cache, .cache/root.keys, .hist-arm
/usr/share/perl5/PVE/AccessControl.pm (replaced)
who/top/w deletedNever expose 8006 port publicly.
I can't say anything about the other parts of the writeup, but the "recommendation" for cleanup/rebuild irritates me:
I can't agree with this. This is a lot of work while you still could overlook something or the bad actor (now that the attack is known) adds thing to stay in the system even after publication of the attack. If you get compromised you have no other option than to wipe your systems and restore from backup, end of story.
Or to be more precise: If up to now you run an EOL system AND exposed it to the Internet you will get hacked again (from this attack or some other, EOL systems usually have a lot of known and unknown vulnerabilities).
So instead of this rather complicated, awkward and obtuse setup the whole part should be rewritten to:
Everything else is just asking for going through the whole trouble again.
I can't agree with this. This is a lot of work while you still could overlook something or the bad actor (now that the attack is known) adds thing to stay in the system even after publication of the attack. If you get compromised you have no other option than to wipe your systems and restore from backup, end of story.
Or to be more precise: If up to now you run an EOL system AND exposed it to the Internet you will get hacked again (from this attack or some other, EOL systems usually have a lot of known and unknown vulnerabilities).
So instead of this rather complicated, awkward and obtuse setup the whole part should be rewritten to:
Everything else is just asking for going through the whole trouble again.
Please stop lecturing everyone from your limited perspective.
If you have Centos 7 LXC containers (CentOS is still supported by tuxcare for a few dollars a month), migrating to PVE 9 from 8 or 7 isn't simply restoring backups because the appropriate cgroups are no longer supported.
PVE 7 and 8 kernels also can have current livepatched kernel fixes from tuxcare, and Debian itself gets free updates from Freexian. The only problem in this case is the Proxmox backend and frontend. It should be behind a firewall, that's obvious, but it's also clear, that this particular error was due to an earlier oversight that existed when version 7 was still supported...
If you have Centos 7 LXC containers (CentOS is still supported by tuxcare for a few dollars a month), migrating to PVE 9 from 8 or 7 isn't simply restoring backups because the appropriate cgroups are no longer supported.
PVE 7 and 8 kernels also can have current livepatched kernel fixes from tuxcare, and Debian itself gets free updates from Freexian. The only problem in this case is the Proxmox backend and frontend. It should be behind a firewall, that's obvious, but it's also clear, that this particular error was due to an earlier oversight that existed when version 7 was still supported...
Last edited:
Yes, my view is indeed quite limited: The Internet would be in a better state if people would not run EOL systems on it.
But you are missing my actual point: It's pointless to clean an infected system since you might have missed an unknown backdoor also planted by the bad actor. So you will need to wipe and reinstall the systen plus restoring backups anyhow, no way around it. This has nothing to do with EOL systems or not, you would need to do the same in case of an compromised Debian13 or ProxmoxVE 9 system.
Just reinstalling won't do much though if you continue with the practice which led to the compromise in the first place (expose EOL software without security patches on the Internet). You will end up getting hacked again. The least you can do is to put some firewall rules and reverse proxy/vpn before it.
You could still run PVE7 inside a VM which also is tightly secured (so not directly exposed to the Internet, strong firewall against it etc) or migrate your CentOS LXCs to CentOS vms for better seperation. This wouldn't be ideal but still a lot better than running bare metal AND exposing it to the internet. And tuxcare supporting CentOS doesn't help you with host issues when you run it inside from lxcs.
I also think that your example shows why it's a bad idea to use containers if you not willing to migrate your workloads to newer containers version/technologies.
Yes but in the end it doesn't matter wheter you get owned through the Kernel, Debian or Proxmox backend. Or are you seriously suggesting that the ProxmoxVE team should provides updates forever for their old software versions?
And the "free updates" by freexian are not for everyone and limited in scope:
The world would also be better without greed, wars, etc., but we have to take things as they are.
All I'm suggesting is that the bug existed while Proxmox 7 was still supported (which was confirmed by devs) and it only became public few days ago.
As for freexian, I doubt that anyone other than individual users and small organizations would comment on this forum and use the community version of proxmox.
The main thing I wanted to point out was that it is possible for someone to have an updated system and kernel despite having EOL PVE 7.x.
All I'm suggesting is that the bug existed while Proxmox 7 was still supported (which was confirmed by devs) and it only became public few days ago.
As for freexian, I doubt that anyone other than individual users and small organizations would comment on this forum and use the community version of proxmox.
The main thing I wanted to point out was that it is possible for someone to have an updated system and kernel despite having EOL PVE 7.x.
Obviously this is possible, since anyone can compile the kernel as he sees fit.
However, I do not believe that Proxmox VE or their community needs to invest time nor resources to maintain (extreme) EOL systems.
I also believe Johannes S is correct, if your system has already been owned, you need to restart with a completely fresh system - no patching & monitoring is going to suffice. I also believe this should include LXCs & VMs which will have to be restored from known (offsite?) snapshots/backups pre-attack, unless extreme-scientific hard evidence is provided that these are untouched (which I see as an unlikely possibility).
This is the nature of bugs, they take time to be be discovered.
Purely theoretically, we don't know how long this bug has been exploited since it was possible to do anything, including clearing logs. Only now have script kiddies gotten their hands on it.
Sure but I think we can agree, that the situation get's much worse (due to a larger scale) as soon as script kiddies gets envolved 
And at least that impact would be avoided by actually not running EOL systems and not exposing hypervisors to the wider Internet.
And at least that impact would be avoided by actually not running EOL systems and not exposing hypervisors to the wider Internet.