Proxmox VE 7 is vulnerable to some type of 0day/RCE non auth

[asdpiasdjoi]

CVE issued now: https://www.cve.org/CVERecord?id=CVE-2023-54391

 

[chrcoluk]

Obviously 7 is EOL, but the big main issue here is, why is something like this not firewalled off with restricted IP access. Also have you done any backups at all in those 4 years?

Even my oldest bodged MBR GRUB install, has been updated over time and behind IP ACL.

Never sit relying on vendor to keep things secure for you (which is why people have given tough love in the replies), assume it can be compromised, assume your server might go on fire one day, take precautions against these events.

 

[Johannes S]

Would Proxmox be taken more seriously when it would take away the user's ability to control and customize it (based on GNU/Linux and OSS)? More like an enterprise appliance where you are used to pay a service technician to update it regularly? At least your setup would be "in the hands of professionals". I'm using hyperbole and satire here but maybe it would match better with the expectations created by other hypervisors? Regardless of my weird ideas, what would help people to stay up to date?

Sadly and most likely compilance theater. If they are in an environment where it's mandatory to have a cyber security insurance and said insurance state in their conditions that they only pay if you patched all of your systems then change managers might finally approve of doing patches. Even then they also might pass the ball to the legal department wheter the "patch or we don't pay"-clause might be taken to court.
Another potential way to enforce this via compilance theater/checkboxing would be a legal obligation like the moronic "you are not allowed to update without recertifying everything for a high amount of cash"-rules in parts of critical infrastructure. It was quite funny to read following report on an talk kernel developer Greg Kroah-Hartmann did on Linux kernels cve issueing process:

One participant in the panel discussion, however, mentioned that the many CVEs are a huge problem in areas where updates are difficult and expensive due to certification regulations –, for example when using Linux in hospitals. Kroah-Hartman explained that US and EU legislators have recognized the problem and are working on solutions, but that this will take time.
https://www.heise.de/en/news/Linux-...s-of-the-CVE-flood-in-the-kernel-9963850.html

So if the same rules state that any updates needs an recertification would also state that any CVE or security issue with a known fix needs to be fixed asap otherwise you loose your clearance for such environments, more people will update.

To be clear: I hate the CVE process and compilance-instead-actual-security debacle of "enterprise it" with passion, but if managment only cares about ticking checkboxes, then that would help. I wouldn't hold my breath though, that lawmakers or insurance companies will actually do this.

 

[MDSOperator]

Proxmox VE 7.1 - cryptojacking rootkit + root SSH backdoor: full teardown

Anonymized write-up based on a real incident on Proxmox VE 7.1-7 (Debian 11 Bullseye), two hosts. All names, IPs, domains, wallets and keys are redacted. Goal: help others recognize this toolchain and verify their own hosts, since the attack is fully stealthy even against a root shell.

TL;DR

Two PVE 7 hosts were compromised and running an XMRig cryptominer hidden by a preload rootkit. The same "install_and_mine.sh / libhide" toolchain planted:

• A miner plus stealth library hiding its process, port and files from every standard tool.
• A separate root SSH key backdoor that survives any miner cleanup (this was the unexpected one).
• A replaced AccessControl.pm (PVE auth module).
• Deleted who / top / w.
• Aggressive log/forensics destruction.

The binaries (sshd, ps, ls, systemd, libc) were not tampered with — the attack is config/file based. Key lessons: don't trust ls/ps/find/ss even as root, and don't trust mtime.

Environment

• Proxmox VE 7.1-7 (pve-manager/7.1-7/df5740ad), kernel 5.13.19-2-pve, Debian 11 Bullseye.
• Two independently-running hosts; both infected identically, ~2 min apart.
• Production VMs (several QEMU guests incl. web/DB) unaffected at application layer.

Timeline (reconstructed)

• T0 00:00-14:12: /var/log/pveproxy/access.log wiped for this whole window — attack entry vector lost.
• T0 12:52: Rogue PVE accounts created: [REDACTED]@pve (Administrator) + root@pam![REDACTED] API token.
• T0 13:14:4x: Rootkit + SSH backdoor installed on both nodes (birth times identical on both).
• T0 13:14-14:05: Root shell profile/history rewritten; logs symlinked to /dev/null; journald.conf changed; auth.log / wtmp / btmp / lastlog / secure -> symlinks.
• T0 14:05: /etc/shadow + /etc/passwd touched on both nodes — root password almost certainly rotated by attacker.
• T0 16:15+: Two VMs found stopped — consistent with host CPU starvation from the miner.

What was planted

1) Miner (identical structure on both hosts)

• Binary /var/lib/systemd/PVE-1 (XMRig), pool [REDACTED]:20004, C2 [REDACTED], wallet [REDACTED], TLS, --donate-level=0.
• Launcher /var/lib/systemd/ld-svc; mtime backdated to 2018-01-15 on both hosts.
• chattr +a / +i not visible as immutable at scan time (only 'e' flag) — but the build clearly intended stealth attrs (state file .stealth-attrs=1).

2) Stealth library — why nothing showed up

• /var/lib/systemd/.hide/libhide.so, injected via LD_PRELOAD in two places: /etc/environment and /etc/profile.d/sysenv.sh.
• Hides: its process name from ps, its port (20004) from ss/netstat, and its files from ls/find — even from a root shell. Only visible after env -u LD_PRELOAD.

3) Persistence — survives reboot and kill

• PVE-1.service (miner, Restart=always)
• PVE-1.timer + PVE-1-maintain.service — restart guard every ~3 min
• PVE-1-update.timer + PVE-1-update.service — daily auto-updater pulling official XMRig, SHA-256-verified, atomic replace

4) SSH root backdoor (separate from miner) — the finding that matters most

/etc/ssh/sshd_config end of file:

Match User root
    AuthorizedKeysCommand /etc/ssh/.sshd-keys-cache
    AuthorizedKeysCommandUser root
    PubkeyAuthentication yes

• /etc/ssh/.sshd-keys-cache -> outputs /etc/ssh/.cache/root.keys
• /etc/ssh/.cache/root.keys = attacker ed25519 key ([REDACTED])
• sshd -T -C user=root confirms it's live: authorizedkeyscommand /etc/ssh/.sshd-keys-cache
• /etc/ssh/.hist-arm + /root/.config/fish/config.fish = self-healing 2h shell-history suppression ("svc-history-guard") to hide their commands

5) PVE auth module replacement

• /usr/share/perl5/PVE/AccessControl.pm replaced, birth T0 13:14:42, mtime faked to 2021-11-25, identical SHA256 on both hosts. Almost certainly the enabler for the rogue-account/console actions, or a GUI re-access backdoor.

6) Remote tools deleted

• who, top, w deleted (not replaced) — so the admin's own checks silently fail. dpkg -V flags them.

Anti-forensics observed

• /var/log/auth.log, wtmp, btmp, lastlog, secure -> symlinks to /dev/null (re-pointed 14:05)
• journald.conf -> Storage=none
• pveproxy access log morning window deleted
• Root .bash_history / .history wiped or never logged; fish + bash history suppressed
• /etc/resolv.conf forced to 8.8.8.8 / 1.1.1.1 (x3) — DNS redirected to attacker-chosen resolvers
• /etc/hosts "donate-block" region added (blocks xmrig donation domains) — generated from data file /var/lib/systemd/.cache/.donate-block
• mtimes backdated everywhere (2018/2021/2022); birth times (crtime) give the truth

How we confirmed / what to check on your own hosts

Set LD_PRELOAD unset or the rootkit hides everything:

env -u LD_PRELOAD ps aux | grep -i -E 'xmrig|PVE-1|ld-svc'
env -u LD_PRELOAD ss -tulpn | grep ':20004'
env -u LD_PRELOAD find /var/lib/systemd -type f
env -u LD_PRELOAD ls -la /etc/environment /etc/profile.d/ /var/lib/systemd/.hide/

Use birth time, not mtime (crtime can't be faked this way):

stat -c '%w %n' /path/to/suspect

Check the SSH authorized-keys command backdoor:

grep -A3 'Match User' /etc/ssh/sshd_config
sshd -T -C user=root | grep -i authorizedkeyscommand
find /etc/ssh -name '*.keys*' -o -name '.cache*'

Package integrity:

dpkg -V openssh-server pve-manager libpve-access-control coreutils procps

Biggest blind spots we hit: find -mtime/-newermt won't see backdated files, and dpkg -V shows config diffs that look benign. Combine birth-time scans + dpkg -V + unsetting LD_PRELOAD.

What we recommend in cleanup/rebuild

1. Remove the SSH backdoor first (it's the real access, separate from the miner): delete the Match User root block, .sshd-keys-cache, /etc/ssh/.cache, /etc/ssh/.hist-arm, set PermitRootLogin no, rotate all host keys + root authorized_keys.
2. apt-get install --reinstall libpve-access-control coreutils procps (restore AccessControl.pm + who/top/w).
3. Kill/remove miner + 4 systemd units + chattr -i/-a where present + delete /var/lib/systemd/{PVE-1,ld-svc,.hide,.cache}, /etc/profile.d/sysenv.sh, restore /etc/environment, /etc/hosts, /etc/resolv.conf.
4. Re-point logs, restore journald.conf.
5. Rotate ALL credentials: node root/passwords, PVE users/tokens, SSH host+user keys, IPMI, backup system, application secrets.
6. Consider migrating VMs off the compromised hosts entirely (full images exist) — cleanest given the forensic unknowns (wiped log window).

Unknowns remaining

• The original entry vector (wipe of the pre-14:12 pveproxy window wiped it)
• Exact payload of the replaced AccessControl.pm

Sample IOCs (normalized)

/var/lib/systemd/PVE-1                      (XMRig binary)
/var/lib/systemd/ld-svc                     (launcher)
/var/lib/systemd/.hide/libhide.so           (preload rootkit)
/var/lib/systemd/.cache/                    (state/donate-block/hugepages state)
/usr/libexec/PVE-1-maintain
/usr/libexec/PVE-1-update
/etc/systemd/system/PVE-1.service|timer
/etc/systemd/system/PVE-1-maintain.service
/etc/systemd/system/PVE-1-update.service|timer
/etc/profile.d/sysenv.sh
/etc/environment                            (LD_PRELOAD export)
/etc/ssh/.sshd-keys-cache, .cache/root.keys, .hist-arm
/usr/share/perl5/PVE/AccessControl.pm       (replaced)
who/top/w deleted

 

[_gabriel]

Unknowns remaining

• The original entry vector

Never expose 8006 port publicly.

Post in thread 'Proxmox Virtual Environment - Security Advisories'

Tuesday at 08:31

Subject: PSA-2026-00043-1: Authentication bypass in EOL Proxmox VE 7 release​

Advisory date: 2026-09-01

Packages: libpve-access-control

Affected: libpve-access-control >= 7.0-7 and < 8.0.4

Roughly corresponding to Proxmox VE 7.0 up to and including 7.4 (end of life since July 2024), and, for completeness, the initial Proxmox VE 8.0 (EOL), as the fix only shipped roughly one month after that release.

Package versions and the overall Proxmox VE version only correlate loosely, since dependency versions are only raised when needed. To determine whether an...

• ProxmoxSecurityAdvisory

 

[Johannes S]

I can't say anything about the other parts of the writeup, but the "recommendation" for cleanup/rebuild irritates me:

What we recommend in cleanup/rebuild

1. Remove the SSH backdoor first (it's the real access, separate from the miner): delete the Match User root block, .sshd-keys-cache, /etc/ssh/.cache, /etc/ssh/.hist-arm, set PermitRootLogin no, rotate all host keys + root authorized_keys.
2. apt-get install --reinstall libpve-access-control coreutils procps (restore AccessControl.pm + who/top/w).
3. Kill/remove miner + 4 systemd units + chattr -i/-a where present + delete /var/lib/systemd/{PVE-1,ld-svc,.hide,.cache}, /etc/profile.d/sysenv.sh, restore /etc/environment, /etc/hosts, /etc/resolv.conf.
4. Re-point logs, restore journald.conf.
5. Rotate ALL credentials: node root/passwords, PVE users/tokens, SSH host+user keys, IPMI, backup system, application secrets.
6. Consider migrating VMs off the compromised hosts entirely (full images exist) — cleanest given the forensic unknowns (wiped log window).

I can't agree with this. This is a lot of work while you still could overlook something or the bad actor (now that the attack is known) adds thing to stay in the system even after publication of the attack. If you get compromised you have no other option than to wipe your systems and restore from backup, end of story.
Or to be more precise: If up to now you run an EOL system AND exposed it to the Internet you will get hacked again (from this attack or some other, EOL systems usually have a lot of known and unknown vulnerabilities).

So instead of this rather complicated, awkward and obtuse setup the whole part should be rewritten to:

What we recommend in cleanup/rebuild

Wipe all your PVE7 hosts, install the latest supported PVE version (PVE 9.2 at the moment) and restore your configuration, VMs and LXCs from your last uncompromised backup before the hack. Don't expose your host systems network to the Internet.

Everything else is just asking for going through the whole trouble again.