[SOLVED] Hacked proxmox server

M

MisterDeeds

Active Member
Nov 11, 2021
143
33
33
35
Dear all

I have the following problem. We are running a 4 node proxmox cluster. A few weeks ago one host of it was hacked. The hosts are hanging behind a firewall. Neither the web GUI nor SSH are open from outside.

When we noticed it, we reinstalled the host. After that it was good for a few weeks. Today I noticed that the /etc/passwd file is read-only again. Also there is a user which is unknown to me.
1677823063875.png

Additionally an entry in the /etc/crontab file. According to my understanding, this always deletes all log file entries.
1677823094817.png
Code: 
* * * * * root bash -c 'for i in $(find /var/log -type f); do cat /dev/null > $i; done' >/dev/null 2>&1

Thus, unfortunately, there are also no entries in /var/log/auth...

It is not clear to me how this can happen. Also what we should do, apparently reinstalling does not bring anything either. Do any of you know of a similar case? how would you proceed?

Thanks alot and best regards
 
I still saw the following tasks. I think these also come from the attacker.
1677824793676.png

What is also unsure is how to proceed with the rest of the cluster. The other nodes are not compromised, but for example the SSH keys are entered, so the compromised node can also access the others.
 
I also saw that a startup program was created with the name "screen-cleanup" and the following content:

Code: 
root@Pve04:~# cat /etc/init.d/screen-cleanup
#!/bin/sh
# $Id: init,v 1.3 2004/03/16 01:43:45 zal Exp $
#
# Script to remove stale screen named pipes on bootup.
#

### BEGIN INIT INFO
# Provides:          screen-cleanup
# Required-Start:    $remote_fs
# Required-Stop:     $remote_fs
# Default-Start:     S
# Default-Stop:
# Short-Description: screen sessions cleaning
# Description: Cleans up the screen session directory and fixes its
#  permissions if needed.
### END INIT INFO

set -e

test -f /usr/bin/screen || exit 0

SCREENDIR=/run/screen

case "$1" in
start)
    if test -L $SCREENDIR || ! test -d $SCREENDIR; then
        rm -f $SCREENDIR
        mkdir $SCREENDIR
        chown root:utmp $SCREENDIR
        [ -x /sbin/restorecon ] && /sbin/restorecon $SCREENDIR
    fi
    find $SCREENDIR -type p -delete
# If the local admin has used dpkg-statoverride to install the screen
# binary with different set[ug]id bits, change the permissions of
# $SCREENDIR accordingly
    BINARYPERM=`stat -c%a /usr/bin/screen`
    if [ "$BINARYPERM" -ge 4000 ]; then
        chmod 0755 $SCREENDIR
    elif [ "$BINARYPERM" -ge 2000 ]; then
        chmod 0775 $SCREENDIR
    else
        chmod 1777 $SCREENDIR
    fi
    ;;
stop|restart|reload|force-reload)
    ;;
esac

exit 0
 
I have been working on setting up my home network security before heavily setting up a ton of servers. Does your network have any open ports? What point of entry do you suspect? I have no idea how to fix the proxmox servers themselves. I'd make sure I don't have an Open NAT to the proxmox servers, and that the network they are on isn't accessible from the outside world using static IP addresses until you get it fixed up. I'd also check any IDS systems you have and see if you have IP address logs but that'd likely come up dry because people use the cloud to hack these days. Still, if you manage to find out the penetrator's IP address, I guess report it to the FBI online.
 
Having a ton of servers makes it seems like maybe someone you know accessed your cluster remotely too (like, maybe someone else helped you set up the network.) If they moved laterally through your network, they may have tripped a VLAN firewall rule as well.
 
Last edited:
What about IPMI? The last cases I saw where a PVE node got hacked the IPMI was outdated and public available. So bots could hack the IPMI and then use it's console to get root access of the server.
 
Hello and thank you for the answers. The hosts are behind a pfSense firewall. NAT ports are open only for selected IP addresses (eg. my static ipv4 address). Further, the servers are located in a locked rack in a data center, so physical access can also be excluded. The only ports that are open for all access are HTTP and HTTPS. These are forwarded to the HAproxy of the pfSense firewall.

The first time it happened, the ILO interface was "compromised". A new user "SystemAdmin" was created there. I deleted the user, changed all passwords and updated the firmware to the latest version. This time there is no such user, but a login from this user was still awarded from today:

1677830770173.png

But a corresponding user does not exist at all.
1677830823847.png

What I have seen that the "Security Override" mode is active. This is a hardware switch in the server. The server had a defect and was replaced. Apparently the technician did not reset this switch.

1677830956231.png

I'll go by the data center this afternoon and reset the switch. After that, I'll reinstall the proxmox server.
 
  • Like
Reactions: HLPCLC
"What's my ip address" says you were hacked by someone in Hong Kong. Maybe they were sending python frames into your system. David Bombal has some interesting videos on youtube on the topic.

https://www.youtube.com/watch?v=YKxKnVE5FaE

I haven't been comfortable enough with opening ports on pfSense yet. There is so much people can do to stuff. Especially with http and https.

Switches are purportedly vulnerable to python attacks according to him.


Like, these videos make me weirded out about using python for the DNS resolver in pfSense.

https://www.youtube.com/watch?v=u5cp_hcwq2c&list=PLhfrWIlLOoKOc3z424rgsej5P5AP8yNKR&index=3

Here, he is saying to disable DTP.

I use NordVPN through OpenVPN in the pfSense firewalls. I don't know if this could help you prevent attacks in the future, but it seems kind of bulletproof if nobody can access your devices through your WAN directly. They offer meshnet services too. Maybe you leaked some data logging into your pfSense.

Switches are purportedly vulnerable to python attacks according to him.
 
Last edited:
MisterDeeds said:
The first time it happened, the ILO interface was "compromised". A new user "SystemAdmin" was created there. I deleted the user, changed all passwords and updated the firmware to the latest version. This time there is no such user, but a login from this user was still awarded from today:
Click to expand...

Did you have the iLO port forwarded out through pfSense? Was it through HAProxy or on some different port?
 
Thank you @HLPCLC and @robertut

The ILO hangs directly on the Internet with public IPv4 address. I can only explain that because of this "security override" switch it is possible to access the ILO... But I will go by today and shut down the server as well as put back the switch.
 
Dunuin said:
What about IPMI? The last cases I saw where a PVE node got hacked the IPMI was outdated and public available. So bots could hack the IPMI and then use it's console to get root access of the server.
Click to expand...
How would you get root access? You would still need the root credentials to log in to console?

Or do you mean by shuting down server and get root access via chroot?
 
efeu said:
How would you get root access? You would still need the root credentials to log in to console?

Or do you mean by shuting down server and get root access via chroot?
Click to expand...
I don't know what vulnerabilities they used, but they got root access over the IPMI. Maybe something like this?:
https://forum.proxmox.com/threads/file-encrypting-trojan.99586/post-430397

And how to then reset the root password, as soon as you get access to the IPMIs serial console, is described here: https://pve.proxmox.com/wiki/Root_Password_Reset
 
Last edited:
I got hacked via IPMI once. That time, the hacker reset the server and then boot to an ISO or sort mounted on ILO's virtual disk, which allowed them to do modifications to the hard disk (adding a backdoor or sort). They will quickly boot back to the system on drive to make this look like a random reboot.

After that, I always set a boot password on grub or bios so I can check IPMI log to see if it's a random reboot or not.
 
Hello together

You are of course right. To hang an IPMI, ILO etc. directly into the internet is not a good solution... However, unfortunately, it can not always be avoided, because when we got our first servers, we also had no firewall and no switches in use. On the server, we then had a virtualized firewall in and so the ILO required direct internet access. In the meantime, we have expanded our infrastructure with a bare-metal firewall and appropriate switches. I have now taken all IPMI and ILO interfaces from the public network behind the firewall. We now hope not to suffer the same attack again. Thank you very much for the numerous feedbacks and inputs!
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom