Hacked System

I

infinityM

Well-Known Member
Dec 7, 2019
179
1
58
31
Hey Guys,

Ok so as the tital suggests... I've got a serious problem that's 2 fold...

So I had a Virtualizor server before moving to proxmox which was hacked and a miner bot kept running on it. Me getting tired of having to try and find and deal with it moved away and formated that server and made it part of my PM cluster...

Now about a year later, the same server is running a miner bot again it seems... What are the ods of the same server being hacked twice? Is it possible that the bot's some how a level beyond the OS?

So after getting hacked again, I decided to setup a vpn with my mikrotik router and blocked the ssh ports for everyone except my internal range...
Now the question becomes... How can I get the miner bot off? How can I find it and remove it. I've got no experience with miner bot's...

Thanks in advance for the assistance :)
 
What about password auth for SSH? Did you had it enabled? Did you change the root password from the old box to the new one?

Its not that hard to brutforce (yeah lots of work) /etc/shadow.
 
All us server has only ssh-key-auth only. Also 2factor @ssh is possible.
 
r.jochum said:
What about password auth for SSH? Did you had it enabled? Did you change the root password from the old box to the new one?

Its not that hard to brutforce (yeah lots of work) /etc/shadow.
Click to expand...
I did have it enabled. But did have fail2ban setup so I thought I was relatively safe #facepalm

The password was changed though aswell as the IP adress
 
H4R0 said:
Format it again, there is no guarantee to remove the botnet, the server will still be backdoored.
Click to expand...
My concern with this is that I would then need to change the hostname of the server... And I have a structure which I would rather keep the hostnames in sequence... Is there a way to format keeping the hostname the same without breaking PM? Read in one of the guides it has to be changed a while back...
 
I would like to suggest to disable root ssh access on your server. Add a local user with an umpredictable long name and a strong password and then use "su", not "sudo", with a strong password too for root. So you will have 2 differents passwords to connect and get admin rights. This will help against brute force attacks as they are mainly for root user.
Deploying fail2ban is a good thing too.
Changing the hostname will only have a limited effect as most of hackers scan the network IPs.
 
fireon said:
All us server has only ssh-key-auth only. Also 2factor @ssh is possible.
Click to expand...

That's what I would suggest, too - so disable ssh login with a password and only use keys. In addition, do the SSH whitelisting with knock or VPN (or both) to disable the access to SSH on the IP level.
 
LnxBil said:
That's what I would suggest, too - so disable ssh login with a password and only use keys. In addition, do the SSH whitelisting with knock or VPN (or both) to disable the access to SSH on the IP level.
Click to expand...
Ok so I have gone so far as to completely reinstall a clean version of proxmox on my server... And low and behold this morning it's running the minerbot again...

I've even completely blocked all and I mean ALL ssh and port 8006 connections with the firewall accept through my VPN...
Somehow this thing keeps getting back on...

What's strange is it will chase up the CPU usage to 50% untill the moment I login, The moment I login it drops down to 1% cpu usage... Anyone have advise on how this is possible and how I can try to resolve it?
 
r.jochum said:
Is there a chance that you have a infected machine on LAN?
Click to expand...
I think highly unlikely since I am blocking all in bound traffic to the server accept from the other proxmox hosts and from my pc on the VPN...
My thought is that it may be something that's rooted in the bios or something maybe?
 
The year break suggests it's not a rootkit but how you were re-infected depends on a lot of factors. What model server and version of iLO are you using? Did you change the default credentials?
 
Virt said:
The year break suggests it's not a rootkit but how you were re-infected depends on a lot of factors. What model server and version of iLO are you using? Did you change the default credentials?
Click to expand...
Well I did not notice it since everything was running smooth. But it's most likely been on the server QUITE a while if not the whole time.
I did notice the CPU running a bit high the whole time, but thought it was the windows VPS on the server. May very well have been the miner all this time...

It's ILO4 HP G8 Server. I did notice there was a systemadministrator user created in ILO,so I think that may very well be how they gained access. Question is how to ensure there's not something hidden in the bios or kernel or something above the os now?
 
Definitely sounds like someone gained unauthorized access. As a general precaution, you should move your iLOs to a management VLAN that has no access to anything other than a few designated VLANs. I would suggest deleting the user and upgrading iLO. The most recent version for 4 can be found here: https://downloads.hpe.com/pub/softlib2/software1/sc-linux-fw-ilo/p192122427/v182737/CP044405.scexe
To install it, extract the scexe file using 7zip and then extract the file that comes out. You'll see a .bin file that you can upload to the iLO via the web portal.
 
Also, be sure to enable logging on the firewall for all rules on that VLAN. You'll be able to track anyone trying to hit your iLO (even if you have it blocked).
 
My 28 cents:

  • Wipe the system again.
  • Reset ILO.
  • Flash BIOS and ilo firmware.
  • Place a firewall in front of the system and monitor all inbound and all outbound connections.

There have been security flaws where attackers could gain access to I'll from
remote.

Since your description indicates you have restricted access there are only two options:
  1. you have an infected system on the network which is the attacker within your network. Hence your firewall rules are useless
  2. Ilo is compromised and something sits within the firmware.​
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom