Possibly got hacked: How to identify listening process

[jelinpons]

I have a standard vmbr0 setup with only containers that use venet devices.

One container with it's own IP address has a webserver where I detected that it answers requests with a javascript redirect and can act like a proxy to the webserver itself.

I thought and hoped only the container is infected and created a new one and changed IPs. But the problem is there. I then completely shutdown the container associated with that IP address.

But if I do web requests to the now offline container it still responds and returns a small HTML and javascript snippet from time to time. If not, it still sends 302 redirect headers.

So I guess it must be the host that is infected. Of course I cannot trust any command in that case (netstat, lsof; checksecurity, chkrootkit, rkhunter). They don't show anything listening on port 80. Of course there is the apache for proxmox GUI itself. But I guess it does this differently. The system is up-to-date apt-wise and the problem persists after rebooting.

Do you have any tips or recommendations how to further investigate? I would like to get to know more before installing everything from scratch and not knowing how they entered the system.

*** UPDATE

"They" replaced spiceproxy with a version of their own that acts as an open proxy. Firewall prohibits port 3128 but they managed to keep it open.

 

[Bedlore]

Did you figure out how you got compromised? I'm always interested in learning from others how to better protect my system.

I hope your back up and running again.

 

[jelinpons]

No. 2 Installations, two compromises within minutes. The only thing I know it is not Proxmox related as I also did another run with plain Debian 7. The problem is there are too much possibilities and to less time to figure things out, as usual.

 

[steffen]

Firewall prohibits port 3128 but they managed to keep it open.

That is interesting. Are you sure that the Firewall is really running?

iptables -L on the console lists all your rules?

 

[jelinpons]

That is interesting. Are you sure that the Firewall is really running?

iptables -L on the console lists all your rules?

Yes, when the setup was up and running, I had closed every port, verified further by nmap, and did only allow incoming 80 and outgoing 80 (egress filtering via apf-firewall). Proxmox ports had only been allowed for my ip. All is possible: Maybe a zero-day between floods against the server, Apache segfaulting during DDOS, GHOST, ... or more simple a misconfiguration of my VLAN setup, leaving unused IP addresses unprotected from the firewall (but also not used by network-interfaces).
Or they found the 2 seconds where the firewall flushes and updates the rules.
I also found changes to ip_forwarding or log_martians sysctl parameters (the first turned on, the latter turned off) from time to time, things i always harden at the beginning.