[HELP] Improve proxmox security

[Diogo Jesus]

Hello everyone,
I got a server on SyS (OVH) with quite good specifications (32gb ram, octa core, etc, etc)
I installed Proxmox on ZFS mode and setup all my containers, everything was working great.
On securtiy i used proxmox firewall (integrated on GUI) and since in our company we have a static IP, we added SSH and 8006 ports to our IP only. (SSH port was changed aswell)

Last thursday our server attacked some chinese IP (we got hacked and someone used our server to attack the other server).

Here is what i don't get. How did they managed to enter on our server? knowing that ssh/proxmox ports were only open to our IP?

From the syslog i can't find anything related to the attack (under our server) so i have no idea how they managed to enter.

Our ISP also had a problem in their datacenter the same day of the attack so i can only have the log from the 12th october (1pm) the attack happened at 6pm.

the only thing that i can find in syslog at 6pm of that day is

kernel: nf_conntrack: nf_conntrack: table full, dropping packet
​

Is there any other way to find older logs? i really need to fix this problem asap.

Thank you,
Diogo Jesus

 

[Alwin]

On securtiy i used proxmox firewall (integrated on GUI) and since in our company we have a static IP, we added SSH and 8006 ports to our IP only. (SSH port was changed aswell)

Depending on your firewall setup, you may have add your companies IP and not limited to it.
Check in '/etc/pve/nodes/<nodename>/host.fw' & '/etc/pve/nodes/<nodename>/host.fw', if there are no rules canceling each other out.
https://pve.proxmox.com/pve-docs/chapter-pve-firewall.html

Last thursday our server attacked some chinese IP (we got hacked and someone used our server to attack the other server).

If there was an access by ssh, then you should find it in the syslog and/or auth.log.

Is there any other way to find older logs? i really need to fix this problem asap.

If your provider reset the machine or the hacker deleted logs, so /var/log/ is empty or is missing those logs, then there are none.

And as a general reminder, is your server up-to-date?

 

[Oliver R]

Hi Diogo!

Sorry to hear that. Can you post the firewall configuration here? Which Proxmox Version do you use?

pveversion -v

Have you installed any third party packages?

 

[Diogo Jesus]

Hi Diogo!

Sorry to hear that. Can you post the firewall configuration here? Which Proxmox Version do you use?

pveversion -v

Have you installed any third party packages?

thank for answering here is the versions

proxmox-ve: 5.0-23 (running kernel: 4.10.17-3-pve)
pve-manager: 5.0-32 (running version: 5.0-32/2560e073)
pve-kernel-4.10.17-3-pve: 4.10.17-23
pve-kernel-4.10.17-1-pve: 4.10.17-18
libpve-http-server-perl: 2.0-6
lvm2: 2.02.168-pve3
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-12
qemu-server: 5.0-15
pve-firmware: 2.0-2
libpve-common-perl: 5.0-18
libpve-guest-common-perl: 2.0-11
libpve-access-control: 5.0-6
libpve-storage-perl: 5.0-15
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-2
pve-docs: 5.0-9
pve-qemu-kvm: 2.9.1-1
pve-container: 2.0-16
pve-firewall: 3.0-3
pve-ha-manager: 2.0-2
ksm-control-daemon: not correctly installed
glusterfs-client: 3.8.8-1
lxc-pve: 2.1.0-2
lxcfs: 2.0.7-pve4
criu: 2.11.1-1~bpo90
novnc-pve: 0.6-4
smartmontools: 6.5+svn4324-1
zfsutils-linux: 0.6.5.11-pve17~bpo90

Also i was able to view other logs from fail2ban auth daemon etc etc. Apparently they found a breach in the proxmox api (ip:8006/ap2/json) the weird thing is that the port 8006 was blocked only to our IP so i don't get it.
from what i understood they enabled the proxmox proxy and from there they used our server as a zombie to attack a chinese telecom

Oct 12 13:20:44 ns3037493 systemd[1]: Started PVE API Proxy Server.
Oct 12 13:20:44 ns3037493 systemd[1]: Starting PVE SPICE Proxy Server...

This is the log from daemon.log
also if i find more stuff around the logs i will keep this updated

 

[Diogo Jesus]

Depending on your firewall setup, you may have add your companies IP and not limited to it.
Check in '/etc/pve/nodes/<nodename>/host.fw' & '/etc/pve/nodes/<nodename>/host.fw', if there are no rules canceling each other out.
https://pve.proxmox.com/pve-docs/chapter-pve-firewall.html


If there was an access by ssh, then you should find it in the syslog and/or auth.log.


If your provider reset the machine or the hacker deleted logs, so /var/log/ is empty or is missing those logs, then there are none.

And as a general reminder, is your server up-to-date?

Sorry for some reason your answer didn't appeared on my screen so i just jumped it.
I think my proxmox is up-to-date as you can see in the above reply.

The firewall seems to be ok, because i can do the port checker (you have plenty of online tools for that) and while i can connect to my ssh port (for example) the online tool will see ssh port offline. So on my point of vue the problem isn't with the firewall itself.

Few days ago i had some problem emails still reading the logs to see if there's something to do with it.

Also under my daemon.log this was found few hours before the actual attack

Oct 12 13:20:37 ns3037493 named[2357]: built with '--prefix=/usr' '--mandir=/usr/share/man' '--libdir=/usr/lib/x86_64-linux-gnu' '--infodir=/usr/share/info' '--sysconfdir=/etc/bind' '--with-python=python3' '--localstatedir=/' '--enable-t$
Oct 12 13:20:37 ns3037493 named[2357]: ----------------------------------------------------
Oct 12 13:20:37 ns3037493 named[2357]: BIND 9 is maintained by Internet Systems Consortium,
Oct 12 13:20:37 ns3037493 named[2357]: Inc. (ISC), a non-profit 501(c)(3) public-benefit
Oct 12 13:20:37 ns3037493 named[2357]: corporation.  Support and training for BIND 9 are
Oct 12 13:20:37 ns3037493 named[2357]: available at isc.org/support
Oct 12 13:20:37 ns3037493 named[2357]: ----------------------------------------------------
Oct 12 13:20:37 ns3037493 named[2357]: adjusted limit on open files from 4096 to 1048576
Oct 12 13:20:37 ns3037493 named[2357]: found 8 CPUs, using 8 worker threads
Oct 12 13:20:37 ns3037493 named[2357]: using 4 UDP listeners per interface
Oct 12 13:20:37 ns3037493 named[2357]: using up to 4096 sockets
Oct 12 13:20:37 ns3037493 systemd[1]: Starting Fail2Ban Service...
Oct 12 13:20:37 ns3037493 named[2357]: loading configuration from '/etc/bind/named.conf'
Oct 12 13:20:37 ns3037493 named[2357]: reading built-in trusted keys from file '/etc/bind/bind.keys'
Oct 12 13:20:37 ns3037493 named[2357]: GeoIP Country (IPv4) (type 1) DB not available
Oct 12 13:20:37 ns3037493 named[2357]: GeoIP Country (IPv6) (type 12) DB not available
Oct 12 13:20:37 ns3037493 named[2357]: GeoIP City (IPv4) (type 2) DB not available
Oct 12 13:20:37 ns3037493 named[2357]: GeoIP City (IPv4) (type 6) DB not available
Oct 12 13:20:37 ns3037493 named[2357]: GeoIP City (IPv6) (type 30) DB not available
Oct 12 13:20:37 ns3037493 named[2357]: GeoIP City (IPv6) (type 31) DB not available
Oct 12 13:20:37 ns3037493 named[2357]: GeoIP Region (type 3) DB not available
Oct 12 13:20:37 ns3037493 named[2357]: GeoIP Region (type 7) DB not available
Oct 12 13:20:37 ns3037493 named[2357]: GeoIP ISP (type 4) DB not available
Oct 12 13:20:37 ns3037493 named[2357]: GeoIP Org (type 5) DB not available
Oct 12 13:20:37 ns3037493 named[2357]: GeoIP AS (type 9) DB not available
Oct 12 13:20:37 ns3037493 named[2357]: GeoIP Domain (type 11) DB not available
Oct 12 13:20:37 ns3037493 named[2357]: GeoIP NetSpeed (type 10) DB not available
Oct 12 13:20:37 ns3037493 named[2357]: using default UDP/IPv4 port range: [32768, 60999]
Oct 12 13:20:37 ns3037493 named[2357]: using default UDP/IPv6 port range: [32768, 60999]
Oct 12 13:20:37 ns3037493 named[2357]: listening on IPv4 interface lo, 127.0.0.1#53
Oct 12 13:20:37 ns3037493 systemd[1]: Started LXC Container Monitoring Daemon.
Oct 12 13:20:38 ns3037493 systemd[1]: Reached target Network is Online.
Oct 12 13:20:38 ns3037493 systemd[1]: apt-daily.timer: Adding 9h 17min 21.862476s random time.
Oct 12 13:20:38 ns3037493 systemd[1]: Started Daily apt download activities.
Oct 12 13:20:38 ns3037493 named[2357]: listening on IPv6 interface lo, ::1#53
Oct 12 13:20:38 ns3037493 named[2357]: generating session key for dynamic DNS
Oct 12 13:20:38 ns3037493 named[2357]: sizing zone task pool based on 5 zones
Oct 12 13:20:38 ns3037493 named[2357]: using built-in root key for view _default
Oct 12 13:20:38 ns3037493 named[2357]: set up managed keys zone for view _default, file 'managed-keys.bind'
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 10.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 16.172.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 17.172.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 18.172.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 19.172.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 20.172.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 21.172.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 22.172.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 23.172.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 24.172.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 25.172.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 26.172.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 27.172.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 28.172.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 29.172.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 30.172.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 31.172.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 168.192.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 64.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 65.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 66.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 67.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 68.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 69.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 70.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 71.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 72.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 73.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 74.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 75.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 76.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 77.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 78.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 79.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 80.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 81.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 82.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 83.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 84.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 85.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 86.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 systemd[1]: apt-daily-upgrade.timer: Adding 33min 58.400128s random time.
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 87.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 systemd[1]: Started Daily apt upgrade and clean activities.
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 88.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 89.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 90.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 91.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 92.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 93.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 94.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 95.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 96.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 97.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 98.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 99.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 100.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 101.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 102.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 103.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 104.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 105.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 106.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 107.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 108.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 109.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 110.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 111.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 112.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 113.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 114.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 115.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 116.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 117.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 118.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 119.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 120.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 121.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 122.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 123.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 124.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 125.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 126.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 127.100.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 254.169.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 2.0.192.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 100.51.198.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 113.0.203.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 255.255.255.255.IN-ADDR.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.IP6.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.IP6.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: D.F.IP6.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 8.E.F.IP6.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 9.E.F.IP6.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: A.E.F.IP6.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: B.E.F.IP6.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: 8.B.D.0.1.0.0.2.IP6.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: automatic empty zone: EMPTY.AS112.ARPA
Oct 12 13:20:38 ns3037493 named[2357]: configuring command channel from '/etc/bind/rndc.key'
Oct 12 13:20:38 ns3037493 named[2357]: command channel listening on 127.0.0.1#953
Oct 12 13:20:38 ns3037493 named[2357]: configuring command channel from '/etc/bind/rndc.key'
Oct 12 13:20:38 ns3037493 named[2357]: command channel listening on ::1#953
Oct 12 13:20:38 ns3037493 named[2357]: managed-keys-zone: journal file is out of date: removing journal file
Oct 12 13:20:38 ns3037493 named[2357]: managed-keys-zone: loaded serial 21
Oct 12 13:20:38 ns3037493 named[2357]: zone 0.in-addr.arpa/IN: loaded serial 1
Oct 12 13:20:38 ns3037493 named[2357]: zone 255.in-addr.arpa/IN: loaded serial 1
Oct 12 13:20:38 ns3037493 systemd[1]: Reached target Timers.
Oct 12 13:20:38 ns3037493 named[2357]: zone localhost/IN: loaded serial 2
Oct 12 13:20:38 ns3037493 named[2357]: zone 127.in-addr.arpa/IN: loaded serial 1
Oct 12 13:20:38 ns3037493 named[2357]: all zones loaded
Oct 12 13:20:38 ns3037493 named[2357]: running
Oct 12 13:20:38 ns3037493 systemd[1]: Starting LXC network bridge setup...

seems that they added root keys for this IPs ? is it right?
thank you so much for trying to support this i feel really lost at this point

i will for sure reinstall proxmox but before i need to know where the breach was so i can fix it before reinstalling all over again

 

[LnxBil]

As a security measure, I' use a separate subnet for all VM traffic and block your hypervisor totally except your public IP.

Any chance they hacked your server over you internal connection?

 

[Diogo Jesus]

As a security measure, I' use a separate subnet for all VM traffic and block your hypervisor totally except your public IP.

Any chance they hacked your server over you internal connection?

i got the xxx.xxx.xxx.xxx for my host (proxmox)
and each container is running a different IP from fail over IPs yyy.yyy.yyy.aaa (i have 4 different containers running and running different IPs)

i don't think they did since internally we have a good security (i assume)

from the logs it gets suspicious when i see

Oct  9 04:09:09 ns3037493 systemd[1]: Stopping Proxmox VE firewall logger...
Oct  9 04:09:09 ns3037493 pvepw-logger[25692]: received terminate request (signal)

i've been searching for this pvepw-logger which i assume it is proxmox password logger? but what does it do? why did it got called at this time? (4am i assume no one is in the office)

and then few hours later this shows up

Oct  9 09:32:11 ns3037493 pveproxy[2133]: worker exit
Oct  9 09:32:11 ns3037493 pveproxy[2131]: worker 2133 finished
Oct  9 09:32:11 ns3037493 pveproxy[2131]: starting 1 worker(s)
Oct  9 09:32:11 ns3037493 pveproxy[2131]: worker 20762 started

Again why is this working with a pveproxy? what does it mean? Is it the log when i access a container via the GUI console?

I can also post the full attack day log but i think it will be a little bit to much

 

[guletz]

i will for sure reinstall proxmox but before i need to know where the breach was so i can fix it before reinstalling all over again

Yes, is important to find how they go into Proxmox, but this is only a part of the problem. As I can see in your post you may ask yourself how you solve others problems, like this(I can provide more):
- at anytime, some bad guy can find and exploit a weakness in your sistem(it is only a matter of time ... not if), so you must have several layers of protection(iptables is only one layer)
- another problem is the fact that is very important to have logs also on a remote server(remote log server), so after a incident, you will be able to see what was happend
- trace your file system activity(like what important files/folders was changed and when), and make alerts about such events
- use a vpn for web-gui access like openvpn
- use port knoking for ssh acces
- use a logchecker that can alert if some suspect activity are happend
- and ... more

Remember, security is a process, not a product!

 

[Diogo Jesus]

Yes, is important to find how they go into Proxmox, but this is only a part of the problem. As I can see in your post you may ask yourself how you solve others problems, like this(I can provide more):
- at anytime, some bad guy can find and exploit a weakness in your sistem(it is only a matter of time ... not if), so you must have several layers of protection(iptables is only one layer)
- another problem is the fact that is very important to have logs also on a remote server(remote log server), so after a incident, you will be able to see what was happend
- trace your file system activity(like what important files/folders was changed and when), and make alerts about such events
- use a vpn for web-gui access like openvpn
- use port knoking for ssh acces
- use a logchecker that can alert if some suspect activity are happend
- and ... more

Remember, security is a process, not a product!

we installed logwatch and rkhunter aswell and used to receive daily emails but one week ago we stopped receiving those emails and we thought it was our smtp domain that was down (and it was) so i was working into getting a smtp server working into a container and assigning to a new domain and start receiving the mails. i also saw a lot of changes under our fail2ban/logwatch logs in /var/logs during these last days knowing that no one except me is working inside of proxmox and even myself didn't touch on proxmox for the past week since i was working only inside a container.


my priority at this point is like how i said that i must find the breach so i can reset my server asap since they can still connect to my server. We can suffer any attack at any time.
Also i'm not even sure if it's a breach made by me or by proxmox itself.

 

[guletz]

i don't think they did since internally we have a good security (i assume)

from the logs it gets suspicious when i see

Oct 9 04:09:09 ns3037493 systemd[1]: Stopping Proxmox VE firewall logger...
Oct 9 04:09:09 ns3037493 pvepw-logger[25692]: received terminate request (signal)

... assume that you have a bad security, this is the right path!

- use monit to watch your important services(IF firewall is down THEN start firewall AND alert me - IF we have 3 such events, then shutdown the server - IF my-firewall-rules are not like THIS, then alert me )

 

[LnxBil]

i don't think they did since internally we have a good security (i assume)

'security' and 'assuming' don't get along :-D

 

[LnxBil]

my priority at this point is like how i said that i must find the breach so i can reset my server asap since they can still connect to my server.

You have stop your system (hard power off), boot a live linux to dd your disk to a off-site storage for later analysis and reinstall immediately to get everything back online. Everything else destroys evidence. There are also people you can call that analyse everything for your.

 

[Diogo Jesus]

'security' and 'assuming' don't get along :-D

Our server is hosted in a data center. And i'm working only inside this server so the company internet security is not up to me. usually the ISP should work on that (i can always limit a firewall of course)

 

[Diogo Jesus]

You have stop your system (hard power off), boot a live linux to dd your disk to a off-site storage for later analysis and reinstall immediately to get everything back online. Everything else destroys evidence. There are also people you can call that analyse everything for your.

if i do it like that it will be a matter of time until they hack me again since the vulnerability is still there (until i fix it) even if we under under attack our services are still working so i will let it on since we cannot just stop the machine like that (a lot of important prod in on at this point and 1h off during the day is a lot of money wasted) i might need to do it during the night but for that i must find the "bug"

 

[Diogo Jesus]

... assume that you have a bad security, this is the right path!

- use monit to watch your important services(IF firewall is down THEN start firewall AND alert me - IF we have 3 such events, then shutdown the server - IF my-firewall-rules are not like THIS, then alert me )

cool is always good to get extra security already noted this one to install with the new server installation

 

[guletz]

i also saw a lot of changes under our fail2ban/logwatch logs in /var/logs during these last days knowing that no one except me is working inside of proxmox and even myself didn't touch on proxmox for the past week since i was working only inside a container.

... and you did not make nothing? This was a HUGE mistake. When you see any suspect activity, you must react. You must check regulary your system(daily base), and not weekly. Use check list ...like:
- ckeck logs
- check firewall
- check files/folders(sha512 hash)
- check services
- REINSTALL critical services like iptables, ssh, ....
- check ....

 

[Diogo Jesus]

... and you did not make nothing? This was a HUGE mistake. When you see any suspect activity, you must react. You must check regulary your system(daily base), and not weekly.

agree on that actually i only saw the changes today even if i had saw them b4 it would be to late.. as i mentioned before they added ssh keys to different IPs (i posted the log on my second or third comment) and that was before the changes under fail2ban etc.. so it would be to late.

But on my point of view they managed to enter from the API, is there something that i can disable the proxmox api since i don't need it?

 

[guletz]

if i do it like that it will be a matter of time until they hack me again since the vulnerability is still there (until i fix it) even if we under under attack our services are still working so i will let it on since we cannot just stop the machine like that (a lot of important prod in on at this point and 1h off during the day is a lot of money wasted) i might need to do it during the night but for that i must find the "bug"

.... nope It could be a wrong path(unknown bug for example)! Use other layers of protections - you can not find the bug directly, but maybe it is easy to find the bug "damage"(like a file/folder changes - ossec is a good tool for this, with iptables blocking sistem ... ).

 

[guletz]

added ssh keys to different IPs

- like I said, use a vpn to access your proxmox server, and the make proxmox web-interface to listen only for vpn connections

 

[LnxBil]

if i do it like that it will be a matter of time until they hack me again since the vulnerability is still there (until i fix it)

Very optimistic thinking to fix it yourself and hardly realizable depending on the hack itself.

our services are still working so i will let it on since we cannot just stop the machine like that (a lot of important prod in on at this point and 1h off during the day is a lot of money wasted)

...and you think running VMs on a compromised server and making money is wise? What if they already hacked your credit card db, injected stuff in your paypal pipeline or installed malware in your software (assuming your're selling software). You have a situation you cannot control.

You're breaking so many forensics 101 rules ... but it's your server.