[SOLVED] VM outside connection suddenly lost

[WordsOfMe]

Hello,
I run a small homelab server with the latest Proxmox VE, and I have now encountered a problem I hope someone can help me with:

Setup
My DMZ is behind a firewall, the interface is 172.27.1.1. The Proxmox node runs on 172.27.100.100, and the VM that troubles me is on 172.27.100.103 (ID 103). This configuration has been running fine for 3 years now.

Issue:
One of the VMs with ID 103 - of course the one with my e-mail server - is not reachable anymore. I can ping the VM IP from the Proxmox node shell and get a reply, but I cannot open any ssh or HTTP connection to the VM. I can open a console from the web interface where the NIC is showing to be properly set up. However, when I try to ping the node I get a "temporary error in name resolution" fault. When I try to ping the gateway (firewall interface) I get a "no route to host" fault.

What I tried:
I have disabled the firewall of the node and the VM, I have removed and re-added the network interface of the VM without success. I can see the tap-Interface entering the forwarding state. I have three more VMs on the node that work flawlessly.

Cause:
I have no idea. I have tried to implement IPv6 an a LXC yesterday, but this did not work as expected and I have reverted everything. IPv6 is disabled via sysctl.conf on the firewall DMZ interface, the node and also the VM.

Any help would be great!
Thanks,
Stefan

 

[WordsOfMe]

...OK, I got it. Here is my answer, mabye this will help others in the future:

• Proxmox node not reachable
  • This was caused by IPv6 being disabled inside the VM. Although IPv6 is also disabled on the Procmox node, apparently the VM (172.27.100.103) cannot ping the node it is running on (172.27.100.100) when inside the VM IPv6 is disabled using /etc/sysctl.conf.
• Gateway not reachable
  • The gateway 172.27.1.1 was actually blocked by a fail2ban instance running inside the VM.
  • I was of course aware of this possibility and had checked the filter status, but nothing showed up in iptables in the VM:
    

    # iptables -L
    Chain INPUT (policy ACCEPT)
    target     prot opt source             destination
    
    Chain FORWARD (policy ACCEPT)
    target     prot opt source             destination
    
    Chain OUTPUT (policy ACCEPT)
    target     prot opt source             destination

  • However, there actually was an active ban against the gateway address 172.27.1.1 as shown by the fail2ban command:
    

    # fail2ban-client status zimbra-smtp
    Status for the jail: zimbra-smtp
    |- Filter
    |  |- Currently failed: 0
    |  |- Total failed:     12
    |  `- File list:        /var/log/zimbra.log
    `- Actions
       |- Currently banned: 1
       |- Total banned:     1
       `- Banned IP list:   172.27.1.1

  • I have no idea why this ban was not shown in the output of iptables, as the banaction configured actually was iptables-multiport. So, if anyone can enlighten me on that bit I would be interested.
  • Restoring connection to the gateway then was easy and just required stopping and disabling fail2ban.
• Root cause
  • The root cause was that I had changed the setup of my firewall a few days before. Instead of fowarding port 465 (SMTPS) to the VM, for certificate renewal issues this port was then handled by HAproxy on the firewall, and from there an unencrypted connection to the VM, port 587 (SUBMISSION), was established.
  • As this is a pure TCP proxy the receiving VM does not know the actual IP of the outside initiating server, instead the connection is always established by the DMZ interface address 172.27.1.1, which is also the gateway for the Promox node and all VMs.
  • The effect of swicthing over to HAproxy was not immediately recognized because it took a few days before the first malicious connection on port 465, probing username / password combinations, was made.