Strange SSH/SFTP behaviour

[taenzerme]

We updated our PVE hosts to latest 4.4 (Enterprise repo) and our pfSense firewall to 2.3.3 last night. All hosts share exactly the same configuration, especially in terms of network hardware.

Some hours after the upgrade SOME of our VMs start showing a strange behaviour:

- Sudden "broken pipe" disconnects during SSH sessions. Even while typing, so it's not a timeout issue.
- SFTP connection to hosts work, directory listings and downloads, too. Uploads: Nada. 32kb of the file, that's it. Then the connection is disconnected (either by the host or the client - I can't find out) and reconnects.

The following things did not make a difference:

- Different VLAN ID for the VM and use of different pfSense VM for routing
- Migrating the VM to another node.
- Restarting SSH daemon in the VM.

What DID work:

- Switching to another VLAN and getting another IP from its DHCP server (i.e. different subnet).

The strange thing: Other Vms on the same subnet/VLAN id work flawlessly. The only difference here: The working hosts are running Debian Wheezy, the not working hosts Jessie.

We did not do ANY modifications on the software side except host and pfSense updates.

No cabling, no pfSense modifications.

Any ideas out there where and what to look for?

 

[taenzerme]

Another sidenote: The VLANs are physically on the same NIC, so I guess I could rule out cabling problems in the rack, can I?

 

[PigLover]

IP address collision on the original VLAN? Some other device(s) using the same address(es) assigned by your DHCP?

 

[taenzerme]

That actually is a good idea. But shouldn't it cause more problems with other services then? Regular FTP or HTTP work just fine with the same VMs.

 

[taenzerme]

I checked the arp entries on every host. To be honest, I have nothing. It's just SFTP upload/connections failing. Everything else works just fine. All hosts (Debian 8) got the latest updates recently. After that this problem appeared. Any known bugs in OpenSSH lately?

 

[taenzerme]

We have been able to narrow it down to some kind of routing problem at least but have not found a solution yet.

SSH and SFTP work just fine when using the public IP and connecting to the server externally through NAT. Connecting to the internal IP via VPN results in the aforementioned problem (affecting SSH/SFTP only, regular HTTP and FTP work fine).

VPN is established via an OpenVPN AS KVM running on one of the cluster nodes in subnet 192.168.100.0/24 with a gateway to pfSense firewall1 in the same subnet. firewall1 is routing the target subnet 10.20.0.0/24 with a nic in that subnet, so SSH/SFTP connections on the target host see firewall1 as connecting client.

All this happened after the latest Proxmox 4.3 => 4.4 and pfSense 2.3.2 => 2.3.4 upgrades.

The configuration was working until the upgrades, but I sense some connection to the pfSense upgrades somehow and guess I post in their forums might be a better audience.

Important information: Going back to pfSense 2.3.2 (snapshot exists) everything works as intended again. So I should be able to rule out Proxmox as culprit here.

If anyone has an idea I'd be happy to hear it of course.

 

[taenzerme]

We found the culprit. On the second virtual firewall (pfSense) pfBlockerNg (an iptables based blocking package) was installed but somehow was broken by the update of pfSense to 2.3.4. I removed the package completely and the connection problems are gone.

Just in case some experiences something similiar with pfSense.