I have encountered a very strange networking behaviour with the latest 4.1 build of Proxmox and I'd like the Pros insight on this if possible.
A (quick) little background first...
I currently run 2 x Proxmox v3 & 5 x Proxmox v4.1 hosts in Online.net [FR] and one Proxmox v4 host in i3D.net [NL]. Each host has a few VMs (CTs to be exact & no more than 6), primarily used for a single customer of mine. Each VM is working just fine in terms of networking: it is accessible from the outside, I can SSH, rsync, SCP to it, I can retrieve its contents to offsite backup servers (located in Germany) and generally, they perform as expected. Especially with the latest Proxmox v4.1 build, setting up the networking defails of a VM is a piece of cake and done in the CT creation process alone, as I don't have to manually modify networking as I had to do with Proxmox v3. So, to sum up, the VMs work as expected in regards to the "outside" world and internally on each host. The problems start when I want to SSH from a VM located in a Proxmox v4 host to a VM on another Proxmox v4 host. The following problems are not present on the 2 Proxmox v3 hosts though.
The issue...
All the Proxmox v4 hosts I have are using the latest 4.1 build at the time of writing. Last night I realized I cannot SSH (and therefore rsync) between 2 VMs located on different Proxmox hosts.
Facts on the VMs:
- They all use Ubuntu 14.04 LTS latest
- They all run a simple LNMP stack, no cPanel or any other control panel
- IPTables is controlled by CSF and the ports I want to control (including port 22) are open and accessible both internally and externally
- All VMs have networking setup according to their hosting provider: online.net uses routes & MAC address assignments while i3D.net uses "traditional" networking (you just use the IPs assigned to you).
- All VMs have a secondary virtual NIC for private networking (inside each Proxmox host obviously) which works as expected.
- Each VM IPs (public & private) are excluded in the firewall of the other same-host VMs.
- VMs on all hosts are not blocked anywhere in their respective f/w rules. So as an example, VM #1 on Proxmox host no. 1 is not blocked at all on VM #3 on Proxmox host no. 2. Same goes for every other VM.
- VMs across Proxmox v4 hosts can ping or curl each other just fine. Telnet verifies ports are accessbile too (especially port 22).
- Traceroute between 2 VMs across Proxmox v4 hosts however times out.
- SSH, rsync etc. between 2 VMs across Proxmox v4 hosts time out.
- SSH, rsync etc. between a VM on a Proxmox v4 host and any other non-Proxmox v4 based VM works just fine. In fact, SSH between a Proxmox v3 based a VM and a Proxmox v4 VM (and vice versa) works just fine.
For the love of god (any god really), SSH and everything over SSH (like rsync) does not work between VMs across different Proxmox v4 hosts.
Any help or insight would greatly be appreciated.
If anyone requires additional info (e.g. interface setup on VMs or host, traceroute or ping results etc.) I'd be more than happy to report them back here.
A (quick) little background first...
I currently run 2 x Proxmox v3 & 5 x Proxmox v4.1 hosts in Online.net [FR] and one Proxmox v4 host in i3D.net [NL]. Each host has a few VMs (CTs to be exact & no more than 6), primarily used for a single customer of mine. Each VM is working just fine in terms of networking: it is accessible from the outside, I can SSH, rsync, SCP to it, I can retrieve its contents to offsite backup servers (located in Germany) and generally, they perform as expected. Especially with the latest Proxmox v4.1 build, setting up the networking defails of a VM is a piece of cake and done in the CT creation process alone, as I don't have to manually modify networking as I had to do with Proxmox v3. So, to sum up, the VMs work as expected in regards to the "outside" world and internally on each host. The problems start when I want to SSH from a VM located in a Proxmox v4 host to a VM on another Proxmox v4 host. The following problems are not present on the 2 Proxmox v3 hosts though.
The issue...
All the Proxmox v4 hosts I have are using the latest 4.1 build at the time of writing. Last night I realized I cannot SSH (and therefore rsync) between 2 VMs located on different Proxmox hosts.
Facts on the VMs:
- They all use Ubuntu 14.04 LTS latest
- They all run a simple LNMP stack, no cPanel or any other control panel
- IPTables is controlled by CSF and the ports I want to control (including port 22) are open and accessible both internally and externally
- All VMs have networking setup according to their hosting provider: online.net uses routes & MAC address assignments while i3D.net uses "traditional" networking (you just use the IPs assigned to you).
- All VMs have a secondary virtual NIC for private networking (inside each Proxmox host obviously) which works as expected.
- Each VM IPs (public & private) are excluded in the firewall of the other same-host VMs.
- VMs on all hosts are not blocked anywhere in their respective f/w rules. So as an example, VM #1 on Proxmox host no. 1 is not blocked at all on VM #3 on Proxmox host no. 2. Same goes for every other VM.
- VMs across Proxmox v4 hosts can ping or curl each other just fine. Telnet verifies ports are accessbile too (especially port 22).
- Traceroute between 2 VMs across Proxmox v4 hosts however times out.
- SSH, rsync etc. between 2 VMs across Proxmox v4 hosts time out.
- SSH, rsync etc. between a VM on a Proxmox v4 host and any other non-Proxmox v4 based VM works just fine. In fact, SSH between a Proxmox v3 based a VM and a Proxmox v4 VM (and vice versa) works just fine.
For the love of god (any god really), SSH and everything over SSH (like rsync) does not work between VMs across different Proxmox v4 hosts.
Any help or insight would greatly be appreciated.
If anyone requires additional info (e.g. interface setup on VMs or host, traceroute or ping results etc.) I'd be more than happy to report them back here.
