Proxmox 5 migrated from 4 lxc networking not working

[John Wright]

Hi,

I have recently installed a new Proxmox 5 node to our Current Proxmox 4 cluster

We migrated one of the lxc VMs and at first it seemed to work fine, then after a few minutes it stopped responding to ping ssh etc

I can console in, but i can not get out to the internet.

The networking card seems to be sending and receiving packages.

I have had the routers and IPs checked by OVH.

Firewall is disabled.

Same thing happens to other machines that I migrate all VMs on Ubuntu 18.04.01 LTS

proxmox-ve: 5.3-1 (running kernel: 4.15.18-11-pve)
pve-manager: 5.3-11 (running version: 5.3-11/d4907f84)
pve-kernel-4.15: 5.3-2
pve-kernel-4.15.18-11-pve: 4.15.18-34
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-47
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-12
libpve-storage-perl: 5.0-38
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-3
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-3
proxmox-widget-toolkit: 1.0-23
pve-cluster: 5.0-33
pve-container: 2.0-35
pve-docs: 5.3-3
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-18
pve-firmware: 2.0-6
pve-ha-manager: 2.0-8
pve-i18n: 1.0-9
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 2.12.1-2
pve-xtermjs: 3.10.1-2
qemu-server: 5.0-47
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3

Any help would be appreciated!

Best Regards

John

Existing Node Proxmox 4 Networking Config



Working Example VM on Proxmox 4 Networking



New Node Proxmox 5 Networking Config (same as proxmox 4 node setup)



Migrated VM Networking (No Changes when it was working on Proxmox 4)



ifconfig output on broken VM

 

[Richard]

Hi,

I have recently installed a new Proxmox 5 node to our Current Proxmox 4 cluster

We migrated one of the lxc VMs and at first it seemed to work fine, then after a few minutes it stopped responding to ping ssh etc

I can console in, but i can not get out to the internet.

The networking card seems to be sending and receiving packages.

I have had the routers and IPs checked by OVH.

AFAICS you use for the VMs bridges which are directly connected to physical networks. In such a case it works only if your provider accept the virtual MAC addresses from the VM. Since it seems that packets are sent out from the NICs (but no response from internet servers) it's very probable that the provider does not accept the MAC addresses - they may have been changed during the migration process (depends on how you made it exactly) or your VMs are now connected to other physical ports.

Try to reconfigure all MAC addresses as they were before migration and all VMs should run in the same node as before.

 

[John Wright]

Hi,

I'm not sure if I understand you correctly, as far as I know the provider has not changed the MAC address policy and the Virtual MAC addresses have not changed either, however just in case I have contacted OVH with your suggestion and am waiting on a response.

I have tried to remove and re-add the Virtual NICs on a different VM with the same issue, however this did not help.

Best Regards

John

 

[Richard]

Follow the packets by tcpdump from (virtual) NIC no NIC and figure out where they get lost.

If packets are sent correctly to the providers network and you don't get any response the reason can be:

- routing (gateway) is not configured properly - even it's less probable
- your provider drops the packets - in that case only he can clarify why

If a packet is sent to virtual network (e.g. from a VM) and not arriving at the other side (e.g. in host) the related configuration in Proxmox has to be investigated. In that case the easiest way is to run

pvereport

and posting the result.

 

[John Wright]

Unfortunately creating the tcpdump command from NIC to NIC i'm not sure how to do

When I did try and run some tcpdump I found that it was the kernel that was dropping all of the packets e.g

Ubuntu Bionic Beaver (development branch) worker tty1

worker login: root
Password:
Last login: Sun Mar 10 22:59:30 UTC 2019 on lxc/tty1
Welcome to Ubuntu Bionic Beaver (development branch) (GNU/Linux 4.15.18-11-pve x86_64)

You have mail.
root@worker:~# tcpdump -v -i eth1
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
^C15:11:00.905409 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 5.39.56.161 tell 5.39.56.167, length 46

1 packet captured
189 packets received by filter
182 packets dropped by kernel
root@worker:~#

Below id a new response from OVH support

From: OVH Support
Dear John,

When IPs are routed to the vRack there is no virtual mac address on the IPs.
We only use vMAC when you route the IP block to the server (not when to the
vRack).

There are no changes to the routing at all on the IPs because nothing has
changed on network side. The only change is that you added 1 extra server to
the vRack. Hence why you see the IPs still work on your other servers in the
vRack.

The only change is that you are using different version of proxmox.

Here is a breakdown on how routing of public IPs work in vRack:
docs. ovh. com /gb /en /dedicated /ip-block-vrack /

On networking side I do not see any changes and as per my last response, you
could see that with iftop, I seen packet flow on eth1 (the interface I
configured 5.39.56.180 and that is the vRack port) including our monitoring
pings.

What I can do if you want, is I can deliver a temporary block of 4 IPs into
your vRack and I can test that with your dedicated server in rescue mode also.

For any other questions or concerns, please feel free to contact us through a
support ticket or through by phone at 0333 370 0425.

Adam O.
OVH UK Support
[1]

Also I have attached the output from pvereport on the host

Best Regards

John Wright

 

[Richard]

Unfortunately creating the tcpdump command from NIC to NIC i'm not sure how to do

Let's assume you are e.g. in container 104 and want to follow pinging to 8.8.8.8

it starts in container (with an arp packet):

tcpdump -e -n -i eth1

Next station is vmbr1-port for this NIC in the host:

tcpdump -e -n -i veth104i1

Then the bridge

tcpdump -e -n -i vmbr1

Then enp61s0f1

tcpdump -e -n -i enp61s0f1

This is the point where the packet leaves your environment. If everything up to here is correct and you don't get any response the problem is in the external environment (router, switch etc.)

If you get a response follow it the other way round.

In the above example I did not take care of the firewall you have - recommended to disable it for those tests.