Hi Alwin,
great to hear from you!
First some Additional Info:
We have a Proxmox cluster with 3 nodes. On one node (pm03) there are 33 VMs. When the 34th VM goes online it gets no network connection. We tried with different virtual network cards. There are some VMs with two (virtual) network interfaces so the number of network interfaces may be higher, here output of
Code:
ip a
## see attachment output_of_ipa.txt, to workaround the 10000 char limit
Each node is on the bridge vmbr0 with the public server IP (<secret>
and there are several vlans for the VMs. Here is the network configuration
Code:
cat /etc/network/interfaces
source /etc/network/interfaces.d/*
auto lo
iface lo inet loopback
iface lo inet6 loopback
iface lo inet loopback
iface eno1 inet manual
iface eno2 inet manual
iface eno3 inet manual
iface eno4 inet manual
#internal vlan for proxmox communication
auto eno1.4001
iface eno1.4001 inet static
address 10.120.1.2/24
mtu 1400
#proxmox
auto vmbr0
iface vmbr0 inet static
address <secret>/25
gateway <secret gw>
bridge-ports eno1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
mtu 1400
up route add -net <secret net> netmask 255.255.255.128 gw <secret gw> dev vmbr0
down route del -net <secret net> netmask 255.255.255.128 gw <secret gw> dev vmbr0
If I move the VM to one of the other nodes with fewer VMs, the network will work immediately.
If I stop one VM on pm03 and activate the new VM it works.
ethtool
from inside the VM shows an active link..
We tried with several VM images (different MACs).
So our suspect was an interface or MAC limit on the bridge
vmbr0
... So the first question is: is there a limit or not? If not, what can we do? We have a lot of experience with Linux networking and can provide you any information that you may need, but here we are stuck :-(
Regarding your question:
Here is the syslog from the proxmox host pm03, where the network does not work, when we activate the vm
Oct 1 10:49:50 pm03-intern pvedaemon[18622]: <root@pam> update VM 159: -net0 virtio=36:78:16:6A:DC:72,bridge=vmbr0,tag=4002
Oct 1 10:49:58 pm03-intern pvedaemon[8623]: <root@pam> starting task UPID:pm03:00007FA4:56F42BEC:5F759836:qmstart:159:root@pam:
Oct 1 10:49:58 pm03-intern pvedaemon[32676]: start VM 159: UPID:pm03:00007FA4:56F42BEC:5F759836:qmstart:159:root@pam:
Oct 1 10:49:58 pm03-intern systemd[1]: Started 159.scope.
Oct 1 10:49:58 pm03-intern systemd-udevd[32685]: Using default interface naming scheme 'v240'.
Oct 1 10:49:58 pm03-intern systemd-udevd[32685]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Oct 1 10:49:58 pm03-intern systemd-udevd[32685]: Could not generate persistent MAC address for tap159i0: No such file or directory
Oct 1 10:49:58 pm03-intern kernel: [14587276.491712] device tap159i0 entered promiscuous mode
Oct 1 10:49:58 pm03-intern kernel: [14587276.514520] vmbr0: port 36(tap159i0) entered blocking state
Oct 1 10:49:58 pm03-intern kernel: [14587276.514613] vmbr0: port 36(tap159i0) entered disabled state
Oct 1 10:49:58 pm03-intern kernel: [14587276.515513] vmbr0: port 36(tap159i0) entered blocking state
Oct 1 10:49:58 pm03-intern kernel: [14587276.515599] vmbr0: port 36(tap159i0) entered forwarding state
Oct 1 10:49:58 pm03-intern pvedaemon[8623]: <root@pam> end task UPID:pm03:00007FA4:56F42BEC:5F759836:qmstart:159:root@pam: OK
Oct 1 10:50:00 pm03-intern systemd[1]: Starting Proxmox VE replication runner...
Oct 1 10:50:00 pm03-intern systemd[1]: pvesr.service: Succeeded.
Oct 1 10:50:00 pm03-intern systemd[1]: Started Proxmox VE replication runner.
Oct 1 10:50:11 pm03-intern pvedaemon[18622]: worker exit
Oct 1 10:50:12 pm03-intern pvedaemon[1789]: worker 18622 finished
Oct 1 10:50:12 pm03-intern pvedaemon[1789]: starting 1 worker(s)
Oct 1 10:50:12 pm03-intern pvedaemon[1789]: worker 32789 started
And here is the syslog from the proxmox host pm02 with less VMs and that works:
Oct 1 13:22:30 pm02-intern qmeventd[1239]: Starting cleanup for 159
Oct 1 13:22:30 pm02-intern qmeventd[1239]: Finished cleanup for 159
Oct 1 13:22:36 pm02-intern pvedaemon[26501]: <root@pam> starting task UPID:pm02:00000EEC:5B966496:5F75BBFC:qmstart:159:root@pam:
Oct 1 13:22:36 pm02-intern pvedaemon[3820]: start VM 159: UPID:pm02:00000EEC:5B966496:5F75BBFC:qmstart:159:root@pam:
Oct 1 13:22:36 pm02-intern systemd[1]: Started 159.scope.
Oct 1 13:22:36 pm02-intern systemd-udevd[3829]: Using default interface naming scheme 'v240'.
Oct 1 13:22:36 pm02-intern systemd-udevd[3829]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Oct 1 13:22:36 pm02-intern systemd-udevd[3829]: Could not generate persistent MAC address for tap159i0: No such file or directory
Oct 1 13:22:36 pm02-intern kernel: [15365246.424596] device tap159i0 entered promiscuous mode
Oct 1 13:22:36 pm02-intern kernel: [15365246.447655] vmbr0: port 4(tap159i0) entered blocking state
Oct 1 13:22:36 pm02-intern kernel: [15365246.447744] vmbr0: port 4(tap159i0) entered disabled state
Oct 1 13:22:36 pm02-intern kernel: [15365246.448554] vmbr0: port 4(tap159i0) entered blocking state
Oct 1 13:22:36 pm02-intern kernel: [15365246.448641] vmbr0: port 4(tap159i0) entered forwarding state
Oct 1 13:22:36 pm02-intern pvedaemon[26501]: <root@pam> end task UPID:pm02:00000EEC:5B966496:5F75BBFC:qmstart:159:root@pam: OK
Any help is greatly appreciated!