Network Interface Limit

tuxflow

New Member
Sep 28, 2020
3
0
1
55
We have a Proxmox cluster with 3 nodes. On one node there are 33 VMs.

Unfortunately I can't run another VM there, every new VM has no network. If I move the VM to an empty node, the network works immediately.

Is there an interface limit per node and why? Are there other solutions to run more VMs on one node?
 
What's in the syslog? Did you restart the affected node?
 
Hi Alwin,

great to hear from you!

First some Additional Info:


We have a Proxmox cluster with 3 nodes. On one node (pm03) there are 33 VMs. When the 34th VM goes online it gets no network connection. We tried with different virtual network cards. There are some VMs with two (virtual) network interfaces so the number of network interfaces may be higher, here output of

Code:
ip a

## see attachment output_of_ipa.txt, to workaround the 10000 char limit

Each node is on the bridge vmbr0 with the public server IP (<secret>
;)
and there are several vlans for the VMs. Here is the network configuration

Code:
cat /etc/network/interfaces

 
source /etc/network/interfaces.d/*

 
auto lo
iface lo inet loopback

 
iface lo inet6 loopback

 
iface lo inet loopback

 
iface eno1 inet manual

 
iface eno2 inet manual

 
iface eno3 inet manual

 
iface eno4 inet manual

 
#internal vlan for proxmox communication
auto eno1.4001
iface eno1.4001 inet static
address 10.120.1.2/24
mtu 1400
#proxmox

 
auto vmbr0
iface vmbr0 inet static
address <secret>/25
gateway <secret gw>
bridge-ports eno1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
mtu 1400
up route add -net <secret net> netmask 255.255.255.128 gw <secret gw> dev vmbr0
        down route del -net <secret net> netmask 255.255.255.128 gw <secret gw> dev vmbr0

If I move the VM to one of the other nodes with fewer VMs, the network will work immediately.


If I stop one VM on pm03 and activate the new VM it works.


ethtool from inside the VM shows an active link..


We tried with several VM images (different MACs).


So our suspect was an interface or MAC limit on the bridge vmbr0... So the first question is: is there a limit or not? If not, what can we do? We have a lot of experience with Linux networking and can provide you any information that you may need, but here we are stuck :-(


Regarding your question:


Here is the syslog from the proxmox host pm03, where the network does not work, when we activate the vm

Oct 1 10:49:50 pm03-intern pvedaemon[18622]: <root@pam> update VM 159: -net0 virtio=36:78:16:6A:DC:72,bridge=vmbr0,tag=4002 Oct 1 10:49:58 pm03-intern pvedaemon[8623]: <root@pam> starting task UPID:pm03:00007FA4:56F42BEC:5F759836:qmstart:159:root@pam: Oct 1 10:49:58 pm03-intern pvedaemon[32676]: start VM 159: UPID:pm03:00007FA4:56F42BEC:5F759836:qmstart:159:root@pam: Oct 1 10:49:58 pm03-intern systemd[1]: Started 159.scope. Oct 1 10:49:58 pm03-intern systemd-udevd[32685]: Using default interface naming scheme 'v240'. Oct 1 10:49:58 pm03-intern systemd-udevd[32685]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable. Oct 1 10:49:58 pm03-intern systemd-udevd[32685]: Could not generate persistent MAC address for tap159i0: No such file or directory Oct 1 10:49:58 pm03-intern kernel: [14587276.491712] device tap159i0 entered promiscuous mode Oct 1 10:49:58 pm03-intern kernel: [14587276.514520] vmbr0: port 36(tap159i0) entered blocking state Oct 1 10:49:58 pm03-intern kernel: [14587276.514613] vmbr0: port 36(tap159i0) entered disabled state Oct 1 10:49:58 pm03-intern kernel: [14587276.515513] vmbr0: port 36(tap159i0) entered blocking state Oct 1 10:49:58 pm03-intern kernel: [14587276.515599] vmbr0: port 36(tap159i0) entered forwarding state Oct 1 10:49:58 pm03-intern pvedaemon[8623]: <root@pam> end task UPID:pm03:00007FA4:56F42BEC:5F759836:qmstart:159:root@pam: OK Oct 1 10:50:00 pm03-intern systemd[1]: Starting Proxmox VE replication runner... Oct 1 10:50:00 pm03-intern systemd[1]: pvesr.service: Succeeded. Oct 1 10:50:00 pm03-intern systemd[1]: Started Proxmox VE replication runner. Oct 1 10:50:11 pm03-intern pvedaemon[18622]: worker exit Oct 1 10:50:12 pm03-intern pvedaemon[1789]: worker 18622 finished Oct 1 10:50:12 pm03-intern pvedaemon[1789]: starting 1 worker(s) Oct 1 10:50:12 pm03-intern pvedaemon[1789]: worker 32789 started


And here is the syslog from the proxmox host pm02 with less VMs and that works:


Oct 1 13:22:30 pm02-intern qmeventd[1239]: Starting cleanup for 159 Oct 1 13:22:30 pm02-intern qmeventd[1239]: Finished cleanup for 159 Oct 1 13:22:36 pm02-intern pvedaemon[26501]: <root@pam> starting task UPID:pm02:00000EEC:5B966496:5F75BBFC:qmstart:159:root@pam: Oct 1 13:22:36 pm02-intern pvedaemon[3820]: start VM 159: UPID:pm02:00000EEC:5B966496:5F75BBFC:qmstart:159:root@pam: Oct 1 13:22:36 pm02-intern systemd[1]: Started 159.scope. Oct 1 13:22:36 pm02-intern systemd-udevd[3829]: Using default interface naming scheme 'v240'. Oct 1 13:22:36 pm02-intern systemd-udevd[3829]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable. Oct 1 13:22:36 pm02-intern systemd-udevd[3829]: Could not generate persistent MAC address for tap159i0: No such file or directory Oct 1 13:22:36 pm02-intern kernel: [15365246.424596] device tap159i0 entered promiscuous mode Oct 1 13:22:36 pm02-intern kernel: [15365246.447655] vmbr0: port 4(tap159i0) entered blocking state Oct 1 13:22:36 pm02-intern kernel: [15365246.447744] vmbr0: port 4(tap159i0) entered disabled state Oct 1 13:22:36 pm02-intern kernel: [15365246.448554] vmbr0: port 4(tap159i0) entered blocking state Oct 1 13:22:36 pm02-intern kernel: [15365246.448641] vmbr0: port 4(tap159i0) entered forwarding state Oct 1 13:22:36 pm02-intern pvedaemon[26501]: <root@pam> end task UPID:pm02:00000EEC:5B966496:5F75BBFC:qmstart:159:root@pam: OK

Any help is greatly appreciated!
 

Attachments

  • output_of_ip_a.txt
    6.7 KB · Views: 1
Did you check your routing table on the node? Is it full?
 
No it's not

root@pm02 ~ # wc -l /proc/net/route 21 /proc/net/route


The node does no routing for the VMs. All the VMs are bridged to the main interface...

root@pm02 ~ # brctl show bridge name bridge id STP enabled interfaces fwbr102i0 8000.aea6c2fa17b4 no fwln102i0 tap102i0 fwbr105i0 8000.c689bdb89cf2 no fwln105i0 tap105i0 vmbr0 8000.246e96ae01e6 no eno1 fwpr102p0 fwpr105p0 tap101i0 tap113i0 tap114i0 tap115i0 tap116i0 tap117i0 tap118i0 tap122i0 tap134i0 tap149i0 tap150i0 tap151i0 tap152i0 tap153i0 tap154i0 tap155i0 tap157i0 tap158i0 tap159i0

the 21 routes are almost all vpn routes... These are the only routes on the bridge

root@pm02 ~ # netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 <gw> 0.0.0.0 UG 0 0 0 vmbr0 <ip> <gw> 255.255.255.128 UG 0 0 0 vmbr0 <ip> 0.0.0.0 255.255.255.128 U 0 0 0 vmbr0

As far as I understand it we run more into a layer 2 problem (bridge/arp), not layer 3 (Routing/IP)...
 
Hello, I am getting the same problem as referenced above. Same error in the log:

Oct 1 10:49:58 pm03-intern systemd-udevd[32685]: Using default interface naming scheme 'v240'.
Oct 1 10:49:58 pm03-intern systemd-udevd[32685]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Oct 1 10:49:58 pm03-intern systemd-udevd[32685]: Could not generate persistent MAC address for tap159i0: No such file or directory
Oct 1 10:49:58 pm03-intern kernel: [14587276.491712] device tap159i0 entered promiscuous mode

Did anyone find a solution to this? Maybe my subnet is restricting the number of connections per bridge?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!