[SOLVED] Second NIC on different Subnet and VLAN

NetgearNinja

New Member
Jun 17, 2020
4
0
1
20
Hi,
I have proxmox VE 6.2-4 installed on an Intel NUC with one ethernet port (eno1). I recently bought an ubs-ethernet dongle (enx00e04) to add one more NIC, so some of the VMs can run on a seperate subnet with different IP addresses in bblan (192.168.48.X range) instead of in aalan (192.168.43.X range).

My physical configuration is:
router port 3 (aalan, 192.168.43.0/24 with DHCP server, VLAN 4) ------ethernet cable-------- NUC ethernet port (interface: eno1)
router port 4 (bblan, 192.168.48.0/24 with DHCP server, VLAN 5) ------ethernet cable-------- NUC usb-ethernet dongle (interface: enx00e04)
Both router ports seem to be working fine, I can access proxmox (in the 192.168.43.X range) on 192.168.43.5 and connecting a laptop to router port 4 assigns an IP address in the 192.168.48.X range, which is what I want.

My proxmox /etc/network/interfaces file:
Code:
auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual

allow-hotplug enx00e04

auto enx00e04
iface enx00e04 inet manual

auto vmbr0
iface vmbr0 inet static
         address 192.168.43.5/24
         gateway 192.168.43.1
         bridge-ports eno1
         bridge-stp off
         bridge-fd 0
         bridge-vlan-aware yes
         bridge-vids 2-4094

auto vmbr1
iface vmbr1 inet static
         address 192.168.48.5/24
         bridge-ports enx00e04
         bridge-stp off
         bridge-fd 0
         bridge-vlan-aware yes
         bridge-vids 2-4094
Now I tried to add a new VM and connected it to vmbr1, but when i get to the networking screen (inside the VM) it says that acquiring an IP address from the DHCP server on bblan has failed. I have already looked at 'Multiple VMs on different subnets with different VLANs' and 'Multiple IPs in different Subnets per NIC on Host', but these solutions don't seem to be working for me (or I made a typo somewhere).

I don't understand why I can't get an IP assigned inside the VM from the bblan DHCP server on the router.
 
Last edited:

Stefan_R

Proxmox Staff Member
Staff member
Jun 4, 2019
566
94
28
Vienna
Can your PVE host acquire an IP with DHCP (e.g. dhclient enx00e04; ip a)? Can your VM acquire a DHCP address if you set it to vmbr0?

To debug, it might help to run 'tcpdump' on your PVE host and trace the DHCP packets. (see here for example)
 

NetgearNinja

New Member
Jun 17, 2020
4
0
1
20
Hi,

Thank you for your reply. I've run the command and my PVE host gets an IP address on the bblan subnet (I got the IP address 192.168.48.126/24, which is in the configured bblan DHCP server range, so it's nice to see that that is at least working correctly). When I set the networking device on a VM to vmbr0 it also gets an IP address from the DHCP server, but on the aalan, which is the wrong subnet.

Something that has been bothering me is that when I connect a monitor to my Intel NUC, I sometimes get the message:
Code:
[ 1X70.433513] usb 2-1: device not accepting address 106, error -71
This has only been showing up since I connected the USB 3.0 Gigabit Ethernet Adapter. I have read that it could have something to do with it broken hardware or it being usb 3.0 and running the dmesg command got me this output:
Code:
[ 1X67.946207] IPv6: ADDRCONF(NETDEV_CHANGE): enx00e04: link becomes ready
[ 1X67.946400] r8152 2-1:1.0 enx00e04: carrier on
[ 1X69.697481] usb 2-1: USB disconnect, device number 105
[ 1X69.697643] xhci_hcd 0000:00:14.0: WARN Set TR Deq Ptr cmd failed due to incorrect slot or ep state.
[ 1X70.017040] usb 2-1: Device not responding to setup address.
[ 1X70.433513] usb 2-1: device not accepting address 106, error -71
[ 1X70.629433] usb 2-1: new SuperSpeed Gen 1 USB device number 107 using xhci_hcd
[ 1X70.650073] usb 2-1: New USB device found, idVendor=XXX, idProduct=XXX, bcdDevice=XXX
[ 1X70.650074] usb 2-1: New USB device strings: Mfr=X, Product=X, SerialNumber=X
[ 1X70.650075] usb 2-1: Product: USB 10/100/1000 LAN
[ 1X70.650076] usb 2-1: Manufacturer: Realtek
[ 1X70.650076] usb 2-1: SerialNumber: 000XXX
[ 1X70.781753] usb 2-1: reset SuperSpeed Gen 1 USB device number 107 using xhci_hcd
The three lines which I don't like and probably contain more information about the problem are:
Code:
[ 1X69.697643] xhci_hcd 0000:00:14.0: WARN Set TR Deq Ptr cmd failed due to incorrect slot or ep state.
[ 1X70.017040] usb 2-1: Device not responding to setup address.
[ 1X70.433513] usb 2-1: device not accepting address 106, error -71
I don't think the adapter's hardware is broken as I just got an IP address in the right subnet, but I still don't know why it doesn't work inside the VMs. I've also switched the adapter to two other usb ports and everytime after a while I get the same error message, but slightly varying as to the usb port configuration. Do you think the main problem lies in these error messages or could it have something to do with the DHCP packets and I should continue debugging using tcpdump? Also, does my /etc/network/interfaces file look correct to you?
 

Stefan_R

Proxmox Staff Member
Staff member
Jun 4, 2019
566
94
28
Vienna
I don't think the adapter's hardware is broken as I just got an IP address in the right subnet, but I still don't know why it doesn't work inside the VMs. I've also switched the adapter to two other usb ports and everytime after a while I get the same error message, but slightly varying as to the usb port configuration. Do you think the main problem lies in these error messages or could it have something to do with the DHCP packets and I should continue debugging using tcpdump? Also, does my /etc/network/interfaces file look correct to you?
Those lines are definitely worrying and might cause network dropouts, but I don't see how they would break DHCP specifically for VMs. I would try to debug further. Do you maybe have the PVE firewall enabled?

Also, your interfaces file looks fine on first glance, I of course don't know if the configuration correctly matches your setup.
 

NetgearNinja

New Member
Jun 17, 2020
4
0
1
20
My PVE firewall is disabled and the options page looks like this:
pve-firewall-setup.PNG

I've run tcpdump -i enx00e04 and this is the output I received:
Code:
root@pve:~# tcpdump -i enx00e04
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enx00e04, link-type EN10MB (Ethernet), capture size 262144 bytes
Then I started up a VM with vmbr1 as it's network interface and I get this output in the same tcpdump window:
Code:
16:32:57.722637 IP6 :: > ff02::XX: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
16:32:57.722698 IP6 :: > ff02::X:ffXX:XX5: ICMP6, neighbor solicitation, who has feXX::Xe0:XXff:feXX:XX5, length 32
16:32:57.866663 IP6 :: > ff02::XX: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
16:32:58.746681 IP6 feXX::Xe0:XXff:feXX:XX5 > ff02::XX: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
16:32:58.746720 IP6 feXX::Xe0:XXff:feXX:XX5 > ip6-allrouters: ICMP6, router solicitation, length 16
16:32:59.194548 IP6 feXX::Xe0:XXff:feXX:XX5 > ff02::XX: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
tcpdump: pcap_loop: The interface went down
6 packets captured
6 packets received by filter
0 packets dropped by kernel
root@pve:~#


EDIT after 5 minutes:
I now have tried it multiple times, but every time my VM seems to try to connect to the network, the interface just shuts down. After a shutdown when I try the tcpdump -i enx00e04 command I get this output:
Code:
root@pve~# tcpdump -i enx00e04
tcpdump: enx00e04: No such device exists
(SIOCGIFHWADDR: No such device)
root@pve:~#
The device only seems to be useful again after reboot. It seems that the interface is up again when trying the command after a min, but then it shuts down again when trying to boot a VM with vmbr1 as it's network device.
 
Last edited:

Stefan_R

Proxmox Staff Member
Staff member
Jun 4, 2019
566
94
28
Vienna
Hm, the device randomly shutting down makes me think it is faulty hardware at play. I don't understand why it only happens when the VM accesses it though... maybe it doesn't like multiple MAC addresses moving over it?

I'd get a different adapter and try again, or maybe if your feeling lucky you can try using your on-board adapter for VMs and your external one for your bridge.
 

NetgearNinja

New Member
Jun 17, 2020
4
0
1
20
Hm, the device randomly shutting down makes me think it is faulty hardware at play. I don't understand why it only happens when the VM accesses it though... maybe it doesn't like multiple MAC addresses moving over it?

I'd get a different adapter and try again, or maybe if your feeling lucky you can try using your on-board adapter for VMs and your external one for your bridge.
Hi, I really want to thank you for your time and help while debugging it. You were right, after all it was faulty hardware. I just got my new ethernet usb adapter and it worked right away after I plugged it in. For those who want to know, it's a ThinkPad USB 3.0 Ethernet adapter.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!