[SOLVED] Problem with USB Network Card - Icy Box - IB-LAN300-C3

[BenjaMwaye]

Hi everybody,

Sorry for my bad English (ask for more information if needed or if I'm not understandable).
And sorry if answer is present on this forum, but I don't find it.

I've got à Proxmox server with an OpnSense VM to control my web access by using my Freebox in bridge mode.
It works correctly even when I change my internet provider offer. Now I'm on the "Free fiber network" (french provider).
This provider gives a box with 3 Ethernet ports (1x 1000 Mb/s POE, 1x 1000 Mb/s and 1x 2500 MB/s).
My web connection could be up to 5Gb/s on all the ports (2.5 GB/s + 1 GB/s + 1 GB/s + 500 MB/s on Wi-Fi).

So to have the best bandwidth on my web access, I buy a USB-Ethernet-2,5GB/s card (Icy Box IB-LAN300-C) and plug it on my ProxMox.
It seems to be well recognize and I've defined a bridge with this card.
This bridge is defined to be the WAN card of the OpnSense.

The problem is that I can't have access to the web when using this usb-network-card.
DHCP (v4 and v6) don't give an IP address (even with IP defined manually, the problem is here).

If I plug the cable on the old Card, IP is sent by DHCP without problem.
But with this card, the connection doesn't work.
If I test to have an IP address by DHCP with a Debian live VM which use the bridge 2.5 GB/s, the Network-Manager don't want to connect (bridge or router mode on the Freebox).
But on my own Debian 11 on another computer (not a VM), the card work without problem to have an IP address and work great.

It looks like a parameter to pass to the network card driver to accept the DHCP or the speed negotiation.
Or a problem of global compatibility of this card with Proxmox and-or OpnSense.

I'm not very good to diagnose ... and solve the problem, so I ask your help to have my USB-Network-Card-2.5GB/s work between my Freebox and my OpnSense which is a VM in Proxmox.
Have you any suggestion and advises to help me to solve this problem?

Thanks a lot in advance for your help.
Best regards,

Benjam

 

[Stefan_R]

Are you using USB passthrough to attach the adapter to a VM? If so, that will not work, as the bandwidth of that is very limited, certainly not at 2.5 Gb/s speeds. Does the adapter work on the PVE host? If so, I'd recommend not using passthrough at all, assigning a static IP to a new bridge interface (connected to the adapter) on the host, and just using a regular virtio NIC for the OpnSense VM. Alternatively, you could try PCIe passthrough of your USB 3.0 controller, that will give you the full available speed in the VM, but will pass through all of the ports on that device.

 

[DerekG]

Hi Benjam,

I have a similar setup/issue with my Fibre connection, this might help to point you in the right direction.

I have a 1GB Fibre connection from a UK community provider. My firewall is a pfsense VM running on Proxmox 7.0

In my case, I experience 2 issues.

1) My ISP has a 'sticky' MAC address issue, whereby if the network card connected to the modem is changed, I can never get a DHCP IP address until a call to support team and request that the 'flush their MAC address table' in the data centre. Sometimes I can overcome this with MAC address spoofing, but it's really a hit and miss affair.

2) I've tested pfsense with 3 different USB3.0 network cards, both Realtek & AXIS chipsets, both will get a DHCP address from my local DHCP server no problem, but neither get an IP address from the Fibre model. The onboard Intel or Realtek cards do get the IP address from fibre DHCP, subject to point 1. above.

My final solution was to set the USB NIC as my LAN port with a static IP address and use the onboard NIC for the Fibre connection. That works perfectly.


On another point, you will not get 5Gb/s from the Fibre connection, you have a single Fibre cable back to the data centre and a single IP address. The Ops team in the data centre WILL rate limit the throughput to whatever they specify, I have 300Mb/s and that has nothing to do with the speed or quantity of my network ports.

I hope that helps

DerekG.

 

[BenjaMwaye]

Hi and thanks for your help

@Stefan_R :
I don't attach the USB network adapter direct to the vm. I try to use it directly as a proxmox bridge attach to my opnsense vm. it appears in the proxmox interface but I don't know if it's correctly detected.

@DerekG :
My problem looks a little similar of your second point. I know that I will not have 2.5 GB/s on the fiber for now, but the Free connection is capable of that sometimes.


lsusb :
Bus 002 Device 003: ID 0bda:8156 Realtek Semiconductor Corp.

dmesg
[56147.349377] usb 3-1: USB disconnect, device number 2
[56147.349479] cdc_ncm 3-1:2.0 enx002427881126: unregister 'cdc_ncm' usb-0000:00:14.0-1, CDC NCM
[56147.349578] vmbr11: port 1(enx002427881126) entered disabled state
[56147.349856] device enx002427881126 left promiscuous mode
[56147.349880] vmbr11: port 1(enx002427881126) entered disabled state
[56192.354374] usb 2-1: new high-speed USB device number 3 using xhci_hcd
[56192.503928] usb 2-1: New USB device found, idVendor=0bda, idProduct=8156, bcdDevice=31.00
[56192.503965] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=6
[56192.503991] usb 2-1: Product: USB 10/100/1G/2.5G LAN
[56192.504009] usb 2-1: Manufacturer: Realtek
[56192.504025] usb 2-1: SerialNumber: 001000001
[56192.527739] cdc_ncm 2-1:2.0: MAC-Address: 00:24:27:88:11:26
[56192.527770] cdc_ncm 2-1:2.0: setting rx_max = 16384
[56192.527873] cdc_ncm 2-1:2.0: setting tx_max = 16384
[56192.528284] cdc_ncm 2-1:2.0 eth0: register 'cdc_ncm' at usb-0000:00:14.0-1, CDC NCM, 00:24:27:88:11:26
[56192.532663] cdc_ncm 2-1:2.0 enx002427881126: renamed from eth0

Chipset looks to be :
RTL8156
On the realtek website the avaible driver for linux looks to compatible only for kernel up to 5.6. (https://www.realtek.com/en/componen...0-100-1000m-gigabit-ethernet-usb-3-0-software)

As it works great on my Debian laptop, I think I should try to find a driver to have the card working correctly.
I'll continue to search if I find a solution but I think it couldn't be solved until pve-kernel upgrade to 5.6.

I'll inform you if I find a solution.
And thanks again for your help.

Best regards,
Benjam

 

[BenjaMwaye]

Hi again everybody,

I do some test and the problem is when using bridge on Proxmox hypervisor :

• Directly on the Debian Proxmox : DHCP gives an IP address (WAN provider DHCP or LAN DHCP) and ping work. So it looks to work. I just can't have detailed information about the device :

IP address (ip add sh) :​

enx002427881126: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr11 state UP group default qlen 1000​

inet 192.168.100.227/24 brd 192.168.100.255 scope global dynamic enx002427881126​

​

Detail of the connection (ethtool) :​

Settings for enx002427881126:​

Current message level: 0x00000007 (7)​

drv probe link​

Link detected: yes​

​

Speed (cat: /sys/class/net/enx002427881126/speed) :​

cat: /sys/class/net/enx002427881126/speed: Argument invalide​

• If I attach the card to a bridge and attach the bridge to a VM, it doesn't work anymore :
  
  • The DHCP request doesn't give an IP and turn indefinitely
  • Problem is here on the OPNsense VM and with a VM with Debian11 freshly installed and up to date.
  • I'll try with the other models of net card that exist in Proxmox.
  • In all the case problem is the same : DHCP request turn indefinitely when "bridging" this network card.

So I don't really understand the problem...

• Hypervisor or bridge compatibility ?
• Bug ?
• Params to pass to the bridge or card ?

...and how to solve it.
If you have idea to solve this problem, I'm very interested

Best regards,
Benjam

 

[Stefan_R]

Can your PVE host request an IP address when the device is connected to a bridge? If not, your device might just be incompatible with bridging. Sometimes there are weird MAC address filters built into the hardware or firmware/driver...

 

[BenjaMwaye]

Hi and thanks for your answer,

I'm not sure if what I just test is what you ask me to do.

Configuration (to be sure that good configuration to test what you ask me) :

Is it good ?

My last test :

• With this config, DHCP request via vmbr11 directly in the Proxmox host works great :

sudo dhclient -v vmbr11

Internet Systems Consortium DHCP Client 4.4.1
Copyright 2004-2018 Internet Systems Consortium.
All rights reserved.]For info, please visit https://www.isc.org/software/dhcp/
Listening on LPF/vmbr11/00:24:27:88:11:26
Sending on   LPF/vmbr11/00:24:27:88:11:26
Sending on   Socket/fallback
DHCPREQUEST for 192.168.100.227 on vmbr11 to 255.255.255.255 port 67
DHCPACK of 192.168.100.227 from 192.168.100.99
bound to 192.168.100.227 -- renewal in 2728 seconds.

ip add sh vmbr11

9: vmbr11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 192.168.100.227/24 brd 192.168.100.255 scope global dynamic vmbr11
valid_lft 5762sec preferred_lft 5762sec
inet6 fe80::224:27ff:fe88:1126/64 scope link
valid_lft forever preferred_lft forever

• But in VMs (both Debian 11 XFCE and OPNsense) DHCP request don't give IP :

sudo dhclient -v enp0s18

...No DHCPOFFERS received
No working release in persistent database - sleeping

So, if my tests are good, it works in bridge on host but not with bridge attached to a VM.
It's weird, I don't understand.

Best regards,
Benjam

 

[Stefan_R]

So, if my tests are good, it works in bridge on host but not with bridge attached to a VM.
It's weird, I don't understand.

As I said, might be MAC related. VMs attached to a bridge do not have their MAC "translated", the bridge sends out the packets to the physical NIC with the source MAC of the VM. So potentially some layer in your stack filters that.

My personal favourite trick to debug anything network related is to run 'tcpdump' on any and all intermediate nodes (VM, PVE, destination, any routers inbetween, etc...) and see how far the packet gets. Try tcpdump -i any udp port 67 and port 68 -vv for DHCP.

 

[BenjaMwaye]

Thanks, I'll try to diagnose with TCPDump.
I'll post the result here to help me to understand them.

 

[BenjaMwaye]

Hi,

I do some tests and trace with tcpdump. I don't understand all the tcpdump output.
But i think DHCP request get out of the VM but server (OPNsense) answer don't get in the VM ?

Here's the output of my tests :

DHCP request on vmbr99 (work great - usb-lan-card RTL8153)
=> Guest VM
sudo dhclient enp0s18 -v

Spoiler: dhclient output

Internet Systems Consortium DHCP Client 4.4.1
Copyright 2004-2018 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
Listening on LPF/enp0s18/fa:9d:f4:4c:98:dc
Sending on LPF/enp0s18/fa:9d:f4:4c:98:dc
Sending on Socket/fallback
DHCPREQUEST for 192.168.100.229 on enp0s18 to 255.255.255.255 port 67
DHCPACK of 192.168.100.229 from 192.168.100.99
RTNETLINK answers: File exists
bound to 192.168.100.229 -- renewal in 2736 seconds.

tcpdump

Spoiler: tcpdump output

16:21:37.346704 ens18 In IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
opnsense.perso.lan.bootps > 192.168.100.229.bootpc: [udp sum ok] BOOTP/DHCP, Reply, length 300, xid 0x391f6262, Flags [none] (0x0000)
Your-IP 192.168.100.229
Client-Ethernet-Address fa:9d:f4:4c:98:dc (oui Unknown)
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: ACK
Server-ID (54), length 4: opnsense.perso.lan
Lease-Time (51), length 4: 6130
Subnet-Mask (1), length 4: 255.255.255.0
Default-Gateway (3), length 4: opnsense.perso.lan
Domain-Name (15), length 15: "perso.lan"
Domain-Name-Server (6), length 4: opnsense.perso.lan
NTP (42), length 4: opnsense.perso.lan

=> Proxmox host
tcpdump
16:26:16.795194 IP (tos 0xc0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 319)

Spoiler: tcpdump output

0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from c6:01:e5:b4:10:ee (oui Unknown), length 291, xid 0x52562e2b, secs 1, Flags [none] (0x0000)
Client-Ethernet-Address c6:01:e5:b4:10:ee (oui Unknown)
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53, length 1: Request
Client-ID Option 61, length 7: ether c6:01:e5:b4:10:ee
Parameter-Request Option 55, length 17:
Subnet-Mask, Time-Zone, Domain-Name-Server, Hostname
Domain-Name, MTU, BR, Classless-Static-Route
Default-Gateway, Static-Route, YD, YS
NTP, Option 119, Classless-Static-Route-Microsoft, Option 252
RP
MSZ Option 57, length 2: 576
Requested-IP Option 50, length 4: debtest.perso.lan
Hostname Option 12, length 7: "debtest"

=> OPNsense
DHCP log

Spoiler: opnsense log

2021-09-20T16:26:16	dhcpd[45398]	DHCPACK on 192.168.100.229 to fa:9d:f4:4c:98:dc (debtest) via vtnet1
2021-09-20T16:26:16	dhcpd[45398]	DHCPREQUEST for 192.168.100.229 from fa:9d:f4:4c:98:dc (debtest) via vtnet1
2021-09-20T16:26:16	dhcpd[45398]	reuse_lease: lease age 1349 (secs) under 25% threshold, reply with unaltered, existing lease for 192.168.100.229

DHCP request on vmbr11 (don’t work - usb-lan-card RTL8156)
=> Guest VM
sudo dhclient enp0s18 -v

Spoiler: dhclient output

Internet Systems Consortium DHCP Client 4.4.1
Copyright 2004-2018 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
Listening on LPF/enp0s18/fa:9d:f4:4c:98:dc
Sending on LPF/enp0s18/fa:9d:f4:4c:98:dc
Sending on Socket/fallback
DHCPREQUEST for 192.168.100.229 on enp0s18 to 255.255.255.255 port 67
DHCPREQUEST for 192.168.100.229 on enp0s18 to 255.255.255.255 port 67
DHCPDISCOVER on enp0s18 to 255.255.255.255 port 67 interval 7
DHCPDISCOVER on enp0s18 to 255.255.255.255 port 67 interval 7
DHCPDISCOVER on enp0s18 to 255.255.255.255 port 67 interval 12
DHCPDISCOVER on enp0s18 to 255.255.255.255 port 67 interval 11
DHCPDISCOVER on enp0s18 to 255.255.255.255 port 67 interval 13
DHCPDISCOVER on enp0s18 to 255.255.255.255 port 67 interval 11
No DHCPOFFERS received.
Trying recorded lease 192.168.100.229
PING 192.168.100.99 (192.168.100.99) 56(84) bytes of data.
--- 192.168.100.99 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms
No working leases in persistent database - sleeping.

tcpdump

Spoiler: tcpdump output

13:01:14.112671 ens18 Out IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from fa:9d:f4:4c:98:dc (oui Unknown), length 300, xid 0x8c549510, secs 50, Flags [none] (0x0000)
Client-Ethernet-Address fa:9d:f4:4c:98:dc (oui Unknown)
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Discover
Hostname (12), length 7: "debtest"
Parameter-Request (55), length 13:
Subnet-Mask (1), BR (28), Time-Zone (2), Default-Gateway (3)
Domain-Name (15), Domain-Name-Server (6), Unknown (119), Hostname (12)
Netbios-Name-Server (44), Netbios-Scope (47), MTU (26), Classless-Static-Route (121)
NTP (42)

=> Proxmox host
tcpdump

Spoiler: tcpdump output

16:16:19.179720 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
opnsense.perso.lan.bootps > debtest.perso.lan.bootpc: [udp sum ok] BOOTP/DHCP, Reply, length 300, xid 0xfc29977d, secs 59, Flags [none] (0x0000)
Your-IP debtest.perso.lan
Client-Ethernet-Address fa:9d:f4:4c:98:dc (oui Unknown)
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53, length 1: Offer
Server-ID Option 54, length 4: opnsense.perso.lan
Lease-Time Option 51, length 4: 6448
Subnet-Mask Option 1, length 4: 255.255.255.0
Default-Gateway Option 3, length 4: opnsense.perso.lan
Domain-Name Option 15, length 15: "perso.lan"
Domain-Name-Server Option 6, length 4: opnsense.perso.lan
NTP Option 42, length 4: opnsense.perso.lan

=> OPNsense
DHCP log

Spoiler: opnsense log

2021-09-20T16:18:00	dhcpd[45398]	DHCPOFFER on 192.168.100.229 to fa:9d:f4:4c:98:dc (debtest) via vtnet1
2021-09-20T16:18:00	dhcpd[45398]	DHCPDISCOVER from fa:9d:f4:4c:98:dc (debtest) via vtnet1
2021-09-20T16:18:00	dhcpd[45398]	reuse_lease: lease age 853 (secs) under 25% threshold, reply with unaltered, existing lease for 192.168.100.229
2021-09-20T16:17:54	dhcpd[45398]	DHCPOFFER on 192.168.100.235 to fa:9d:f4:4c:98:dc (debtest) via vtnet1
2021-09-20T16:17:53	dhcpd[45398]	DHCPDISCOVER from fa:9d:f4:4c:98:dc (debtest) via vtnet1
2021-09-20T16:17:48	dhcpd[45398]	DHCPOFFER on 192.168.100.234 to fa:9d:f4:4c:98:dc (debtest) via vtnet1
2021-09-20T16:17:47	dhcpd[45398]	DHCPDISCOVER from fa:9d:f4:4c:98:dc (debtest) via vtnet1
2021-09-20T16:17:29	dhcpd[45398]	DHCPOFFER on 192.168.100.233 to fa:9d:f4:4c:98:dc (debtest) via vtnet1
2021-09-20T16:17:28	dhcpd[45398]	DHCPDISCOVER from fa:9d:f4:4c:98:dc (debtest) via vtnet1
2021-09-20T16:17:15	dhcpd[45398]	DHCPOFFER on 192.168.100.228 to fa:9d:f4:4c:98:dc (debtest) via vtnet1
2021-09-20T16:17:14	dhcpd[45398]	DHCPDISCOVER from fa:9d:f4:4c:98:dc via vtnet1
2021-09-20T16:16:58	dhcpd[45398]	DHCPOFFER on 192.168.100.232 to fa:9d:f4:4c:98:dc (debtest) via vtnet1
2021-09-20T16:16:57	dhcpd[45398]	DHCPDISCOVER from fa:9d:f4:4c:98:dc via vtnet1
2021-09-20T16:16:51	dhcpd[45398]	DHCPOFFER on 192.168.100.231 to fa:9d:f4:4c:98:dc (debtest) via vtnet1
2021-09-20T16:16:50	dhcpd[45398]	DHCPDISCOVER from fa:9d:f4:4c:98:dc via vtnet1
2021-09-20T16:16:49	dhcpd[45398]	DHCPOFFER on 192.168.100.230 to fa:9d:f4:4c:98:dc (debtest) via vtnet1
2021-09-20T16:16:48	dhcpd[45398]	DHCPDISCOVER from fa:9d:f4:4c:98:dc via vtnet1
2021-09-20T16:16:19	dhcpd[45398]	DHCPOFFER on 192.168.100.229 to fa:9d:f4:4c:98:dc (debtest) via vtnet1
2021-09-20T16:16:19	dhcpd[45398]	DHCPDISCOVER from fa:9d:f4:4c:98:dc (debtest) via vtnet1
2021-09-20T16:16:19	dhcpd[45398]	reuse_lease: lease age 752 (secs) under 25% threshold, reply with unaltered, existing lease for 192.168.100.229
2021-09-20T16:16:10	dhcpd[45398]	DHCPOFFER on 192.168.100.229 to fa:9d:f4:4c:98:dc (debtest) via vtnet1
2021-09-20T16:16:10	dhcpd[45398]	DHCPDISCOVER from fa:9d:f4:4c:98:dc (debtest) via vtnet1
2021-09-20T16:16:10	dhcpd[45398]	reuse_lease: lease age 743 (secs) under 25% threshold, reply with unaltered, existing lease for 192.168.100.229

I don't understand really the logs.
Have you any idea about the problem and how to solve?

Best regards,
Benjam

 

[Stefan_R]

Hard to tell from the excerpts you posted, but I'd say this looks like your USB NIC filters incoming packets to match it's own MAC address. This is not really feasible for VM usage, where the VM has it's own MAC on the linux bridge interface. Either try a different NIC, or look into NATing the VMs on your PVE host.

 

[BenjaMwaye]

Thanks for your reply.

I'm not sûre to understand all what you said about MAC bloking in Proxmox Linux bridge.
I'll continue o search in the forum how to use this card or how to use it via NATing (as you say) in OPNsense (with provider ip).

Best regards,
Benjam

 

[BenjaMwaye]

Hi everybody,

It's work.
I upgrade Proxmox to the version 7 and it works without any problem.

I think it's the kernel which is 5.11, as I read somewhere that the Realtek 8156 fully compatible Linux since 4.9, but perhaps I'm wrong when I say that.
(or it could be the change in bridge support between Proxmox 6 and 7)

So it works great now.
Thanks for all.

Best regards,
Benjam