USB Ethernet 2.5Gbit problem

X

xcj

Member
Jul 6, 2021
34
7
13
47
Hi there,

Hi have two proxmod nodes with gigabit ethernet where everything runs, management, VM's services , replication between nodes and backups, these are on the network 192.168.1.0/24, have a GW and DNS defined, and were working as intended.

I received two USB 2.5Gbit adapters that i plan to use to conect the two nodes B2B (for future use in replication). I Enabled the NIC's on both nodes and configured an interconnect network 10.1.1.0/30, node 1 is 10.1.1.2, node 2 is 10.1.1.3, the network config does not have a GW defined since it's a B2B config just for these two nodes.

Screenshot 2021-07-13 at 18.19.14.pngScreenshot 2021-07-13 at 18.19.03.png

Both of them apear to be UP (ip addr output):

3: enx00e0a0100325: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:e0:a0:10:03:25 brd ff:ff:ff:ff:ff:ff
inet 10.1.1.2/30 scope global enx00e0a0100325
valid_lft forever preferred_lft forever
inet6 fe80::2e0:a0ff:fe10:325/64 scope link
valid_lft forever preferred_lft forever

3: enx00e0a01000d6: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:e0:a0:10:00:d6 brd ff:ff:ff:ff:ff:ff
inet 10.1.1.3/30 scope global enx00e0a01000d6
valid_lft forever preferred_lft forever
inet6 fe80::2e0:a0ff:fe10:d6/64 scope link
valid_lft forever preferred_lft forever

But i cant ping node2 from node1 and vice versa, but they give different reasons:

root@node1:~# ping 10.1.1.3
Do you want to ping broadcast? Then -b. If not, check your local firewall rules.

root@node2:~# ping 10.1.1.2
PING 10.1.1.2 (10.1.1.2) 56(84) bytes of data.
From 10.1.1.3 icmp_seq=1 Destination Host Unreachable
From 10.1.1.3 icmp_seq=2 Destination Host Unreachable
From 10.1.1.3 icmp_seq=3 Destination Host Unreachable

Regarding firewall rules i haven't set any on the hosts.

And both nodes have the syslog flooded with this message:


Jul 13 18:07:31 node1 kernel: cdc_ncm 2-5:2.0 enx00e0a0100325: 2500 mbit/s downlink 2500 mbit/s uplink
Jul 13 18:07:32 node1 kernel: cdc_ncm 2-5:2.0 enx00e0a0100325: 2500 mbit/s downlink 2500 mbit/s uplink
Jul 13 18:07:32 node1 kernel: cdc_ncm 2-5:2.0 enx00e0a0100325: 2500 mbit/s downlink 2500 mbit/s uplink
Jul 13 18:07:33 node1 kernel: cdc_ncm 2-5:2.0 enx00e0a0100325: 2500 mbit/s downlink 2500 mbit/s uplink
Jul 13 18:07:33 node1 kernel: cdc_ncm 2-5:2.0 enx00e0a0100325: 2500 mbit/s downlink 2500 mbit/s uplink
Jul 13 18:07:34 node1 kernel: cdc_ncm 2-5:2.0 enx00e0a0100325: 2500 mbit/s downlink 2500 mbit/s uplink
Jul 13 18:07:34 node1 kernel: cdc_ncm 2-5:2.0 enx00e0a0100325: 2500 mbit/s downlink 2500 mbit/s uplink
Jul 13 18:07:35 node1 kernel: cdc_ncm 2-5:2.0 enx00e0a0100325: 2500 mbit/s downlink 2500 mbit/s uplink
Jul 13 18:07:35 node1 kernel: cdc_ncm 2-5:2.0 enx00e0a0100325: 2500 mbit/s downlink 2500 mbit/s uplink
Jul 13 18:07:36 node1 kernel: cdc_ncm 2-5:2.0 enx00e0a0100325: 2500 mbit/s downlink 2500 mbit/s uplink
Jul 13 18:07:36 node1 kernel: cdc_ncm 2-5:2.0 enx00e0a0100325: 2500 mbit/s downlink 2500 mbit/s uplink
Jul 13 18:07:37 node1 kernel: cdc_ncm 2-5:2.0 enx00e0a0100325: 2500 mbit/s downlink 2500 mbit/s uplink


I am pretty sure that i am missing a step, what did i do wrong? Help a rookie out! :)
 
Hi,

10.1.1.3 should be the broadcast address of 10.1.1.0/30. So I'd try to use
  • 10.1.1.1 and 10.1.1.2 or
  • something simpler/bigger like 10.10.10.0/24

See some IPv4 subnet calculator
 
Hi, i'm kind of embarressed about that (using the broadcast IP as a usable IP) , it was one thing that i didn't give any second thoughts :oops:.

Correcting the IP's, the hosts can now ping each other, but my syslog keeps getting flooded with the message:

"Jul 13 18:07:31 node1 kernel: cdc_ncm 2-5:2.0 enx00e0a0100325: 2500 mbit/s downlink 2500 mbit/s uplink"

I have that message using the /30 network or the /24 network as suggested.

Is this because i didn't set up a gateway, this connection is not meant to have a gateway, does anyone know why proxmox floods the system log with a message that it found a new NIC?

Thanks!
 
  • Like
Reactions: Dominic
I'm guessing that the NIC is losing connection based on the USB cable... I had similar issues when using USB-to-5GbE adapters. Try the following:
  • Are you in a USB-3.2 port?
  • Can you swap the USB with another one?
  • Check dmesg for entries from the USB NIC. Make sure you don't see it repeating.
Code: 
root@sin-dc-196m-hv-cpu-001:/var/log/ceph# dmesg | grep usb
[    0.613559] usbcore: registered new interface driver usbfs
[    0.613559] usbcore: registered new interface driver hub
[    0.613559] usbcore: registered new device driver usb
[    2.284145] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.11
[    2.284148] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.284150] usb usb1: Product: xHCI Host Controller
[    2.284152] usb usb1: Manufacturer: Linux 5.11.22-1-pve xhci-hcd
[    2.284154] usb usb1: SerialNumber: 0000:00:14.0
[    2.299435] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 5.11
[    2.299439] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.299441] usb usb2: Product: xHCI Host Controller
[    2.299443] usb usb2: Manufacturer: Linux 5.11.22-1-pve xhci-hcd
[    2.299445] usb usb2: SerialNumber: 0000:00:14.0
[    2.305344] usb: port power management may be unreliable
[    2.645508] usb 2-2: new SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[    2.674906] usb 2-2: New USB device found, idVendor=2eca, idProduct=c101, bcdDevice= 1.01
[    2.674913] usb 2-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    2.674916] usb 2-2: Product: Pacific
[    2.674918] usb 2-2: Manufacturer: Aquantia
[    2.674920] usb 2-2: SerialNumber: 00000001
[    5.085916] usbcore: registered new interface driver cdc_ether
[    5.090048] usbcore: registered new interface driver aqc111
[    5.102580] aqc111 2-2:1.0 eth0: register 'aqc111' at usb-0000:00:14.0-2, Aquantia AQtion USB to 5GbE Controller, mac:address:removed
 
Last edited:
Hi there, these are USB 3.0, i have three available, i tried all, sometimes the adapter lowers the speed all the way down to 10Mbt.

Here is the DSMEG from node1, please note that the USB being detected severall times is precicely because i was trying all ports, and the times it detected a USB1 should be when the Ethernet speed lowered to 10Mbit.

Below the the DSMEG i pasted the syslog flooding continuing, please note that while this happens both nodes can ping each other trough this connection.

root@node1:/mnt/pve/Backup/dump# dmesg | grep usb [189491.106797] usb 2-5: USB disconnect, device number 4 [189491.106845] cdc_ncm 2-5:2.0 enx00e0a0100325: unregister 'cdc_ncm' usb-0000:00:14.0-5, CDC NCM [189503.854913] usb 1-6: new high-speed USB device number 4 using xhci_hcd [189556.027110] usb 1-6: new high-speed USB device number 6 using xhci_hcd [189556.176409] usb 1-6: New USB device found, idVendor=0bda, idProduct=8156, bcdDevice=31.00 [189556.176411] usb 1-6: New USB device strings: Mfr=1, Product=2, SerialNumber=7 [189556.176412] usb 1-6: Product: USB 10/100/1G/2.5G LAN [189556.176413] usb 1-6: Manufacturer: Realtek [189556.176413] usb 1-6: SerialNumber: 001300E0A0100325 [189556.196530] cdc_ncm 1-6:2.0 usb0: register 'cdc_ncm' at usb-0000:00:14.0-6, CDC NCM, 00:e0:a0:10:03:25 [189556.251691] cdc_ncm 1-6:2.0 enx00e0a0100325: renamed from usb0 [189559.380650] usb 1-6: USB disconnect, device number 6 [189559.380683] cdc_ncm 1-6:2.0 enx00e0a0100325: unregister 'cdc_ncm' usb-0000:00:14.0-6, CDC NCM [189566.947185] usb 1-6: new high-speed USB device number 8 using xhci_hcd [189567.096512] usb 1-6: New USB device found, idVendor=0bda, idProduct=8156, bcdDevice=31.00 [189567.096514] usb 1-6: New USB device strings: Mfr=1, Product=2, SerialNumber=7 [189567.096515] usb 1-6: Product: USB 10/100/1G/2.5G LAN [189567.096516] usb 1-6: Manufacturer: Realtek [189567.096516] usb 1-6: SerialNumber: 001300E0A0100325 [189567.117099] cdc_ncm 1-6:2.0 usb0: register 'cdc_ncm' at usb-0000:00:14.0-6, CDC NCM, 00:e0:a0:10:03:25 [189567.124407] cdc_ncm 1-6:2.0 enx00e0a0100325: renamed from usb0 [189567.172430] usb 1-6: USB disconnect, device number 8 [189567.172474] cdc_ncm 1-6:2.0 enx00e0a0100325: unregister 'cdc_ncm' usb-0000:00:14.0-6, CDC NCM [189658.527796] usb 2-4: new SuperSpeed Gen 1 USB device number 5 using xhci_hcd [189658.552470] usb 2-4: New USB device found, idVendor=0bda, idProduct=8156, bcdDevice=31.00 [189658.552471] usb 2-4: New USB device strings: Mfr=1, Product=2, SerialNumber=7 [189658.552472] usb 2-4: Product: USB 10/100/1G/2.5G LAN [189658.552473] usb 2-4: Manufacturer: Realtek [189658.552473] usb 2-4: SerialNumber: 001300E0A0100325 [189658.572926] cdc_ncm 2-4:2.0 usb0: register 'cdc_ncm' at usb-0000:00:14.0-4, CDC NCM, 00:e0:a0:10:03:25 [189658.575462] cdc_ncm 2-4:2.0 enx00e0a0100325: renamed from usb0 [190116.522387] usb 2-4: USB disconnect, device number 5 [190116.522454] cdc_ncm 2-4:2.0 enx00e0a0100325: unregister 'cdc_ncm' usb-0000:00:14.0-4, CDC NCM [190140.514513] usb 2-4: new SuperSpeed Gen 1 USB device number 6 using xhci_hcd [190140.535115] usb 2-4: New USB device found, idVendor=0bda, idProduct=8156, bcdDevice=31.00 [190140.535116] usb 2-4: New USB device strings: Mfr=1, Product=2, SerialNumber=7 [190140.535117] usb 2-4: Product: USB 10/100/1G/2.5G LAN [190140.535118] usb 2-4: Manufacturer: Realtek [190140.535119] usb 2-4: SerialNumber: 001300E0A0100325 [190140.558553] cdc_ncm 2-4:2.0 usb0: register 'cdc_ncm' at usb-0000:00:14.0-4, CDC NCM, 00:e0:a0:10:03:25 [190140.561467] cdc_ncm 2-4:2.0 enx00e0a0100325: renamed from usb0 [190183.722786] usb 2-4: USB disconnect, device number 6 [190183.722817] cdc_ncm 2-4:2.0 enx00e0a0100325: unregister 'cdc_ncm' usb-0000:00:14.0-4, CDC NCM [190189.326743] usb 1-6: new high-speed USB device number 10 using xhci_hcd [190189.480856] usb 1-6: New USB device found, idVendor=0bda, idProduct=8156, bcdDevice=31.00 [190189.480858] usb 1-6: New USB device strings: Mfr=1, Product=2, SerialNumber=7 [190189.480858] usb 1-6: Product: USB 10/100/1G/2.5G LAN [190189.480859] usb 1-6: Manufacturer: Realtek [190189.480860] usb 1-6: SerialNumber: 001300E0A0100325 [190189.504472] cdc_ncm 1-6:2.0 usb0: register 'cdc_ncm' at usb-0000:00:14.0-6, CDC NCM, 00:e0:a0:10:03:25 [190189.507353] cdc_ncm 1-6:2.0 enx00e0a0100325: renamed from usb0 [190229.311552] usb 1-6: USB disconnect, device number 10 [190229.311584] cdc_ncm 1-6:2.0 enx00e0a0100325: unregister 'cdc_ncm' usb-0000:00:14.0-6, CDC NCM [190233.623083] usb 2-4: new SuperSpeed Gen 1 USB device number 7 using xhci_hcd [190233.647698] usb 2-4: New USB device found, idVendor=0bda, idProduct=8156, bcdDevice=31.00 [190233.647699] usb 2-4: New USB device strings: Mfr=1, Product=2, SerialNumber=7 [190233.647700] usb 2-4: Product: USB 10/100/1G/2.5G LAN [190233.647701] usb 2-4: Manufacturer: Realtek [190233.647702] usb 2-4: SerialNumber: 001300E0A0100325 [190233.669005] cdc_ncm 2-4:2.0 usb0: register 'cdc_ncm' at usb-0000:00:14.0-4, CDC NCM, 00:e0:a0:10:03:25 [190233.672131] cdc_ncm 2-4:2.0 enx00e0a0100325: renamed from usb0


Jul 15 23:00:22 node2 kernel: cdc_ncm 2-4:2.0 enx00e0a01000d6: 2500 mbit/s downlink 2500 mbit/s uplink Jul 15 23:00:22 node2 kernel: cdc_ncm 2-4:2.0 enx00e0a01000d6: 2500 mbit/s downlink 2500 mbit/s uplink Jul 15 23:00:23 node2 kernel: cdc_ncm 2-4:2.0 enx00e0a01000d6: 2500 mbit/s downlink 2500 mbit/s uplink Jul 15 23:00:23 node2 kernel: cdc_ncm 2-4:2.0 enx00e0a01000d6: 2500 mbit/s downlink 2500 mbit/s uplink Jul 15 23:00:24 node2 kernel: cdc_ncm 2-4:2.0 enx00e0a01000d6: 2500 mbit/s downlink 2500 mbit/s uplink Jul 15 23:00:24 node2 kernel: cdc_ncm 2-4:2.0 enx00e0a01000d6: 2500 mbit/s downlink 2500 mbit/s uplink Jul 15 23:00:25 node2 kernel: cdc_ncm 2-4:2.0 enx00e0a01000d6: 2500 mbit/s downlink 2500 mbit/s uplink Jul 15 23:00:25 node2 kernel: cdc_ncm 2-4:2.0 enx00e0a01000d6: 2500 mbit/s downlink 2500 mbit/s uplink Jul 15 23:00:26 node2 kernel: cdc_ncm 2-4:2.0 enx00e0a01000d6: 2500 mbit/s downlink 2500 mbit/s uplink Jul 15 23:00:26 node2 kernel: cdc_ncm 2-4:2.0 enx00e0a01000d6: 2500 mbit/s downlink 2500 mbit/s uplink Jul 15 23:00:27 node2 kernel: cdc_ncm 2-4:2.0 enx00e0a01000d6: 2500 mbit/s downlink 2500 mbit/s uplink Jul 15 23:00:27 node2 kernel: cdc_ncm 2-4:2.0 enx00e0a01000d6: 2500 mbit/s downlink 2500 mbit/s uplink Jul 15 23:00:28 node2 kernel: cdc_ncm 2-4:2.0 enx00e0a01000d6: 2500 mbit/s downlink 2500 mbit/s uplink


Thanks for the help!
 
In other words, DSMEG stops identifying new USB Ethernet adapters when i stop messing with them, but the syslog flooding continues, altough the connection works.
 
You will likely need to move the NIC into a USB3.1 or better, port as well.... I think you're likely to run into issues when pushing traffic.
 
Hi can someone provide any info on how to build the driver in proxmox? This a bit out of my depth.

I installed "build-essentials" trough APT.

I have the driver on a directory:

root@node2:~/r8152-2.15.0# ls -l
total 628
-rw-rw-r-- 1 1000 1000 3823 Mar 2 08:40 50-usb-realtek-net.rules
rw-rw-r-- 1 1000 1000 19761 Apr 20 04:37 compatibility.h
-rw-rw-r-- 1 1000 1000 1041 Sep 14 2020 Makefile
-rw-rw-r-- 1 1000 1000 609871 Apr 20 03:42 r8152.c
-rw-rw-r-- 1 1000 1000 962 Aug 4 2020 ReadMe.txt

From what i read i shoud just run "make" inside that directory, and "make install" after.

Make gives the error:

root@node2:~/r8152-2.15.0# make
make -C /lib/modules/5.4.124-1-pve/build M=/root/r8152-2.15.0 modules
make[1]: *** /lib/modules/5.4.124-1-pve/build: No such file or directory. Stop.
make: *** [Makefile:24: modules] Error 2

Thanks in advance!
 
Last edited:
@branto , Regarding USB 3.1, these two machines only have USB3.0.
USB3.0 has 5Gbit bandwidth, why do you think i'll run into issues?
 
Last edited:
I found in another post that i needed to install linux headers, i installed trough apt "pve-headers".

"make" and "make-install" went OK:

Code: 
root@node1:~/r8152-2.15.0# make
make -C /lib/modules/5.4.124-1-pve/build M=/root/r8152-2.15.0 modules
make[1]: Entering directory '/usr/src/linux-headers-5.4.124-1-pve'
  CC [M]  /root/r8152-2.15.0/r8152.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC [M]  /root/r8152-2.15.0/r8152.mod.o
  LD [M]  /root/r8152-2.15.0/r8152.ko
make[1]: Leaving directory '/usr/src/linux-headers-5.4.124-1-pve'
root@node1:~/r8152-2.15.0# make install
make -C /lib/modules/5.4.124-1-pve/build M=/root/r8152-2.15.0 INSTALL_MOD_DIR=kernel/drivers/net/usb modules_install
make[1]: Entering directory '/usr/src/linux-headers-5.4.124-1-pve'
  INSTALL /root/r8152-2.15.0/r8152.ko
  DEPMOD  5.4.124-1-pve

The readme said to run to additional commands "depmod -a" and "update-initramfs -u", the first didn't give any output, and the second i'm not sure if it's sucess or fail:

Code: 
root@node1:~/r8152-2.15.0# depmod -a
root@node1:~/r8152-2.15.0# update-initramfs -u
update-initramfs: Generating /boot/initrd.img-5.4.124-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.

Afterwards i just reconnected the ethernet cable and i , activaded the interface and still get the syslog spammed.

Can anyone hint if the output of the "update-initramfs -u" is what is expected and if i need to reboot the hosts?

Thanks again!
 
xcj said:
Can anyone hint if the output of the "update-initramfs -u" is what is expected and if i need to reboot the hosts?
Click to expand...
That output is fine. "No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync." is normal if running grub (BIOS) and not systemd boot (EFI).
 
Following your reply, i just rebooted both hosts, and it seems to work!!!

I can ping the IP on the USB NIC's, no syslog SPAM!, and both report correct speed!

HAPPY!! Now to the next battle, get replication to work trough this B2B connection!


Code: 
root@node1:~# ethtool enx00e0a0100325
Settings for enx00e0a0100325:
        Supported ports: [ MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
                                2500baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
                                2500baseT/Full
        Advertised pause frame use: Symmetric Receive-only
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Full
                                             2500baseT/Full
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 2500Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 32
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: pumbg
        Wake-on: g
        Current message level: 0x00007fff (32767)
                               drv probe link timer ifdown ifup rx_err tx_err tx_queued intr tx_done rx_status pktdata hw wol
        Link detected: yes


Code: 
root@node2:~# ethtool enx00e0a01000d6
Settings for enx00e0a01000d6:
        Supported ports: [ MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
                                2500baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
                                2500baseT/Full
        Advertised pause frame use: Symmetric Receive-only
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Full
                                             2500baseT/Full
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 2500Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 32
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: pumbg
        Wake-on: g
        Current message level: 0x00007fff (32767)
                               drv probe link timer ifdown ifup rx_err tx_err tx_queued intr tx_done rx_status pktdata hw wol
        Link detected: yes
 
  • Like
Reactions: Dominic
xcj said:
@branto , Regarding USB 3.1, these two machines only have USB3.0.
USB3.0 has 5Ghz bandwidth, why do you think i'll run into issues?
Click to expand...

Granted, I was using 5GbE USB adapters, but I was having all kinds of issues running on 3.0 ports. Once traffic got above 2.5 on my machine, but this wasn't on ProxMox; it was a CentOS machine with the latest 5.11 kernel at the time. Once I moved the interfaces to the 3.2 ports, they stabilized.

I hope I'm wrong.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back