Network card missing

A

ademirk

New Member
Proxmox Subscriber
Jun 6, 2024
20
0
1
I did setup a new promox node, and after setting up nic to a VM as hostpci other nics are gone from my setup. they should work as bond1 and vmbr1.
Code: 
root@pve2:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 34:48:ed:f3:fb:9c brd ff:ff:ff:ff:ff:ff
    altname enp24s0f0
3: eno2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 34:48:ed:f3:fb:9c brd ff:ff:ff:ff:ff:ff permaddr 34:48:ed:f3:fb:9d
    altname enp24s0f1
10: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 34:48:ed:f3:fb:9c brd ff:ff:ff:ff:ff:ff
11: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 34:48:ed:f3:fb:9c brd ff:ff:ff:ff:ff:ff
    inet 172.16.0.36/16 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::3648:edff:fef3:fb9c/64 scope link
       valid_lft forever preferred_lft forever
12: tap127i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr127i0 state UNKNOWN group default qlen 1000
    link/ether 42:27:0f:e7:b3:c0 brd ff:ff:ff:ff:ff:ff
13: fwbr127i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 6e:a5:59:a6:8a:7e brd ff:ff:ff:ff:ff:ff
14: fwpr127p0@fwln127i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether be:9d:34:62:77:5b brd ff:ff:ff:ff:ff:ff
15: fwln127i0@fwpr127p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr127i0 state UP group default qlen 1000
    link/ether 6e:a5:59:a6:8a:7e brd ff:ff:ff:ff:ff:ff
16: tap126i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether 5a:1f:24:0d:b8:a7 brd ff:ff:ff:ff:ff:ff

But there are other nics missing:
Code: 
root@pve2:~# dmesg | grep  -i 0000:3b
[    1.960936] pci 0000:3b:00.0: [14e4:1657] type 00 class 0x020000 PCIe Endpoint
[    1.960968] pci 0000:3b:00.0: BAR 0 [mem 0xab090000-0xab09ffff 64bit pref]
[    1.960990] pci 0000:3b:00.0: BAR 2 [mem 0xab0a0000-0xab0affff 64bit pref]
[    1.961012] pci 0000:3b:00.0: BAR 4 [mem 0xab0b0000-0xab0bffff 64bit pref]
[    1.961027] pci 0000:3b:00.0: ROM [mem 0xfffc0000-0xffffffff pref]
[    1.961365] pci 0000:3b:00.0: PME# supported from D0 D3hot D3cold
[    1.961506] pci 0000:3b:00.1: [14e4:1657] type 00 class 0x020000 PCIe Endpoint
[    1.961538] pci 0000:3b:00.1: BAR 0 [mem 0xab060000-0xab06ffff 64bit pref]
[    1.961560] pci 0000:3b:00.1: BAR 2 [mem 0xab070000-0xab07ffff 64bit pref]
[    1.961583] pci 0000:3b:00.1: BAR 4 [mem 0xab080000-0xab08ffff 64bit pref]
[    1.961598] pci 0000:3b:00.1: ROM [mem 0xfffc0000-0xffffffff pref]
[    1.961809] pci 0000:3b:00.1: PME# supported from D0 D3hot D3cold
[    1.961909] pci 0000:3b:00.2: [14e4:1657] type 00 class 0x020000 PCIe Endpoint
[    1.961941] pci 0000:3b:00.2: BAR 0 [mem 0xab030000-0xab03ffff 64bit pref]
[    1.961963] pci 0000:3b:00.2: BAR 2 [mem 0xab040000-0xab04ffff 64bit pref]
[    1.961986] pci 0000:3b:00.2: BAR 4 [mem 0xab050000-0xab05ffff 64bit pref]
[    1.962001] pci 0000:3b:00.2: ROM [mem 0xfffc0000-0xffffffff pref]
[    1.962210] pci 0000:3b:00.2: PME# supported from D0 D3hot D3cold
[    1.962316] pci 0000:3b:00.3: [14e4:1657] type 00 class 0x020000 PCIe Endpoint
[    1.962348] pci 0000:3b:00.3: BAR 0 [mem 0xab000000-0xab00ffff 64bit pref]
[    1.962370] pci 0000:3b:00.3: BAR 2 [mem 0xab010000-0xab01ffff 64bit pref]
[    1.962392] pci 0000:3b:00.3: BAR 4 [mem 0xab020000-0xab02ffff 64bit pref]
[    1.962407] pci 0000:3b:00.3: ROM [mem 0xfffc0000-0xffffffff pref]
[    1.962617] pci 0000:3b:00.3: PME# supported from D0 D3hot D3cold
[    2.027555] pci 0000:3b:00.0: ROM [mem 0xfffc0000-0xffffffff pref]: can't claim; no compatible bridge window
[    2.027557] pci 0000:3b:00.1: ROM [mem 0xfffc0000-0xffffffff pref]: can't claim; no compatible bridge window
[    2.027559] pci 0000:3b:00.2: ROM [mem 0xfffc0000-0xffffffff pref]: can't claim; no compatible bridge window
[    2.027562] pci 0000:3b:00.3: ROM [mem 0xfffc0000-0xffffffff pref]: can't claim; no compatible bridge window
[    2.027999] pci 0000:3b:00.0: ROM [mem 0xab200000-0xab23ffff pref]: assigned
[    2.028001] pci 0000:3b:00.1: ROM [mem 0xab240000-0xab27ffff pref]: assigned
[    2.028002] pci 0000:3b:00.2: ROM [mem 0xab280000-0xab2bffff pref]: assigned
[    2.028004] pci 0000:3b:00.3: ROM [mem 0xab2c0000-0xab2fffff pref]: assigned
[    2.028038] pci_bus 0000:3b: resource 1 [mem 0xab200000-0xab2fffff]
[    2.028040] pci_bus 0000:3b: resource 2 [mem 0xab000000-0xab0fffff 64bit pref]
[    2.029751] pci 0000:3b:00.0: Adding to iommu group 1
[    2.029812] pci 0000:3b:00.1: Adding to iommu group 1
[    2.029879] pci 0000:3b:00.2: Adding to iommu group 1
[    2.029941] pci 0000:3b:00.3: Adding to iommu group 1
[    2.951694] tg3 0000:3b:00.0 eth4: Tigon3 [partno(BCM95719) rev 5719001] (PCI Express) MAC address b0:26:28:9c:ad:2c
[    2.951700] tg3 0000:3b:00.0 eth4: attached PHY is 5719C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
[    2.951703] tg3 0000:3b:00.0 eth4: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
[    2.951705] tg3 0000:3b:00.0 eth4: dma_rwctrl[00000001] dma_mask[64-bit]
[    2.962754] tg3 0000:3b:00.1 eth5: Tigon3 [partno(BCM95719) rev 5719001] (PCI Express) MAC address b0:26:28:9c:ad:2d
[    2.962758] tg3 0000:3b:00.1 eth5: attached PHY is 5719C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
[    2.962761] tg3 0000:3b:00.1 eth5: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
[    2.962763] tg3 0000:3b:00.1 eth5: dma_rwctrl[00000001] dma_mask[64-bit]
[    2.973752] tg3 0000:3b:00.2 eth6: Tigon3 [partno(BCM95719) rev 5719001] (PCI Express) MAC address b0:26:28:9c:ad:2e
[    2.973757] tg3 0000:3b:00.2 eth6: attached PHY is 5719C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
[    2.973760] tg3 0000:3b:00.2 eth6: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
[    2.973762] tg3 0000:3b:00.2 eth6: dma_rwctrl[00000001] dma_mask[64-bit]
[    2.984662] tg3 0000:3b:00.3 eth7: Tigon3 [partno(BCM95719) rev 5719001] (PCI Express) MAC address b0:26:28:9c:ad:2f
[    2.984666] tg3 0000:3b:00.3 eth7: attached PHY is 5719C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
[    2.984669] tg3 0000:3b:00.3 eth7: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
[    2.984671] tg3 0000:3b:00.3 eth7: dma_rwctrl[00000001] dma_mask[64-bit]
[    3.423913] tg3 0000:3b:00.1 ens1f1: renamed from eth5
[    3.433006] tg3 0000:3b:00.3 ens1f3: renamed from eth7
[    3.440954] tg3 0000:3b:00.2 ens1f2: renamed from eth6
[    3.450948] tg3 0000:3b:00.0 ens1f0: renamed from eth4
[   17.183509] tg3 0000:3b:00.2 ens1f2: Link is up at 1000 Mbps, full duplex
[   17.183520] tg3 0000:3b:00.2 ens1f2: Flow control is on for TX and on for RX
[   17.183523] tg3 0000:3b:00.2 ens1f2: EEE is disabled
[   17.516603] tg3 0000:3b:00.3 ens1f3: Link is up at 1000 Mbps, full duplex
[   17.516613] tg3 0000:3b:00.3 ens1f3: Flow control is on for TX and on for RX
[   17.516616] tg3 0000:3b:00.3 ens1f3: EEE is disabled

[ 3.433006] tg3 0000:3b:00.3 ens1f3: renamed from eth7
[ 3.440954] tg3 0000:3b:00.2 ens1f2: renamed from eth6
this two interfaces are not showing anyware. ANY HELP ?
 
Using lspci -k | sed -n '/Ethernet/,/driver in use/p' I can find the devices but not using the correct driver and without DeviceName, how can I solve this?


Code: 
root@pve2:/# lspci -k | sed -n '/Ethernet/,/driver in use/p'
18:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
        DeviceName: NIC1
        Subsystem: Dell NetXtreme BCM5720 Gigabit Ethernet PCIe
        Kernel driver in use: tg3
18:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
        DeviceName: NIC2
        Subsystem: Dell NetXtreme BCM5720 Gigabit Ethernet PCIe
        Kernel driver in use: tg3
19:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
        DeviceName: NIC3
        Subsystem: Dell NetXtreme BCM5720 Gigabit Ethernet PCIe
        Kernel driver in use: vfio-pci
19:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
        DeviceName: NIC4
        Subsystem: Dell NetXtreme BCM5720 Gigabit Ethernet PCIe
        Kernel driver in use: vfio-pci
3b:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
        Subsystem: Broadcom Inc. and subsidiaries 4-port 1Gb Ethernet Adapter
        Kernel driver in use: vfio-pci
3b:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
        Subsystem: Broadcom Inc. and subsidiaries 4-port 1Gb Ethernet Adapter
        Kernel driver in use: vfio-pci
3b:00.2 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
        Subsystem: Broadcom Inc. and subsidiaries 4-port 1Gb Ethernet Adapter
        Kernel driver in use: vfio-pci
3b:00.3 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
        Subsystem: Broadcom Inc. and subsidiaries 4-port 1Gb Ethernet Adapter
        Kernel driver in use: vfio-pci
 
After search a lot for a solution, I found there's an issue when you connect any pci interface to the VM and then remove it. When you connect it start to use the vfio driver, but when you remove it from VM, it doesn't unbind from vfio-pci.
I did try a lot of solutions to unbind it, but after reboot, always came back to vfio-pci, and so my ethernet doesn't work as it should.
So my only solution was to format PVE and try again, taking caution not to connect the wrong PCI nic to VM, because it doesn't came back anymore in proxmox.



https://superuser.com/questions/1477455/unable-to-bind-nic-to-driver-vfio-pci
https://askubuntu.com/questions/150...to-vfio-pci-properly-on-demand-without-reboot
https://forum.proxmox.com/threads/p...-pcie-attached-nvme-disks.154229/#post-702456
 
You must log in or register to reply here.
Top Bottom
Back