Adding sata-PciEx1 expansion card disables host's integrated network adapter

[kutuzov-san]

First of all:
- I'm not native english speaker, so sorry if something's off;
- I'm new to proxmox, but i'm learning.

Here's what happened: I installed proxmox, played with it, made several VM's and one VM with GPU passthrough. Till this moment everything is fine (and what was not fine, was fixed). Recently I bought Pci-Ex1 to Sata adapter to expand my HHD number beyond standard 6 (chinese noname). I connected adapter to motherboard's slot pci4 (number from motherboard's user manual) and tried to start PC. It worked, motherboard's network card worked (there was bipping light) till proxmox start. When the system was supposed to be fully loaded I found out, that I couldn't enter webgui. Hopefully, i checked through monitor connected to GPU that guest VM is fully loaded.
Disabling:
- firewall gave me nothing (it was not activated in the first place, but I checked it out either way);
- virtualization during loading also left me with nothing.
I tried:
- to change Pci-e configuration in bios from "auto" to specific one and got nothing;
- run Ubuntu Live and it told me "Man! Here is your internet and this are your disks. All 10 of them (8 mounted, 1 USB and 1 connected via adapter). What else?" From this I conclude that the problem is not with my hardware.

So here is my questions:
- is it even possible to use aforementioned adapter with proxmox? (thou it is highly possible that there is no problem as proxmox runs on the same core as debian (or so I heard), but still)
- what causes motherboards integrated network card to disable during proxmox loading?
- as I remember, adding Pci-E device may change nvme partition identificators. May be problem somehow connected?
- what information can I give and what I need to do to fix this strange problem?

my PC (host):
- Processor: ryzen 3700x
- Motherboard: MSI b550 Tomahawk;
- RAM 64G (don't remember the vendor. Something with "Viper" in model name);
- GPU: Nvidia RTX 3060lhr 12G
- HDD: x5 (WD, Seagate)
- SSD: 1 Kingspec
- NVME m2: 2 (netac, reletek)

Oh! And congratulate you all with the coming of the new year =)

 

[mr44er]

chinese noname

That could be really the brand or the cheapest s...tuff.

More likely:
Dive into the wonderful world of limited PCI-E Lanes.
ryzen 3700x -> has 24 lanes
Nvidia RTX 3060lhr -> makes 16 lanes go poof
1x NVMe-Slot -> makes 4 lanes go poof (or already throttled@2 because nothing is left, onboard NIC wants lanes and SATA-Controller too)

Nothing is left, another card wants some lanes...and mobo decides to kick off the NIC. Be it intended, be it an error, but not unusual.

If you really want/need more lanes, use Threadripper or Epyc platform.

https://www.reddit.com/r/VFIO/comments/m8aw5z/pcie_lanes_explained/
https://www.cgdirector.com/guide-to-pcie-lanes/


Check PCI-E IDs via lspci with and without the card. compare with ip -a, compare nano /etc/network/interfaces
If the IDs change with the card, edit nano /etc/network/interfaces + systemctl restart networking or reboot.

If you don't see any device with lspci you don't have enough lanes and mobo does not throttle accordingly.
Check lspci -vv where downgrades are shown.

Example good:

LnkCap:    Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+

Example bad:

LnkSta:    Speed 8GT/s, Width x4 (downgraded)
            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

 

[_gabriel]

https://pve.proxmox.com/wiki/Networ...ice names,latest network device naming scheme

 

[kutuzov-san]

I checked everything except for "lspci -vv" output as it is too big and I couldn't scroll it in server command line.
But aside from it there is no change in files beyond additions on adding a card. For example command "ip -a" and content of /etc/network/interfaces don't change.
There is a change in simple lspci. The difference is adding new line of "03:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43ea" as it becomes 4 lines instead of 3.
With and without card there is no change in that output shows

05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)

2a:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller

 

[mr44er]

Is this behaviour the same on Ubuntu live (with and without that card?)

Or the other way around...if you don't see an entry (ubuntu and proxmox) with "Chinese SATA Whatever-Controller" checking lspci you have no lanes left for it. Maybe the bridge "03:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43ea" tries to get lanes (steal from RTL8125), but then errors out and because of this the NIC resets...sort of.

Maybe it helps to dig through dmesg . At least it should tell as minimum "pci4: error", but the explanation about the lanes is the "why".

 

[sidnakvee]

Hi , Please note I am facing the same issue when added second WD western digital 1 TB nvme. However runing the command lspci shows lots of entries what to do next and what to look for kindly explain a bit to reslve this issue . thansk again

see entry for new nvme drive as Non -voliatle memory controller : sandisk corp WD black Sn770 Nbme SSD (rev 01)