Unable to load the system with a u.2 drive

[hexrebuilt]

Hi everyone, i'm having an issue that i don't know how to troubleshoute.

I boucht an Intel optane u.2 disk, with the aim of passing it trough to Truenas. The u.2 drive works even if it is an ebay-used one (94% life left, so i will probably die before the drive).
To test it, i've tried on a different Windows machine, with a PCI X4 adapter i'm able to fully use it and format it. On the server i have to use an aliexpress m.2 to u.2 adapter. It seems to be working since it is visible in the BIOS, along with all the other disks, including another 260gb PCI intel drive.

The issue:
But as soon as I start Proxmox, it goes into lock for some reason. While checking the log, after an hard reset, unplugging the drive, i'm able to see an I/O error on the other PCI disk and I don't know what else to do, since i don't have anymore a display output thanks to the GPU passthrough.

Any ideas? The only one that comes to mind is the absence of enough PCI lines on the AMD 2600, but from the mobo manual I should be at the full limit (8x on the GPU, 8x on the hba, 4x for the Intel PCI disk and the remaining 4x available are those of the m.2 that are converted to u.2 4x). In fact I should have a spare 1x available.

The drive is not in the "compatible" u.2 drive list, however i'm able to see every detail in the bios so i think that it's fine on that side.

My specs:​

• Motherboard: GA-AX370-Gaming 5
• Ram: 64GB corsair vengeance 3000Mhz
• CPU: AMD Ryzen 5 2600X
• GPU: RTX A2000 in the PCI x16 slot (x8)
• HBA Fujitsu LSI SAS3008 in IT mode in the PCI x8 slot
• Intel SSD DC P4600 via m.2 to u.2 adapter
• 900P INTEL 280GB PCIE SSD in the PCI x4 slot
• boot drive: Kingston 120gbx2

Somebody has any idea about what to check? the system works just fine without the U.2 connected

 

[BobhWasatch]

Disable the VM with the passthrough so you can see what the error is. Most likely the PCI ID's are changing and something (commonly the network device) is no longer where the configuration thinks it is.

ETA: There are several threads here about AMD boxes and passthrough. Basically, most AMD mother boards aren't very friendly to passthrough in terms of the way their IOMMU groups are laid out. What you want to do may or may not be possible.

 

[hexrebuilt]

I fell so dumb for not having thought about it. I have to disable the "start at boot" for all of the VMS. Since they are all inside the 900p drive.
I'll take a look and marked as solved if that's the case.
Thanks in the meantime

 

[gfngfn256]

If you leave the m.2 adapter connected but disconnect the u.2 drive from the adapter, does it boot normally? That m.2 adapter from the far-east maybe shady.
Another test - try live-booting a regular Linux distro (Debian as its closest to Proxmox?) & see if it boots & picks up the m.2 adapter + u.2 drive.

 

[hexrebuilt]

If you leave the m.2 adapter connected but disconnect the u.2 drive from the adapter, does it boot normally? That m.2 adapter from the far-east maybe shady.
Another test - try live-booting a regular Linux distro (Debian as its closest to Proxmox?) & see if it boots & picks up the m.2 adapter + u.2 drive.

It does.
I've checked the PCI IDs. The U.1 is the 00:01.0, so i don't think that it does interfere.

even while removing all the devices from the VM, the system will not show the WebUI.

I attatch the Log for one of those startup, before killing it and unplugging the u.2 drive, restoring the normal operations.
The drive appears to be visible

Jul 08 22:19:52 hex kernel: pci 0000:01:00.0: Adding to iommu group 14

and with it's own group. Also it appears to be visible in the NVME disks

Jul 08 22:19:52 hex kernel: nvme nvme0: pci function 0000:01:00.0

Tomorrow i'll try the live test to see if the u.2 creates some issues also there.

 

[gfngfn256]

the system will not show the WebUI.

You mean GUI interface from another terminal accessing the IP address?

I guess you probably meant the host CLI login screen on the local monitor attached to the host. The GUI (Web interface) never shows on the local terminal/monitor.

If in fact you do mean the GUI, then your NW maybe misconfigured / changed due to PCI renumeration etc.

 

[hexrebuilt]

You mean GUI interface from another terminal accessing the IP address?

Yes, that's the issue. i'm unable to see what's going on. i just know that none of the VMs is available on the network, just like proxmox.

I guess you probably meant the host CLI login screen on the local monitor attached to the host. The GUI (Web interface) never shows on the local terminal/monitor.

No, i'm not. the CLI screen is unavailable for me since i'm passing the GPU to a VM. after the "loading initial ramdisk" the screen gets "normally frozen". The only difference is that with the U.2 drive, and this issue, the screen gets disconnected, while under normal operations it stays on.
I've renounced to use the CLI in the moment that i've decided to recycle the 2600x

If in fact you do mean the GUI, then your NW maybe misconfigured / changed due to PCI renumeration etc

I've checked and it seems like it's not. the U.2 has its own group and the ID appears to not shift the others. Also i've removed all the pci devices from the vm to avoid possible conflicts, and nothing changes.

In the LOG it's visible that the drive is showing up, but i don't understand what could cause the system to lock in that way.

 

[gfngfn256]

What's left to look at:

1. Set all VM's not to start at run (you've probably done this already).
2. You may have a PSU power problem, some of those MB/PSU simply can't cope with all that PCI gear. What I would try is removing the RTX A2000 (usually the GPU is power hungry) & see the results. Some MBs won't post with no GPU, so YMMV.
3. Get yourself a different adapter/solution for the U.2 drive.
4. Live media booting a Debian OS.
5. Installing an older/different version of Proxmox, from here.

 

[hexrebuilt]

Hi, i have left to try options 4&. nothing changes, i think that's also because the u.2 is sata powered.

if by live booting to debian, everything shows up, what can i do?

 

[hexrebuilt]

Well, it turns out that was just a shitty adapter from m.2 to u.2. two of them actually.

Solved with another shitty adapter m.2 to PCI3x4 with another decent adapter from PCI3x4 to u.2. saw right away.

Lesson learned: Chinese adapters are questionable but if lucky they works, if not, you will be in pain for months