[SOLVED] U.2 and 2 x HDD disappeared

T

taskman

New Member
Jan 11, 2023
9
0
1
Virtual Environment 7.3-3
Not sure where to start. I ran a bunch of commands and posted the result here in the hopes someone will understand the output

I have a U.2 connected to an M.2 adapter and another connected to PCI. The M.2 U.2 disappeared after a reboot.
I have 5 x HDD connected to a Silverstone FS305. Two of the five drives disappeared after a reboot.

I was able to see the HDD on the BIOS screen. I didn't look for the U.2.



This is lsblk
Code: 
NAME                                MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                                   8:0    0  14.6T  0 disk
└─Disk1_16TB-vm--109--disk--0       253:6    0  14.6T  0 lvm
sdb                                   8:16   0  14.6T  0 disk
└─Disk2_16TB-vm--109--disk--0       253:4    0  14.6T  0 lvm
sde                                   8:64   0 931.5G  0 disk
├─sde1                                8:65   0  1007K  0 part
├─sde2                                8:66   0   512M  0 part /boot/efi
└─sde3                                8:67   0   931G  0 part
  ├─pve-swap                        253:13   0     8G  0 lvm  [SWAP]
  └─pve-root                        253:14   0   923G  0 lvm  /
sdf                                   8:80   0  14.6T  0 disk
└─Disk5_16TB-vm--109--disk--0       253:5    0  14.6T  0 lvm
sdk                                   8:160  1  28.7G  0 disk
├─sdk1                                8:161  1    20M  0 part
├─sdk2                                8:162  1     1M  0 part
├─sdk3                                8:163  1    40M  0 part
└─sdk4                                8:164  1  28.6G  0 part
nvme2n1                             259:1    0   1.5T  0 disk
└─Intel1.6_TB-vm--200--disk--0      253:0    0   1.5T  0 lvm
nvme0n1                             259:2    0   3.6T  0 disk
├─Flux_SSD_4TB-Flux_SSD_4TB_tmeta   253:2    0  15.8G  0 lvm
│ └─Flux_SSD_4TB-Flux_SSD_4TB-tpool 253:8    0   3.6T  0 lvm
│   ├─Flux_SSD_4TB-Flux_SSD_4TB     253:9    0   3.6T  1 lvm
│   ├─Flux_SSD_4TB-vm--102--disk--0 253:10   0    32G  0 lvm
│   ├─Flux_SSD_4TB-vm--102--disk--1 253:11   0   225G  0 lvm
│   ├─Flux_SSD_4TB-vm--103--disk--0 253:15   0   225G  0 lvm
│   ├─Flux_SSD_4TB-vm--104--disk--0 253:16   0   225G  0 lvm
│   ├─Flux_SSD_4TB-vm--106--disk--1 253:17   0   225G  0 lvm
│   ├─Flux_SSD_4TB-vm--107--disk--0 253:18   0   225G  0 lvm
│   ├─Flux_SSD_4TB-vm--108--disk--0 253:19   0   225G  0 lvm
│   └─Flux_SSD_4TB-vm--200--disk--0 253:20   0   1.5T  0 lvm
└─Flux_SSD_4TB-Flux_SSD_4TB_tdata   253:3    0   3.6T  0 lvm
  └─Flux_SSD_4TB-Flux_SSD_4TB-tpool 253:8    0   3.6T  0 lvm
    ├─Flux_SSD_4TB-Flux_SSD_4TB     253:9    0   3.6T  1 lvm
    ├─Flux_SSD_4TB-vm--102--disk--0 253:10   0    32G  0 lvm
    ├─Flux_SSD_4TB-vm--102--disk--1 253:11   0   225G  0 lvm
    ├─Flux_SSD_4TB-vm--103--disk--0 253:15   0   225G  0 lvm
    ├─Flux_SSD_4TB-vm--104--disk--0 253:16   0   225G  0 lvm
    ├─Flux_SSD_4TB-vm--106--disk--1 253:17   0   225G  0 lvm
    ├─Flux_SSD_4TB-vm--107--disk--0 253:18   0   225G  0 lvm
    ├─Flux_SSD_4TB-vm--108--disk--0 253:19   0   225G  0 lvm
    └─Flux_SSD_4TB-vm--200--disk--0 253:20   0   1.5T  0 lvm

Code: 
[    0.000000]   Intel GenuineIntel
[   14.668589] Btrfs loaded, crc32c=crc32c-intel, zoned=yes, fsverity=yes
[   15.392463] snd_hda_intel 0000:01:00.1: enabling device (0000 -> 0002)
[   15.392545] snd_hda_intel 0000:01:00.1: Disabling MSI
[   15.392548] snd_hda_intel 0000:01:00.1: Handle vga_switcheroo audio client
[   15.392634] snd_hda_intel 0000:3a:00.1: enabling device (0000 -> 0002)
[   15.392664] snd_hda_intel 0000:3a:00.1: Handle vga_switcheroo audio client
[   15.392723] snd_hda_intel 0000:3a:00.6: enabling device (0000 -> 0002)
[   15.397030] snd_hda_intel 0000:3a:00.6: no codecs found!
[   15.486746] intel_rapl_common: Found RAPL domain package
[   15.486747] intel_rapl_common: Found RAPL domain core

dmesg | grep -i 'error\|warn\|exception'
Code: 
[    0.445795] ACPI BIOS Error (bug): Failure creating named object [\_GPE._L02], AE_ALREADY_EXISTS (20210730/dswload2-326)
[    0.445802] ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20210730/psobject-220)
[    0.446658] ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PCI0.GPP7.UP00.DP40.UP00.DP68], AE_NOT_FOUND (20210730/dswload2-162)
[    0.446660] ACPI Error: AE_NOT_FOUND, During name lookup/catalog (20210730/psobject-220)
[    0.833961] mce: [Hardware Error]: Machine check events logged
[    0.833962] mce: [Hardware Error]: CPU 1: Machine Check: 0 Bank 29: ffffffffffffffff
[    0.833966] mce: [Hardware Error]: TSC 0 MISC ff1fffffffffffff SYND ffffffffffffffff IPID ffffffffffffffff
[    0.833968] mce: [Hardware Error]: PROCESSOR 2:a60f12 TIME 1692610240 SOCKET 0 APIC 2 microcode a601201
[    0.837754] RAS: Correctable Errors collector initialized.
[   15.144300] EXT4-fs (dm-14): re-mounted. Opts: errors=remount-ro. Quota mode: none.
[   15.278755] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
[   15.297305] bluetooth hci0: Direct firmware load for mediatek/BT_RAM_CODE_MT7922_1_1_hdr.bin failed with error -2

fdisk -l
Code: 
Disk /dev/nvme2n1: 1.46 TiB, 1600321314816 bytes, 3125627568 sectors
Disk model: INTEL SSDPE2KE016T8
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/Intel1.6_TB-vm--200--disk--0: 1.46 TiB, 1599875317760 bytes, 3124756480 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/nvme0n1: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: CT4000P3PSSD8
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sda: 14.55 TiB, 16000900661248 bytes, 31251759104 sectors
Disk model: ST16000NM000J-2T
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdb: 14.55 TiB, 16000900661248 bytes, 31251759104 sectors
Disk model: ST16000NM000J-2T
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sde: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: CT1000MX500SSD1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: C0F6BC0D-8E6C-46E1-BA8C-256A99136C27

Device       Start        End    Sectors  Size Type
/dev/sde1       34       2047       2014 1007K BIOS boot
/dev/sde2     2048    1050623    1048576  512M EFI System
/dev/sde3  1050624 1953525134 1952474511  931G Linux LVM

Partition 1 does not start on physical sector boundary.


Disk /dev/sdf: 14.55 TiB, 16000900661248 bytes, 31251759104 sectors
Disk model: ST16000NM001G-2K
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/Disk2_16TB-vm--109--disk--0: 14.55 TiB, 15999826919424 bytes, 31249661952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 2EF7D19D-8FA9-420B-87B6-CD7FDA0D10F8

Device                                          Start         End     Sectors  Size Type
/dev/mapper/Disk2_16TB-vm--109--disk--0-part1     128     4194304     4194177    2G Linux swap
/dev/mapper/Disk2_16TB-vm--109--disk--0-part2 4194432 31249661918 31245467487 14.5T Solaris /usr & Apple ZFS


Disk /dev/mapper/Disk5_16TB-vm--109--disk--0: 14.55 TiB, 15999826919424 bytes, 31249661952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 9E680D17-621C-45FF-8F80-63035F86DAA1

Device                                          Start         End     Sectors  Size Type
/dev/mapper/Disk5_16TB-vm--109--disk--0-part1     128     4194304     4194177    2G Linux swap
/dev/mapper/Disk5_16TB-vm--109--disk--0-part2 4194432 31249661918 31245467487 14.5T Solaris /usr & Apple ZFS


Disk /dev/mapper/Disk1_16TB-vm--109--disk--0: 14.55 TiB, 15999826919424 bytes, 31249661952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: D692FFF3-C375-4602-91E4-24F3946C405C

Device                                          Start         End     Sectors  Size Type
/dev/mapper/Disk1_16TB-vm--109--disk--0-part1     128     4194304     4194177    2G Linux swap
/dev/mapper/Disk1_16TB-vm--109--disk--0-part2 4194432 31249661918 31245467487 14.5T Solaris /usr & Apple ZFS


Disk /dev/mapper/Flux_SSD_4TB-vm--102--disk--0: 32 GiB, 34359738368 bytes, 67108864 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes


Disk /dev/mapper/pve-swap: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/pve-root: 923.01 GiB, 991072092160 bytes, 1935687680 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/Flux_SSD_4TB-vm--102--disk--1: 225 GiB, 241591910400 bytes, 471859200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: gpt
Disk identifier: 99BFA961-4F1A-4BD6-A63B-5F953721042B

Device                                          Start       End   Sectors  Size Type
/dev/mapper/Flux_SSD_4TB-vm--102--disk--1-part1  2048      4095      2048    1M BIOS boot
/dev/mapper/Flux_SSD_4TB-vm--102--disk--1-part2  4096 471857151 471853056  225G Linux filesystem


Disk /dev/mapper/Flux_SSD_4TB-vm--103--disk--0: 225 GiB, 241591910400 bytes, 471859200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: gpt
Disk identifier: 99BFA961-4F1A-4BD6-A63B-5F953721042B

Device                                          Start       End   Sectors  Size Type
/dev/mapper/Flux_SSD_4TB-vm--103--disk--0-part1  2048      4095      2048    1M BIOS boot
/dev/mapper/Flux_SSD_4TB-vm--103--disk--0-part2  4096 471857151 471853056  225G Linux filesystem


Disk /dev/mapper/Flux_SSD_4TB-vm--104--disk--0: 225 GiB, 241591910400 bytes, 471859200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: gpt
Disk identifier: 80D7FE2C-2F5E-4700-8A82-A7D8C88750BD

Device                                          Start       End   Sectors  Size Type
/dev/mapper/Flux_SSD_4TB-vm--104--disk--0-part1  2048      4095      2048    1M BIOS boot
/dev/mapper/Flux_SSD_4TB-vm--104--disk--0-part2  4096 471857151 471853056  225G Linux filesystem


Disk /dev/mapper/Flux_SSD_4TB-vm--106--disk--1: 225 GiB, 241591910400 bytes, 471859200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: gpt
Disk identifier: 1A6F284F-CE36-402F-A446-2A12C4E5C2EA

Device                                          Start       End   Sectors  Size Type
/dev/mapper/Flux_SSD_4TB-vm--106--disk--1-part1  2048      4095      2048    1M BIOS boot
/dev/mapper/Flux_SSD_4TB-vm--106--disk--1-part2  4096 471857151 471853056  225G Linux filesystem


Disk /dev/mapper/Flux_SSD_4TB-vm--107--disk--0: 225 GiB, 241591910400 bytes, 471859200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: gpt
Disk identifier: 379F98C1-365D-4543-AE04-437A2E88EF0D

Device                                          Start       End   Sectors  Size Type
/dev/mapper/Flux_SSD_4TB-vm--107--disk--0-part1  2048      4095      2048    1M BIOS boot
/dev/mapper/Flux_SSD_4TB-vm--107--disk--0-part2  4096 471857151 471853056  225G Linux filesystem


Disk /dev/mapper/Flux_SSD_4TB-vm--108--disk--0: 225 GiB, 241591910400 bytes, 471859200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: gpt
Disk identifier: ECBACB60-5438-45F9-87C8-9B2159E2A8FB

Device                                          Start       End   Sectors  Size Type
/dev/mapper/Flux_SSD_4TB-vm--108--disk--0-part1  2048      4095      2048    1M BIOS boot
/dev/mapper/Flux_SSD_4TB-vm--108--disk--0-part2  4096 471857151 471853056  225G Linux filesystem


Disk /dev/mapper/Flux_SSD_4TB-vm--200--disk--0: 1.46 TiB, 1599875317760 bytes, 3124756480 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes


Disk /dev/sdk: 28.67 GiB, 30784094208 bytes, 60125184 sectors
Disk model:  SanDisk 3.2Gen1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 75EC96FF-FEA6-4E82-8270-2359AA33BF56

Device      Start      End  Sectors  Size Type
/dev/sdk1    2048    43007    40960   20M Microsoft basic data
/dev/sdk2   43008    45055     2048    1M BIOS boot
/dev/sdk3   45056   126975    81920   40M EFI System
/dev/sdk4  126976 60125150 59998175 28.6G Linux filesystem
 
Last edited:
It would help a lot if you could post the output of the commands inside [CODE][/CODE] tags. There are also buttons in the editor for inline code and full code blocks.

Besides that, have you tried to reseat the adapters and if the cables are connected well / tried different cables?
 
aaron said:
It would help a lot if you could post the output of the commands inside [CODE][/CODE] tags. There are also buttons in the editor for inline code and full code blocks.

Besides that, have you tried to reseat the adapters and if the cables are connected well / tried different cables?
Click to expand...
I tried to edit the post, but it only edits the subject.

I removed the M.2 adapter and installed it again, twice
I removed all the HDD in the HDD holder and installed them again.
I checked the SATA cables from the motherboard and the HDD holder by pulling and pushing on them.

The HDD are being picked up in the BIOS but not in Proxmox.
 
Some more information. I updated my bios and it caused some issues sending me to grub but on the bright side I could take photos of all the drives. The drives are being detected by the motherboard, just Proxmox isn't liking them.

https://imgur.com/a/syhIBbl

https://imgur.com/a/Vfh9GMt

The MSI bios seem a bit weird. The line that has AHCI/Raid is always on but when you click it, it changes from AHCI to Raid. I have not changed that setting before or after the bios update. Proxmox still detects 3 of the 5 16TB HDD and it still detects the single U.2 on the PCI card, but not the U.2 on the M.2 adapter. This happened after a reboot.

https://imgur.com/a/T7r0SVr
 
Last edited:
This solution worked to get the drives back, but now my TrueNas doesn't start giving an error that IOMMU isn't present
https://imgur.com/a/Ck4Y9iR


https://forum.proxmox.com/threads/d...h-marvell-88se9230-chipset.38102/#post-239646

"
I followed your suggestions and after trying different combinations I was able to get the drives to show up with the following config change in /etc/default/grub without disabling VT-d in the BIOS (I need this for passthrough to my VM and LXCs):
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
GRUB_CMDLINE_LINUX="iommu=soft"
Then update grub and reboot the server and the drives are present:
sudo update-grub
"
 
Looks like if my PCIE passthrough is switched on then some of my HDD's disappear and the one U.2 on M.2 also disappear

No HDD
TrueNas can use the HBA plugged into PCIE
TrueNas has error of MPT Bios fault
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on video=efifb:off"

No HDD
TrueNas can use the HBA plugged into PCIE
TrueNas has error of MPT Bios fault
GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt"

No HDD
TrueNas can use the HBA plugged into PCIE
TrueNas has error of MPT Bios fault
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amd_iommu=force_enable iommu=pt"


HDD detected
TrueNas can't detect the HBA plugged into PCIE
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=off"
 
I figured it out. Just took 5 hours. I don't understand what difference it makes to Proxmox in which PCIE slot the devices are but seems it does in my case


Motherboard
PRO X670-P WIFI

HBA according to Proxmox information
Broadcom/LSI
SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]


Using my original passthrough config string
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt nomodeset initcall_blacklist=sysfb_init vfio-pci.ids=10de:2204,10de:1aef disable_vga=1"


This was the motherboard setup that failed
PCIE_E1 x16 = GPU
PCIE_E2 x1 = empty
PCIE_E3 x4 = Intel U.2 with PCIE adapter
PCIE_E4 x2 = Dell HBA

M2_1 = empty
M2_2 = SSD
M2_3 = M.2 to U.2 adapter
M2_4 = empty


I swapped the U.2 and the HBA and now everything is working.
PCIE_E1 x16 = GPU
PCIE_E2 x1 = empty
PCIE_E3 x4 = Dell HBA
PCIE_E4 x2 = Intel U.2 with PCIE adapter

M2_1 = empty
M2_2 = SSD
M2_3 = M.2 to U.2 adapter
M2_4 = empty
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom