PCIe NVME Drive for VM Storage

B

Brailyn

New Member
Oct 27, 2023
9
1
3
I just purchased a PCI adapter to add an NVME drive to an old server. I thought it wasn't supported as lsblk doesn't show it, but I can see the drive with lspci...

How can I add this drive to proxmox so it can keep VMs on it? Only thing I can find online is how to do passthrough with it but I am not interested in doing that at the moment...

root@proxmox2:~# lspci
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
00:06.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5)
00:1c.2 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 3 (rev b5)
00:1c.3 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 4 (rev b5)
00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 (rev b5)
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5)
00:1f.0 ISA bridge: Intel Corporation C204 Chipset LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller (rev 05)
00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 05)
01:00.0 Non-Volatile memory controller: Kingston Technology Company, Inc. SNVS2000G [NV1 NVMe PCIe SSD 2TB] (rev 01)
02:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
05:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
06:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 02)
07:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 10)


root@proxmox2:~# lsblk -a
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 0B 0 loop
loop1 7:1 0 0B 0 loop
loop2 7:2 0 0B 0 loop
loop3 7:3 0 0B 0 loop
loop4 7:4 0 0B 0 loop
loop5 7:5 0 0B 0 loop
loop6 7:6 0 0B 0 loop
loop7 7:7 0 0B 0 loop
sda 8:0 0 9.1T 0 disk
├─sda1 8:1 0 2G 0 part
└─sda2 8:2 0 9.1T 0 part
sdb 8:16 0 9.1T 0 disk
├─sdb1 8:17 0 2G 0 part
└─sdb2 8:18 0 9.1T 0 part
sdc 8:32 0 9.1T 0 disk
├─sdc1 8:33 0 2G 0 part
└─sdc2 8:34 0 9.1T 0 part
sdd 8:48 0 9.1T 0 disk
├─sdd1 8:49 0 2G 0 part
└─sdd2 8:50 0 9.1T 0 part
sde 8:64 1 57.8G 0 disk
├─sde1 8:65 1 100M 0 part
├─sde5 8:69 1 4G 0 part
├─sde6 8:70 1 4G 0 part
└─sde7 8:71 1 49.7G 0 part
sdf 8:80 0 232.9G 0 disk
├─sdf1 8:81 0 1007K 0 part
├─sdf2 8:82 0 1G 0 part
└─sdf3 8:83 0 231.9G 0 part
├─pve-swap 252:0 0 8G 0 lvm [SWAP]
├─pve-root 252:1 0 68G 0 lvm /
├─pve-data_tmeta 252:2 0 1.4G 0 lvm
│ └─pve-data-tpool 252:4 0 137.1G 0 lvm
│ ├─pve-data 252:5 0 137.1G 1 lvm
│ ├─pve-vm--101--disk--0 252:6 0 140G 0 lvm
│ ├─pve-vm--102--disk--0 252:7 0 32G 0 lvm
│ ├─pve-vm--103--disk--1 252:8 0 48G 0 lvm
│ ├─pve-vm--103--disk--0 252:9 0 4M 0 lvm
│ └─pve-vm--103--disk--2 252:10 0 4M 0 lvm
└─pve-data_tdata 252:3 0 137.1G 0 lvm
└─pve-data-tpool 252:4 0 137.1G 0 lvm
├─pve-data 252:5 0 137.1G 1 lvm
├─pve-vm--101--disk--0 252:6 0 140G 0 lvm
├─pve-vm--102--disk--0 252:7 0 32G 0 lvm
├─pve-vm--103--disk--1 252:8 0 48G 0 lvm
├─pve-vm--103--disk--0 252:9 0 4M 0 lvm
└─pve-vm--103--disk--2 252:10 0 4M 0 lvm
 
Nevermind, in doing this my PCI passthrough device IDs were incremented so the NVME drive I wanted was actually being passed though to Truenas... Sorry about that. Just had to re-assign and it showed up.
 
I recently added another pci device and I lost this storage. I cant see any devices using it in passthrough settings and the PCI address has not changed since my last post. Is there anything I can do get my volumes back into proxmox? Has the drive possibly failed?

Code: 
root@proxmox2:~# lsblk -a
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0                          7:0    0     0B  0 loop
loop1                          7:1    0     0B  0 loop
loop2                          7:2    0     0B  0 loop
loop3                          7:3    0     0B  0 loop
loop4                          7:4    0     0B  0 loop
loop5                          7:5    0     0B  0 loop
loop6                          7:6    0     0B  0 loop
loop7                          7:7    0     0B  0 loop
sde                            8:64   0 232.9G  0 disk
├─sde1                         8:65   0  1007K  0 part
├─sde2                         8:66   0     1G  0 part
└─sde3                         8:67   0 231.9G  0 part
  ├─pve-swap                 252:0    0     8G  0 lvm  [SWAP]
  ├─pve-root                 252:1    0    68G  0 lvm  /
  ├─pve-data_tmeta           252:2    0   1.4G  0 lvm 
  │ └─pve-data-tpool         252:6    0 137.1G  0 lvm 
  │   ├─pve-data             252:7    0 137.1G  1 lvm 
  │   ├─pve-vm--101--disk--0 252:8    0   140G  0 lvm 
  │   ├─pve-vm--102--disk--0 252:9    0    32G  0 lvm 
  │   ├─pve-vm--103--disk--1 252:10   0    48G  0 lvm 
  │   ├─pve-vm--103--disk--0 252:11   0     4M  0 lvm 
  │   └─pve-vm--103--disk--2 252:12   0     4M  0 lvm 
  └─pve-data_tdata           252:3    0 137.1G  0 lvm 
    └─pve-data-tpool         252:6    0 137.1G  0 lvm 
      ├─pve-data             252:7    0 137.1G  1 lvm 
      ├─pve-vm--101--disk--0 252:8    0   140G  0 lvm 
      ├─pve-vm--102--disk--0 252:9    0    32G  0 lvm 
      ├─pve-vm--103--disk--1 252:10   0    48G  0 lvm 
      ├─pve-vm--103--disk--0 252:11   0     4M  0 lvm 
      └─pve-vm--103--disk--2 252:12   0     4M  0 lvm 
root@proxmox2:~# lspci
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
00:06.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5)
00:1c.2 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 3 (rev b5)
00:1c.3 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 4 (rev b5)
00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 (rev b5)
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5)
00:1f.0 ISA bridge: Intel Corporation C204 Chipset LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller (rev 05)
00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 05)
01:00.0 Non-Volatile memory controller: Kingston Technology Company, Inc. SNVS2000G [NV1 NVMe PCIe SSD 2TB] (rev 01)
02:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
02:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
03:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
05:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
06:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
07:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 02)
08:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 10)
 
I have not checked all of your posted info carefully, but if you do any PCI-passthrough, you need to make sure of the IOMMU groupings as they cannot be split. In other words, if you passthrough a PCI device, anything in the same IOMMU group that that device is a part of, will become unavailable to the host, even if it is not being explicitly passed through.

You will need to check the IOMMU groupings for correct IOMMU isolation - as shown here in the Proxmox wiki.

Here is an easy CLI method for a full IOMMU groups listing with ports etc.
Code: 
for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
 
Very good to know, thank you!


Code: 
root@proxmox2:~# for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
IOMMU group 0 00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller [8086:0158] (rev 09)
IOMMU group 10 00:1f.0 ISA bridge [0601]: Intel Corporation C204 Chipset LPC Controller [8086:1c54] (rev 05)
IOMMU group 10 00:1f.2 SATA controller [0106]: Intel Corporation 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller [8086:1c02] (rev 05)
IOMMU group 10 00:1f.3 SMBus [0c05]: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller [8086:1c22] (rev 05)
IOMMU group 11 05:00.0 Ethernet controller [0200]: Intel Corporation 82574L Gigabit Network Connection [8086:10d3]
IOMMU group 12 06:00.0 Ethernet controller [0200]: Intel Corporation 82574L Gigabit Network Connection [8086:10d3]
IOMMU group 13 07:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 02)
IOMMU group 13 08:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 10)
IOMMU group 1 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)
IOMMU group 1 00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0155] (rev 09)
IOMMU group 1 01:00.0 Non-Volatile memory controller [0108]: Kingston Technology Company, Inc. SNVS2000G [NV1 NVMe PCIe SSD 2TB] [2646:500e] (rev 01)
IOMMU group 1 02:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)
IOMMU group 1 02:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)
IOMMU group 2 00:06.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:015d] (rev 09)
IOMMU group 2 03:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
IOMMU group 3 00:1a.0 USB controller [0c03]: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 [8086:1c2d] (rev 05)
IOMMU group 4 00:1c.0 PCI bridge [0604]: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 [8086:1c10] (rev b5)
IOMMU group 5 00:1c.2 PCI bridge [0604]: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 3 [8086:1c14] (rev b5)
IOMMU group 6 00:1c.3 PCI bridge [0604]: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 4 [8086:1c16] (rev b5)
IOMMU group 7 00:1c.6 PCI bridge [0604]: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 [8086:1c1c] (rev b5)
IOMMU group 8 00:1d.0 USB controller [0c03]: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 [8086:1c26] (rev 05)
IOMMU group 9 00:1e.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev a5)
 
You must log in or register to reply here.
Top Bottom