2nd NVME drive stops 1st working?

[dbs22]

I'm trying to add a second NVME drive to my Proxmox 7.2-3 setup to create another LVM-Thin drive for addition VM disk storage, however when I add the drive physically it stops VMs being able to start on the original nvme drive with the following error "Could not open '/dev/nvme1/vm-101-disk-0': Not a directory".

I've included some pictures of the drive setup (with the 2nd NVME) but I'll try and explain the config too. This is all setup on a Ryzen B450 system with a single m.2 slot and 4x SATA connectors. Proxmox in installed on /dev/sdd (sata 4) with 15gb partition and the remainder of the SSD is a LVM-Thin lvm-local/data/pve. I also have 2x4tb and 1x2tb sata HDDs in a 10tb LVM as hdd (sata connector 1-3 /dev/sda -c). Finally I have an NVME (m.2) 1tb as nvme1 VG.

I have another 1tb NVME drive on PCIEx4 adaptor board and I have tried it in both of the PCIEx16 slots. However when I boot with it installed I can no longer start any of the VMs on the NVME drive nvme1 VG with the the following error "Could not open '/dev/nvme1/vm-101-disk-0': Not a directory". When the second NVME drive is installed
Proxmox boots fine (I run it headless) and it shows both NVME drives. I can create a LVM Thin VG on the second NVME drive and add and start VMS on it. HDD VG is not impacted and works as normal in all configurations. On nvme1 I can browse the VM drives, I can query the SMART status for drive temps and wear status. Looking at PVSCAN, VGSCAN and LVSCAN it all looks ok. I'm only a beginner with LVM. I can't copy the vm drives off nvme1, nor does qm rescan help. Once I remove nvme2 everything goes back to how it was.

I'm not sure where to go with this so any help would be appreciated. The last thing I can think to do is backup or move all the vms to hdd insert the second nmve drive and try to setup the lvm thin again from nvme1 on it to see if that fixes it, but it will take a loooooong time to move all the VMs to the hdd.

 

[VictorSTS]

I would apt install nvme-cli and run nvme list with and without the second drive. Maye your server lists the second nvme drive with the same id of the nvme drive that holds your VM data.

 

[dbs22]

I get the following before and after adding the second nvme

 

[dbs22]

The LVM seems to be ok too with before the 2nd NVME was added and after in each series of pictures,

 

[dbs22]

I ended up using a pcie sata adaptor to add another drive and that worked without any issues, not the same performance as a x4 nvme drive, but sufficient

 

[superbunny54]

Hi ! I have exactly the same issue:
one nvme drive -> everything good,
add a second one, my pool is broken even tho the second disk is not event formatted and there is nothing on it
Did someone find a real solution?

some additional info:
like @dbs22 everything looks fine from LVM
my storage appear like normal on the proxmox gui.

the strange thing is, I can create new VM and the volume is created successfully:
scsi0: successfully created disk 'nvme1:vm-106-disk-0,iothread=1,size=32G,ssd=1'
checking with lvs | grep 106 show:
vm-106-disk-0 nvme1 Vwi-a-tz-- 32.00g nvme1

but when I try to start the vm:
kvm: -drive file=/dev/nvme1/vm-106-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on: Could not open '/dev/nvme1/vm-106-disk-0': Not a directory
TASK ERROR: start failed: QEMU exited with code 1

What has been tested so far:
- compared lvs vgs with and without second disk -> everything looks the same
- create lvm pool with the new drive -> nothing changed however note that this pool work and I can run VM with disks on it

I will eventually try to migrate my machine to the new pool, then recreate a pool with the original drive and copy everything back to it. ( this is a long process so I'd like to keep this option as a last resort )

 

[dbs22]

I switched to a sata drive and never found a solution, the same drive works fine in a windows machine.

 

[superbunny54]

I switched to a sata drive and never found a solution, the same drive works fine in a windows machine.

Thank you for your reply ! however I'd like to avoid changing drive !