pass through nvme OS drive and nvme storage?

[dankkster]

I have an old dell r720, and I added 2 nvme drives for a work vm I use (it windows 10 if that matters). It seems to still be sluggish (just like it was in esxi/RDP on two SSD drives) and I am thinking that perhaps the abstraction layers are getting in the way since I am not seeing any performance gains from the nvme hardware I added. While the older dell servers dont necessarily have m.2 ports, you can buy pcie adapters that the nvme drive goes into, and then you add it like a pcie card. these are single adapters and they are in 2 separate pcie slots.

Wouldn't passthrough be pretty much the same as pcie passthrough or is there something more that I might need to do?

Also, I wanted to make sure that if indeed, it is possible to pass these housed/adapted nvme drives, would I also be able to add the OS drive itself as well? Thanks for any help regarding this matter as I am coming from vmware esxi/vcs and finding this experience very refreshing to actually have hardware I can use without too much fuss.

Regards

 

[leesteken]

Yes, NVMe drives are PCIe devices. Whether you can get PCIe passthrough working depends on your motherboard and it's IOMMU groups (as you cannot share devices from the same group between VMs or the Proxmox host), the adapter you are going to buy and the devices themselves (whether they reset properly when you (re)start the VM).
If it's an older system, you might not see any gains as the PCIe generation and PCIe layout of the CPU(s) and motherboard might be the bottleneck.

 

[dankkster]

Fair enough... I will take a look again at the IOMMU grouping and make sure they are separated. I will read up on the link you added and see how it goes. Thanks for the help

 

[leesteken]

Fair enough... I will take a look again at the IOMMU grouping and make sure they are separated. I will read up on the link you added and see how it goes. Thanks for the help

See if you can find out if someone on the internet had success doing passthrough of that particular SATA controller. There are several threads on this forum about motherboard SATA controller not resetting properly and being useless for passthrough.

 

[dankkster]

See if you can find out if someone on the internet had success doing passthrough of that particular SATA controller. There are several threads on this forum about motherboard SATA controller not resetting properly and being useless for passthrough.

Oh great lol - Ill do that. Thanks for the heads up. I did find out they are separately grouped and I dont necessarily even care about bottlenecked speeds tbh. I just want it to be snappier and more responsive overall. I have this win10 vm running with 64GB of RAM and 10 cores of a dual xeon server. Something is getting in this vm's way to perform properly and it may have nothing at all to do with disk drives or abstraction layers at all. Whatever it is, it is persistent in VMWare and Proxmox.

 

[leesteken]

I just want it to be snappier and more responsive overall. I have this win10 vm running with 64GB of RAM and 10 cores of a dual xeon server. Something is getting in this vm's way to perform properly and it may have nothing at all to do with disk drives or abstraction layers at all. Whatever it is, it is persistent in VMWare and Proxmox.

VMs usually get snappier when you give it less cores and less memory. That way, it leaves more resources for Proxmox and background processing and emulation, so that will interfere less with the VM. Also enable NUMA if so the VM is aware of the dual socket (and give it two virtual sockets) and that memory might be remote to the CPU. Also use VirtIO SCSI single and enable IO Threat to more the disk I/O off the main emulation thread used for other virtual devices.

 

[dankkster]

the machine this is on has 256GB of memory and 2 10-core xeon 2680v2 2.8GHz. I'll dial it back as far as I need to for this machine to stop lagging so badly in RDP. having to wait upwards of 5 seconds for a menu to drop down from a right click seems excessive, however this may be expected behavior. Im no expert for sure. I will make those changes immediately and report back after a bit to let you know if any of them helped.

 

[dankkster]

I cant believe i didnt have numa enabled :| so that is done now. I changed it down to 2sock/4core from 2/5 on the cpu as well. I set ballooning for the RAM and set it to 4096/16384. I already use virtio, but not single and I am not sure how to change that as this drive was converted from a VMWARE drive.

Here are the lines for the OS nvme and data nvme respectively in the conf file.
virtio0: workstation:vm-200-disk-0,cache=writeback,discard=on,iothread=1,size=210G <--- OS
virtio1: stg-nvme:200/vm-200-disk-1.qcow2,discard=on,iothread=1,size=950G <---- Data

Anything here I should alter?

thank you for the advice!

 

[leesteken]

I cant believe i didnt have numa enabled :| so that is done now. I changed it down to 2sock/4core from 2/5 on the cpu as well. I set ballooning for the RAM and set it to 4096/16384. I already use virtio, but not single and I am not sure how to change that as this drive was converted from a VMWARE drive.

Here are the lines for the OS nvme and data nvme respectively in the conf file.
virtio0: workstation:vm-200-disk-0,cache=writeback,discard=on,iothread=1,size=210G <--- OS
virtio1: stg-nvme:200/vm-200-disk-1.qcow2,discard=on,iothread=1,size=950G <---- Data

Anything here I should alter?

Switch from VirtIO to SCSI for the virtual disks (detach and re-attach or something like that) and use the VirtIO SCSI Single controller and enable IO Thread for each virtual disk. I don't know how your VM will handle such a change.

 

[dankkster]

hm, I think the only way I can make that change now is to detach the drive and then manually add the controller type in the conf file and save. That will readd the drive with the proper controller I believe... I will look around for the syntax and hope I dont bork the VM in the process. I can say that the VM at the moment does appear to be somewhat snappier than it was before.

 

[leesteken]

... and then manually add the controller type in the conf file and save. That will readd the drive with the proper controller I believe... I will look around for the syntax and hope I dont bork the VM in the process.

This is all possible via the Proxmox GUI and you don't need to edit the VM configuration file. Set the SCSI controller type in Options. And re-add the drives under Hardware.

 

[dankkster]

I wasnt able to find anywhere in the options where I could change to VirtIO SCSI Single.

However, under Hardware, I was able to detach the drive and then edit it to let me choose VirtIO SCSI, but there wasnt an option for Single in there. With a little bit of headache I was able to boot to the OS. If you know where to go to select VirtIO SCSI Single. I would appreciate that as I couldnt figure that out.

--edit: I found a guide that did have Virtio SCSI Single selected and saw the conf file for it. I was able to change it that way.

I have not been able to pass through the OS drive as of yet, but I am still trying. I have made a few attempts that land me right in the EFI shell. I dont know if this is obvious or not, but I do already have an OS on the nvme and a files on the data drive if that matters.

I have tried adding to the conf file:
scsi0: /dev/disk/by-id/nvme-WDS500G3X0C-00SJG4_21025A701361
scsi1: /dev/disk/by-id/nvme-Samsung_SSD_970_EVO_1TB_S468NT0KA10754K

I have tried just adding:
scsi0: /dev/nvme0n1
scsi1: /dev/nvme1n1

None of these have worked though. Still hopeful I can get this sorted.

 

[leesteken]

Maybe I misremembered. Are you using an old version of Proxmox (pveversion -v)?

I think you should just change

virtio0: workstation:vm-200-disk-0,cache=writeback,discard=on,iothread=1,size=210G <--- OS
virtio1: stg-nvme:200/vm-200-disk-1.qcow2,discard=on,iothread=1,size=950G <---- Data

to
scsi0: workstation:vm-200-disk-0,cache=writeback,discard=on,iothread=1,size=210G scsi1: stg-nvme:200/vm-200-disk-1.qcow2,discard=on,iothread=1,size=950G
and update the VM boot order accordingly.

 

[dankkster]

Maybe I misremembered. Are you using an old version of Proxmox (pveversion -v)?

no, using latest version.

I think you should just change

to
scsi0: workstation:vm-200-disk-0,cache=writeback,discard=on,iothread=1,size=210G scsi1: stg-nvme:200/vm-200-disk-1.qcow2,discard=on,iothread=1,size=950G
and update the VM boot order accordingly.

I may or may not give that a shot, I ended up having to completelty reinstalll the vm because it started running so poorly after these changes. Matter of fact we will just call this one closed as I dont want any more headaches with it. Thanks for the help