Accessing SSD plugged into mobo SATAs from the VM

[Startide]

Hi gang,

So I got proxmox running on an NVME disk, I tried adding my ssds as a RAID1 ZFS, but it turns out that's just more storage for proxmox to use, and not an actual disk I can view and use in the VM (ubuntu). At least I couldn't make it work.

What's the proper way to go about this ?

I found this, but I am a bit confused with the instructions : https://pve.proxmox.com/wiki/Passthrough_Physical_Disk_to_Virtual_Machine_(VM)

Can someone help ? THX

 

[Dunuin]

I found this, but I am a bit confused with the instructions : https://pve.proxmox.com/wiki/Passthrough_Physical_Disk_to_Virtual_Machine_(VM)

Can someone help ? THX

I would also have posted a link to this. Whats the thing you don't understand there?

 

[Startide]

I would also have posted a link to this. Whats the thing you don't understand there?

that part :
qm set 592 -scsi2 /dev/disk/by-id/ata-ST3000DM001-1CH166_Z1F41BLC

Is 592 supposed to be the VM ID thing ?

 

[Dunuin]

qm set 592 -scsi2 /dev/disk/by-id/ata-ST3000DM001-1CH166_Z1F41BLC

"qm set" is the command.
"592" is the VMID you want to passthrough the disk into (but keep in mind that this is no physical passthrough, your VM is still working with a virtual disk that is mapped to the physical blockdevice)
"-scsi2" is the virtual port the VM should be using for this. So in this case "scsi2" (the third scsi slot) when virtio SCSI should be used. Make sure scsi2 is not aleady in use by the VM.
"/dev/disk/by-id/ata-ST3000DM001-1CH166_Z1F41BLC" is the path to the block device you want to passthrough. Don't use /dev/sda and so on but the output based on ls -la /dev/disk/by-id.

Also have a look at the manual for the qm command to understand how that command works: https://pve.proxmox.com/pve-docs/qm.1.html

 

[Startide]

> your VM is still working with a virtual disk that is mapped to the physical blockdevice)

Is that a bad thing ? What would be the alternative, use one of those PCIE SATA hardware controllers ?

 

[Dunuin]

> your VM is still working with a virtual disk that is mapped to the physical blockdevice)

Is that a bad thing ? What would be the alternative, use one of those PCIE SATA hardware controllers ?

Depends...
Additional overhead (but isn't that bad in my opinion), VM can't manage the power of the disk (can't send it to sleep/spindown), VM can't monitor the disks health via SMART (but PVe host can), not the real physical block size is reported, VM won't start with a failed disk, ...

Only option for the VM to access the real physical disk would be PCI passthrough. But in case of SATA that would mean passing through the whole SATA controller with all disks attached to it and usually this isn't possible without buying a dedicated HBA card because the SATA controller is part of the chipset and therefore sharing a IOMMU group with all the other onboard devices (and you can only passthrough whole IOMMU groups with all devices in it).

 

[Startide]

I assume the NVME disk counts as sata, despite the physical connector being different ?

The card is decently cheap on amazon, I guess I could buy one.


EDIT : ha, they don't often support raid it seems (well at least the ones I find rn)

 

[Dunuin]

Some M.2 SSDs are SATA, some are NVMe. SATA != NVMe

 

[Startide]

Bonus question : I am adding them with the qm command, I need one SCSI2 and one SCSI3 I assume ? Gotta bind them both ?

 

[Dunuin]

Bonus question : I am adding them with the qm command, I need one SCSI2 and one SCSI3 I assume ? Gotta bind them both ?

If you want to passthrough two disks and scsi2 and scsi3 aren't already in use by the VM, yes.

 

[Startide]

If you want to passthrough two disks and scsi2 and scsi3 aren't already in use by the VM, yes.

Indeed that works, I can see them both ! Now I need to set them as a raid1

 

[Startide]

If you want to passthrough two disks and scsi2 and scsi3 aren't already in use by the VM, yes.

Ok I got myself a raid card to make things clean (and future proof some extra storage).



What do I do so that they get setup as a raid array and viewable in my VM ? I need to mount them as a raid and then do the passthrough trick again ?

Found this online : https://www.linuxbabe.com/linux-server/linux-software-raid-1-setup

 

[Dunuin]

Then you need to PCI passthrough the raid card (and this might not be possible depending on the mainboard and PCIe slot used): https://pve.proxmox.com/wiki/PCI_Passthrough

 

[Startide]

I am getting an error when doing the Verify IOMMU isolation step :

 

[Startide]

Anyone got ideas of how I can passthrough that raid device ?