[SOLVED] Best practice for Truenas virtualized on Proxmox

[Lancer]

Hello everyone

I am planning to use Truenas as a VM inside Proxmox.
My planned usage is hosting handful of VMs and containers, and I want to share some files and do backups.
Also I love ZFS !
But I did read that write amplification will hurt your ssd if you install ZFS vm inside ZFS hypervisor.

So after searching and reading, I plan to:
1. Install Proxmox on my first ssd, with ZFS as file system (I know about mirror but I don't have much disks or money, Insted I am planning to do daily backups of proxmox to another disk).
2. Install Truenas as a VM on the second ssd which will be formatted as ext4 maybe. also maybe will use it for iso storage and vm's backups.
3. Use my last ssd's for the rest of VMs & backups.

Notes:
- I will use Sun F80 which contain four independent ssd drives on a pcie card with PLP.
- I don't have the luxury to do mirrors or raidz on the meantime, maybe in future. also I think doing zfs mirror on f70 is not a real mirror due to chance of card failure.
- As for backups: I hope to do daily backups of Proxmox and Truenas to a raidz on some spinning rust soon.

So is this the best direction?
And is it the best way of enjoying ZFS without much risks?
If not, Please give me your suggesions/Ideas.

Thank you all

 

[shanreich]

You could also passthrough the disks [1] you want to use for TrueNAS directly, so TrueNAS can handle the real disks for you.

[1] https://pve.proxmox.com/wiki/Passthrough_Physical_Disk_to_Virtual_Machine_(VM)

 

[Dunuin]

You could also passthrough the disks [1] you want to use for TrueNAS directly, so TrueNAS can handle the real disks for you.

Not really the "real" disks. TrueNAS will still handle QEMU virtual disks with that. For real physical disk access PCI passthrough of a NVMe SSD or HBA is required: https://pve.proxmox.com/wiki/PCI_Passthrough
But still better than ZFS on top of ZFS or nested filesystems.
When using Disk Passthrough make sure to set a Serial for the QEMU disk as TrueNAS will complain otherwise.

 

[Lancer]

You could also passthrough the disks [1] you want to use for TrueNAS directly, so TrueNAS can handle the real disks for you.

[1] https://pve.proxmox.com/wiki/Passthrough_Physical_Disk_to_Virtual_Machine_(VM)

Yes this is what I intended, in fact I want to pass my hba controller to Truenas.

Anyway I hope that means everything else is OK : )

 

[Lancer]

Not really the "real" disks. TrueNAS will still handle QEMU virtual disks with that. For real physical disk access PCI passthrough of a NVMe SSD or HBA is required: https://pve.proxmox.com/wiki/PCI_Passthrough
But still better than ZFS on top of ZFS or nested filesystems.
When using Disk Passthrough make sure to set a Serial for the QEMU disk as TrueNAS will complain otherwise.

Thanks for this advice.
Just to be sure: I think my plan above will not involve (ZFS on top of ZFS) since Truenas will be on another drive.
Also I did understand from your reply that giving truenas my whole drive is better than making an ext4 drive then make a virtual drive inside it for Truenas.

Right ?

 

[Dunuin]

Thanks for this advice.
Just to be sure: I think my plan above will not involve (ZFS on top of ZFS) since Truenas will be on another drive.

Yes.

Also I did understand from your reply that giving truenas my whole drive is better than making an ext4 drive then make a virtual drive inside it for Truenas.


Right ?

Yes, because you avoid nested filesystems as the ext4 and qcow2 would add unneccessary additional overhead.

Yes this is what I intended, in fact I want to pass my hba controller to Truenas.

Keep in mind that you:
a) can only passthrough whole IOMMU groups, so the HBA needs to be connected to a PCIe port that is directly connected to the CPU and not to the chipset as it otherwise probably wouldn't be alone in a IOMMU group and stuff like NICs, USB, onboard disk controller and so on might be passthroughed too.
b) can only passthrough the whole HBA with all of its ports. So all disks attached to it will be passed through and won't be availalble for the host or other VMs anymore

You don't have this limitations when using disk passthrough instead of PCI passthrough of the HBA, but then TrueNAS will only work with virtual disks and not the real physical disks (so TrueNAS can't monitor the disk health, can't send them to sleep and so on).

 

[Pandalorian]

The thing I don't understand is if your HBA handles all of your drives, won't giving TrueNAS exclusive access to the HBA mean giving over the drive(s) that Proxmox is installed on?

 

[Dunuin]

The thing I don't understand is if your HBA handles all of your drives, won't giving TrueNAS exclusive access to the HBA mean giving over the drive(s) that Proxmox is installed on?

Yes, thats why you usually buy a HBA card to passthrough, so you don't have to passthrough the onboard disk controller where your system disks might be attached to. Or you install PVE on NVMe SSDs that aren'T using the disk controller anyway.

 

[JensF]

And just keep in mind that the passed through disks are not in the Proxmox backup.

 

[spacemonQee]

And just keep in mind that the passed through disks are not in the Proxmox backup.

how would you go about and backup those?

 

[Johannes S]

how would you go about and backup those?

With a backup tool inside the VM. In case of truenas it makes sense to utilize it's own included backup/sync/replication tasks. But in the end you can use any backup tool as long as it gets the job done. I use restic together with resticprofile inside a docker container to backup my nas data to a hetzner storagebox and the native sync/backup tasks to sync my nas data to an USB drive.. borgbackup, rsync or Proxmox own backup client would also do the job in both cases.