[SOLVED] Best practice for Truenas virtualized on Proxmox

[Lancer]

Hello everyone

I am planning to use Truenas as a VM inside Proxmox.
My planned usage is hosting handful of VMs and containers, and I want to share some files and do backups.
Also I love ZFS !
But I did read that write amplification will hurt your ssd if you install ZFS vm inside ZFS hypervisor.

So after searching and reading, I plan to:
1. Install Proxmox on my first ssd, with ZFS as file system (I know about mirror but I don't have much disks or money, Insted I am planning to do daily backups of proxmox to another disk).
2. Install Truenas as a VM on the second ssd which will be formatted as ext4 maybe. also maybe will use it for iso storage and vm's backups.
3. Use my last ssd's for the rest of VMs & backups.

Notes:
- I will use Sun F80 which contain four independent ssd drives on a pcie card with PLP.
- I don't have the luxury to do mirrors or raidz on the meantime, maybe in future. also I think doing zfs mirror on f70 is not a real mirror due to chance of card failure.
- As for backups: I hope to do daily backups of Proxmox and Truenas to a raidz on some spinning rust soon.

So is this the best direction?
And is it the best way of enjoying ZFS without much risks?
If not, Please give me your suggesions/Ideas.

Thank you all

 

[shanreich]

You could also passthrough the disks [1] you want to use for TrueNAS directly, so TrueNAS can handle the real disks for you.

[1] https://pve.proxmox.com/wiki/Passthrough_Physical_Disk_to_Virtual_Machine_(VM)

 

[Dunuin]

You could also passthrough the disks [1] you want to use for TrueNAS directly, so TrueNAS can handle the real disks for you.

Not really the "real" disks. TrueNAS will still handle QEMU virtual disks with that. For real physical disk access PCI passthrough of a NVMe SSD or HBA is required: https://pve.proxmox.com/wiki/PCI_Passthrough
But still better than ZFS on top of ZFS or nested filesystems.
When using Disk Passthrough make sure to set a Serial for the QEMU disk as TrueNAS will complain otherwise.

 

[Lancer]

You could also passthrough the disks [1] you want to use for TrueNAS directly, so TrueNAS can handle the real disks for you.

[1] https://pve.proxmox.com/wiki/Passthrough_Physical_Disk_to_Virtual_Machine_(VM)

Yes this is what I intended, in fact I want to pass my hba controller to Truenas.

Anyway I hope that means everything else is OK : )

 

[Lancer]

Not really the "real" disks. TrueNAS will still handle QEMU virtual disks with that. For real physical disk access PCI passthrough of a NVMe SSD or HBA is required: https://pve.proxmox.com/wiki/PCI_Passthrough
But still better than ZFS on top of ZFS or nested filesystems.
When using Disk Passthrough make sure to set a Serial for the QEMU disk as TrueNAS will complain otherwise.

Thanks for this advice.
Just to be sure: I think my plan above will not involve (ZFS on top of ZFS) since Truenas will be on another drive.
Also I did understand from your reply that giving truenas my whole drive is better than making an ext4 drive then make a virtual drive inside it for Truenas.

Right ?

 

[Dunuin]

Thanks for this advice.
Just to be sure: I think my plan above will not involve (ZFS on top of ZFS) since Truenas will be on another drive.

Yes.

Also I did understand from your reply that giving truenas my whole drive is better than making an ext4 drive then make a virtual drive inside it for Truenas.


Right ?

Yes, because you avoid nested filesystems as the ext4 and qcow2 would add unneccessary additional overhead.

Yes this is what I intended, in fact I want to pass my hba controller to Truenas.

Keep in mind that you:
a) can only passthrough whole IOMMU groups, so the HBA needs to be connected to a PCIe port that is directly connected to the CPU and not to the chipset as it otherwise probably wouldn't be alone in a IOMMU group and stuff like NICs, USB, onboard disk controller and so on might be passthroughed too.
b) can only passthrough the whole HBA with all of its ports. So all disks attached to it will be passed through and won't be availalble for the host or other VMs anymore

You don't have this limitations when using disk passthrough instead of PCI passthrough of the HBA, but then TrueNAS will only work with virtual disks and not the real physical disks (so TrueNAS can't monitor the disk health, can't send them to sleep and so on).

 

[Pandalorian]

The thing I don't understand is if your HBA handles all of your drives, won't giving TrueNAS exclusive access to the HBA mean giving over the drive(s) that Proxmox is installed on?

 

[Dunuin]

The thing I don't understand is if your HBA handles all of your drives, won't giving TrueNAS exclusive access to the HBA mean giving over the drive(s) that Proxmox is installed on?

Yes, thats why you usually buy a HBA card to passthrough, so you don't have to passthrough the onboard disk controller where your system disks might be attached to. Or you install PVE on NVMe SSDs that aren'T using the disk controller anyway.

 

[JensF]

And just keep in mind that the passed through disks are not in the Proxmox backup.

 

[spacemonQee]

And just keep in mind that the passed through disks are not in the Proxmox backup.

how would you go about and backup those?

 

[Johannes S]

how would you go about and backup those?

With a backup tool inside the VM. In case of truenas it makes sense to utilize it's own included backup/sync/replication tasks. But in the end you can use any backup tool as long as it gets the job done. I use restic together with resticprofile inside a docker container to backup my nas data to a hetzner storagebox and the native sync/backup tasks to sync my nas data to an USB drive.. borgbackup, rsync or Proxmox own backup client would also do the job in both cases.

 

[curana]

I am in a similar situation and would like to add a few questions for clarification.

Scenario: I am running Proxmox on 2 NVMes with ZFS as the filesystem. On this pool I have a virtual disk which is the filesystem for TrueNAS, also using ZFS. I plan to add more HDDs to the system, most likely to be managed by TrueNAS.

1. I understood that I should not nest filesystems at all (even ext4) or just not ZFS? Because a virtual disk enforces nested filesystems by nature, no?
2. Today I added one of the HDDs via passthrough to the TrueNAS VM and use ZFS on it. I understood, that it would be better to passthrough the complete SATA controller instead and let TrueNAS handle the disks? This would mean the HDDs I connect to the Proxmox host would not be accessible by Proxmox anymore. The SATA controller is listed in the dropdown for PCI passthrough, not sure if it would actually work.
3. Regarding Backup I realized that snapshots are not possible with Proxmox anymore (because of the passthrough), a backup via Proxmox would only include the virtual harddrive running TrueNAS itself (in my case on the NVMes).

What would you recommend? Is nested ZFS just an overhead in resources or is there a risk going along with it?

Thanks for your help.

 

[FernandoRD]

I've been using OpenMediaVault with Proxmox for a while, and I'm thinking start using TrueNas... Is it mandatory to use passthrough? Or it is only a matter of performance? Because OpenMediaVault is working very well without it, in fact I'm using it with Ceph to have redundancy... And I Want to keep this way with TrueNas...

 

[Johannes S]

I've been using OpenMediaVault with Proxmox for a while, and I'm thinking start using TrueNas... Is it mandatory to use passthrough? Or it is only a matter of performance? Because OpenMediaVault is working very well without it, in fact I'm using it with Ceph to have redundancy... And I Want to keep this way with TrueNas...

Not mandatory but highly recommended. The issue is, that the NAS OS won't access the disks otherwise. Even with disk passthrough there is some abstraction layer between them so the NAS OS can't access them for S.M.A.R.T-Monitoring and other functions depending on direct access.
You can search the forum for earlier discussions on this (they are for TrueNAS, but the argument is true for every NAS OS):
https://forum.proxmox.com/threads/passing-through-disks-to-truenas-vm-properly.109804/
https://forum.proxmox.com/threads/truenas-on-proxmox-is-passthrough-to-hds-needed.134102/