Best approach for a TrueNAS VM

[zecas]

Hi,

I'm thinking about setting a truenas VM on a proxmox server and scenarios which I can choose for that job.

On proxmox installation I have a ZFS raid10 pool for VM storage.

I would create a VM for the truenas, but what would be the best option for the truenas storage?

Scenario #1:​

One possibility would be to add say 4 disks to the system, pass them to the truenas VM (pass-through disks by their ids).

I would then be able to set the truenas storage on a ZFS raid10, for instance.

With this approach:
1- I would have to set enough memory on the truenas VM in order to manage the ZFS data;
2- I would require 4 additional disks for this storage.

The advantage is that I would have physical disks directly associated with truenas storage.

Scenario #2:​

Since I have my proxmox configured with a storage pool on a ZFS raid10, I could just add a single virtual disk to truenas, since it would already be stored in that ZFS pool.

With this solution:
1- I need no additional disks;
2- Truenas memory requirements would be less demanding, since the proxmox server is the one maintaining the main ZFS pool, as it does already.



Do you see any other scenario better than any of the above ones?
Or would you discard any of the above solutions?


Thank you.

 

[Dunuin]

The advantage is that I would have physical disks directly associated with truenas storage.

No. With disk pasthrough your TrueNAS is still using virtual disks. To be able to directly access the real physical disks you need to PCI passthrough a dedicated HBA. With disk passthrough there is still virtualization overhead and stuff like SMART won't work.

Since I have my proxmox configured with a storage pool on a ZFS raid10, I could just add a single virtual disk to truenas, since it would already be stored in that ZFS pool.

That way you get ZFS on top of ZFS. ZFS got terrible overhead and this overhead will multiply, not add up. Also, with a ZFS pool 20% of space should be kept free for best performance and less fragmentation. So with ZFS on top of ZFS you get 80% of 80% = only 64% of capacity.

2- Truenas memory requirements would be less demanding, since the proxmox server is the one maintaining the main ZFS pool, as it does already.

Not really. Both ZFS pools would need a ARC in RAM. So more wasted memory.

 

[zecas]

Thank you for your quick reply.

No. With disk pasthrough your TrueNAS is still using virtual disks. To be able to directly access the real physical disks you need to PCI passthrough a dedicated HBA. With disk passthrough there is still virtualization overhead and stuff like SMART won't work.

With disk passthrough I was thinking that I could just remove the disks and connect to another PC in case of such need, and would be able to re-mount the storage as they are all part of the same ZFS pool. But you're right, having them set as virtual disks, I would miss smart data on truenas side.

I wasn't thinking about PCI passthrough because, since I have all disks connected on a backplane, they all share the same controller and same backplane connectors. Adding another HBA controller would be of course possible and I'll take some time considering it, but I have to figure out a way of physically install and connect the disks to that additional HBA controller.

That way you get ZFS on top of ZFS. ZFS got terrible overhead and this overhead will multiply, not add up. Also, with a ZFS pool 20% of space should be kept free for best performance and less fragmentation. So with ZFS on top of ZFS you get 80% of 80% = only 64% of capacity.

Not really. Both ZFS pools would need a ARC in RAM. So more wasted memory.

Yes I understand what you're saying, I would not want to have ZFS on top of ZFS.

When I thought about adding a single virtual disk, I was thinking of using it inside truenas as a single disk with no ZFS nor anything to have some redundancy, because the ZFS on proxmox would already have that responsibility.

So my truenas VM would have one disk for the OS and a second disk for the storage, all of them stored on proxmox ZFS pool. It would be just a simple way of "sharing" that single disk available on truenas but with all features that truenas can provide.

 

[Dunuin]

When I thought about adding a single virtual disk, I was thinking of using it inside truenas as a single disk with no ZFS nor anything to have some redundancy, because the ZFS on proxmox would already have that responsibility.

Not sure about TrueNAS SCALE, but TrueNAS Core only works with ZFS. So you are forced to use ZFS inside it.

 

[zecas]

Not sure about TrueNAS SCALE, but TrueNAS Core only works with ZFS. So you are forced to use ZFS inside it.

Oh, ok ... that changes the game around. I'll take a better look at that, but suddenly the PCI passthrough starts looking as a better approach...
Thanks.

 

[Dunuin]

Oh, ok ... that changes the game around. I'll take a better look at that, but suddenly the PCI passthrough starts looking as a better approach...
Thanks.

Jup, PCI passthrough should be the best approach as long as you don't want to migrate that TrueNAS VM between PVE nodes. But your hardware has to support it. If you can't do that (for example because your backplane won't support multiple HBAs or your mainboard won't offer isolated IOMMU groups) I would try it with disk passthrough. Disk passthrough wouldn`t be that great, but is working for other people here (if I remeber right you will have to pass the disks with some additional arguments because TrueNAS doesn't like virtual disks without unique IDs). And still better than running ZFS on top of ZFS.