One single zvol hosting all the VMs' images or individual zvols?

[Axl_fzetafs]

Hello,

When preparing VM's images is better to have all of them hosted on a one big zvol exposed through one iSCSI LUN or have as many zvols, and hence LUNs, as the number of VMs?
When I firstly worked on iSCSI/zvols/hypervisors (it was VMWare but it is not significant), 8 years ago, my boss wanted me to create one zvol for each VM and to send deltas to a backup server (via SSH).
I guess the reason was that we would be able to restore any snapshot of any machine independently.
By reading what people do and also Wasim Ahmed's book, I see that instead one single LUN is used for all the machines.

What would be the benefits of each solution? Can I achieve redundancy among PVE members when going for individual zvols?

Thanks,

Alex

 

[bbgeek17]

What is providing iSCSI and the ZFS behind it?

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Axl_fzetafs]

@bbgeek17 , It'll be TrueNAS

 

[bbgeek17]

Lets forget for a minute about ZFS in the backend. As far as PVE is concerned - all it sees is a raw iSCSI disk. There are a few things you can do with it:
HA: place thick LVM, use directly to pass to VM
NON-HA: place thick LVM, use directly to pass to VM, place thin LVM

LVM and Direct disk are seen as raw disks by the final VM. You will need to place a file system on that disk.
Finally, there will be an application that will actually generate the IO using some block size.

As you can see, by the time the IO gets back to ZFS layer on TrueNAS it will go through several layers: some are direct pass-through, others may combine/merge IO for optimization.
In short, its impossible to provide one size fits all guidance without analyzing the entire chain. Only you can do it.

As far as how you expose storage to PVE, i.e. one large LUN or many smaller ones: both are valid approaches, both have drawbacks and pluses.

Single LUN is a create it and forget it. You will need to LVM the resulting disk on PVE if you wish to share it between VMs. If you need HA, you will have to use Thick LVM and loose LVM snapshots. You could use backend ZFS, but it will be a single snapshot for entire LUN.

Multiple LUNs are more flexible, allow for easier HA, individual backend ZFS snapshots, more precise disk management. However, its a more manual approach in that for every new disk you will need to create a new LUN, expose it via iSCSI, etc. There are some plugins that are designed to help automate this, however you will have to decide how suitable they are for production.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Axl_fzetafs]

Many thanks @bbgeek17 , interesting reply. I need to double check a couple of things to better understand what you said. Especially thick-LVM and thin-LVM. It's true that I was exposed to LVM, but not that in depth. Thanks again.

Alex