Hi,
the idea is to have 2 Storage VMs in a dedicated pacemaker/corosync cluster which re-exports the SAN block storage via NFS, ZFS over iSCSI or LVMThin over iSCSI back to the PVE cluster nodes
why ? - beside cluster file systems like GFS2/OCFS2 there are imho not many working open source solutions which support thin provisioning, snapshots and HA on SANs in a storage vendor agnostic / PVE (host) centric way
Please correct me if I'm wrong in the following statements/configs ! this is a draft/evaluation - don't run in production !
sample setup:
- 2 PVE node cluster + qdevice - existing storage: iSCSI SAN
- 2 x 25Gbit per PVE node via DAC connected - switchless (for frontend protocol traffic)
- for more then 2 PVE nodes in a cluster -> 10Gbit/25Gbit switch
- Storage VM corosync on the same link(s) as for PVE corosync + the same qdevice can be used/configured as for the PVE cluster
- 2 (e.g.) Debian/Ubuntu Storage VMs running on 2 PVE nodes - only one Storage VM may run on 1 PVE node at any time
- the Storage VMs get access to the shared SAN LUN(s) - there are multiple access possibilities at least for iscsi backend -> tbd: direct LUN access, LUN(scsi) passthru, shared thick LV + multipath
- pacemaker/corosync Storage VM cluster configured
front end protocol to the PVE nodes:
Concept 1 - NFS (kernel) server within Storage VM HA cluster
pros:
- already existing PVE NFS client/plugin
- together with qcow2 file format it provides thin provisioning and snapshots
- easy to setup and configure with pacemaker for HA
- fast cluster failover possible (within seconds - depending on workload and for v3 only)
- good documentation especially in combination with pacemaker clusters
- works with nearly every Linux filesystem
cons:
- only working in active/passive mode ... it's not possible to run 2 or more NFS (kernel) instances on one node (without special configs and containers) -> ev. solution: NFS Ganesha (user space) for active/active instances/storage pools
- NFS v4.x not really cluster/failover friendly (see grace/lease periods) - so focus on "legacy" NFSv3 and therefore no session trunking and other new v4.x features available
- bridging/converting a block storage protocol to NFS can maybe slower then iscsi <-> iscsi ... tbc
- internal qcow2 snapshots can freeze VMs during snapshot creation/deletion but with snapshots as volume chains this should be solved
- only sync option is save - which costs performance
Concept 2 - ZFS over iSCSI server within Storage VM HA cluster
pros:
- nearly all the power and features of ZFS but in a shared HA storage config .... thin provisioning, performant snapshots, compression - you name it ....
- iSCSI as front-end protocol - ev. more performant then NFS ... but with ZFS as file backend and sync=always not the case anymore ?!
- already existing/integrated PVE plugin
- multiple ZFS over iSCSI instances / Pools possible ("active/active")
neutral:
- ZFS over iSCSI works currently only with LIO as (Linux) iSCSI backend ... all other backends are obsolete .. SPDK or SCST would be interesting extensions to the plugin
cons:
- you have to run ZFS inside the VM .... at least from a performance view it needs more resources then other filesystems
- the ZFS over iSCSI "SSH API" is not compatible with the way pacemaker handles LIO configuration via resource agents - it's resolvable but needs more investigation
- not a real API existing ... ssh via root + shell commands to the VM Storage server(s)
- no multipathing in the QEMU iscsi initiator available
- imho would only make sense with dedicated NVMe/SSD SLOGs on the SAN -> sync=always necessary
- ZFS can detect but cannot automatically heal corrupted file blocks without ZFS mirror or parity disks/LUNs
Concept 3 - LVMThin over iSCSI server within Storage VM HA cluster
pros:
- the lightweight and maybe more performant version to ZFS over iSCSI
cons:
- less features then ZFS but extendable with different dm-targets
- no PVE Storage plugin exists currently - feature request sent - could work similiar as ZFS over iSCSI plugin with LIO and LVM(thin)
- see also the other cons on ZFS over iSCSI (multipath, ssh API)
the idea is to have 2 Storage VMs in a dedicated pacemaker/corosync cluster which re-exports the SAN block storage via NFS, ZFS over iSCSI or LVMThin over iSCSI back to the PVE cluster nodes
why ? - beside cluster file systems like GFS2/OCFS2 there are imho not many working open source solutions which support thin provisioning, snapshots and HA on SANs in a storage vendor agnostic / PVE (host) centric way
Please correct me if I'm wrong in the following statements/configs ! this is a draft/evaluation - don't run in production !
sample setup:
- 2 PVE node cluster + qdevice - existing storage: iSCSI SAN
- 2 x 25Gbit per PVE node via DAC connected - switchless (for frontend protocol traffic)
- for more then 2 PVE nodes in a cluster -> 10Gbit/25Gbit switch
- Storage VM corosync on the same link(s) as for PVE corosync + the same qdevice can be used/configured as for the PVE cluster
- 2 (e.g.) Debian/Ubuntu Storage VMs running on 2 PVE nodes - only one Storage VM may run on 1 PVE node at any time
- the Storage VMs get access to the shared SAN LUN(s) - there are multiple access possibilities at least for iscsi backend -> tbd: direct LUN access, LUN(scsi) passthru, shared thick LV + multipath
- pacemaker/corosync Storage VM cluster configured
front end protocol to the PVE nodes:
Concept 1 - NFS (kernel) server within Storage VM HA cluster
pros:
- already existing PVE NFS client/plugin
- together with qcow2 file format it provides thin provisioning and snapshots
- easy to setup and configure with pacemaker for HA
- fast cluster failover possible (within seconds - depending on workload and for v3 only)
- good documentation especially in combination with pacemaker clusters
- works with nearly every Linux filesystem
cons:
- only working in active/passive mode ... it's not possible to run 2 or more NFS (kernel) instances on one node (without special configs and containers) -> ev. solution: NFS Ganesha (user space) for active/active instances/storage pools
- NFS v4.x not really cluster/failover friendly (see grace/lease periods) - so focus on "legacy" NFSv3 and therefore no session trunking and other new v4.x features available
- bridging/converting a block storage protocol to NFS can maybe slower then iscsi <-> iscsi ... tbc
- internal qcow2 snapshots can freeze VMs during snapshot creation/deletion but with snapshots as volume chains this should be solved
- only sync option is save - which costs performance
Concept 2 - ZFS over iSCSI server within Storage VM HA cluster
pros:
- nearly all the power and features of ZFS but in a shared HA storage config .... thin provisioning, performant snapshots, compression - you name it ....
- iSCSI as front-end protocol - ev. more performant then NFS ... but with ZFS as file backend and sync=always not the case anymore ?!
- already existing/integrated PVE plugin
- multiple ZFS over iSCSI instances / Pools possible ("active/active")
neutral:
- ZFS over iSCSI works currently only with LIO as (Linux) iSCSI backend ... all other backends are obsolete .. SPDK or SCST would be interesting extensions to the plugin
cons:
- you have to run ZFS inside the VM .... at least from a performance view it needs more resources then other filesystems
- the ZFS over iSCSI "SSH API" is not compatible with the way pacemaker handles LIO configuration via resource agents - it's resolvable but needs more investigation
- not a real API existing ... ssh via root + shell commands to the VM Storage server(s)
- no multipathing in the QEMU iscsi initiator available
- imho would only make sense with dedicated NVMe/SSD SLOGs on the SAN -> sync=always necessary
- ZFS can detect but cannot automatically heal corrupted file blocks without ZFS mirror or parity disks/LUNs
Concept 3 - LVMThin over iSCSI server within Storage VM HA cluster
pros:
- the lightweight and maybe more performant version to ZFS over iSCSI
cons:
- less features then ZFS but extendable with different dm-targets
- no PVE Storage plugin exists currently - feature request sent - could work similiar as ZFS over iSCSI plugin with LIO and LVM(thin)
- see also the other cons on ZFS over iSCSI (multipath, ssh API)
Last edited: