Best filesystem for small cluster?

[yaric]

Hi, I am new to Proxmox, and as many of us also tasked to create production cluster.

I have 2 identical servers and disks in 2 separate hardware RAIDS10, SSD and HDD separately (I dont trust software raids).

I did make choice not to use Ceph and keep servers in cluster more independent. One server down is better than 2. We can afford downtime 1h in case of serious server crash, but in case of distributed filesystem crash, downtime can take days to recover for small company without secondary cluster. Therefore 1 server runs VMs, second is on standby with replicated images.

So installer silently created LVM and LVS-thin storage. Great! All I need to do is place my image and setup replication, how convenient! Well, not so fast, it appears replication is not supported neither at file based storage, nor in LVS-thin?! To have that I have to setup ZFS.

And then I like... DUH... so why did you setup me LVM+LVM-thin in first place??? If installer used non-LVM partition, it will be easy for me just to add ZFS to unpartitioned space, or why installer not installed ProxMox directly into ZFS?

What is the proper action now for this setup? Migrate to ZFS? Reinstall?
Or keep LVS and setup replication of volumes manually?

Thank you.

 

[esi_y]

Hi, I am new to Proxmox, and as many of us also tasked to create production cluster.

I have 2 identical servers and disks in 2 separate hardware RAIDS10, SSD and HDD separately (I dont trust software raids).

I did make choice not to use Ceph and keep servers in cluster more independent. One server down is better than 2.

Proxmox VE is really not meant to use 2 nodes alone as a "cluster", especially not with High Availability. The cluster needs to keep a virtually synchronised configuration state and there's no concept of primary / secondary, instead a quorum (you can think of as "majority") of nodes is required to have the cluster manageable (VMs would not go down, but operations would be block upon lost quorum). As you can imagine, if you have total of 2 members, the majority cannot be 1. Such cluster would always be inquorate when one of the nodes is down.

I suggest to read on the cluster network [1] and beyond (e.g. some corosync related information [2]). What might be interesting for you is the "Corosync External vote support" (i.e. Q device) [1], if that's not possible, some alternative configuration options [2], however these (two_node, auto_tie_breaker, wait_for_all) are not officially something Proxmox will be happy to provide support on production setup.

We can afford downtime 1h in case of serious server crash, but in case of distributed filesystem crash, downtime can take days to recover for small company without secondary cluster. Therefore 1 server runs VMs, second is on standby with replicated images.

You always have the option of running 2 standalone instances of PVE, i.e. not a cluster.

So installer silently created LVM and LVS-thin storage. Great! All I need to do is place my image and setup replication, how convenient! Well, not so fast, it appears replication is not supported neither at file based storage, nor in LVS-thin?! To have that I have to setup ZFS.

Sadly, PVE only support replication on ZFS and you have mentioned above that you like hardware RAIDs, in that respect, I can only recommend to read on that [3] too before making further decisions.

And then I like... DUH... so why did you setup me LVM+LVM-thin in first place??? If installer used non-LVM partition, it will be easy for me just to add ZFS to unpartitioned space, or why installer not installed ProxMox directly into ZFS?

You can absolutely run your own way, e.g. I do not even like ZFS on root (but can create ZFS pool later). Or you can install on top of Debian [4], which gives you much greater flexibility.

What is the proper action now for this setup? Migrate to ZFS? Reinstall?

I would start by deciding if I want a PVE cluster (2 nodes + Q device) or 2 standalone PVE installations. And consider the ZFS on top of HW RAID topic seriously as well. Others will take me apart here at this last remark, but to answer your title question - I would run 2 separate i.e. non-cluster setups, keep my HW RAID and roll XFS. Disaster recovery with 1h time to spare should be fine with backups.

PS There is a CLI tool for external host migration, considered "Technology Preview", so if you do not want to migrate through backups and are happy to accept its status (I have not run into issues with it and it's been around since v7), see threads regarding qm remote-migrate, e.g. [5].

[1] https://pve.proxmox.com/wiki/Cluster_Manager
[2] https://manpages.debian.org/bookworm/corosync/votequorum.5.en.html
[3] https://openzfs.org/wiki/System_Administration#Hardware_RAID
[4] https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm
[5] https://forum.proxmox.com/threads/m...n-hosts-too-big-to-backup.152017/#post-689203

 

[waltar]

You can indeed snapshot your lvm/lvm-thin hw-raid storage and replicate the snapshots to the second server as with zfs. DR time just depends on your waiting time until you press bulk start on DR server and vm/lxc image actuality depends on your snapshot and replication rythmus.

 

[yaric]

Proxmox VE is really not meant to use 2 nodes alone as a "cluster", especially not with High Availability. The cluster needs to keep a virtually synchronised configuration state and there's no concept of primary / secondary, instead a quorum (you can think of as "majority") of nodes is required to have the cluster manageable (VMs would not go down, but operations would be block upon lost quorum). As you can imagine, if you have total of 2 members, the majority cannot be 1. Such cluster would always be inquorate when one of the nodes is down.

I suggest to read on the cluster network [1] and beyond (e.g. some corosync related information [2]). What might be interesting for you is the "Corosync External vote support" (i.e. Q device) [1], if that's not possible, some alternative configuration options [2], however these (two_node, auto_tie_breaker, wait_for_all) are not officially something Proxmox will be happy to provide support on production setup.



You always have the option of running 2 standalone instances of PVE, i.e. not a cluster.



Sadly, PVE only support replication on ZFS and you have mentioned above that you like hardware RAIDs, in that respect, I can only recommend to read on that [3] too before making further decisions.



You can absolutely run your own way, e.g. I do not even like ZFS on root (but can create ZFS pool later). Or you can install on top of Debian [4], which gives you much greater flexibility.



I would start by deciding if I want a PVE cluster (2 nodes + Q device) or 2 standalone PVE installations. And consider the ZFS on top of HW RAID topic seriously as well. Others will take me apart here at this last remark, but to answer your title question - I would run 2 separate i.e. non-cluster setups, keep my HW RAID and roll XFS. Disaster recovery with 1h time to spare should be fine with backups.

PS There is a CLI tool for external host migration, considered "Technology Preview", so if you do not want to migrate through backups and are happy to accept its status (I have not run into issues with it and it's been around since v7), see threads regarding qm remote-migrate, e.g. [5].

[1] https://pve.proxmox.com/wiki/Cluster_Manager
[2] https://manpages.debian.org/bookworm/corosync/votequorum.5.en.html
[3] https://openzfs.org/wiki/System_Administration#Hardware_RAID
[4] https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm
[5] https://forum.proxmox.com/threads/m...n-hosts-too-big-to-backup.152017/#post-689203

Thank you, this is brilliant and extremely informative.