hardware RAID; ZFS and replication

0

0x2c4

New Member
Jul 16, 2023
8
2
1
hi,

I am aiming to have replication for my VMs set on my proxmox cluster. Currently the file system is ext4 for VM drives and ZSF for boot drives, so I will be changing this to ZFS but...I realized that ZFS requires from me to set a software RAID...I have a hardware RAID on the drives (RAID1 on two boot drives and RAID 5 on 6 VM drives), In my opinion hardware RAID is much better than software, but maybe I am wrong?

What should I do to achieve the best result, should I disable the hardware RAID and set up software RAID with ZSF and then get replication, or is there a way to get replication with hardware RAID and maybe XFS....?

Thanks is advance.
 
  • Like
Reactions: Johannes S and UdoB
hi @Lukas Moravek

Thank you so much for responding, yes I do use SAS SSD Enterprise grade.
Thank you for the links, I will have a read.

Thank you

MM
 
Thanks to Lukas for the very helpful links. I'm delving into the HA and replication aspects of ProxMox, and today I was doubly shocked!

1°. ProxMox only supports ZFS for replication between two servers; otherwise, you'll need a specific high-end like Blockbridge storage solution (which also makes coffee, given the cost).
2. Many users here say ZFS is better than hardware RAID.

I won't argue much about the second point because, IMHO, it's like asking whether it's better to have virtual or real sex. The difference lies in whether you accept a potential risk of bit rot but have hardware acceleration, or rely on software (in this area, I see that if everything were software-based, without 3D graphics cards, the world would be different today). Of course, with the skyrocketing cost of RAM these days, perhaps a CTRL RAID costs less than a 32GB stick of ECC Reg RAM, which is necessary, given that ZFS uses system RAM to perform its calculations, and the more parity you need, like RaidZ2, the more it requires.
What I can't tolerate is that ProxMox doesn't replicate between two hardware disks connected in directory mode -> Hardware RAID Group <- and replication is not possible. For me, this is a major limitation, because it's getting out of hand on what's better, espousing the argument that software redundancy is better than hardware redundancy. With the old competition, which now has exorbitant prices, you could either use vCenter with the license that supports the replicator, or backup software with replication (Veeam, Nakivo, Hornet) that works via snapshots and detects changed blocks.
I see that Veeam R&D is interested in the matter, not that this software drives me crazy because PBS works very well on ProxMox, I don't know... maybe this could be used as a starting point to replicate the datastore from one side to the other, but I'm not a developer so I'm waiting with hope that something will move as it did when the success of this product occurred after reckless commercial choices for VMWare.
 
Frank-75 said:
Thanks to Lukas for the very helpful links. I'm delving into the HA and replication aspects of ProxMox, and today I was doubly shocked!

1°. ProxMox only supports ZFS for replication between two servers; otherwise, you'll need a specific high-end like Blockbridge storage solution (which also makes coffee, given the cost).
2. Many users here say ZFS is better than hardware RAID.

I won't argue much about the second point because, IMHO, it's like asking whether it's better to have virtual or real sex. The difference lies in whether you accept a potential risk of bit rot but have hardware acceleration, or rely on software (in this area, I see that if everything were software-based, without 3D graphics cards, the world would be different today). Of course, with the skyrocketing cost of RAM these days, perhaps a CTRL RAID costs less than a 32GB stick of ECC Reg RAM, which is necessary, given that ZFS uses system RAM to perform its calculations, and the more parity you need, like RaidZ2, the more it requires.
What I can't tolerate is that ProxMox doesn't replicate between two hardware disks connected in directory mode -> Hardware RAID Group <- and replication is not possible. For me, this is a major limitation, because it's getting out of hand on what's better, espousing the argument that software redundancy is better than hardware redundancy. With the old competition, which now has exorbitant prices, you could either use vCenter with the license that supports the replicator, or backup software with replication (Veeam, Nakivo, Hornet) that works via snapshots and detects changed blocks.
I see that Veeam R&D is interested in the matter, not that this software drives me crazy because PBS works very well on ProxMox, I don't know... maybe this could be used as a starting point to replicate the datastore from one side to the other, but I'm not a developer so I'm waiting with hope that something will move as it did when the success of this product occurred after reckless commercial choices for VMWare.
Click to expand...

Yes, I would seriously recommend giving Proxmox Backup Server a try. It can create snapshots during backup regardless of the storage backend you use, and it also supports live restore, which allows you to start a VM while it is still being restored.

Another big advantage is that you can restore backups to any Proxmox Virtual Environment installation, whether it is part of a cluster or just a standalone node.

In my case, I mostly run Hewlett Packard Enterprise (DL360, DL380) servers with hardware RAID controllers like HPE Smart Array, and I’m not willing to give up the Smart Array in order to move to a filesystem that requires direct disk access and also consumes additional RAM and CPU.

So for now I’m waiting for BTRFS support in Proxmox storage, since it supports snapshots natively. Until then, PBS works perfectly well for my needs.
 
Last edited:
Hi kmentzelos
I'm already using PBS and I'm very happy because it works very well. With a few tweaks, I can restore the infrastructure from scratch in a few hours (all you need is another server, enough storage, and a backup repository). If it could also be used as replication software, that would be fantastic!
Meanwhile, I'm continuing to study the cluster with two servers, an ML380 and a DL160. VM migration between them works well. The VM disks are on directory-type storage on hardware CTRL. I also have a VM using a NAS with iSCSI for storage. With this VM, migration is instantaneous because there are no disks to move. This second option is like having a SAN + two nodes, but that's not my current goal.

Last night, after the post, I searched for replication software and found that Nakivo performs replication. Now I'll try a trial license to see if everything works as I initially planned with our storage type.
Like you, I have no intention of giving up the features of HP Smart Storage, as it offers many advantages, but the downside is that it requires updates, up-to-date firmware, and an array with the right settings.
 
Hi Frank,

I also saw their website, but I haven’t tried anything yet. I’d be interested to hear about your findings with Nakivo Backup & Replication.

For now I’m focusing on completing my Proxmox Virtual Environment + DRBD testing and setup. For my use case it’s quite straightforward and feels safe, especially in primary/secondary mode.
I’m also experimenting with a primary/primary setup, which so far is going very well.

My current layout is roughly:
server (hardware RAID) → physical disk (raid5 ld1) → VG1 → LV1 → DRBD → VG2 → Proxmox storage
VG2 is replicated between the nodes. Combined with Proxmox Backup Server, I feel quite safe with this setup.
 
Last edited:
You must log in or register to reply here.
Top Bottom
Back