3 Node, 2 Storage and ZFS over iscsi

[AlteSocke]

Hello all,

I currently have a 3 node (worker) and 1 storage setup.
The storage server is a Debian which is connected to the three workers via ZFS over iscsi.
Currently I have about 30 virtual systems running on the proxmox cluster.

The setup is working fine.
Now I want to connect a second storage server so that it can be updated and rebooted without having to set a longer downtime for all VMs. This would then also have to be connected via ZFS over iscsi. But since the workers "only" see the ZFS and not the iscsi targed, I can't use multipath.

:~# ls -l /dev/disk/by-path/
total 0
lrwxrwxrwx 1 root root 9 Aug 26 10:46 pci-0000:00:1f.2-ata-6 -> ../../sr0
lrwxrwx 1 root root 9 Aug 26 10:46 pci-0000:00:1f.2-ata-6.0 -> ../../sr0
lrwxrwx 1 root root 9 Aug 26 10:46 pci-0000:03:00.0-scsi-0:1:0:0 -> ../../sda
lrwxrwxrwx 1 root root 10 Aug 26 10:46 pci-0000:03:00.0-scsi-0:1:0:0-part1 -> ../../sda1
lrwxrwx 1 root root 10 Aug 26 10:46 pci-0000:03:00.0-scsi-0:1:0:0-part2 -> ../../sda2
lrwxrwx 1 root root 10 Sep 13 14:41 pci-0000:03:00.0-scsi-0:1:0:0-part3 -> ../../sda3

I had in mind to synchronize both ZFS systems with zfs-sync, that would be a failover already. The question now is, how can I automate the failover in proxmox, so that if one storage server fails the other one takes over automatically?

Thanks and greetings
AlteSocke

 

[bbgeek17]

I had in mind to synchronize both ZFS systems with zfs-sync, that would be a failover already.

AFAIK ZFS replication offered in Proxmox is not synchronous, meaning the target is always somewhat behind and RPO (recovery point objective) is never 0.

The question now is, how can I automate the failover in proxmox, so that if one storage server fails the other one takes over automatically?

If you have two independent storage nodes that perform async backend replication it would be extremely hard to create a robust automatic failover.
iSCSI is run over TCP - a stateful protocol. That means that it expects all the same services and IP addresses to be present when a failure happens. That also has to happen within the timeouts (Application, iSCSI, TCP).

The solution you are looking for requires shared storage between the two storage nodes and use of clustering software (pacemaker/corosync) that would control IP/ZFS/iSCSI services. There are a few articles on the net on how to create such a cluster. For a proper HA you would also need a vote/witness/quorum, ie 3rd member.

In summary: Can you build something like this? Yes. Should you? That depends. If its a home sandbox - go for it. Is it for production system - dont.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[AlteSocke]

Thx @bbgeek17 for you replay!

We have now solved our sorage problem not with ZFS, but with Ceph.

Storage 1 was already connected to the Proxmox cluster as "ZFS over ISCSI" and the VM's were in operation and productive.

On Storage Server 2 we also installed Proxmox and added the system to the cluster.
Ceph was installed and configured on all nodes (3+1).

All disks of the Storage 2 server were added as OSD. A pool was created over them with the values
* Size: 2
* Min Size: 1
After that the disks of all VM's could be moved to the Ceph storage. We then installed Proxmox on the Storage 1 server and added this server to the cluster. This results in the following setup

3 x PVE Worker
* Virtual machines only
* Ceph Monitors
* Ceph Manager

2 x PVE Storage with 8 x 2 TB each
* No Virtual Machines (only in case of emergency)
* Ceph Monitors
* Holder of the OSD disks

So far this works very well

We have also done some testing (in a smaller test environment) in which we have
* Removed and wiped disks
* crashed storage and nodes
* stopped and restarted the whole cluster

The Ceph survived everything and there was no data loss yet.

Best regards

 

[bbgeek17]

Glad you worked it out. Ceph is definitely is better fit for you than ZFS/iSCSI.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox