Stand-by replication scenario

[Deleted member 93625]

Hi all,

Hope I can explain this well. I have attached a diagram for easy understanding.

My company wants to have a kind of stand-by rack. One rack has Proxmox nodes with a ZFS over iSCSI storage (right on the diagram). Another rack (left on the diagram) will have the same setup but no VMs running on and VM disks are not stored on the left storage, hence, stand-by. All PVE nodes are in a single cluster.

What the company wants is, the whole right rack is replicated continuously (or with some intervals) to the left so when the right rack gets a real extreme disaster (e.g. right rack got bomb), we can use the left rack maybe not immediately but with the minimal hassle. In this scenario, I described as a rack but eventually these racks will be in different data centres.

If I want to achieve this, what would be the best way supposed no need to worry about networking connections? Is this something possible via Proxmox GUI? Hope I explained clearly. Thanks a lot.

Eoin

 

[Stoiko Ivanov]

for clean failover you have 2 possibilities:
* use a shared storage (Ceph, NFS, CIFS, gluster) - keep in mind that you would need to keep that with the same availability
* use storage replication - https://pve.proxmox.com/pve-docs/chapter-pvesr.html (note that you might lose some date between the replication intervalls)

also keep in mind that partitioning everything exactly in half usually does not work (quorum problems)

I hope this helps!

 

[Deleted member 93625]

for clean failover you have 2 possibilities:
* use a shared storage (Ceph, NFS, CIFS, gluster) - keep in mind that you would need to keep that with the same availability
* use storage replication - https://pve.proxmox.com/pve-docs/chapter-pvesr.html (note that you might lose some date between the replication intervalls)

also keep in mind that partitioning everything exactly in half usually does not work (quorum problems)

I hope this helps!

Hi Stoiko,

Thanks for that. I'd like to ask a bit further if you don't mind.

1. When you say 'use a shared storage', what does it mean exactly? Are you saying the ZFS over iSCSI storage should be known to all cluster nodes? Isn't it already true if I set up the environment like the attached diagram?
2. The storage replication page (the link) says, it supports local ZFS storage. Does that mean ZFS over iSCSI is considered as a local storage not a shared storage?
3. I had an impression that the storage replication looks similar to ZFS replication. Does Proxmox GUI is working as a configuration tool for ZFS replication on backend? Is this the essence of storage replication?

On my second thought, if #3 is correct, can I just connect those two storage boxes and set up regular replication without Proxmox nodes being aware of? And when the left rack should be used, can I just hook the storage as a ZFS over iSCSI storage?

I am getting confused now as someone just said maybe VM configurations also should be replicated to other nodes. Hmm.... Or, is it possible to create two separate clusters and just do the whole cluster replication? Thanks again.

Eoin

 

[LnxBil]

I am getting confused now as someone just said maybe VM configurations also should be replicated to other nodes.

Yes, that is a task that has to be done separately, so your picture is right, but you need both arrows. The PVE arrow is for the VM configuration (/etc/pve/nodes/*/{lxc,qemu-server}) and the storage is also replicated with ZFS send/receive. That should do the trick, but you have to do this manually, because there is no current implementation of this in PVE.

 

[Deleted member 93625]

Yes, that is a task that has to be done separately, so your picture is right, but you need both arrows. The PVE arrow is for the VM configuration (/etc/pve/nodes/*/{lxc,qemu-server}) and the storage is also replicated with ZFS send/receive. That should do the trick, but you have to do this manually, because there is no current implementation of this in PVE.

Hi LnxBil,

Thanks for that. I am still a bit confused, sorry.

1. If all the VMs existing on the right rack only, shouldn't the VM configuration copy be also one way unless there was a live migration or HA fail-over? Also, copying these is also requiring manual job?
2. For ZFS storage replication, when you saying that I have to do manually, it sounds like I have to set up kind of cron job on the storage box itself. If so, PVE nodes will not be aware of replication, is this correct? Meaning, PVE nodes don't need to be involved with this replication?
3. If I am thinking correctly about #2, do I still register the left storage into PVE cluster?

Thanks very much.

Eoin

 

[LnxBil]

If all the VMs existing on the right rack only, shouldn't the VM configuration copy be also one way unless there was a live migration or HA fail-over? Also, copying these is also requiring manual job?

You should regularly copy the files manually. If do not want to have a unused PVE cluster on one side, you should also run VMs there but you should distinct the VMs e.g. with ID ranges, e.g. left side only VMs with 1xxxx, right side only 2xxxx. With this, you can utilize both clusters and have no dead hardware. You can then replicate left to right and vice versa.

For ZFS storage replication, when you saying that I have to do manually, it sounds like I have to set up kind of cron job on the storage box itself. If so, PVE nodes will not be aware of replication, is this correct? Meaning, PVE nodes don't need to be involved with this replication?

Yes, the storage is replicated, as long as your iSCSI target setup is aware of the LUNs, everything should be fine. This is the simplest way to do this and most enterprise SANs do replication on this level.

If I am thinking correctly about #2, do I still register the left storage into PVE cluster?

No, every cluster has its own storage, replication takes care of the rest.

In general, there is no out-of-the-box solution to this (yet), so you have to do everything by yourself and create the automated scripts and monitoring, but if should work without any problems.

 

[Deleted member 93625]

You should regularly copy the files manually. If do not want to have a unused PVE cluster on one side, you should also run VMs there but you should distinct the VMs e.g. with ID ranges, e.g. left side only VMs with 1xxxx, right side only 2xxxx. With this, you can utilize both clusters and have no dead hardware. You can then replicate left to right and vice versa.



Yes, the storage is replicated, as long as your iSCSI target setup is aware of the LUNs, everything should be fine. This is the simplest way to do this and most enterprise SANs do replication on this level.



No, every cluster has its own storage, replication takes care of the rest.

In general, there is no out-of-the-box solution to this (yet), so you have to do everything by yourself and create the automated scripts and monitoring, but if should work without any problems.

Hi,

Thanks for your detailed explanation. On your second comment,

as long as your iSCSI target setup is aware of the LUNs, everything should be fine

: could I ask a bit more if that's okay?

I'd like to know what that means exactly. Are you talking about the recovery scenario? For example, when the right rack got a bomb so left rack is going to be used, do I have to hook the left storage with the same iSCSI target configuration that right storage had? Is this what you meant? Thanks again.

Eoin

 

[LnxBil]

I'd like to know what that means exactly. Are you talking about the recovery scenario? For example, when the right rack got a bomb so left rack is going to be used, do I have to hook the left storage with the same iSCSI target configuration that right storage had? Is this what you meant? Thanks again.

Yes, the setup on each side should be the same, so you have to check if your storage can do that. In order to have a working failover, you need to have the same lun names and all have to be exported so that the VM can start directly. This is more a ha san problem and heavily depends on your used software.

Hopefully, we'll get some cluster2cluster HA with PVE in the future, but in the moment, everything is homebrew.

 

[Deleted member 93625]

Yes, the setup on each side should be the same, so you have to check if your storage can do that. In order to have a working failover, you need to have the same lun names and all have to be exported so that the VM can start directly. This is more a ha san problem and heavily depends on your used software.

Hopefully, we'll get some cluster2cluster HA with PVE in the future, but in the moment, everything is homebrew.

Thanks very much. It gave me quite good information. Do you know by chance, if it's possible to make this with ZFS on Linux box? What my company did is just bought a storage server and installed Ubuntu Linux and made zpool on it.

Eoin

 

[LnxBil]

Thanks very much. It gave me quite good information. Do you know by chance, if it's possible to make this with ZFS on Linux box? What my company did is just bought a storage server and installed Ubuntu Linux and made zpool on it.

I personally stay away from Ubuntu, but it should be possible. Why don't you just use PVE for the storage as well? With all PVE, your infrastructure is simpler, because you only have one operating system, not two to maintain etc. But in the end, it does not matter.

 

[Deleted member 93625]

I personally stay away from Ubuntu, but it should be possible. Why don't you just use PVE for the storage as well? With all PVE, your infrastructure is simpler, because you only have one operating system, not two to maintain etc. But in the end, it does not matter.

@LnxBil Thanks for that. I am not a huge fan of Ubuntu either (I am more Debian person but I have to admit sometimes driver issue drives me crazy). I also was thinking about installing PVE as an option but since you mentioned things should be configured manually anyway, I guess it won't be a big matter. Thanks again.

Eoin

 

[LnxBil]

@LnxBil Thanks for that. I am not a huge fan of Ubuntu either (I am more Debian person but I have to admit sometimes driver issue drives me crazy). I also was thinking about installing PVE

PVE has the Debian Userland and Ubuntu Kernel, so the driver stuff should be taken care of, and the rest is almost vanilla debian :-D