Understanding Storage Replication Stack

[gdi2k]

Very interested in the new Storage Replication functionality in PVE 5.00, and I had a few questions regarding the documentation page on the wiki:
https://pve.proxmox.com/wiki/Storage_Replication

• It seems like only 2 nodes would be required for this to work well, but in the documentation it also talks about quorate, which makes me think that a minimum of 3 nodes is needed to achieve proper quorum. What is the minimum recommended number of nodes for this type of replication?
  
  
• There is a comment on the wiki: "redistributing services after a more preferred node comes online will lead to errors."
  - what does this mean exactly? Does it refer to the VM being moved back to the original node once it comes back online after a failure? What are the errors this will lead to?

Thanks!

 

[dcsapak]

It seems like only 2 nodes would be required for this to work well, but in the documentation it also talks about quorate, which makes me think that a minimum of 3 nodes is needed to achieve proper quorum. What is the minimum recommended number of nodes for this type of replication?

you only need 2 nodes for the replication, but quorum is always needed for all cluster actions, so either you ensure you have quorum (with e.g. a third node/corosync qdevice) or you follow the guide and set the expected votes (as it says in the article; not recommended)

There is a comment on the wiki: "redistributing services after a more preferred node comes online will lead to errors."
- what does this mean exactly? Does it refer to the VM being moved back to the original node once it comes back online after a failure? What are the errors this will lead to?

yes, this means if ha relocates your vms, the start on the new node, but when trying to migrate them back after the original node is online, this will fail, either because live migration of replicated vms does not work currently, or on containers, because the replica snapshots/replica state is not correct

 

[gdi2k]

Thanks Dominik, that's understood. Looks like a great solution for my needs.

 

[mailinglists]

So what looks like a process to migrate VM back to the originating node?
(Scenario: PM host #1 fails. HA goes to work starting it's VMs on host #2. Host #1 is repaired and comes back. What now?)

 

[gdi2k]

I don't know the answer to that yet either, but I'll need to find out soon as we are implementing this in the coming weeks.

Any feedback from those in the know?

 

[gdi2k]

Also, is it possible to somehow use a separate replication network (in the same way you can use a separate network for ceph)? In my scenario, the servers have 10G NICs that could connect directly to one another, but I have no 10G switch available.

 

[Andy_Screen]

yes, this means if ha relocates your vms, the start on the new node, but when trying to migrate them back after the original node is online, this will fail, either because live migration of replicated vms does not work currently, or on containers, because the replica snapshots/replica state is not correct

Hi, quick question about that: Can I make an offline migration after the original node comes backup up again?