Hi,
I'm actively testing Proxmox to move out from VMware, due to recent changes ...
So far so good, I've been able to deploy the following Ceph stretch cluster configuration :
- 2 Production datacenter (10kms, <1ms RTT, 2x100GB PFON), with each 3 nodes (full NVME OSD)
- 1 Witness datacenter (200kms, 20ms RTT, 1x100GB PFON from DC1 and DC2), with 1 PVE virtualized, acting as witness and a NAS to receive backups
It just works great !
Now, I'd like to upgrade the third Datacenter as DR site, adding 3 same nodes as production datacenters, to receive async replica from production site.
That's where I'm stuck, I cannot find a valid Ceph configuration:
- The stretch pool (based on the stretch crush map rule) is automatically mapped to replica nodes. I'd like the stretch pool to be only added to my 6 production nodes
- The replica pool (based on the replica crush map rule) is automatically mapped to production nodes. I'd like the stretch pool to be only added to my 3 replica nodes
Moreover, I've read Ceph stretched cluster official documentation stating :
In the future, stretch mode might support erasure-coded pools and might support deployments that have more than two data centers.
May I understand the setup I want to deploy is simply impossible in the current version ? Then if exclusively using Ceph, I would have to choose between :
- Stretch Ceph on DC1 & DC2 + 1 witness on DC3
- Regular Ceph on DC1 + async replica on DC2
But not a combination of both ?
A workaround would be to not use Ceph but a SAN in the DR site, but it would avoid such configuration.
Thanks !
Edit : I've been able to present each Ceph pool to its respective node (Datacenter > Storage > Edit > nodes)
Edit2 : RTFM, first. Native replication offered by Proxmox is only possible with ZFS ... Not Ceph. I'll dig further to find a suitable solution. If anyone has the answer, you're welcom
I'm actively testing Proxmox to move out from VMware, due to recent changes ...
So far so good, I've been able to deploy the following Ceph stretch cluster configuration :
- 2 Production datacenter (10kms, <1ms RTT, 2x100GB PFON), with each 3 nodes (full NVME OSD)
- 1 Witness datacenter (200kms, 20ms RTT, 1x100GB PFON from DC1 and DC2), with 1 PVE virtualized, acting as witness and a NAS to receive backups
It just works great !
Now, I'd like to upgrade the third Datacenter as DR site, adding 3 same nodes as production datacenters, to receive async replica from production site.
That's where I'm stuck, I cannot find a valid Ceph configuration:
- The stretch pool (based on the stretch crush map rule) is automatically mapped to replica nodes. I'd like the stretch pool to be only added to my 6 production nodes
- The replica pool (based on the replica crush map rule) is automatically mapped to production nodes. I'd like the stretch pool to be only added to my 3 replica nodes
Moreover, I've read Ceph stretched cluster official documentation stating :
In the future, stretch mode might support erasure-coded pools and might support deployments that have more than two data centers.
May I understand the setup I want to deploy is simply impossible in the current version ? Then if exclusively using Ceph, I would have to choose between :
- Stretch Ceph on DC1 & DC2 + 1 witness on DC3
- Regular Ceph on DC1 + async replica on DC2
But not a combination of both ?
A workaround would be to not use Ceph but a SAN in the DR site, but it would avoid such configuration.
Thanks !
Edit : I've been able to present each Ceph pool to its respective node (Datacenter > Storage > Edit > nodes)
Edit2 : RTFM, first. Native replication offered by Proxmox is only possible with ZFS ... Not Ceph. I'll dig further to find a suitable solution. If anyone has the answer, you're welcom
Last edited:
