Hello everyone!
We've recently upgraded our backbone to 50G and are having some interesting findings in our (3 node) cluster . We're running on latest Proxmox 8.3 with Ceph 18.2.
Ceph VM-Pool is configured with 3x replication over all 3 nodes (so one copy resides on each node).
When we're running backups (both LXC and KVM), CEPH reads the VM-image from the blockdevice / placement group which has been set as primary. This primary group my reside in either the local server or one of the other two.
To prevent this behaviour from happening, we've now set
to
. The default behaviour prefers the primary placement groups, the localize setting prefers the location closest to the server with the VM residing on.
For a 3-node 3x replication cluster this eliminates any network-usage while doing backups (reads are all done locally), on our bigger clusters (20-50 nodes) have a noticeably lower network usage while doing backups.
Question: Why is this setting set to default and not localize? @fabian (sorry for tagging you directly here but we're doing awesome playing ping-pong together) ;-)
Cheerio
Florian
We've recently upgraded our backbone to 50G and are having some interesting findings in our (3 node) cluster . We're running on latest Proxmox 8.3 with Ceph 18.2.
Ceph VM-Pool is configured with 3x replication over all 3 nodes (so one copy resides on each node).
When we're running backups (both LXC and KVM), CEPH reads the VM-image from the blockdevice / placement group which has been set as primary. This primary group my reside in either the local server or one of the other two.
To prevent this behaviour from happening, we've now set
Code:
rbd_read_from_replica_policy
Code:
localizeFor a 3-node 3x replication cluster this eliminates any network-usage while doing backups (reads are all done locally), on our bigger clusters (20-50 nodes) have a noticeably lower network usage while doing backups.
Question: Why is this setting set to default and not localize? @fabian (sorry for tagging you directly here but we're doing awesome playing ping-pong together) ;-)
Cheerio
Florian
