Dear Community.
We are currently in the process of building a Proxmox cluster with Ceph as the underlying storage.
We have 4 nodes, with 4 x 100TB OSDs attached to each node (16 OSDs total) and plan to scale this out by adding another 4 nodes with the same number of OSDs on attached to each node.
We would like to use an EC Pool of 4+2 across the 4 nodes, with a custom crush rule (as below), to ensure that shards, parity are distributed on separate nodes in the cluster and separate OSDs within the nodes.
Additionally, the EC 4+2 with the custom crush rule should allow us to scale out when we add the additional 4 nodes, whilst keeping the data in the same pool to mitigate the need to migrate the data to a new pool with the desired EC profile.
We have built and applied the crush rule and the cluster is healthy. However, current tests reveal that when we down a node, the cluster fails to rebalance and the VMs running in the cluster freeze, operations performed on the pool fail. When the node is restored, Ceph rebalances as expected, VMs and operations on the pool run as normal.
If anyone has any experience of implementing a pool in Proxmox, with the resilience / redundancy that allows the failure of one node and placement of the shards / parity as per the above, it would be great if you could offer any advise on achieving our goal.
Custom Crush Rule:
rule 4_2_ec {
id x
type erasure
min_size 6
max_size 6
step set_chooseleaf_tries 5
step set_choose_tries 100
step take default class HDD
step choose indep 3 type host
step choose indep 2 type osd
step emit
}
Thanks in advance for any advise or assistance with this matter.
We are currently in the process of building a Proxmox cluster with Ceph as the underlying storage.
We have 4 nodes, with 4 x 100TB OSDs attached to each node (16 OSDs total) and plan to scale this out by adding another 4 nodes with the same number of OSDs on attached to each node.
We would like to use an EC Pool of 4+2 across the 4 nodes, with a custom crush rule (as below), to ensure that shards, parity are distributed on separate nodes in the cluster and separate OSDs within the nodes.
Additionally, the EC 4+2 with the custom crush rule should allow us to scale out when we add the additional 4 nodes, whilst keeping the data in the same pool to mitigate the need to migrate the data to a new pool with the desired EC profile.
We have built and applied the crush rule and the cluster is healthy. However, current tests reveal that when we down a node, the cluster fails to rebalance and the VMs running in the cluster freeze, operations performed on the pool fail. When the node is restored, Ceph rebalances as expected, VMs and operations on the pool run as normal.
If anyone has any experience of implementing a pool in Proxmox, with the resilience / redundancy that allows the failure of one node and placement of the shards / parity as per the above, it would be great if you could offer any advise on achieving our goal.
Custom Crush Rule:
rule 4_2_ec {
id x
type erasure
min_size 6
max_size 6
step set_chooseleaf_tries 5
step set_choose_tries 100
step take default class HDD
step choose indep 3 type host
step choose indep 2 type osd
step emit
}
Thanks in advance for any advise or assistance with this matter.
Last edited: