Ceph pool size 2/2 on 3-server Cluster

[Deepen Dhulla]

Want to checkout what happen if Ceph pool size 2/2 instead of 2/3 on 3-server Proxmox-Ceph-Cluster , and if one Server goes down, would it be still able work in production like 2/3 setup...or will it halt all.

Reason is that we would be having PBS Setup to take daily backup. for Data Safety.
so today its 3 Server, when its 4 server, we would have more space, and better iops and bit low load on ceph network.

Want Proxmox team & community feedback on it. While creating a test setup did-not find any down-time effect on any VM & LXC. yet would like to know. so that if any thing is missed.

Thanks in advance., your feedback would help me if something is missed.
-Deepen.

 

[shanreich]

2/2 means that there are 2 copies of each object, and you need at least 2 copies of each object for I/O to work. This means that as soon as you lose one OSD, I/O is completely blocked for some objects. This is also why I would strongly recommend against this setup, since as soon as something bad happens, I/O is (at least partly) blocked until recovery / rebalance has finished. If during this downtime something else happens (another OSD on another node goes down for instance), you might even lose some data completely.

 

[Deepen Dhulla]

Ya I too had same thought about : as soon as you lose one OSD, I/O is completely blocked for some objects...But to my surprise it did not, atleast in my test senerios...where heavy data Read/write/update via custom script was done. thus thought to get community feedback on same.

 

[shanreich]

It will work after recovering, when you have 3 nodes. So maybe that's why this worked in your case? Depending on how much data needs to be recovered, recovery can be quite fast. Newly written data will also be immediately available.

 

[Deepen Dhulla]

With 2/3 option no issue. Proxmox and Ceph works great. Having in production for now 6+ years. Was trying to checkout 2/2 option on 3 server clusters