can I still use Ceph to distribute storage with size=1 and min_size=1?

[QuentinCh]

Hi everyone,

I'm currently running a lab with four ESXi hosts. Only two of them have local storage capabilities, each with 2 x 4 TB drives and 2 x 128 GB drives. The other two hosts don't have a disk backplane.

At the moment, storage is provided by two DataCore servers (in HCI mode) and distrubuted accross all hosts.

I'm considering migrating the entire lab to PVE. Since this is purely a lab environment with no critical data, I'm fine with wiping everything and rebuilding it from scratch.

Since this is only a lab environment, I understand the risk of data loss in the event of a disk or node failure.

My question is: can I still use Ceph to distribute storage with size=1 and min_size=1?

Alternatively, would size=2 and min_size=1 be a better approach, considering that I only have two nodes with disks?

Thanks in advance for your help!

PS : my english isn't that good and I used a translator

 

[bbgeek17]

Hi @QuentinCh , welcome to the forum.

Yes, you can use Ceph with size=1 and min_size=1. All four PVE nodes can access the Ceph storage over the network, even if only two nodes have disks for OSDs.

However, size=1 provides no data redundancy, so a disk or node failure can result in data loss. With size=2 and min_size=1, you get two copies of the data, but both copies must be placed on separate nodes with OSDs. With only two storage nodes, this is a workable lab setup, but I would not place any data there that I cant afford to loose.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[alexskysilk]

At the moment, storage is provided by two DataCore servers (in HCI mode) and distrubuted accross all hosts.

Since you already paid for datacore, why not just use that? https://docs.datacore.com/FAQ/HVSANBPG/Hyper-converged-Virtual-SAN/Creating-a-cluster.htm

 

[QuentinCh]

Hi,

Thanks @bbgeek17 for your answer. Since this is a lab, there is no critical data, so I’ll give it a try soon, probably with size=1 and min_size=1 to make the most of the available storage capacity.

@alexskysilk If possible, I won’t reuse DataCore on this lab. The main reason is that the two nodes with disk backplanes only have 8 GB available for their boot disks (SD cards), so I have to dedicate the two 128 GB disks to the DataCore VM.

 

[tschmidt]

Make sure to try rebooting the osd hosts. I my experience VMs are likely to crash or experience other interesting effects if their drives become unavailable.

 

[alexskysilk]

The main reason is that the two nodes with disk backplanes only have 8 GB available for their boot disks

If you'd rather not use it thats fine, but be aware that you really dont want to run PVE on SDcards either; no matter what you do if you intend to use these machines you'd need to dedicate space to the OS.

 

[tschmidt]

splitting root from boot works fine for me btw

 

[gurubert]

However, size=1 provides no data redundancy, so a disk or node failure can result in data loss.

Correction: will result in data loss.
And there is no room for any maintenance without shutting down the VMs.

 

[Johannes S]

As others said: Don't. Even in a lab environment I wouldn't risk data loss, which you will most likely get this way.
Just because it's technically possible to do change the defaults, doesn't make this good idea.
See also https://forum.proxmox.com/threads/ceph-pool-size-is-2-1-really-a-bad-idea.68939/ and https://blog.noc.grnet.gr/2016/10/18/surviving-a-ceph-cluster-outage-the-hard-way/ for reference. If you have only two nodes with discs I would consider using NFS, ISCSI or ZFS over ISCSI instead. Alternatively ( if possible) spreading the disc's over all four nodes.