Use Disk PassThrough or Raid-0 for PVE with Ceph (Testsystem)

[Martin.B.]

I know that there are several threads and ressources about not using a raid HBA for PVE with Ceph.

I have some OLD Systems to do some testing and, of course, will not get any money for new hardware for tests.
Those systems have an Areca 1883LP controller. All Disks (Seagate Nytro SSDs 2TB) are in the front enclosure that is directly connected to the Areca controller.

I now have the possibility to create passthrough disks in the controller, in this case no smart data is shown in PVE.
Or i can create a raid-0 with just 1 disk for every hardware disk, then i get smart data shown in PVE, but it is not the real smart data of the disk and does not contain very much information.

What would be the (better) way to get PVE with Ceph running, just for testing. Data loss does not matter in this case, just the basic functionality.

As a workaround: could i install multiple PVE+Ceph as VMs on a PVE server?

 

[UdoB]

As a workaround: could i install multiple PVE+Ceph as VMs on a PVE server?

Sure! For teaching/learning/debugging this works great.

Example: in my Homelab I have one specific cluster member with 64 GiB Ram and 2 TB local storage for VMs.

I was able to create six virtual PVE nodes with 8 GiB Ram each, construct a cluster, add another two virtual 32 GB disks to each of them and form a Ceph installation. Now I got 12 OSDs, multiple MONs and MGRs.

It worked surprisingly well. I was able to utilize different Ceph pools to run several (small) Test-VMs. And it allowed me to test Ceph failure behavior / auto-repairing and also to experiment with Erasure Coding.

Pro tip: before injecting artificial failures make a snapshot - all "hosts" are virtual


Disclaimer: I am not using any Ceph currently. My story is here: https://forum.proxmox.com/threads/fabu-can-i-use-ceph-in-a-_very_-small-cluster.159671/

Also: https://pve.proxmox.com/wiki/Nested_Virtualization

 

[aaron]

As a workaround: could i install multiple PVE+Ceph as VMs on a PVE server?

This is what we use for many internal tests, and also in the hands-on labs for our trainings.
Good for functionality and behavior tests as long as performance is not a factor!

 

[Martin.B.]

Thanks for your fast answers, looks like i will go this way and only use 2 of the hardware machines and build the ceph cluster virtual. The machines do only have 32GB RAM and an older 6 core CPU, so maybe using 2 hardware machines will be better. I am sure i will get it when the hardware comes to its limit
If i use all disks as raid-5 i should have ~10TB per node free for VM usage.

It is just for testing and learning and comparing it to the "non ceph" environment with "diskless" server and redundant iSCSI storage as backend.

@UdoB : Thanks for the story and the hint with the snapshots.

 

[alexskysilk]

It is just for testing and learning and comparing it to the "non ceph" environment with "diskless" server and redundant iSCSI storage as backend.

Good for functionality and behavior tests as long as performance is not a factor!

This is a major point. using ceph on this sort of POC will not teach you how to actually use ceph in production since you wont be able to actually put the type of load on it that would be meaningful. Is your intention to use this knowledge to put together a production cluster, or just for funsies?

 

[Martin.B.]

It will be to see the difference of using ceph on lokal storage vs. PVE nodes without local storage and some kind of remote storage.
If any of this will be used in future production environment there will not be much "load" on it. Not CPU nor IOPs wise.
But i think we will get the idea what should or should not be done in which built.

Reading the story of UdoB i may tend to not use ceph in our production environment later, as we will have not more than 3 nodes. But maybe a bit testing and playing with it, will convince me of the opposite.

In any case i don't think it will be a bad thing to see how a ceph cluster is configured and how it reacts to different "losses" and outages.

 

[Johannes S]

Reading the story of UdoB i may tend to not use ceph in our production environment later, as we will have not more than 3 nodes. But maybe a bit testing and playing with it, will convince me of the opposite.

It depends on your usecase. If you want to have still a working cluster if two nodes get down (due to maintenance or failure) you will need more than three nodes for the reasons Udo explained in his writeup. But if you are fine with just being able to update one node at a time or to only be able to allow one node to fail a three-node-cluster might still be ok for your usecase. Several of the Pros in this forum mentioned that they have customers who are quite happy with a three-node PVE+Ceph clusters. Others use ZFS Storage replication (which is asynchronous and so by design has a (although minimal) loss of data (might be ok if you can live with loosing 1-15 minutes of data) or storage hardware connected via NFS/ISCSI or FibreChannel (which might end up being a single point of failure or in case of FC/ISCSI doesn't have all feature ZFS or Ceph have). So it boils down to your requirements, which tradeoffs you are comfortable with and which riscs you can accept. Pick your poison

 

[UdoB]

So it boils down to your requirements, which tradeoffs you are comfortable with and which riscs you can accept. Pick your poison

This!

It's great to have choices. To make a good decision you need to know the disadvantages in beforehand and in detail.

 

[Martin.B.]

Thanks for your input, Johannes S.
This is exactly why we are testing as much possibilities as we can.
And, at the moment, we are learning very much every day.
In the past we would have installed a small VMWare cluster with shared storage (cluster), as this is what everyone in our IT team knows. But Broadcom does not like small business customers anymore....
The possible solutions we are currently looking at:
- Hyper-V in any form with or without clustering
- Proxmox (2 nodes + qdevice) with ZFS (asynchronous as you mentioned)
- Proxmox (2 nodes + qdevice?) with shared (iSCSI/NFS) storage as backend
- Proxmox with ceph (3 nodes)

Cannot say much about the sizing at the moment, as there are still many things unsure. Can be anything from 10-25 VMs (mostly Windows) and 10-20TB needed storage.
Only thing that is pretty sure at the moment: nodes will be in seperate rooms in different buildings connected by (multiple) fibres ~100m in length (already existing). Even with a switch (or 2 for HA) on each side the latency will be <1ms between those locations.

I am sure there are more possibilities/variants we did not think of.

Edit: UdoB posted while i was writing
Exactly this is why we are testing as many solutions as possible. Every solution has its disadvantages, for VMWare this is: Broadcom.

 

[Johannes S]

Regarding Ceph: The PVE wiki has a page how to set up a mesh network for three nodes. This saves some bucks because you will need less switches: https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server

The reference doc chapter on deploying hci is also a good read to get an idea which Hardware Investments might be needed:
https://pve.proxmox.com/wiki/Deploy...r#_recommendations_for_a_healthy_ceph_cluster