various questions about proxmox

[pazzoide76]

HI,
I'm new to proxmox.
I need a couple of clarifications.
I have to configure a proxmox cluster with external san fc and therefore the VMs will be installed on this storage.
What is the minimum number of hosts I need to have to guarantee HA?
How many network cards should I use and what speed should they have?
If I wanted to use Ceph what is the minimum number of hosts to use?
I didn't really understand how to calculate disk space in Ceph.
Let's say I have 3 hosts.
Each host has 3 TB of disk space and I set 3 replicas, will the usable space be 1 TB?
Does 3 replicas mean that the VM's disk space will be copied to the disks of all three hosts and in the event of a host downtime the VM will continue to run?

Thank you

Greetings

 

[B.Otto]

I have to configure a proxmox cluster with external san fc and therefore the VMs will be installed on this storage.

then your only choice will be to use LVM, which means you won't be able to use Snapshots

What is the minimum number of hosts I need to have to guarantee HA?

three hosts. It is possible to make an HA with two Proxmox hosts, but then you need to configure a QDevice on a third node as tiebreaker.

How many network cards should I use and what speed should they have?

This depends on your workload. In addition to the network cards used for your VM network, you want at least one dedicated port for Crorosync (cluster network), ideally two dedicated cards/networks. The Corosync network doesnt have to be high-through, but needs to be low-latency and reliable (so no WAN, VPN or other shenanigans). Additional network ports might be necessary for storage, but since you are planning to use FC, you wont need them in this case.

If I wanted to use Ceph what is the minimum number of hosts to use?

Do note that Ceph is a different kind of storage style, so you wont be able to use your (single shared) FC storage with a Ceph installation on Proxmox.
The minimum number of Proxmox hosts needed for hyperconverded Ceph is three. Unlike the HA requirement, this is a must, there is nothing like a Qdevice for Ceph.

I didn't really understand how to calculate disk space in Ceph.
Let's say I have 3 hosts.
Each host has 3 TB of disk space and I set 3 replicas, will the usable space be 1 TB?
Does 3 replicas mean that the VM's disk space will be copied to the disks of all three hosts and in the event of a host downtime the VM will continue to run?

Pretty much, yeah. Each block gets replicated to two more hosts, so your total raw storage needs to be three times the useable storage. This may sound much as first, but note that unlike other storage solutions there is no extra storage need through configured RAIDs.

There are more things to note when setting up an enterprise-level HCI-environment with Ceph, like decent Network, enough OSDs, enough RAM/CPU, or calculating enough extra free space in the case of a node failure.

 

[pazzoide76]

Thanks for the replies.
If I wanted to use ceph instead of a san fc, how many nics should I install and what speed should they have?
Should the space be SSD/NVME or can I use normal SAS disks?
Considering the previous example, i.e. 3 hosts with 3 TB of space and 3 replicas, in the event of failure of a node does everything continue to work correctly or do I need to save more additional disk space?
If I need additional space how do I calculate it?

Thank you

 

[B.Otto]

Ceph really likes high-throughput networking. I would do at least 2x 10Gbit, but 25/40/100Gbit would be better. With three Hosts you could do something like a Meshed Network, so you could omit the expensive 100Gbit-Switches.

SAS or SATA SSDs work well with Ceph, I dont think Ceph can properly use the full potential of NVMe storage though (I read about environments where people get more out of their NVMes by placing several OSDs on each NVMe, but that requires manual configuration).

Do note that Ceph should be run without a raid controller, so you need a HBA that will present the disks directly to the OS (JBOD).

The way I usually rule-of-thumb enterprise Ceph storage is as follows:
- needed effective storage for your VMs: 3TiB
- accounting for TB <-> TiB conversion: 3 * 1.1 = 3.3TB
- 3-replication: 3.3 TB * 3 = 9.9TB
- Ability to compensate one failed node: 9.9 * 3 / (3 - 1) = 14.85TB
- 10% buffer because you never want Ceph to be completely full: 14.85 * 1.1 = 16.34 TB
So you would need a total of 16.34TB raw storage, split between 3 Nodes makes this 5.45TB of disks on each node.

The thing is, for proper balancing you want at least 4 OSDs per Host, so at least 4 disk per Host. More disks make for better balancing/recovery but also requires more RAM/CPU for Ceph. Thus its harder to be cost-effective when you only need a few TB of storage.

You see, Ceph does need quite a bit of resources for a proper setup, and more hosts = less overhead.

 

[pazzoide76]

thank you