cluster performance degradation

[toto-ets]

I wonder why all the guides talk about a minimum of 3 nodes? I could get to 4 at most and associate an SSD for each node, I have to think then that my solution should be ZFS HA, but I wouldn't do the migrations live right? or does it just take more time to migrate them?

 

[ness1602]

1. Yes 1min is replication time, but you need to spin up machine on a different node when first one dies, right?
2. For ceph, 3 is min and it works great if you have fast disks. With more nodes it is even more great

 

[toto-ets]

yes exactly, if the node dies I have to run the VMs on another node, but how can I replicate the VM without ceph if the node is off?

 

[kellogs]

yes exactly, if the node dies I have to run the VMs on another node, but how can I replicate the VM without ceph if the node is off?

https://www.techtutorials.tv/sections/promox/proxmox-zfs-replication/

 

[kellogs]

please make sure you are not using any raid card for your disks.

 

[kellogs]

also please make sure your memory is going to be enough since ZFS uses alot of memory for itself ... if you are doing hyperconverged there might be resource contention between your VMs and the ZFS needs.

 

[kellogs]

also dont setup your total nodes = 4 in a cluster config. you would have split brain issue. always stick with odd number setup.

 

[toto-ets]

assicurati di non utilizzare alcuna scheda raid per i tuoi dischi.

Si, lo sapevo, io non uso RAID HW

 

[toto-ets]

also please make sure your memory is going to be enough since ZFS uses alot of memory for itself ... if you are doing hyperconverged there might be resource contention between your VMs and the ZFS needs.

I have 256 of RAM for each node

 

[kellogs]

Si, lo sapevo, io non uso RAID HW

haha you are italian? i dont read Italy but i just came back from Venice and Rome

 

[toto-ets]

also dont setup your total nodes = 4 in a cluster config. you would have split brain issue. always stick with odd number setup.

my intention is to always use 3 nodes, and then use the PBS for the bkp, am I doing well or not?

 

[kellogs]

I have 256 of RAM for each node

Maybe not enough if you need to share this amount with your VMs also. please do some research on the required RAM for ZFS.

 

[kellogs]

my intention is to always use 3 nodes, and then use the PBS for the bkp, am I doing well or not?

Yes that is perfectly fine and good choice. The mistake is using HDD and large size too 16TB per disk for your ceph and 3 nodes only IMO.

 

[ness1602]

also dont setup your total nodes = 4 in a cluster config. you would have split brain issue. always stick with odd number setup.

This is not true,if you have 4 nodes, you can only lose one, since 3 is needed for minimum quoroum.

 

[kellogs]

This is our setup for new ceph production and once everything tested ok, we would purchase the support from Proxmox too. we have 4 nodes for compute and 5 nodes for ceph only using enterprise NVME and 2x 100G per node.

 

[ness1602]

Yes that is perfectly fine and good choice. The mistake is using HDD and large size too 16TB per disk for your ceph and 3 nodes only IMO.

Three is okay for start, but those hdd and this use case is a engineer error.

 

[toto-ets]

It takes a lot of space, how did I mount 16TB SSDs? I don't think there are 16TB ones or am I wrong?

 

[UdoB]

It takes a lot of space, how did I mount 16TB SSDs? I don't think there are 16TB ones or am I wrong?

No problem: 100 TB per single SSD seem to be available: https://www.techradar.com/best/large-hard-drives-and-ssds#section-the-largest-ssd-on-the-market - listing also 30TB and 60TB "small" ones...

;-)

 

[ness1602]

Usually now (in the west) , you use 3.xTB and 7.xTB ssds or nvmes, and then you can get good density and performance.

 

[toto-ets]

yes, but now I bought 12 16 TB HDDs, so if I use ZFS could I have just 2 nodes? It doesn't make sense to put the third node, so I also save electricity, I had bought the third ninth to use ceph