proxmox converged infrastructure planning - input please - thank you

[newmember]

I am planning a proxmox deployment to replace 9 esxi blade servers.

Operational Goals:
Move away from RAID Arrays (always running out of space)
Move to storage that can grow easily in the future (we hate rebuilding raid arrays)
Move VM hot moves between nodes (currently cant move live VMs with ESXi)
Host Windows 2008/2012/2016/2019, Ubuntu, Centos
Schedule nightly or on demand VM snapshots (currently using a third party software to so this)
Nightly VM backups each night, full backup each week (currently using a third party software to so this)

User Goals:
Create own VMs
Not see other users/teams VMs
See only Team's own storage ( ie Dev Storage for Dev Team, Staging Storage for Storage Team etc)

40GB LAN minimum we thought would be better to support ceph replication traffic. We thought 10GB LAN would be too slow.
128GB RAM per host to host multiple VMs using 32GB RAM each
2x8core CPU Xeon 2.6Ghz

There would be an initial deployment. Then we would add additional nodes as needed. We thought that we would have a rolling upgrade. As we add new nodes they would be faster and better then we would VM loads to new servers and downgrade older nodes for lower priority services.

Proxmox Support - is is suggested to purchase support for every node in the architecture? What is the best purchase plan. We think that with fail-over builtin we would not need 2hr support. Suggestions please.

We have three options for servers (just as a budget and feature check)

May I have your thoughts and comments please.

Thank you

 

[wolfgang]

Hi,

1.) Network:
I would use a 25 GBit Network for Ceph.
Because of 25GBit as a lower latency than 40Gbit.
Ceph benefits more from latency than from bandwidth.

2.)

We thought that we would have a rolling upgrade. As we add new nodes they would be faster and better then we would VM loads to new servers and downgrade older nodes for lower priority services.

Yes this is possible.

3.)

Proxmox Support - is is suggested to purchase support for every node in the architecture? What is the best purchase plan. We think that with fail-over builtin we would not need 2hr support. Suggestions please.

You have to use the same subscription level cluster-wide.
The ticked are additive so with every new node you get more tickets.

Generall I would recommend you single socket systems.
Because it makes life easier with ceph. With NUMA you have to take care that the ceph processes are on the same node as the nic and the disk controller.

4.) FC can be used with Proxmox VE.
You can use the Ubuntu HCL to check if your HW is supported.

https://certification.ubuntu.com/


5.) Ceph is not compatible with Raid Controller. So you have to use HBA or the onboard controller.

 

[Mikepop]

Hi, We have had a lot of problems with R620, high osd ceph latency make them useless/had to change to Supermicro hardware, same HBAs than Dell, and no problems since that. We purchased support licences licences to every node in one of our clusters trying to fix this performance issue tunning proxmox somehow with support but despite we could not fix our issue I think Proxmox it's a great product that deserves support if you can afford it. Proxmox support was nice, Dell support was absolutely awful/useless.

Regards

 

[newmember]

Thanks for the feedback.

Should I consider ZFS over Ceph?

 

[wolfgang]

Should I consider ZFS over Ceph?

No, this will become very slow and brings no benefit.