New to Proxmox. Need suggestions

[sabari]

Hi All. Am new to Proxmox and need suggestion on a proper setup. We have the following servers which we are planning to reuse.

CPU	No of CPUs	Physical Cores/ CPU	Total Physical Cores	RAM	RAID Controller	HDD	Quantity	RAID	Model
Xeon E5 2690 @ 2.90 GHz	2​	8​	16​	64​	H710P Mini	Samsung EVO 860 500GB	2​	1​	Dell R 620
Xeon E5 2690 @ 2.90 GHz	2​	8​	16​	64​	H710P Mini	Seagate SAS - 278GB	2​	1​	Dell R 620
Xeon E5 2690 @ 2.90 GHz	2​	8​	16​	64​	S110 Embedded	Seagate SATA - 465.75GB	2​	0​	Dell R 620
Xeon E5 2667 @ 3.30 GHz	2​	8​	16​	64​	H710P Mini	Samsung SSD EVO 500GB	2​	1​	Dell R 620
						WD SATA 931GB	1​	0​
Xeon X5650 @ 2.67 GHz	2​	6​	12​	56​	H200 Integrated	Seagate SAS - 278GB	6​	0​	Dell R 610

Except for the fourth server all others have Solarflare 10Gig network card with 2 ports and integrated 4 1Gig ports in them.
The fourth server has only the integrated 4 1Gigs on it.

We can spend a little on the storage to get a 8TB SAS on all the nodes for Ceph. Since these are servers with mixed configurations, can some one suggest if this would result in a stable Proxmox setup?

We are planning to run 20 to 25 VMs in them which are going to be a mixture of Ubuntu, Centos and WIndows

As far as the networking, I understand that one 1Gig goes to the management, 1 10 Gig dedicated to Ceph, 1 10 Gig should be dedicated to the VM traffic all belonging to separate VLANs. Can some one please confirm.

 

[LnxBil]

Drop all SSD EVO disks, you will not have fun with them on CEPH. CPUs are OK, go with the fasted ones. Normally you don't need a lot of cores for virtualization (at least not for so little as you described). You need to up the RAM tremendously, 64 GB RAM is not much, my 12W Odroid H3+ has already 64 GB RAM.

 

[sabari]

Thanks for the response. Will see if we can manage some upgrades to the memory. Will consider moving all the SAS drives that I have on S5 and replace the SSDs. Now does the mix of the RAID controllers have any effect on the Ceph. Planning to add few 8TB SAS drives and configure RAID 5 on each of the nodes.

 

[LnxBil]

Thanks for the response. Will see if we can manage some upgrades to the memory. Will consider moving all the SAS drives that I have on S5 and replace the SSDs.

good! You will also need good enterprise SSDs for the CEPH OSD.

Now does the mix of the RAID controllers have any effect on the Ceph. Planning to add few 8TB SAS drives and configure RAID 5 on each of the nodes.

Unless you don't use RAID at all, no. You need to set those controllers - at least for the disks for CEPH - to HBA-mode. If you can't do this, don't run CEPH.

 

[czechsys]

My tips, select what is possible for you. All tips are based on 2x switch in stack for network redundancy:

0] buy another 2x(4x) 10Gbps card so you have 4x10Gbps
1] use 2x10Gbps LACP for VM
2] use 2x10Gbps LACP for Ceph
or you can mix 1] 2] on 2x10Gbps LACP, if acceptable performance
3] 2x1Gbps corosync link - different switches than those 10Gbps. Or primary 1x1Gbps link + secondary on any 10Gbps LACP.
4] If you prefer HW raid, use it for OS disks
5] connect extra HBA card to backplane for Ceph disks (for example Intel S4xxx, Kingston DCxxx, etc)
6] if you don't prefer HW raid, use HBA cards even for OS (proxmox zfs - check documentation)
6] pve management can be on those 10Gbps links in own vlan

 

[sabari]

I now after reading understood that ZFS and RAID controllers don't go well together and RAID 0 is not equivalent of an independent hard drive connected to SAS HBA. Now I need to flash the controllers to IT Mode for me to proceed. Any good working resource suggestions please

 

[LnxBil]

Any good working resource suggestions please

google ... servethehome has some good tutorial for vanilla HBAs, maybe also HP. I just bought HBAs that I knew have good tutorials (everything LSI/broadcom/Mega-based controllers do).

 

[Dunuin]

You also often find people selling preflashed HBAs. Just bought this week a preflashed SAS2308 with cables for 8x SATA for 35€ incl. Shipping.