looking for advise/ceph help

[si458]

Hi All

im looking for advise and also ceph help

sadly this is all the hardware i have to hand without buying more stuff:

3 main servers, 24 cores, 96gb ram, quad 1gb networks, 60gb boot SSD, 500gb OSD HDD, SATA 2 speeds sadly
4 mini servers, quad cores, 8gb ram, dual 1gb networks, 60gb boot SSD, 1TB SSD, SATA 3 speeds
1 x 24port 1gb switch

my plan was to run 3 main servers with proxmox with subscriptions in cluster with ceph

i have already created the Cluster and CEPH with the 3 main servers, works brilliantly!

i still have the 4 mini servers which i can run beside them, but my questions follow:

1. do i need to install proxmox on each of the mini servers, to add the 1TB SSDs into the ceph cluster?
as i cant find anything online that shows a guide how to set up the OSDs on there own and then add them into the cluster?

2. if i do need to install proxmox on each of the mini servers, do i need a subscription for each of them?

3. is this setup actually a good setup or can i do some more tweaking? (PREFER WITHOUT SPENDING MONEY)
ALSO I ALREADY UNDERSTAND ABOUT HAVING ONLY A SINGLE SWITCH, i will sort out another one if i can

 

[t.lamprecht]

Hi,

1. do i need to install proxmox on each of the mini servers, to add the 1TB SSDs into the ceph cluster?
as i cant find anything online that shows a guide how to set up the OSDs on there own and then add them into the cluster?

Theoretically no, but it really will make things harder as you won't profit from the native ceph server management integration into Proxmox VE and won't see the OSD and other ceph services of the non-PVE nodes, so restart after upgrade can be easily forgotten, you need to ensure the non-PVE nodes runs the same and compatible ceph version as the PVE ones, ... So yes, but will turn into a bit of an headache fast.

2. if i do need to install proxmox on each of the mini servers, do i need a subscription for each of them?

from https://www.proxmox.com/en/proxmox-ve/pricing:

In a Proxmox VE Cluster you need to have the same subscription level for all your servers.

So yes, to be able to ask for enterprise support that would be required.

Note, that ceph won't be supported either if you run it split on PVE and non-PVE notes, as that'd mean that our enterprise support team has to support arbitrary distros and arbitrary ceph packages combined with ours, which cannot be really done.

3. is this setup actually a good setup or can i do some more tweaking? (PREFER WITHOUT SPENDING MONEY)

No real suggestions without spending money. Some other points in your setup would be:
* the network - IMO this really will limit you and break clustering (never run cluster network on ceph private one) so I really would try to get another network in there 10G for ceph could to wonders and bring much more stability in the cluster.
If that's really not an option you could also look into getting additionally 1G double port nics for each node - if the switch has the switching capacity you could use still just that one (or get a separate one, 1G switches are pretty cheap nowadays) and separate the traffic here a bit, could even do a bond for ceph and a single nic network for corosync cluster traffic.
* lack of OSDs, normally one has a few OSDs per host, that's not a real issue in itself though - could be expanded later on.

 

[si458]

Hi

Thank you for your help/advise

I have already looked into 10G networking but to get 3 cards for main servers only and a 8 port 10G switch is a little expensive at the moment (about £800)

Also I didn’t see much point in 10G if the main servers are only sata 2 speeds anyways so I wouldn’t see much different even if I put the SSDs into the main servers instead of the HDDs

I’m also not to bothered about the read/write speeds either, it’s more the HA I’m interested in and the migrations to keep the machines online and available
I have also looked into bonding 2x1gb ports each per server in the past but never got it to work, but I might of been doing something wrong

the other issue is the 7 licences of support which adds up as well is very expensive, considering I wouldn’t even use the 4 mini servers for any virtual stuff just ceph (around £2k)

just another idea too, if I setup a external ceph cluster with the 4 mini servers, then setup Proxmox on the 3 main ones but not in the ceph cluster and then linked the Proxmox servers into the external ceph storage using the rdb, would Proxmox staff support this under the enterprise subscriptions for help if something happened in future?
Like if Proxmox couldn’t read/write to the external ceph?

 

[t.lamprecht]

I’m also not to bothered about the read/write speeds either, it’s more the HA I’m interested in and the migrations to keep the machines online and available

if you want HA get that extra network, separate ceph off from the other things, else you'll have a bad time on failure - as then ceph recovery kicks in, may overload your link and break corosync cluster traffic and thus quorum (it's low in bandwidth but quite time sensitive)

the other issue is the 7 licences of support which adds up as well is very expensive, considering I wouldn’t even use the 4 mini servers for any virtual stuff just ceph (around £2k)

Proxmox does not sells licenses, you're free to use Proxmox VE as is, it's open source after all. We sell support + enterprise repository subscriptions only.
Using them for ceph managed by Proxmox VE is using them, plus you always can fallback on putting some VMs there in an outage.
If you have further questions about this write to office@proxmox.com this is the community forum after all

just another idea too, if I setup a external ceph cluster with the 4 mini servers, then setup Proxmox on the 3 main ones but not in the ceph cluster and then linked the Proxmox servers into the external ceph storage using the rdb, would Proxmox staff support this under the enterprise subscriptions for help if something happened in future?
Like if Proxmox couldn’t read/write to the external ceph?

you can interface with external storage just fine, but ceph server itself won't be supported then - only things on PVE itself.
If there's an issue specifc to the RBD client connection it will be looked into by the enterprise staff, if it's clearly related to ceph server (i.e., an outage or the like) it won't.

If in doubt check the support agreement ( https://www.proxmox.com/en/downloads/item/proxmox-ve-subscription-agreement ) and/or ask office@proxmox.com with specific questions.