How to decide on Proxmox set-up (requirements)?

[sjjh]

Hi everybody, new here and to virtualization management solutions, and a step before "installation and configuration" (but did not find a better category, feel free to move the topic).

I need some advice to find the best system set-up for us, e.g. how many servers, where to locate, ZFS vs. CEPH, amount of CPU, RAM, ... In a more generic approach (and to make this thread helpful fort others as well), I'm curious about which criteria play a role in deciding on a good solution, and which value is relevant to choosing one over the other (e.g. criteria: number of users, value: <20 = alternative A, >20 alternative B). If there is a good wizard/algorithm/explanation existing how to choose a good set-up, please point it out to me.

For my specific situation/question, I'll sum up some facts, that I assume are relevant for the decision. Please feel free to ask for more details/other information, to be able to give good advice.
About the organization the system set-up is intended for
An intentional community, an ecovillage (value-wise Free and Open Source software is thus a good match, as well as there is a bias for self-hosting on premise). 150 people work and live together in a small village (~15 buildings), additional employees come here to work. The community consists of a school (~80 children, ~15 teachers), a seminar house (~50 beds, ~5 employees), a farm (~30ha land, ~6 employees), a kitchen with an canteen (3 meals x 340 days/year; ~10 employees), an administration (~3 employees), facility management (~4 employees), car sharing, a village shop, a workshop/office house (>1.000qm, tiny house building, machine building, online shop warehouse, architects, ... ; ~25 people) a give-away shop, a small fablab, a district heating system, a youth project, ... The whole community is self-organized, lots of activities are handled voluntarily and many people work part-time. So there is a big BYOD culture. The comunity has three legal entities: a collective, an association, and a foundation. Budget is (very) limited.

About the existing IT infrastructure
One AC'ed room with one server rack, currently containing:
- OPNsense with firewall, VPN, web proxy, DHCP, DNS, ...
- synology RS812+ (HW EOL) with 4x4TB HDDs as NAS (used for data storage of all the projects, and also for backups)
- proprietary PBX appliance (EOL)
- desktop PC for district heating control & visualization SW
- desktop PC for book keeping SW
- 3x servers for another association (having an office on premise + national wide VPN users)
- desktop PC with an Proxmox VE test installation.
- 3 switches (1Gbit/s, one fiber switch, one switch with 10Gbit uplink)
The different buildings are connected via optical fibers. In total 28 managed switches (all 1GBit/s links), plus some unmanaged (desktop) switches. Downlink from the internet currently 2x 100Mbit/s + 20Mbit/s (for VoIP).
5 WiFi AP's in two buildings.
In a separate building a backup for the NAS (synology DS414).
Roughly 20 office desktop PCs, a growing number of laptops, and approx. 20 computers/laptops in the school.
Three network multi-function printers (two of them with local configured cost centers).
network attached sub-distribution stations for the district heating, 14 network attached solar inverters, a rising number of network attached solar thermal data loggers.
A weather station, using an Raspberry Pi with weewx to process the data. A digital scale with a self-programmed web-app to analyze the harvest results.
Each building, as well as guest net, heating, solar, have their own VLAN.
Website, email and mailing lists are hosted externally.


About the contemplated IT services
Upgrade internet connection to 1GBit/s
Virtualization of book keeping server, district heating server, PBX to save HW (and energy).
Introduction of some "infrastructure IT", e.g.
- central account management (for computers & laptops), SSO service
- monitoring (incl. SNMP)
- directory service (LDAP)
- UPS
Some new services, also for end-users
- energy management system, with visualization
- forum
- polls
- calendar
- address book
- ...
In general a rising number of end devices is expected.

 

[Dunuin]

Your existing infrastrucure sounds really terrible for the job (I think some poeple here in the forums run more professional homelabs at home just for a single household). Sounds like stuff isn't redundant enough to run critical services like heating of a whole village. I would highly suggest to get that more professional, eliminating all the single-point-of-failures. Stacked switches and every NIC as a bonded dual NIC, so switches/NICs can fail without interuption. Get rid of all the desktop computers and buy proper (second hand) servers with redundant PSUs (connected to redundant UPSs), ECC RAM, every storage at least mirrored and enterprise/datacenter grade. Create a cluster of at least 5 PVE nodes with ceph for high availability. You will probably want additional fast (40+ Gbit) stacked switches just for Ceph communication. At least two Proxmox Backup Servers (one onsite and one offsite or in another building). Dedicated OPNsense servers that you run HA using pfsync.

Best you really hire a professional IT company to have a look at your infrastructure and tell your what to buy and how to set everything up that you get a reliable and secure infrastructure.

 

[sjjh]

Thanks for your feedback! I believe some people here in the forum pay higher focus to their homelab IT infrastructure, than our ecovillage/community does on our IT infrastructure. The heating luckily works autonomous, but out of convenience and complete data logging I indeed would want to put the server on a cluster. Currently stacked switches in all buildings seem (money wise) out of scope and not needed, as they rarely fail and we at least always have a spare one at hand (and we do not have enough glass fibers to connect the buildings redundantly anyway). But coming back to the topic of Proxmox: Why do you recommend at least 5 nodes? And why CEPH and not e.g. ZFS? And out out of failure prevention reasons (e.g. fire), is it sensible to place the PVE nodes (if I go with just 3) in three different buildings? I'm e.g. wondering about latency on the other hand.

More general, what criteria should I look at, to find a fitting solution?

 

[Dunuin]

Why do you recommend at least 5 nodes?

You always need quorum. With 5 nodes any 2 of them may fail and your cluster would still be working. Especially when on low budget you might not be able to have whole spare servers or you are running really old hardware that isn't as reliable anymore as you would like. So you might need to power down a node from time to time to fix problems, maybe it's down for some days because you need to order replacement hardware. With 5 nodes your cluster would still work if one node is powered down because of maintenance and then another server fails.
Additionally, if you shut down a server or a server just fails, all the CEPH storage and VMs will have to be migrated to the remaining nodes. So the remaining nodes always need enough spare resources to compensate the failed servers. The more nodes you got, the easier it is to compensate this.

And why CEPH and not e.g. ZFS?

ZFS is not a real shared storage. Its local storage that gets replicated at fixed intervals and is never perfectly synced. Lets say a VM is running on node A and gets replicated to nodes B and C every minute. 55 seconds after the last replication node A fails and the VM will crash. With HA the VM will be started on node B or C but the last replication was 55 seconds ago. So the VM on node B or C will lose all the data that the VM on node A wrote in those 55 seconds. So with ZFS you will have to live with losing some data on server failures.
Another point is that ZFS replication will scale really bad. Lets say you want 5 nodes with a total of 16TB of usable storage. As ZFS is a local storage, each of those 5 nodes needs 16TB of SSDs, so 80TB of SSDs in total so you can store 5 copies of everything.
With ceph you just need 3 copies of everything ( + some empty space to compensate failed servers/OSDs) spread across those 5 nodes. So the more nodes you got, the less attractive ZFS will be.

And out out of failure prevention reasons (e.g. fire), is it sensible to place the PVE nodes (if I go with just 3) in three different buildings? I'm e.g. wondering about latency on the other hand.

Corosync and ceph both need low latency and it's also recommended to use dedicated switches and NICs for these. Different buildings connected by fibre shouldn`t be a problem for corosync. For ceph you want the latency as small as possible, as every write needs to be transmitted to two other servers.

 

[sjjh]

Thanks for the comments!

So the VM on node B or C will lose all the data that the VM on node A wrote in those 55 seconds. So with ZFS you will have to live with losing some data on server failures.

That's a lot better than today! The NAS currently makes 3x/day a backup, e.g. the book keeping server only a nightly backup. Thus everything smaller 12h of data loss is an improvement, only 3 or 5 minutes would mean a HUGE improvement. ZFS seems thus sufficient to me.

So the more nodes you got, the less attractive ZFS will be.

I see. So maybe 2 nodes + quorum device with ZFS is enough. Both nodes sized in a way, that they can in case of failure of one node (at least temporarily) handle all services. I have not really a number of required disk space yet, but will try to collect some data. E.g. the NAS so far consumes 3.4TB.

I will try to collect a list of services to virtualize and their hw requirements to allow a better idea for the specific sizing.

Collecting all the input, so far I have following aspects/procedures to decide on sizing/architecture:

• accepted data loss: none: CEPH cluster; some minutes (~3-15min, depending on amount of data and network bandwith): ZFS cluster
• cluster type: CEPH: min. 3 nodes (more data, more nodes); ZFS: min. 2 nodes + quorum device
• desired availability: ??
• hw-sizing (RAM, HDD, CPU) of nodes: Sum of the hw requirements of the services to be virtualized, plus Proxmox system requirements (the statement "1 GB memory for every TB used storage" might be more relevant for CEPH than for ZFS??)
• network connection: depending on cluster type and transfered data, for CEPH: every IO-command, generally >10GBit/s, <1ms latency; ZFS: generally 10GBit/s sufficent (can be controlled a little by the sync-interval)

Looking forward to any additional input and recommendations!