high performance high availabilty advice needed

[tenfoldinaus]

ok guys, got tired of paying for web hosting and piss poor IT help.
so i decided to invest some real money and host my websites and business databases/reverse etls from home
ive dropped some serious cash, and now i need the BEST PERFORMANCE setup, but im so torn about which way to go,
ill list what i have and what i need and im hoping you can give me the best performance initial install and setup advice possible.
ive been using proxmox for many years to run my own services fro home and home assistant etc, but this beast is a bit differernt

------------------------------------
Internet connection
Primary ... 10gb enterprise grade high cos ethernet connection 10gbUP/10gbDOWN ......installed ready to go
Back up ... 1gb up/down enterprise grade high cos ......installed ready to go
------------------------------------------------
network hardware
1X UBIQUITI udm SE dream machine ROUTER
2X UBIQUITI USW-EnterpriseXG-24

setup like below

Router 1x ubiquiti UDM SE dream machine with 10gb WAN1 and 1gb wan 2

10gb connection to
2x USW-EnterpriseXG-24 with a -- sfp28-- 25GB--connection-between those

-------------------------------------------------------------------------

EACH PROXMOX SERVER CONTAINS THE FOLLOWING
3x LENOVO P620 thinkstations each containing
-5995WX threadripper cpu
-256gb ram 3200mhz
1x samsung 990pro nvme 2tb
1x wdblue SN570 2tb gen4 nvme
2x quad nvme riser adaptors gen 3 (so 8x samsung evo970 2tb ea so 16tb in total)
2x Seagate 18TB Exos 7200 HDD so 36tb in total
1x 10gb onboard sfp+ 10gb connection
1x dual 10gb sfp+ 10gb pci xt550 intel (so 3x 10gb connections in total PER SERVER)

i was thinking
of setting up a proxmox cluster with the 3 servers.

7 websites all running on WORDPRESS ( i know dont get me started) but this is less about hosting the actual sites. becuase you can only squeeze so much out of the dogs breakfast that is wordpress.

its more about,
i self host JITSU (a open source tracking solution which deploys analytics js, and transfer data to my own data warehouses, primarily run on clickhouse.
all on currently using ubuntu server VMs and microk8s, but they dont strictly need to gbe on k8s, i was just experimenting (to be honest ive found the whole k8s on LXC to be super difficult and gave up instead using the ubuntu vms)., and i run many other different little services in LXC containers.
etc. and i use NGINX proxy manager for those , as an easy solution, and handle ingress for

i use POWER bi to run alot of queries and reports from these warehouses.

here is the issue.
for the last 6 months i was waiting for the final server to arrive so i played around with different storage and setups
like zfs, raid , lvm, etc., then MAYASTOR on kubernates etc
but i cannot for the life of me figure out which way to go.

so im hoping you can give some advise.
what i want is
obviously redundancy,
but i want the fastest i can do for everything.

how would you setup the storage,
1. which drive to install proxmox on , and what storage method, given 3 server cluster
1a, should i even use a proxmox cluster, i struggle to understand the difference between using k8s vs proxmox cluster is it essentially the same thing, and k8s wont offer any advantages vs just using a proxmox cluster. apart from scaling
2. which drives and storage method to setup for websites (just running on LXC webmins)
3. which devices and storage method to setup for clickhouse DB (need to get the fastest queries but also dependable redundancy)
4. i can install jitsu and DBs on kubernates or in LXC containers (will i get any performance benefits using kubernates and mayastor vs lxc and docker versions)
5. would like to automate proxmox backups to the HDD's using the backup server , thinking this will be ok?.

zfs seems a waste of resources and pretty slow, but i havent tried raid 10 etc,
my ego wants the fastest of everything but i know redundancy is super important too.....urgh!!

any help will be appreciated.

 

[VictorSTS]

IMHO, you should have planned for which architecture you want, then buy the appropriate hardware for it.

- Those switches do not support stacking, so no LACP for your bonds.
- Besides the EXOS, every other disk is consumer grade, you will get (maybe much) lower performance and worst resiliency (they all lack PLP and have low endurance) than with enterprise grade disks, specially with ZFS[1] or Ceph[2]. This limits your choices to provide disk redundancy.
- Your backups must be outside your servers: if the whole server breaks, your backups will be unavailable too. A fourth computer running PBS would be the recommended setup.

Have not used mayastor, clickhouse or Jitsu, barely used k8s, so can't really recommend on that.

[1] https://openzfs.github.io/openzfs-docs/Performance and Tuning/Hardware.html#hard-drives
[2] https://docs.ceph.com/en/quincy/start/hardware-recommendations/#data-storage

 

[tenfoldinaus]

actually,
to be fair,i havent paid for any of the disks,
they were all free
happy to purchase enterprise grade disks and 4th server for pbs
all the samsung evos
were given by a friend who closed his shop, but i think they will be ok for running PVE and other containers,
maybe no good for databases etc.

 

[tenfoldinaus]

IMHO, you should have planned for which architecture you want, then buy the appropriate hardware for it.

- Those switches do not support stacking, so no LACP for your bonds.
- Besides the EXOS, every other disk is consumer grade, you will get (maybe much) lower performance and worst resiliency (they all lack PLP and have low endurance) than with enterprise grade disks, specially with ZFS[1] or Ceph[2]. This limits your choices to provide disk redundancy.
- Your backups must be outside your servers: if the whole server breaks, your backups will be unavailable too. A fourth computer running PBS would be the recommended setup.

Have not used mayastor, clickhouse or Jitsu, barely used k8s, so can't really recommend on that.

[1] https://openzfs.github.io/openzfs-docs/Performance and Tuning/Hardware.html#hard-drives
[2] https://docs.ceph.com/en/quincy/start/hardware-recommendations/#data-storage

oh and forgot to mention i also have a USW-Pro-Aggregation
so that fixes another problem

and access to 3x quad sfp28 nics if i need , which to be honest i probably will just swap them out intel E810, but wasnt sure if they were compatible with proxmox 8 yet, being so new.. so didnt want to confuse the topic too much

 

[sb-jw]

You seem to be very resistant to advice in this regard. I actually thought that you wanted to do better than everyone else and that you depend on these services. But apparently that's not the case, because you're using hardware that you'll sooner or later regret using. You'll potentially lose money and time, and if you're unlucky, data too.

What @VictorSTS wrote there is absolutely no nonsense but rather state of the art if you want to do it seriously. Your SSDs are certainly sufficient for a home lab, I wouldn't use something like that in my machines for serious things.

 

[wbk]

Hi tenfoldinaus,

so i decided to invest some real money and host my websites and business databases/reverse etls from home
ive dropped some serious cash, and now i need the BEST PERFORMANCE setup, but im so torn about which way to go,

While your splashing on hardware, I have some more useless advice: invest in a keyboard with a working "shift" and "point" key, besides caps lock and comma.

It will you when you try to write your story, and give people who might help you the idea that you take them serious.

 

[tenfoldinaus]

You seem to be very resistant to advice in this regard. I actually thought that you wanted to do better than everyone else and that you depend on these services. But apparently that's not the case, because you're using hardware that you'll sooner or later regret using. You'll potentially lose money and time, and if you're unlucky, data too.

What @VictorSTS wrote there is absolutely no nonsense but rather state of the art if you want to do it seriously. Your SSDs are certainly sufficient for a home lab, I wouldn't use something like that in my machines for serious things.

im not resistant to anything, happy to take advise and happy to replace the ssd's was just saying they were free,
if they are no good, then in the bin they go guys,
i just need to know what to get.
and ill go shopping, apart from the ssd, i dont think there is too much drama withe the remaining hardware right?

 

[tenfoldinaus]

Hi tenfoldinaus,

While your splashing on hardware, I have some more useless advice: invest in a keyboard with a working "shift" and "point" key, besides caps lock and comma.

It will you when you try to write your story, and give people who might help you the idea that you take them serious.

wow your not helpful at all

 

[tenfoldinaus]

just to clarify,
im completely open to people telling me to go buy new hardware,
no problems,
i didnt post the topic to argue,
i need help, and dont want to regret things later,
so im all ears.

 

[LnxBil]

So, let us recap:

• buy better (=enterprise) SSDs
• buy better network equipment that can do at least LACP, I recommend two switches with MLAG, no stacked switch (too much bad experience up to total disaster and cheaper)
• buy hardware for PBS
• use CEPH for high availability

 

[sb-jw]

To make it a little more concrete, here are my preferences as to which hardware I would choose.

buy better (=enterprise) SSDs

I would recommend Samsung SM/PM 863a/883 or 893.

buy better network equipment that can do at least LACP, I recommend two switches with MLAG, no stacked switch (too much bad experience up to total disaster and cheaper)

I would recommend Arista switches here. The 10G SFP+ switches are already available at a reasonable price, at least in Germany. The switches are definitely Enterprise Grade and bring you maximum low latency in CEPH and support LACP with Layer3+4 They also have MLAG support (not VC like older Juniper, maybe a QFX5100 would be an alternative for you).

buy hardware for PBS

Something like an E3-1220 v5 with 16 GB RAM and SSD Only is sufficient (e.g. Dell PowerEdge R330 with 8x SFF). As SSDs, you should use similar ones as mentioned above.

use CEPH for high availability

I absolutely support this statement!

 

[tenfoldinaus]

To make it a little more concrete, here are my preferences as to which hardware I would choose.

I would recommend Samsung SM/PM 863a/883 or 893.

I would recommend Arista switches here. The 10G SFP+ switches are already available at a reasonable price, at least in Germany. The switches are definitely Enterprise Grade and bring you maximum low latency in CEPH and support LACP with Layer3+4 They also have MLAG support (not VC like older Juniper, maybe a QFX5100 would be an alternative for you).

Something like an E3-1220 v5 with 16 GB RAM and SSD Only is sufficient (e.g. Dell PowerEdge R330 with 8x SFF). As SSDs, you should use similar ones as mentioned above.

I absolutely support this statement!

thank god for you my man!