hiya guys,
long story short
i own a successful online ecommerce site
but i have downscaled considerably and semi retired,
but still needed things to keep me busy at home. i enjoy tinkering and decided to host all myself from home and get rid of paid hosts and (hidden agenda i wanted an excuse to fit a 10gb symmetric high cos fibre internet connection to my home
along with a dedicated UPS to keep my business and cheekily my HIGHLY automated home running in the event of power outage.
currently things are running all over the place across various nodes (no failsafe and no HA....i know it makes me cringe when i think about it too, considering its my family source of income
but i have been procrastinating for months because i cant make a decision on best way to proceed for OPTIMAL PERFORMANCE/and HIGH AVAILABILTY
i was hoping to setup 2 nodes and run a 3rd qdevice to keep the quorum , but i could incorporate a third (however i really dont want to if i can help it)
what i cant decide is the best storage and network setup for provisioning each node.
i have some zfs pools currently and lvm /lvmthin but nodes are now just running with default host file system in ex4
my questions are
1. which device on each node should i install the host and what is the best file system for it.
2. how should i provision the remaining storage components, zfsraid or ceph , or should i do something differnt.
3. the qdevice does have capacity enough to run proxmox backup server or should i do it another way.
4. 2 nodes are side by side, and so is the available upstream aggregation layer3 switch.. so what the best way to provision/maximise my networking capacity for saturation of available links should i aggregate, or maybe make a direct link from node1 to node2 given each node has 5LAN ports
i can handle a small amount of data loss by going the zfs route as the majority is database/statistical data , so a small loss between replication is fine. but if can be avoided would be ideal.
but are there are any drawbacks/complexities im unaware of having (not used ceph before)
it makes me a bit apprehensive and all the mixed info out there
anyways here is the kit.
2x HIGHLY CUSTOMISED LENOVO P620 THINKSTATIONS
[[cpu and RAM]]
-5995WX threadripper cpu
-512gb ram 3200mhz over all 8 lanes (possible to upgrade to 1tb if needed
[[storage]]
nvme slot 1 -4tb micron 7450 m2 GEN4 ENTERPRISE GRADE
nvme slot 2 -4tb micron 7450 m2 GEN4 ENTERPRISE GRADE ENTERPRISE GRADE
PCIE 16x slot 1 1x quad nvme riser with 4x 2tb gen3 nvme { so essentially } ENTERPRISE GRADE
PCIE 16x slot 2 1x quad nvme riser with 4x 2tb gen3 nvme { 16TB over 8cards ^ } ENTERPRISE GRADE
2x Seagate 18TB Exos 7200 HDD so (36tb in total) mainly nvr footage and proxmox backups but can as recommended
[[LAN]]
PCIE 16X slot 3 CX4121A MCX4121A-ACAT DUAL port 25gb sfp28
onboard LAN 1x 10gb intel copper
PCIE SLOT 4 1x x550t2 intel DUAL 10gb copper
essentially 2x 25gb sfp28 & 3x10gb coper
[[NON ESSENTIALS]] (1400w power supply, nvidia RTX A4500 20GB GDDR6
upstream available connections
10gb copper ports---available including aggregation switch LADCP LAYER 3 SWITCH ---(dedicated 10gb/10gb symmetric upstream fibreHIGH COS)
&
4x 25gb sfp28 ports available into same as above aggregation switch LADCP LAYER 3 SWITCH ---(dedicated 10gb/10gb symmetric upstream fibreHIGH COS)
[[current vms and containers ( all containers are runnning across different nodes currently)]]
preffered ---[business and HA critical]
1x lxc wordpress site
1x maridb lxc container for website db
1x lxc running alpine docker for Jitsu (essentially i self host segment/analyticis js and serve this subdomain from here)
1x lxc container running nginx proxy manager (various subdomains and other ingestions etc]
1x lxc running alpine docker (clickhouse db WEBSITE ANALYTICS DATA)
3x ubuntu server VM (for running 3x microk8s)
1x lxc alpine docker running MONGO DB
[[personal container]]
1x vm for home assistant in supervisor setup
1x alpine docker container for frigate nvr
long story short
i own a successful online ecommerce site
but i have downscaled considerably and semi retired,
but still needed things to keep me busy at home. i enjoy tinkering and decided to host all myself from home and get rid of paid hosts and (hidden agenda i wanted an excuse to fit a 10gb symmetric high cos fibre internet connection to my home
along with a dedicated UPS to keep my business and cheekily my HIGHLY automated home running in the event of power outage.currently things are running all over the place across various nodes (no failsafe and no HA....i know it makes me cringe when i think about it too, considering its my family source of income
but i have been procrastinating for months because i cant make a decision on best way to proceed for OPTIMAL PERFORMANCE/and HIGH AVAILABILTY
i was hoping to setup 2 nodes and run a 3rd qdevice to keep the quorum , but i could incorporate a third (however i really dont want to if i can help it)
what i cant decide is the best storage and network setup for provisioning each node.
i have some zfs pools currently and lvm /lvmthin but nodes are now just running with default host file system in ex4
my questions are
1. which device on each node should i install the host and what is the best file system for it.
2. how should i provision the remaining storage components, zfsraid or ceph , or should i do something differnt.
3. the qdevice does have capacity enough to run proxmox backup server or should i do it another way.
4. 2 nodes are side by side, and so is the available upstream aggregation layer3 switch.. so what the best way to provision/maximise my networking capacity for saturation of available links should i aggregate, or maybe make a direct link from node1 to node2 given each node has 5LAN ports
i can handle a small amount of data loss by going the zfs route as the majority is database/statistical data , so a small loss between replication is fine. but if can be avoided would be ideal.
but are there are any drawbacks/complexities im unaware of having (not used ceph before)
it makes me a bit apprehensive and all the mixed info out there
anyways here is the kit.
2x HIGHLY CUSTOMISED LENOVO P620 THINKSTATIONS
[[cpu and RAM]]
-5995WX threadripper cpu
-512gb ram 3200mhz over all 8 lanes (possible to upgrade to 1tb if needed
[[storage]]
nvme slot 1 -4tb micron 7450 m2 GEN4 ENTERPRISE GRADE
nvme slot 2 -4tb micron 7450 m2 GEN4 ENTERPRISE GRADE ENTERPRISE GRADE
PCIE 16x slot 1 1x quad nvme riser with 4x 2tb gen3 nvme { so essentially } ENTERPRISE GRADE
PCIE 16x slot 2 1x quad nvme riser with 4x 2tb gen3 nvme { 16TB over 8cards ^ } ENTERPRISE GRADE
2x Seagate 18TB Exos 7200 HDD so (36tb in total) mainly nvr footage and proxmox backups but can as recommended
[[LAN]]
PCIE 16X slot 3 CX4121A MCX4121A-ACAT DUAL port 25gb sfp28
onboard LAN 1x 10gb intel copper
PCIE SLOT 4 1x x550t2 intel DUAL 10gb copper
essentially 2x 25gb sfp28 & 3x10gb coper
[[NON ESSENTIALS]] (1400w power supply, nvidia RTX A4500 20GB GDDR6
upstream available connections
10gb copper ports---available including aggregation switch LADCP LAYER 3 SWITCH ---(dedicated 10gb/10gb symmetric upstream fibreHIGH COS)
&
4x 25gb sfp28 ports available into same as above aggregation switch LADCP LAYER 3 SWITCH ---(dedicated 10gb/10gb symmetric upstream fibreHIGH COS)
[[current vms and containers ( all containers are runnning across different nodes currently)]]
preffered ---[business and HA critical]
1x lxc wordpress site
1x maridb lxc container for website db
1x lxc running alpine docker for Jitsu (essentially i self host segment/analyticis js and serve this subdomain from here)
1x lxc container running nginx proxy manager (various subdomains and other ingestions etc]
1x lxc running alpine docker (clickhouse db WEBSITE ANALYTICS DATA)
3x ubuntu server VM (for running 3x microk8s)
1x lxc alpine docker running MONGO DB
[[personal container]]
1x vm for home assistant in supervisor setup
1x alpine docker container for frigate nvr
Last edited: