Shared-nothing Architecture with proxmox ?
- Thread starter atec666
- Start date
i think i will subscribe to your product, many goodies since 2009 ! (first truy of proxmox).
The aim is to build internet service provider architecture (very little , ok ;-)).
i have constraint with hardware.
i don't want to deploy many and many nodes for redundancy ! (begins with 6 or 8)
Actually i plan to design a node like this :
- MicroATX with core i5 3470, 16Go/32Go ram,
- two HDD/SDD of 500GB each,
- atx psu
- rack shelf 2U
And nothing else for nodes, they must be independent each other.
I only plan to do HA for strategic service such as DNS, LDAP. (not apache, mysql etc)
This hardware inspire me :
https://www.thingiverse.com/thing:3024082
The guy is building this solution , is working with shared-nothing architecture !
Here is his "how to" (google traduc sorry)
We operate in "grid computing" mode, we do not use a tiered architecture because they create single point of failure (SPoF).
Depending on the client and the expected load, we allocate a number of nodes (1 node = 1 server). Each node is front-end web and database. It's 'shared-nothing', ie all the nodes that make up a grid have all the data.
In this configuration, all the intelligence is in the database, the cache, the inter-node exchanges. We developed everything in C.
Once we have this: 'each node is independent in resource', we end up with an architecture where the load distribution and the 'fail-over' are very simple. It can be done in pure software with, for example under linux, IPVS.
The nodes of the same grid are distributed over several bays see several datacenters.
The beauty in all this is that a grid can then discuss with another grid, etc ... each cluster is reliable and all SPoF are eliminated.
We are left with an architecture where the physical servers are "disposable", they can leave the production in a simple command, be allocated in another cluster and resynchronized, ...
If it is in file server mode, to simply raid-1 network, there are many solutions that exist: GlusterFS, Luster, drdb, .. or if the replication should not be instantaneous, just do of rsync.
The aim is to build internet service provider architecture (very little , ok ;-)).
i have constraint with hardware.
i don't want to deploy many and many nodes for redundancy ! (begins with 6 or 8)
Actually i plan to design a node like this :
- MicroATX with core i5 3470, 16Go/32Go ram,
- two HDD/SDD of 500GB each,
- atx psu
- rack shelf 2U
And nothing else for nodes, they must be independent each other.
I only plan to do HA for strategic service such as DNS, LDAP. (not apache, mysql etc)
This hardware inspire me :
https://www.thingiverse.com/thing:3024082
The guy is building this solution , is working with shared-nothing architecture !
Here is his "how to" (google traduc sorry)
We operate in "grid computing" mode, we do not use a tiered architecture because they create single point of failure (SPoF).
Depending on the client and the expected load, we allocate a number of nodes (1 node = 1 server). Each node is front-end web and database. It's 'shared-nothing', ie all the nodes that make up a grid have all the data.
In this configuration, all the intelligence is in the database, the cache, the inter-node exchanges. We developed everything in C.
Once we have this: 'each node is independent in resource', we end up with an architecture where the load distribution and the 'fail-over' are very simple. It can be done in pure software with, for example under linux, IPVS.
The nodes of the same grid are distributed over several bays see several datacenters.
The beauty in all this is that a grid can then discuss with another grid, etc ... each cluster is reliable and all SPoF are eliminated.
We are left with an architecture where the physical servers are "disposable", they can leave the production in a simple command, be allocated in another cluster and resynchronized, ...
If it is in file server mode, to simply raid-1 network, there are many solutions that exist: GlusterFS, Luster, drdb, .. or if the replication should not be instantaneous, just do of rsync.
This defies the shared-nothing principal. For HA you need to cluster the nodes together and use a shared storage (of sorts).
If you have disposable services for a limited amount of time, then such a approach could be taken. Long running services, as you seem to plan, should be run on enterprise grade hardware (eg. ECC, 24/7 runtime).
Sounds like refurb Dell Optiplex 9010 with Intel AMT could be a cheaper option for you
, but how to fit them into a 2U rack shelf would be another question.I think the OP intends to do something like hyper-converged infrastructure in Windows 2016/2019 (Hyper-V + S2D without traditional shared storage)?
I'm under the impression Promox with Ceph can achieve something similar as well, but about the HA part, I think it depends on what type of HA you're after, or to what extend. If it's just storage, then Ceph also does the job, but if you're after fault-tolerance at the VM level like how VMware vSphere can do it, then neither Proxmox nor Hyper-V with MS failover clustering is for you (please correct me if I'm on the wrong track).
What HA at the hypervisor level does is if a host in the cluster is down, then the VMs will be restarted back onto another host. Depending on one's requirements, but to me (purely based off my experience), this doesn't really bring much practical benefit as sometimes a file system check may kick in due to different reasons (I know this can be disabled) or other unexpected problems due to the crash.
Instead of spending all the effort of setting up/maintaining an HA cluster at the hypervisor level, I strongly recommend one consider achieving redundancy at the application/code level. For example, the native/recommended way of achieving redundancy for MS Active Directory is simply by creating multiple domain controllers in different places, no fancy HA/clustering and shared storage required (in fact, it should be avoided). Also with DNS, redundancy should be done at the DNS level as well.
That said, granted the hyper-converged infrastructure you're after may not be able to do fault-tolerance at the VM level, it is still an interesting concept for people who wish to utilise relatively cheaper and less hardware.
Hi,
I am a fan of using ...not so good hardware with proxmox, however I can't see any use for proxmox here.
Your nodes are not running VMs so why proxmox? Also, with 6-8 nodes you'll be OK but you will not have a spare ethernet to run the clustering on. If you try to expand (on success) then you will get stuck at 16-32 nodes I think.
You want something like FOG - free open source ghost to provision your nodes. You can create images on the nodes and then store them in FOG for later deployment.
If you want something more enterprisey then packstack (simpler openstack).
Just my opinion.
Andrew
I am a fan of using ...not so good hardware with proxmox, however I can't see any use for proxmox here.
Your nodes are not running VMs so why proxmox? Also, with 6-8 nodes you'll be OK but you will not have a spare ethernet to run the clustering on. If you try to expand (on success) then you will get stuck at 16-32 nodes I think.
You want something like FOG - free open source ghost to provision your nodes. You can create images on the nodes and then store them in FOG for later deployment.
If you want something more enterprisey then packstack (simpler openstack).
Just my opinion.
Andrew
I don't see, where you want to go with your setup. From my point of view, you didn't define a clear goal and requirements. I suggest that you may read up a little more on Proxmox VE, to get a feel for what it could help you to achieve.
https://pve.proxmox.com/pve-docs/pve-admin-guide.html
https://pve.proxmox.com/pve-docs/pve-admin-guide.html