Shared-nothing Architecture with proxmox ?

In general yes, but it depends on your project.
 
i think i will subscribe to your product, many goodies since 2009 ! (first truy of proxmox).
The aim is to build internet service provider architecture (very little , ok ;-)).
i have constraint with hardware.
i don't want to deploy many and many nodes for redundancy ! (begins with 6 or 8)
Actually i plan to design a node like this :
- MicroATX with core i5 3470, 16Go/32Go ram,
- two HDD/SDD of 500GB each,
- atx psu
- rack shelf 2U

And nothing else for nodes, they must be independent each other.
I only plan to do HA for strategic service such as DNS, LDAP. (not apache, mysql etc)

This hardware inspire me :
https://www.thingiverse.com/thing:3024082
The guy is building this solution , is working with shared-nothing architecture !

Here is his "how to" (google traduc sorry)

We operate in "grid computing" mode, we do not use a tiered architecture because they create single point of failure (SPoF).
Depending on the client and the expected load, we allocate a number of nodes (1 node = 1 server). Each node is front-end web and database. It's 'shared-nothing', ie all the nodes that make up a grid have all the data.
In this configuration, all the intelligence is in the database, the cache, the inter-node exchanges. We developed everything in C.
Once we have this: 'each node is independent in resource', we end up with an architecture where the load distribution and the 'fail-over' are very simple. It can be done in pure software with, for example under linux, IPVS.
The nodes of the same grid are distributed over several bays see several datacenters.
The beauty in all this is that a grid can then discuss with another grid, etc ... each cluster is reliable and all SPoF are eliminated.
We are left with an architecture where the physical servers are "disposable", they can leave the production in a simple command, be allocated in another cluster and resynchronized, ...

If it is in file server mode, to simply raid-1 network, there are many solutions that exist: GlusterFS, Luster, drdb, .. or if the replication should not be instantaneous, just do of rsync.
 
I only plan to do HA for strategic service such as DNS, LDAP. (not apache, mysql etc)
This defies the shared-nothing principal. For HA you need to cluster the nodes together and use a shared storage (of sorts).

This hardware inspire me :
https://www.thingiverse.com/thing:3024082
The guy is building this solution , is working with shared-nothing architecture !
If you have disposable services for a limited amount of time, then such a approach could be taken. Long running services, as you seem to plan, should be run on enterprise grade hardware (eg. ECC, 24/7 runtime).
 
Actually i plan to design a node like this :
- MicroATX with core i5 3470, 16Go/32Go ram,
- two HDD/SDD of 500GB each,
- atx psu
- rack shelf 2U

Sounds like refurb Dell Optiplex 9010 with Intel AMT could be a cheaper option for you ;), but how to fit them into a 2U rack shelf would be another question.

I only plan to do HA for strategic service such as DNS, LDAP. (not apache, mysql etc)
This defies the shared-nothing principal. For HA you need to cluster the nodes together and use a shared storage (of sorts).

I think the OP intends to do something like hyper-converged infrastructure in Windows 2016/2019 (Hyper-V + S2D without traditional shared storage)?

I'm under the impression Promox with Ceph can achieve something similar as well, but about the HA part, I think it depends on what type of HA you're after, or to what extend. If it's just storage, then Ceph also does the job, but if you're after fault-tolerance at the VM level like how VMware vSphere can do it, then neither Proxmox nor Hyper-V with MS failover clustering is for you (please correct me if I'm on the wrong track).

What HA at the hypervisor level does is if a host in the cluster is down, then the VMs will be restarted back onto another host. Depending on one's requirements, but to me (purely based off my experience), this doesn't really bring much practical benefit as sometimes a file system check may kick in due to different reasons (I know this can be disabled) or other unexpected problems due to the crash.

I only plan to do HA for strategic service such as DNS, LDAP. (not apache, mysql etc)

Instead of spending all the effort of setting up/maintaining an HA cluster at the hypervisor level, I strongly recommend one consider achieving redundancy at the application/code level. For example, the native/recommended way of achieving redundancy for MS Active Directory is simply by creating multiple domain controllers in different places, no fancy HA/clustering and shared storage required (in fact, it should be avoided). Also with DNS, redundancy should be done at the DNS level as well.

That said, granted the hyper-converged infrastructure you're after may not be able to do fault-tolerance at the VM level, it is still an interesting concept for people who wish to utilise relatively cheaper and less hardware.
 
Hi,
I am a fan of using ...not so good hardware with proxmox, however I can't see any use for proxmox here.
Your nodes are not running VMs so why proxmox? Also, with 6-8 nodes you'll be OK but you will not have a spare ethernet to run the clustering on. If you try to expand (on success) then you will get stuck at 16-32 nodes I think.

You want something like FOG - free open source ghost to provision your nodes. You can create images on the nodes and then store them in FOG for later deployment.
If you want something more enterprisey then packstack (simpler openstack).

Just my opinion.
Andrew
 
i read hyper-converged...
so, is my hardware, with cephfs (or rbd), a descent solution ?
will ceph kill my 16 GB node, with core i5 cpu ? ...

this schema will be fine ? according to you ?
 

Attachments

  • Infra-002.png
    Infra-002.png
    196.9 KB · Views: 10
Last edited:
i really like raid1 with mdadm ... very easy to repair and maintain ...
(i have only two , HDD per node)
So i don't know if i must choose glustefFS , cephfs or stay with LVM-THIN (so local FS) but without cold migration possibility
 
Last edited:
I don't see, where you want to go with your setup. From my point of view, you didn't define a clear goal and requirements. I suggest that you may read up a little more on Proxmox VE, to get a feel for what it could help you to achieve.
https://pve.proxmox.com/pve-docs/pve-admin-guide.html
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!