Searching for the right setup (Hardware and Software).

raffael

Member
May 10, 2017
12
2
8
40
Hi

I'm going to rebuild the server infrastructure of our companies and could use some input in what direction i should go.
The main goal of this is to remove single points of failure.

What runs on the infrastructure:

Critical:
  • Samba Domain Controller (about 40 users with appData on share)
  • Samba shares (1.5TB in use)
  • File indexing/searching Service
  • Mailserver (Linux 500GB storage in use)
  • Groupware (Linux)
  • Jabber Server
  • Webservers (Company/project websites)
  • MySQL server (No heavy load)
  • DNS server
  • Ldap Server
  • Firewall (ipfire)
  • ArchiCAD Bim Server (Windows large files 200GB in use)
  • JEE Aplication Server (linux)
  • Windows Remote Login VM

Not so critical (May be installed on the old hardware):
  • Owncloud (100GB in use)
  • Java build Server.
  • Maven Repository Proxy (100GB in use)

Weekend/Night maintenance downtime is a small to no problem.
It is not a big problem if e.g. the samba shares/mail servers are offline for a have an hour (or even an hour) twice a year because of a crash.
It is a huge problem if we would have a unplanned downtime of several hours/days.

What I'm now thinking of is two of this machines: Supermicro 1028R-WTNRT
Or this one: Supermicro TwinPro 2028TP-DNCTR
And build a Proxmox HA cluster with one of the old Machines or a small additional one.
Each node with 32/64 GB Ram.
Each node with one 6/8 Core 2.4 GHz cpu.
Each node will have an Intel NVMe SSD 1.2TB (or 2 TB) for System, VMs and Fast Storage (e.g. users AppData)
For Storage about 4 disks per node 2TB SAS3 Seagate Enterprise Capacity 2.5 (7200RPM)

The nodes will be directly connected through 10GBs, for storage sync, and each is connected to the network by the other 10GBs port. For additional networks (DMZ, WLAN, Internet) I plan to install an additional 4 Port 1GBs card.


My biggest field of insecurity is the storage system (and syncing).
My first idea was to go with drbd over lvm over raid, but after reading this forum and some other resources I'm not sure if proxmox and drbd will work together good in the future.
Ceph seems to be interesting (also to learn more about) but I have the feeling that it is overkill for my needs and I would need another node and a 10GBs switch. And what I read is that even with 3 nodes ceph has to much overhead and is more for a separated storage networks.
I'm starting to read about GlusterFS but do not know much about it yet.

There are also other questions:

  • Should i go with an all SSD approach? Would it be much faster or just cost much more?
  • Is it really a good Idea to run Firewall VM, DMZ VMs and main network VMs on the same Hosts?
  • Is it a good idea to use LXC containers for DMZ services or should I use KVM vms?
  • Should I fill all CPU sockets or will the cores just be bored?
  • Am I missing something super important?

I'm thankful for any tips and hints.

Sorry I wanted to post links to the hardware parts but I'm not allowed to post them yet
 
Please stop spamming the forum with these useless messages without any content.
It's not useless. I am actually following this with interest. My response just bumped the thread, so hopefully someone could give an answer.
 
It's not useless. I am actually following this with interest. My response just bumped the thread, so hopefully someone could give an answer.

you can follow a thread without posting it again and again, please stop these postings.
 
Hello,

For HA you need a cluster FS (file system), and at least 3 nodes. For FS you can use NFS, gluster or ceph, as you mention. You must think how your setup will need to scale in the near future (1-2 years).

DRDB is an option but is tricky. Maybe you can try NFS. In this case you can go with this variants:
1. you can use glusterfs(as a mirror between 2 nodes) - then on glusterfs you can enable the nfs server
2. you can use on each of your 2 micro servers the AoE (ata over ethernet) - one one micro server you can create a mirror. This mirror will be used by nfs. You will need then to use ucarp (cluster ip) who will serve the nfs server and VIP for any proxmox container/kvm guest. In case of the primary ucarp node will be broken, the ucarp vip will go to the second micro server, then it will start the nfs server. This process will take about 2-6 seconds. Then the proxmox guests will be started on this node.

Gluster is simple to setup but it is no so fast. The second option will be better in terms of performance.
 
Why not buy an used NetApp? Then you simple use NFS and have HA without any problem. You will also save space because of compression and deduplication. You could run cifs on the NetApp directly.
 
Hi

I'm going to rebuild the server infrastructure of our companies and could use some input in what direction i should go.
The main goal of this is to remove single points of failure.

What runs on the infrastructure:

Critical:
  • Samba Domain Controller (about 40 users with appData on share)
  • Samba shares (1.5TB in use)
  • File indexing/searching Service
  • Mailserver (Linux 500GB storage in use)
  • Groupware (Linux)
  • Jabber Server
  • Webservers (Company/project websites)
  • MySQL server (No heavy load)
  • DNS server
  • Ldap Server
  • Firewall (ipfire)
  • ArchiCAD Bim Server (Windows large files 200GB in use)
  • JEE Aplication Server (linux)
  • Windows Remote Login VM

Not so critical (May be installed on the old hardware):
  • Owncloud (100GB in use)
  • Java build Server.
  • Maven Repository Proxy (100GB in use)

Weekend/Night maintenance downtime is a small to no problem.
It is not a big problem if e.g. the samba shares/mail servers are offline for a have an hour (or even an hour) twice a year because of a crash.
It is a huge problem if we would have a unplanned downtime of several hours/days.

What I'm now thinking of is two of this machines: Supermicro 1028R-WTNRT
Or this one: Supermicro TwinPro 2028TP-DNCTR
And build a Proxmox HA cluster with one of the old Machines or a small additional one.
Each node with 32/64 GB Ram.
Each node with one 6/8 Core 2.4 GHz cpu.
Each node will have an Intel NVMe SSD 1.2TB (or 2 TB) for System, VMs and Fast Storage (e.g. users AppData)
For Storage about 4 disks per node 2TB SAS3 Seagate Enterprise Capacity 2.5 (7200RPM)

The nodes will be directly connected through 10GBs, for storage sync, and each is connected to the network by the other 10GBs port. For additional networks (DMZ, WLAN, Internet) I plan to install an additional 4 Port 1GBs card.


My biggest field of insecurity is the storage system (and syncing).
My first idea was to go with drbd over lvm over raid, but after reading this forum and some other resources I'm not sure if proxmox and drbd will work together good in the future.
Ceph seems to be interesting (also to learn more about) but I have the feeling that it is overkill for my needs and I would need another node and a 10GBs switch. And what I read is that even with 3 nodes ceph has to much overhead and is more for a separated storage networks.
I'm starting to read about GlusterFS but do not know much about it yet.

There are also other questions:

  • Should i go with an all SSD approach? Would it be much faster or just cost much more?
  • Is it really a good Idea to run Firewall VM, DMZ VMs and main network VMs on the same Hosts?
  • Is it a good idea to use LXC containers for DMZ services or should I use KVM vms?
  • Should I fill all CPU sockets or will the cores just be bored?
  • Am I missing something super important?

I'm thankful for any tips and hints.

Sorry I wanted to post links to the hardware parts but I'm not allowed to post them yet

Hi, I have a similar setup also with ArchiCAD and BIM4U on a Proxmox Host.

Could you tell me wich Fileindexing Service you use? After that I'm posting the hardware setup I'm using for my customer :)

cheers Mac
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!