Proxmox 3 and DRBD?

[ericmachine]

Hi everyone,

Earlier I had a discussion with a friend and suggested to use this setup for Proxmox DRBD.

Server 1 (active) -> Server 2 (passive) -> Server 3 (passive)

Each server will have 2 VMs.
- App Server VM
- DB VM

Therefore,

Server 1's proxmox running - VM1 Start, VM2 Start
Server 2's proxmox running - VM1 Stop, VM2 Stop
Server 3's proxmox running - VM1 Stop, VM2 Stop

My questions:-
i. Why do I need 3 servers? Why can't I use Server 1 -> Server 2?
ii. If my Server 1 breaks down, will it automatically make Server 2 as primary? I assume it will auto-start my Server 2 VMs right?
iii. The tech specs for my servers as below.

Server 1 - 2 x processors, 32GB RAM and 1TB RAID 6 + 1 Hot Swap
Server 2 & 3 - 1 x processor, 32GB RAM and 1TB RAID 1

I assume DRBD will work for the above specs right? Or Do I have to get all 3 servers to be same specs?

Any help? Thanks.

 

[udo]

Hi everyone,

Earlier I had a discussion with a friend and suggested to use this setup for Proxmox DRBD.

Server 1 (active) -> Server 2 (passive) -> Server 3 (passive)

Each server will have 2 VMs.
- App Server VM
- DB VM

Therefore,

Server 1's proxmox running - VM1 Start, VM2 Start
Server 2's proxmox running - VM1 Stop, VM2 Stop
Server 3's proxmox running - VM1 Stop, VM2 Stop

My questions:-
i. Why do I need 3 servers? Why can't I use Server 1 -> Server 2?
ii. If my Server 1 breaks down, will it automatically make Server 2 as primary? I assume it will auto-start my Server 2 VMs right?
iii. The tech specs for my servers as below.

Server 1 - 2 x processors, 32GB RAM and 1TB RAID 6 + 1 Hot Swap
Server 2 & 3 - 1 x processor, 32GB RAM and 1TB RAID 1

I assume DRBD will work for the above specs right? Or Do I have to get all 3 servers to be same specs?

Any help? Thanks.

Hi,
ontop of the drbd-volume must be an lvm-storage. pve need an active vg on each node, which has access to this storage. This mean, that you can't go for an active-passive scenario.

With pve you have allways an active-active drbd over two nodes.
And why you wan't 3 nodes running for only 2 VMs on one node? That's a big waste of resources.

The third node is only for quorum and don't need very powerfull. On the other nodes you should use two drbd-ressources (each for one node, because of split brain).

For speed go for raid 10 and fast interconnect between the two nodes (10GB Ethernet or Infiniband).

Udo

 

[ericmachine]

Hi,
ontop of the drbd-volume must be an lvm-storage. pve need an active vg on each node, which has access to this storage. This mean, that you can't go for an active-passive scenario.

With pve you have allways an active-active drbd over two nodes.
And why you wan't 3 nodes running for only 2 VMs on one node? That's a big waste of resources.

The third node is only for quorum and don't need very powerfull. On the other nodes you should use two drbd-ressources (each for one node, because of split brain).

For speed go for raid 10 and fast interconnect between the two nodes (10GB Ethernet or Infiniband).

Udo

Hi Udo,

Thanks for the tips

Just to clarify further.

a. When you say 1 node? Do you mean 1 physical server?
b. When you say with pve, always on active-active drbd over 2 nodes.

I assume you mean 2 servers.

Thus,

server 1's proxmox running - VM1 start, VM2 start (active 1)
server 2's proxmox running - VM1 start, VM2 start (active 2)
server 3's proxmox running - VM1 stop, VM2 stop (passive)

is that what you meant?

c. When you say this And why you wan't 3 nodes running for only 2 VMs on one node? That's a big waste of resources.

I assume you mean 3 physical servers. Actually I have 2 VMs with specs like this:-

Server

VM1 - Application Server - 100GB HDD, 15GB vmemory
VM2 - Postgresql DB - 700GB HDD, 15GB vmemory

Note: I will be installing OpenERP 7 here, thus I need to make sure the ERP works. This servers will manage around 4 - 5 businesses each site.

Note: Each site will have the same servers mentioned above

Note: each physical server will have 1TB physical disk and 32GB physical memory. The only difference is Server 1 has 2 x processors while other 2 has 1 processor.

Is there anyway I can optimize further to reduce costs? Say reduce servers, etc?

d. When you say The third node is only for quorum and don't need very powerfull

Sorry I am not that sure what's a quorum. But I read it in google and it stated as "The cluster configuration database, also called the quorum, tells the cluster which physical server(s) should be active at any given time."

Does that mean my Server 3 doesn't need to even have any VMs? Just a normal server will do?

say

1 processor, 4GB RAM, 1TB HDD can do the job? Just proxmox, but no VMs like Server 1 and 2?

e. When you say For speed go for raid 10 and fast interconnect between the two nodes (10GB Ethernet or Infiniband).

Do you mean I need to have RAID10 for all my servers - Server 1, Server 2 & Server 3?

First I don't think my database will even go up to 100GB that soon. Plus even though it's an important ERP system, but I doubt it's as mission critical compares to bank system. Is RAID10 a bit overkilled? Plus I assume RAID10 I need to have more HDD for each server?

I do prefer to reduce costs instead of increase hopefully raid6 can still do the job

Any help? Thanks.

 

[udo]

Hi Udo,

...
a. When you say 1 node? Do you mean 1 physical server?
b. When you say with pve, always on active-active drbd over 2 nodes.

I assume you mean 2 servers.

node mean pve-clusternode (physical server)

...
server 1's proxmox running - VM1 start, VM2 start (active 1)
server 2's proxmox running - VM1 start, VM2 start (active 2)
server 3's proxmox running - VM1 stop, VM2 stop (passive)

is that what you meant?

This is not the real viewing if you use shared storage (like DRBD).
You can start an VM on each node, which have access to the shared storage - the VM-config is available for all clusternodes (on the virtual filesystem /etc/pve).
In yourcase on server3 isn't possible to start the VMs which are on a drbd-volume.

c. When you say this And why you wan't 3 nodes running for only 2 VMs on one node? That's a big waste of resources.

I assume you mean 3 physical servers. Actually I have 2 VMs with specs like this:-

Server

VM1 - Application Server - 100GB HDD, 15GB vmemory
VM2 - Postgresql DB - 700GB HDD, 15GB vmemory

Note: I will be installing OpenERP 7 here, thus I need to make sure the ERP works. This servers will manage around 4 - 5 businesses each site.

Note: Each site will have the same servers mentioned above

Note: each physical server will have 1TB physical disk and 32GB physical memory. The only difference is Server 1 has 2 x processors while other 2 has 1 processor.

Is there anyway I can optimize further to reduce costs? Say reduce servers, etc?

d. When you say The third node is only for quorum and don't need very powerfull

Sorry I am not that sure what's a quorum. But I read it in google and it stated as "The cluster configuration database, also called the quorum, tells the cluster which physical server(s) should be active at any given time."

With virtualisation normaly IO (and RAM) is the bottleneck (before CPU-power) but this depends on the usage.
But you can normaly run much more than 2 VMs on one node.

Does that mean my Server 3 doesn't need to even have any VMs? Just a normal server will do?

say

1 processor, 4GB RAM, 1TB HDD can do the job? Just proxmox, but no VMs like Server 1 and 2?

yes - like this http://www.amazon.com/HP-ProLiant-MicroServer-704941-001-Server/dp/B00AKWUZ58/ref=sr_1_1?ie=UTF8

e. When you say For speed go for raid 10 and fast interconnect between the two nodes (10GB Ethernet or Infiniband).

Do you mean I need to have RAID10 for all my servers - Server 1, Server 2 & Server 3?

why you want to use an raid10 on node3 whitout VMs??

First I don't think my database will even go up to 100GB that soon. Plus even though it's an important ERP system, but I doubt it's as mission critical compares to bank system. Is RAID10 a bit overkilled? Plus I assume RAID10 I need to have more HDD for each server?

raid-10 isn't overkill. I have good experiences with 4 * SAS 600GB 15K with raid10 (and areca raidcontroller).

I do prefer to reduce costs instead of increase hopefully raid6 can still do the job

Depends on your raidcontroller and the disks - do benchmarking and tests with your setup. But be sure, raid-6 with sata disks are not the real thing for databases.
SAS-drives (not the near-line (sata disks with sas-controller)) can handle multible access much better.

Udo

 

[ericmachine]

Hi everyone,

I plan to buy IBM servers due to the on-site support services (it has to be IBM). I check with them and they said not tested proxmox from their end. Just need to make sure it does work. Can this work with Proxmox 3.0/3.1 DRBD?

Any help?

Primary Server

x3650-M4 (7915-C4A)

4

x3650 M4, Xeon 6C E5-2620 95W 2.0GHz/1333MHz/15MB, 1x8GB, O/Bay HS 3.5in SAS/SATA, SR M5110e, 550W p/s, Rack

Intel Xeon 6C Processor Model E5-2620 95W 2.0GHz/1333MHz/15MB W/Fan

8GB (1x8GB, 2Rx4, 1.35V) PC3L-10600 CL9 ECC DDR3 1333MHz LP RDIMM

Internal Storage

IBM 2TB 7.2K 6Gbps NL SAS 3.5in G2HS HDD

ServeRAID M5100 Series 1GB Flash/RAID 5 Upgrade for IBM System x

ServeRAID M5100 Series RAID 6 Upgrade for IBM System x

IBM Integrated Management Module Advanced Upgrade

IBM System x 550W High Efficiency Platinum AC Power Supply

IBM UltraSlim Enhanced SATA Multi-Burner

x3650 M4 ODD Cable Tape Enablement Kit for x3650 M4 IBM RDX Internal USB 3.0 Dock with 1TB Cartridge

Secondary Server (fail over for DRBD)

x3650-M4 (7915-C4A) - 1P6C, 32GB, 4x 2TB HDD, 1x 2TB Spare HDD, R10 Ready
x3650 M4, Xeon 6C E5-2620 95W 2.0GHz/1333MHz/15MB, 1x8GB, O/Bay HS 3.5in SAS/SATA, SR M5110e, 550W p/s, Rack
8GB (1x8GB, 2Rx4, 1.35V) PC3L-10600 CL9 ECC DDR3 1333MHz LP RDIMM
Internal Storage
IBM 2TB 7.2K 6Gbps NL SAS 3.5in G2HS HDD
IBM Integrated Management Module Advanced Upgrade
IBM UltraSlim Enhanced SATA Multi-Burner
IBM System x 550W High Efficiency Platinum AC Power Supply
x3650 M4 ODD Cable

Quorum Server

x3250-M4 (2583-F2A) - 1P4C, 8GB, 2x 1TB HDD, R1 Ready, RPS
x3250 M4, Xeon 4C E3-1270v2 69W 3.5GHz/1600MHz/8MB,1x4GB, O/Bay HS 2.5in SAS/SATA, SR H1110, 460W p/s, Rack
4GB (1x4GB, 2Rx8, 1.5V) PC3-12800 CL11 ECC DDR3 1600MHz LP UDIMM
Internal Storage
IBM 1TB 2.5in SFF HS 7.2K 6Gbps NL SATA HDD
IBM 460W Redundant Power Supply Unit with 80+ certified
IBM UltraSlim Enhanced SATA Multi-Burner
IBM Integration Management Module Standard Upgrade
ServeRAID M1015 SAS/SATA Controller
ServeRAID M1000 Series Advance Feature Key

Thanks in advance.

 

[e100]

I know nothing about the systems you listed so I can not directly answer your question.
The Proxmox kernel is based on RedHat EL 6 Kernel.
If the hardware works with RedHat (which I bet IBM would know) then it *should* be able to run Proxmox with no issues.

Your quorum server specs seem like overkill. eg. a 10GB drive would suffice, no need for 1TB, only need enough RAM to run proxmox (1GB?)
Unless you plan to do something other than quorum with it.

Did not read every single post in this thread but I did not see this mentioned anywhere.
I always recommend creating TWO DRBD volumes.

Your "Primary" and "Secondary" servers need to have enough resources to run all of the VMs on a single node.
Really you should have two identical servers where a single server has enough RAM/CPU/Disk space/Disk IO Speed to run all of your VMs.
Then you split the disk space into two DRBD volumes.
Then split up your VMs across the two nodes and DRBD volumes.
VMs running on server A have their data stored on DRBD0
VMs running on server B have their data stored on DRBD1

If it were me I would not run all the VMs on one node and have to failover all the VMs when that node fails.
I would run some on nodeA some on nodeB.
Do you really want to deal with bringing up the web server and the DB server when stuff breaks or just one of them?

Even if you decide to run them all on one node I still recommend two DRBD volumes and divide the VMs among the two volumes.
If you do not do this you will wish you had when some weird split-brain problem comes up.