Mysterious Proxmox Cluster Freeze - Help Please!

[z33k3r]

Hello,

We have a two node (master/slave) cluster utilizing DRBD for the image share. This morning I was greeted by frantic employees saying the web sites were down. Went to SSH into our VM's to see what was wrong and I couldn't reach the images. I then went to SSH the master node. Session was denied. I could ping the cluster nodes, but could not reach them for an SSH session.

After driving to the data center, I was able to login to the clusters via crash cart, but no activity was happening. A reboot and restart of images etc got everything back online, but I am now trying to figure out what exactly happened.

Checking the syslog, there was normal entries and then it just stopped and started back with the reboot messages.

Mar 29 21:35:03 server1  proxwww[32401]: Starting new child 32401
Mar 29 21:35:04 server1 pvemirror[2977]: starting cluster syncronization
Mar 29 21:35:04 server1  pvemirror[2977]: syncing templates
Mar 29 21:35:04 server1  pvemirror[2977]: cluster syncronization finished (0.08 seconds (files 0.00, config 0.00))
Mar 29 21:35:18 server1  proxwww[32403]: Starting new child 32403
Mar 29 21:35:33 server1  proxwww[32410]: Starting new child 32410
Mar 30 07:38:07 server1  kernel: imklog 3.18.6, log source = /proc/kmsg started.

I can't seem to find any type of Proxmox error log so if any body has any ideas to which I can use to trouble shoot the event and prevent it happening again, it would be greatly appreciated!

Edit: Using bare metal installations of PVE 1.7

 

[dietmar]

I can't seem to find any type of Proxmox error log so if any body has any ideas to which I can use to trouble shoot the event and prevent it happening again, it would be greatly appreciated!

This usually indicates some hardware problem - try to run a memory test to see if the memory is still OK.

 

[e100]

Went to SSH into our VM's to see what was wrong and I couldn't reach the images. I then went to SSH the master node. Session was denied. I could ping the cluster nodes, but could not reach them for an SSH session.

This sounds like an IO issue, had a similar problem happen to me once.
Multiple disks in the RAID array stopped working at the same time causing the array to fail.

IO to DRBD volumes, even on the other node, was stalled until I turned off the broken node.
After the broken node was turned off the VMs on the other node started started working fine all on their own.

 

[z33k3r]

I went down to the data center and was able to console in and reboot the nodes, so it wasn't the RAID array... but what I do have some suspisions about now is that when we followed the Proxmox guide to DRBD setup, their configuration is set to 30mb sync rate...

On high traffic VMs, this seems like it would be too slow to do anything else with. For instance, our sites fly when nothing else is happening. If i create a couple VM's that have themselves a DRBD file drive that needs to sync, the live sites slow to a crawl!

Our servers are running on 15k RPM SAS2's in RAID10 on a dedicated LSI/3ware controller... these drives scream through anything I throw at them, but yet we are getting very high IOdelay...

Am I correct in assuming, I need to step up the sync rate to 1000mb as the ports allow on my Master/Slave nodes!? ...and is there a way to do this live without having to shut down my cluster and all of it's sites?

 

[z33k3r]

This usually indicates some hardware problem - try to run a memory test to see if the memory is still OK.

Is there no error logs or activity logs of some kind with Proxmox?

 

[e100]

The DRBD Sync rate limits the speed of copying data to a node that is out of sync.(ie the initial sync or resynchronisation after a node failure)
It does not limit the speed of the real time replication.
Yes, you an change it on the fly using drbdadm

In my experience DRBD has very little overhead.

If you are seeing worse performance on DRBD vs not using DRBD then I would suggest looking into the items below:

Are you trying to write more data than your replication network can handle?
Typically the only limiting factor is the replication network which will limit the write speed.
Some 1Gb ports are better than others, Intel cards typically max out around 115MB/sec with two of them bonded I have seen up to 225MB/sec from DRBD.


Do you have battery backed write cache for your RAID controllers?
Without the write cache DRBD performance can suffer because the contant updates to the DRBD metadata cause more random IO.

 

[z33k3r]

If you are seeing worse performance on DRBD vs not using DRBD...

Not with vs. without... Just that if I am trying to create more VM's or do other things on that node that has one of X number of VMs with high load, working with the web KVM applet just doesn't respond well at all, practically useless.

Also for instace. Yesterday, one VM was under high load and the rest just sitting doing basic static file serving. I am working on a pair of load balanced VM's that required the DRBD file system to have synced files. During the DRBD process of syncing, the rest of that cluster's node IODelay went up about 10%... this morning, they were'nt done, loads were 30-80+ on that 'one VM' and so I canceled the two LB'ed VM sync process and turned them off. Instant return to load of 2-5.

Are you trying to write more data than your replication network can handle? Typically the only limiting factor is the replication network which will limit the write speed. Some 1Gb ports are better than others, Intel cards typically max out around 115MB/sec with two of them bonded I have seen up to 225MB/sec from DRBD.

They are both linked to the back bone 1gb, then linked together directly on a second NIC for DRBD dedication. All using CAT-6 (lols appreciated ;D). The servers are using Intel 82576 controllers @ 1gb.

Do you have battery backed write cache for your RAID controllers?
Without the write cache DRBD performance can suffer because the contant updates to the DRBD metadata cause more random IO.

Yes