Serious problem of Quorum - Servers are in Production

C

cesarpk

Well-Known Member
Mar 31, 2012
770
3
58
Hi guys

Urgent, please can anyone help me, my servers are in Production

I have a serious problem with lost of Quorum, and I have 4 servers with PVE 2.3.

Suddenly i lost quorum on all nodes PVE, the VMs on DELL servers follow running but the only VM installed on the workstation with "HA" is off, and I can't turn it on.

After of this problem, I have executed: "pvecm e 1 ; /etc/init.d/pve-cluster restart" for get Quorum and take control again of all PVE Nodes. Now all PVEs Nodes work well, but this not correct my initial problem, ie if I restart the PVE Nodes, the problem returns again.

If anybody can help me and want that i see the log files, as I am no expert for this (I feel like a fool), please tell me in an easy to understand.

Waiting for the generous help from someone, I say see you soon

Cesar
 
spirit said:
Hi,
maybe it's a mutlicast problem.
Do you see something in
/var/log/cluster/corosync.log ?
Another question : what's is your hardware network switches models ?
Click to expand...

Hi spirit and thank you very much for your fine attention

I use unmanaged switches to 1 Gb / s Planet brand, and if it worked my PVE cluster for several days, I think it is no problem of "Multicast", right?. For now i don't remember the model, but if is necessary know, i will ask it to my partner.

This is my corosync.log on a Node:
Apr 25 10:58:48 corosync [TOTEM ] Retransmit List: dd1da dd1db dd1dc dd1dd dd1de dd1df dd1e0 dd1e1
Apr 25 10:58:48 corosync [TOTEM ] Retransmit List: dd1da dd1db dd1dc dd1dd dd1de dd1df dd1e0 dd1e1
Apr 25 10:58:48 corosync [TOTEM ] Retransmit List: dd1da dd1db dd1dc dd1dd dd1de dd1df dd1e0 dd1e1
Apr 25 10:58:48 corosync [TOTEM ] Retransmit List: dd1da dd1db dd1dc dd1dd dd1de dd1df dd1e0 dd1e1
....etc...
Apr 25 15:02:44 corosync [TOTEM ] Retransmit List: 68e 68f 690 691 692 684 685 686 687 6a2 6a3 6a6 67b 67c 67d 67e 67f 680 681 682 683 68c 68d 693 694 695 696 697 698 699
Apr 25 15:02:44 corosync [TOTEM ] Retransmit List: 689 68a 68b 6a0 6a1 6a4 6a5 67b 67c 67d 67e 67f 680 681 682 683 68c 68d 68e 68f 690 691 692 693 694 695 696 697 698 699
.....etc..
Apr 25 15:42:34 corosync [TOTEM ] Retransmit List: 2f6 2de 2e0 2eb 2ed 2e1 2e2 2e3 2e4 2e5 2e6 2e7 2e8 2e9 2ef 2f0 2f1 2f2 2f3 2f4 2f5 2f7
Apr 25 15:42:34 corosync [TOTEM ] Retransmit List: 2f5 2f7 2df 2ea 2ec 2ee 2e1 2e2 2e3 2e4 2e5 2e6 2e7 2e8 2e9 2ef 2f0 2f1 2f2 2f3 2f4 2f6
Apr 25 15:42:34 corosync [TOTEM ] FAILED TO RECEIVE
....etc..

Waiting for your reply I say see you soon

Best regards
cesar
 
I like to add:

The file /var/log/cluster/corosync.log have ASCII characters, i don't see simple text, so I had to look at the .gz files.

Best regards
Cesar
 
Hi,
your corosync logs show that corosync could not communicate for some seconds/minutes between nodes
So something bad has happen on network layer. It could be multicast problem (But if you have an unmanaged switch, they are no filtering or something like that).

It also could be a network load which use all bandwith or cpu on your switch.
Does it happen when backups are running ?
 
spirit said:
Hi,
your corosync logs show that corosync could not communicate for some seconds/minutes between nodes
So something bad has happen on network layer. It could be multicast problem (But if you have an unmanaged switch, they are no filtering or something like that).

It also could be a network load which use all bandwith or cpu on your switch.
Does it happen when backups are running ?
Click to expand...

The problem was permanent (24 hours a day) until it got the update. The automatic Backups are scheduled for 00:00 Hrs.

Best regards
Cesar
 
Last edited:
spirit said:
Hi,
your corosync logs show that corosync could not communicate for some seconds/minutes between nodes
So something bad has happen on network layer. It could be multicast problem (But if you have an unmanaged switch, they are no filtering or something like that).

It also could be a network load which use all bandwith or cpu on your switch.
Does it happen when backups are running ?
Click to expand...

Oh Nooo, the problem persist, PVE looses sync (turns red) in the web interface on all others Nodes.
The update not helped me

My log file:
Apr 29 07:39:31 corosync [TOTEM ] Retransmit List: 114c4 114c5 114c6 114ab 114ac 114ad 114ae 114af 114b0 114b1 114b2 114b3 114b4 114c3 114c7 114c8 114c9 114d4 114d5 114b5 114b6 114b7 114b8 114b9 114ba 114bb 114bc 114ca 114cb 114cc
Apr 29 07:39:31 corosync [TOTEM ] Retransmit List: 114b3 114b4 114c3 114c7 114c8 114c9 114c0 114c1 114ab 114ac 114ad 114ae 114af 114b0 114b1 114b2 114c2 114c4 114c5 114b5 114b6 114b7 114b8 114b9 114ba 114bb 114bc 114c6 114ca 114cb
Apr 29 07:39:31 corosync [TOTEM ] Retransmit List: 114cc 114cd 114ce 114cf 114d0 114d1 114d2 114d3 114ab 114ac 114ad 114ae 114af 114b0 114b1 114b2 114b3 114b4 114c3 114b5 114b6 114b7 114b8 114b9 114ba 114bb 114bc 114c4 114c5 114c6
Apr 29 07:39:31 corosync [TOTEM ] FAILED TO RECEIVE

Is there a solution?

Re-Edit:
1- In 2 Nodes i use:
- vmbr1 (not vmbr0) with the IP of PVE Cluster
- vmbr1 is connected to my switch LAN
2- In other 2 Nodes i use:
- mdadm RAID1 only for PVE (not for VMs)
- only 1 VM (KVM)
- HA
- For HA I use failover domain
- rgmanager service
- DRBD without mdadm RAID1
 
Last edited:
I again

After of see my problem again, in a Node I execute:
pvecm e 1 ; /etc/init.d/pve-cluster restart
cman_tool: Cannot open connection to cman, is it running ?
Restarting pve cluster filesystem: pve-cluster.

service cman status
Found stale pid file <----??? (the cman service is hang up each middle or one hour)

service cman restart
Stopping cluster:
Stopping dlm_controld... [ OK ]
Stopping fenced... [ OK ]
Stopping cman... [ OK ]
Unloading kernel modules... [ OK ]
Unmounting configfs... [ OK ]
Starting cluster:
Checking if cluster has been disabled at boot... [ OK ]
Checking Network Manager... [ OK ]
Global setup... [ OK ]
Loading kernel modules... [ OK ]
Mounting configfs... [ OK ]
Starting cman... [ OK ]
Waiting for quorum... Timed-out waiting for cluster
[FAILED]

pvecm e 1 ; /etc/init.d/pve-cluster restart
Restarting pve cluster filesystem: pve-cluster.

This is becoming a nightmare
 
Last edited:
Please to anyone who can answer me:

Does anyone have a PVE cluster of four nodes or more without problems of corosync?

If the answer is correct, Please show the result of the execute "pveversion -v"
 
You use a separate network for the cluster communication? Or DRBD, VMs and cluster use the same network?
 
dietmar said:
You use a separate network for the cluster communication? Or DRBD, VMs and cluster use the same network?
Click to expand...

Hi Dietmar

I don't use separate network for the cluster communication
I use separate network for DRBD communication

Best regards
Cesar

Re-Edit:
But rrd tools don't show excessive use of network
 
Last edited:
dietmar said:
How is the network configured - do you use bonding? What kind of network card?
Click to expand...

I use:
2 Servers DELL PowerEdge 2900 (with DELL 2900 comes broadcom) + 2 NIC extra broadcom too:
- 2 BCM5721 Gigabit Ethernet PCI Express
- 2 NetXtreme II BCM5708 Gigabit Ethernet
Bonds and bridges:
2 NICs use: 1 bond active-backup + 1 bridge (vmbr0) for connect to Switch LAN
2 NICs use: 1 bond balance-rr + 1 bridge (vmbr1) for connect to another Server DELL (crossover-cable)
Servers DELL don't have DRBD,
balance-rr is for use of a Mirrror of Data Base on only a VM.

2 Workstartion ASUS P8H77-M PRO with Intel Core I7
- On board 1 NIC: Realtek 8111F, 1 x Gigabit LAN Controller
- On slot 4 NICs : RTL8111/8168 PCI Express Gigabit (TP-LINK brand)
Bonds and bridges:
1 NIC use: vmbr0 = WAN / eth3 (NIC card)

2 NICs use:
bond1 = active-backup / eth0 eth1 (eth0=onboard, eth1=NIC card)
vmbr1 = LAN / bond1 (vmbr1 = are connect to Switch LAN)

2 NICs use:
bond4000 = balance-rr for DRBD / eth2 eth4 (both are NIC cards, connect to another Workstation - crossover-cable)
 
Last edited:
Hi Again

Dietmar or anyone, can help me with my problem of Quorum? ??? :(
 
Last edited:
dietmar said:
Maybe there is a high load on some of the nodes when that happens?
Click to expand...

Thanks dietmar for your fine atention, and please, let me to do four questions:

1- If I make changes like these on all PVE nodes, will be correct this configuration for avoid problems of cluster comunications?:

Configurations into PVE Nodes:
Bridge vmbr2 to NIC-1: with IP/mask 10.0.0.0/24 for the PVE Nodes and his cluster comunication > To Switch LAN unmanaged
Bridge vmbr1 to NIC-2:without IP for the PVE Nodes, and only for network comunications with the VMs with IP/mask 10.0.0.0/24 > To Switch LAN unmanaged

Ie. VMs and PVE Nodes are connected to the same switch Gb. unmanaged and belong to the same segment

2- Or will be better separate the segments and/or switches of PVE Nodes from the segments and/or switches of the VMs?

3- Make bond active-backup of pair NICs for all these cases will be a problem?

4- Additional recommendations?

I will be very grateful if you can dispel my doubts

Best regards
Cesar

This post was edited to make best the questions
 
Last edited:
spirit said:
Hi,I'm running active-backup in production without problem.
Just to be sure, are all your active network cards on the same switch ?
Click to expand...

Hi Spirit

Yes, active and pasive NICs are on the same switch, except the that I use for DRBD, DRBD is in balance-rr direct of NIC to NIC
Spirit, can you tell me your scenario regarding my questions?

Best regards
Cesar
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back