New Cluster/New Error

[adamb]

Haven't seen this one before. Just setup a new cluster. All was going well until I decided to reboot one of the nodes. Here is what I got in the kern.log.

Oct 4 10:20:02 fiosprox1 kernel: INFO: task pvedaemon:5016 blocked for more than 120 seconds.
Oct 4 10:20:02 fiosprox1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 4 10:20:02 fiosprox1 kernel: pvedaemon D ffff881870300ec0 0 5016 1 0 0x00000004
Oct 4 10:20:02 fiosprox1 kernel: ffff881868f6bc98 0000000000000086 0000000000000000 0000000000000287
Oct 4 10:20:02 fiosprox1 kernel: ffff881868f6bc28 ffffffff81095d77 ffff881868f6bc38 ffff88307126d568
Oct 4 10:20:02 fiosprox1 kernel: ffff881868f6bc38 ffff881870301460 ffff881868f6bfd8 ffff881868f6bfd8
Oct 4 10:20:02 fiosprox1 kernel: Call Trace:
Oct 4 10:20:02 fiosprox1 kernel: [<ffffffff81095d77>] ? finish_wait+0x67/0x90
Oct 4 10:20:02 fiosprox1 kernel: [<ffffffffa01bb095>] fuse_request_send+0xe5/0x290 [fuse]
Oct 4 10:20:02 fiosprox1 kernel: [<ffffffff81095be0>] ? autoremove_wake_function+0x0/0x40
Oct 4 10:20:02 fiosprox1 kernel: [<ffffffffa01bdddc>] create_new_entry+0xac/0x250 [fuse]
Oct 4 10:20:02 fiosprox1 kernel: [<ffffffffa01be161>] fuse_mkdir+0xc1/0xf0 [fuse]
Oct 4 10:20:02 fiosprox1 kernel: [<ffffffff811a288e>] vfs_mkdir+0x8e/0xf0
Oct 4 10:20:02 fiosprox1 kernel: [<ffffffff811a5a1e>] sys_mkdirat+0xfe/0x120
Oct 4 10:20:02 fiosprox1 kernel: [<ffffffff81072a13>] ? do_setitimer+0x1b3/0x220
Oct 4 10:20:02 fiosprox1 kernel: [<ffffffff81072b8a>] ? alarm_setitimer+0x3a/0x70
Oct 4 10:20:02 fiosprox1 kernel: [<ffffffff811a5a58>] sys_mkdir+0x18/0x20
Oct 4 10:20:02 fiosprox1 kernel: [<ffffffff8100b182>] system_call_fastpath+0x16/0x1b

 

[dietmar]

Most likely, nodes are trying to fence other nodes, but fencing does not work. You lost quorum on the cluster?

 

[adamb]

Most likely, nodes are trying to fence other nodes, but fencing does not work. You lost quorum on the cluster?

Just tested fencing manually without any issues. I was able to reboot each node using the fence method.

 

[dietmar]

The question is if you lost quorum (on all partitions)?

Also see https://bugzilla.proxmox.com/show_bug.cgi?id=105