kernel panic ? how to restore shutdown only solution?

[yaboc]

hi

three of my 4 cluster nodes show up with red marks but all the VMs are operational

i var/log/messages i see

Sep 22 16:17:31 px3 kernel: [8359563.040449] pvestatd D ffff88010970bdf8 0 18963 1 0x00000004
Sep 22 16:17:31 px3 kernel: [8359563.040452] ffff88010970bdf8 ffff8805fe84e3c0 ffff880603faa640 ffff8805fefd0cc0
Sep 22 16:17:31 px3 kernel: [8359563.040455] ffff88010970c000 ffff8805f04043ac ffff8805fefd0cc0 00000000ffffffff
Sep 22 16:17:31 px3 kernel: [8359563.040457] ffff8805f04043b0 ffff88010970be10 ffffffff81845185 ffff8805f04043a8
Sep 22 16:17:31 px3 kernel: [8359563.040460] Call Trace:
Sep 22 16:17:31 px3 kernel: [8359563.040462] [<ffffffff81845185>] schedule+0x35/0x80
Sep 22 16:17:31 px3 kernel: [8359563.040465] [<ffffffff8184543e>] schedule_preempt_disabled+0xe/0x10
Sep 22 16:17:31 px3 kernel: [8359563.040467] [<ffffffff81847149>] __mutex_lock_slowpath+0xb9/0x130
Sep 22 16:17:31 px3 kernel: [8359563.040469] [<ffffffff818471df>] mutex_lock+0x1f/0x30
Sep 22 16:17:31 px3 kernel: [8359563.040471] [<ffffffff8121ceda>] filename_create+0x7a/0x160
Sep 22 16:17:31 px3 kernel: [8359563.040474] [<ffffffff8121de73>] SyS_mkdir+0x53/0x100
Sep 22 16:17:31 px3 kernel: [8359563.040476] [<ffffffff818492b6>] entry_SYSCALL_64_fastpath+0x16/0x75
Sep 22 16:19:31 px3 kernel: [8359683.040266] pvestatd D ffff8801471dfdf8 0 18559 1 0x00000004
Sep 22 16:19:31 px3 kernel: [8359683.040272] ffff8801471dfdf8 ffff8805fe84e3c0 ffff880603fa9980 ffff8805ff37d940
Sep 22 16:19:31 px3 kernel: [8359683.040275] ffff8801471e0000 ffff8805f04043ac ffff8805ff37d940 00000000ffffffff
Sep 22 16:19:31 px3 kernel: [8359683.040277] ffff8805f04043b0 ffff8801471dfe10 ffffffff81845185 ffff8805f04043a8
Sep 22 16:19:31 px3 kernel: [8359683.040295] Call Trace:
Sep 22 16:19:31 px3 kernel: [8359683.040304] [<ffffffff81845185>] schedule+0x35/0x80
Sep 22 16:19:31 px3 kernel: [8359683.040306] [<ffffffff8184543e>] schedule_preempt_disabled+0xe/0x10
Sep 22 16:19:31 px3 kernel: [8359683.040309] [<ffffffff81847149>] __mutex_lock_slowpath+0xb9/0x130
Sep 22 16:19:31 px3 kernel: [8359683.040311] [<ffffffff818471df>] mutex_lock+0x1f/0x30
Sep 22 16:19:31 px3 kernel: [8359683.040315] [<ffffffff8121ceda>] filename_create+0x7a/0x160
Sep 22 16:19:31 px3 kernel: [8359683.040318] [<ffffffff8121de73>] SyS_mkdir+0x53/0x100
Sep 22 16:19:31 px3 kernel: [8359683.040321] [<ffffffff818492b6>] entry_SYSCALL_64_fastpath+0x16/0x75

@px4:~# pveversion -v
proxmox-ve: 4.2-54 (running kernel: 4.4.10-1-pve)
pve-manager: 4.2-15 (running version: 4.2-15/6669ad2c)
pve-kernel-4.4.6-1-pve: 4.4.6-48
pve-kernel-4.2.6-1-pve: 4.2.6-36
pve-kernel-4.4.8-1-pve: 4.4.8-52
pve-kernel-4.4.10-1-pve: 4.4.10-54
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 1.0-1
pve-cluster: 4.0-42
qemu-server: 4.0-81
pve-firmware: 1.1-8
libpve-common-perl: 4.0-68
libpve-access-control: 4.0-16
libpve-storage-perl: 4.0-55
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.5-19
pve-container: 1.0-68
pve-firewall: 2.0-29
pve-ha-manager: 1.0-32
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u2
lxc-pve: 1.1.5-7
lxcfs: 2.0.0-pve2
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve9~jessie
openvswitch-switch: 2.5.0-1


i cant execute service restart commands.

when i do it just goes to next line and sits there until ctrlx

how can safely bring them back "online"

thanks

 

[yaboc]

i had to offline the VMs and hard reset the node. after that everything came back up. i also upgraded my cluster to the latest from GUI on each node.