Node show all items in gray with question marks

[khaled.j.hussein]

I am running Proxmox 5.4-10 on three nodes with share storage connected to DELL MD over ISCSI, when the connection lost with ISCSI storage during switch reboot then all node status show gray with question marks, one of the node I reboot it and it works fine, the first one did not show this, third one I did not reboot it and I tried to reboot pve services

pve-cluster
corosync
pvestatd
pveproxy
pvedaemon

after reboot VMs and containers show normal status but all storage still has question mark and after 5 minutes all VMs and container back to gray and question mark, also I tried to restart iscsid service and did not work

is there any solution for this problem without rebooting my machine

Thanks

 

[matrix]

Hi,

Any hints in dmesg or syslog?


post the output of: pveversion -v as well

 

[khaled.j.hussein]

Hi,
thanks for your reply matrix, please check attached files

 

[matrix]

Hi,

You could restart pvestatd service systemctl pvestatd restart, then see the status if there an error systemctl pvestatd status

 

[matrix]

The problem is from pve-kernel-4.15[1] try to update proxmox-ve 6.1-2 and pve-kernel 5.3

[1] https://bugzilla.proxmox.com/show_bug.cgi?id=1943#c41

 

[khaled.j.hussein]

Hi,

I do restart the pvestatd and multipathd services but the same, syslog say

Feb 11 12:38:32 pve8 pveproxy[26445]: proxy detected vanished client connection
Feb 11 12:38:32 pve8 pveproxy[22278]: proxy detected vanished client connection
Feb 11 12:38:38 pve8 multipathd[7725]: md3820: load table [0 3481755648 multipath 4 pg_init_retries 50 queue_if_no_path retain_attached_hw_handler 1 rdac 2 1 round-robin 0 1 1 8:16 1 round-robin 0 1 1 8:32 1]
Feb 11 12:38:38 pve8 multipathd[7725]: md3820: event checker started
Feb 11 12:39:00 pve8 systemd[1]: Starting Proxmox VE replication runner...
Feb 11 12:39:01 pve8 systemd[1]: Started Proxmox VE replication runner.
Feb 11 12:39:08 pve8 multipathd[7725]: md3820: sdc - rdac checker reports path is down: inquiry failed
Feb 11 12:39:08 pve8 multipathd[7725]: path checkers took longer than 30 seconds, consider increasing max_polling_interval



Feb 11 12:39:39 pve8 multipathd[7725]: md3820: sdc - rdac checker reports path is down: inquiry failed
Feb 11 12:39:39 pve8 multipathd[7725]: path checkers took longer than 30 seconds, consider increasing max_polling_interval
Feb 11 12:40:00 pve8 systemd[1]: Starting Proxmox VE replication runner...
Feb 11 12:40:01 pve8 pvesr[11588]: trying to acquire cfs lock 'file-replication_cfg' ...
Feb 11 12:40:02 pve8 systemd[1]: Started Proxmox VE replication runner.
Feb 11 12:40:10 pve8 multipathd[7725]: md3820: sdc - rdac checker reports path is down: inquiry failed
Feb 11 12:40:10 pve8 multipathd[7725]: path checkers took longer than 30 seconds, consider increasing max_polling_interval

 

[khaled.j.hussein]

Hi matrix,

I rebooted the node and it worked fine, and I am looking for upgrade to 6.1 but I am afraid to have big problems with my cluster,

and I have another issue, why when I lost the connection with my shared storage which is Dell MD3820 even it is few seconds , all VMs and containers stuck and i must restart them to come back up, I am using multipathd in my setup

Thanks

 

[matrix]

Hi,

I rebooted the node and it worked fine

Great

but I am afraid to have big problems with my cluster

Just follow the instructions in wiki, should work fine [1]

all VMs and containers stuck

Did you see this post? maybe it will help you [2]


[1] https://pve.proxmox.com/wiki/Upgrade_from_5.x_to_6.0
[2]https://forum.proxmox.com/threads/lost-connections-with-vms.57225/#post-265380