status: unknown

[akanarya]

Hi,
I have a 3 node cluster.
Last saturday I updated them all.
After upgrade (not immediatelly, at saturday night i think), one machine went into status unknown.
That machine and its drives-vms are shown as unknown status.
However, I can access to the server, and ssh it.
Ceph shows there is no problem.
I dig a bit but i couldnt find aything usefull, restarted it several times.
I attached some photos.
What can be the problem?

Thanks
Ali

 

[Moayad]

Hi,

Have tried to restart the below services on the vs5 node?

systemctl restart pvedaemon
systemctl restart pvestatd
systemctl restart pveproxy
systemctl restart corosync

 

[akanarya]

Thanks Moayad,
These restarts resolved the status. But when i reboot i have to run them again.

 

[Moayad]

Please provide us with the Syslog since the issue occurs

 

[akanarya]

I attached last syslog of vs5, I had done several reboots. I hope it helps.

 

[Moayad]

Hello,
Thank you for the Syslog! is the MTU set correct for all nodes switches?

 

[akanarya]

I checked the mtu sizes of all interfaces of all nodes. They are all same and set to 1500.
My switches dont have mtu change option, there is only jumbo frame support but it is disabled.

 

[akanarya]

I checked the node with "systemctl status pve*", i am getting:
-----------------
pve-firewall.service - Proxmox VE firewall
Loaded: loaded (/lib/systemd/system/pve-firewall.service; enabled; vendor preset: enabled)
Active: failed (Result: timeout) since Mon 2022-03-14 17:06:28 +03; 1 day 1h ago

Mar 14 17:04:39 vs5 systemd[1]: Starting Proxmox VE firewall...
Mar 14 17:06:28 vs5 systemd[1]: pve-firewall.service: Start operation timed out. Terminating.
Mar 14 17:06:28 vs5 systemd[1]: pve-firewall.service: Control process exited, code=killed, status=15/TERM
Mar 14 17:06:28 vs5 systemd[1]: pve-firewall.service: Failed with result 'timeout'.
Mar 14 17:06:28 vs5 systemd[1]: Failed to start Proxmox VE firewall.
-----------------

at the other node firewall service is active.
Interestingly i dont remember that i did anything with proxmox firewall.

does this message mean something to my issue?

 

[Moayad]

Does the time is the same between the nodes?
Is all storage running on all nodes? I have seen in the provided Syslog that the ISCSI can't connect to the storage.

Mar 14 16:51:15 vs5 pvestatd[5006]: command '/usr/bin/iscsiadm --mode node --targetname iqn.2000-01.com.synology:KHSVR.test.cf847dff18 --login' failed: exit code 8
Mar 14 16:51:15 vs5 iscsid: Connection-1:0 to [target: iqn.2000-01.com.synology:KHSVR.test.cf847dff18, portal: fe80::211:32ff:fe40:931,3260] through [iface: default] is shutdown.
Mar 14 16:51:15 vs5 iscsid: Connection-1:0 to [target: iqn.2000-01.com.synology:KHSVR.test.cf847dff18, portal: fe80::211:32ff:fe40:932,3260] through [iface: default] is shutdown.
Mar 14 16:51:21 vs5 pvestatd[5006]: status update time (199.146 seconds)

 

[akanarya]

yes, time is get by ntp server and it looks like it is working
main storage is ceph
iscsi target is defined on 2 nodes only.

infact this node is a low profile machine, there may be some hardware issues also,
for example this machine comes to online in almost 12-13 minutes after reboot. Frankly this machine is not that bad either.
however this "status unknown" state hasnt occur before the update, i dont know there may be coincidence,
because i havent need to reboot it before this update for a long time.

if there are some diagnostic tools inside proxmox, it can be helpfull to localise the problem.

 

[andersdd]

Hi,

Have tried to restart the below services on the vs5 node?

systemctl restart pvedaemon
systemctl restart pvestatd
systemctl restart pveproxy
systemctl restart corosync

I had same issue and the above restarts resolved the problem for me.