Strange cluster behavior after upgrade from 7 to 8

[jungsky]

Hello,

We have a three node cluster, and yesterday we performed an upgrade from 7 to 8.1.4. The upgrade process went smoothly and cluster is up and running, but is behaving a bit strangely. Everything is normal for a few minutes, and then random one or two nodes start showing question marks instead of normal icons. At the same time, the subscription status switches to "No Subscription".




After a minute or two, everything switches back to normal and the same process keeps repeating over and over since the upgrade.

While the question marks are on, we can perform almost every operation on the cluster as always. The only thing we can't do is migrate machines to the node which has question marks on. Migration from a node with question marks to one with normal icons works fine.

Thanks in advance for any suggestion how to resolve the issue.

Best regards,
Vedran

 

[fiona]

Hi,
please check your system logs/journal on the nodes with the issues. Is there any storage that might not be responding quickly? Unfortunately, that can currently interfere with gathering the status as a whole: https://bugzilla.proxmox.com/show_bug.cgi?id=3714

 

[jungsky]

Hi Fiona,
Thank you for your quick response. You are right, there are syslog messages about errors in iSCSI connections. Here is one of them:

2024-04-26T12:36:51.169502+02:00 prox3 iscsid: connection-1:0 cannot make a connection to fe80::9209:d0ff:fe27:4b89:3260 (-1,22)
2024-04-26T12:36:54.146592+02:00 prox3 pvestatd[1239]: command '/usr/bin/iscsiadm --mode node --targetname iqn.2000-01.com.synology:HQ-DataCluster.Target-13.8ae72ef83be --login' failed: exit code 15
2024-04-26T12:36:54.161178+02:00 prox3 pvestatd[1239]: status update time (366.919 seconds)
2024-04-26T12:36:54.169700+02:00 prox3 iscsid: Connection-1:0 to [target: iqn.2000-01.com.synology:HQ-DataCluster.Target-13.8ae72ef83be, portal: fe80::9209:d0ff:fe27:4b89,3260] through [iface: default] is shutdown.

However, the VMs running on those disks are running without problems, even when the question marks are displayed. Since none of us have much experience with iscsiadm, if you know a quick fix for it, we would appreciate it.

We are also considering moving our last few iSCSI disks to NFS. However, when we try to choose new Target storage, it doesn't display anything and connection times out (596). Yesterday we were able to choose storage without a problem, while creating an new VM.

 

[fiona]

Please share the output of the commands requested here: https://bugzilla.proxmox.com/show_bug.cgi?id=5173#c1
It might be the same issue, i.e. your target advertises a portal where you cannot login.

 

[jungsky]

iscsiadm -m node:

192.168.5.222:3260,1 iqn.2000-01.com.synology:HQ-DataCluster.Target-1.8ae72ef83be
[fe80::9209:d0ff:fe27:4b89]:3260,1 iqn.2000-01.com.synology:HQ-DataCluster.Target-1.8ae72ef83be
192.168.5.222:3260,1 iqn.2000-01.com.synology:HQ-DataCluster.Target-11.8ae72ef83be
[fe80::9209:d0ff:fe27:4b89]:3260,1 iqn.2000-01.com.synology:HQ-DataCluster.Target-11.8ae72ef83be
192.168.5.222:3260,1 iqn.2000-01.com.synology:HQ-DataCluster.Target-12.8ae72ef83be
[fe80::9209:d0ff:fe27:4b89]:3260,1 iqn.2000-01.com.synology:HQ-DataCluster.Target-12.8ae72ef83be
192.168.5.222:3260,1 iqn.2000-01.com.synology:HQ-DataCluster.Target-13.8ae72ef83be
[fe80::9209:d0ff:fe27:4b89]:3260,1 iqn.2000-01.com.synology:HQ-DataCluster.Target-13.8ae72ef83be

iscsi -m session:

tcp: [1] 192.168.5.222:3260,1 iqn.2000-01.com.synology:HQ-DataCluster.Target-11.8ae72ef83be (non-flash)
tcp: [2] 192.168.5.222:3260,1 iqn.2000-01.com.synology:HQ-DataCluster.Target-13.8ae72ef83be (non-flash)
tcp: [3] 192.168.5.222:3260,1 iqn.2000-01.com.synology:HQ-DataCluster.Target-1.8ae72ef83be (non-flash)

We get the same error for all three connections from session command. And node command shows one target (12.8) that was deleted from cluster storage long ago.

 

[fiona]

You could try removing the outdated target like suggested here https://bugzilla.proxmox.com/show_bug.cgi?id=5173#c17
But I'm not sure if that is enough, or if you have to fix the server-side configuration.