[SOLVED] After 8.1.3 upgrade Summary Graphs are blank

[Rickb]

This does not seem to be a upgrade issue or related to Synology. I have been running brand new loads of 8.1.4 and I enabled ISCSI to an open-e ISCSI target yesterday and then it start for me as well. We had IP address restriction on the target because it kept trying to add the open-e management interface as a target since there is multiple ip addresses on the open-e server. Removed the iSCSI target, logged out of all ISCSI session, stopped iscsid, restarted rrdcached and graphs came back shortly after. So it seems to be that when there is multiple ip address to the target and it can't reach one of them is where the issue is. The graphs got cut of right at the time that I configure the ISCSI target in the cluster. No IPV6 involved.

 

[Elfy]

Thank you for sharing your solution @iay. I faced a similar issue and resolved it by disabling IPv6 on my Synology device. I was able to kick RRD back into gear without having to reboot my hosts by logging out of the iSCSI session then logging back in. Here are the basic commands (can also be done in GUI but I'm a cli guy):

1. Determine the iSCSI Qualified Name (IQN) of your target. Look for the IQN in the output of the command:
   iscsiadm -m discovery -t sendtargets -p <target_address> #Replace <target_address> with the IPv4 address of your iSCSI target.
2. Logout of the iSCSI node:
   iscsiadm -m node -T <IQN> --logout #Replace <IQN> with the iSCSI Qualified Name
3. Login to the iSCSI node to refresh the connection:
   iscsiadm -m node -T <IQN> --login #Replace <IQN> with the iSCSI Qualified Name

Hope you get this figured out @Rickb. Have you checked syslogs? Mine was flooded with iscsid[2557]: connection-1:0 cannot make a connection to {ipv6_address}:3260 (-1,22).

 

[Rickb]

@Elfy The fix was to make sure that there are no unreachable iscsi targets ipv4 and or ipv6 in PVE after target discovery. Because PVE defaults to scan for targets and adds ALL ip address even if they are not reachable is the issue. Why the graphs stop makes no sense but I will defer that to the developers. We had to scan the targets and then remove all invalid target ip addresses that where added from the /etc/iscsi/nodes/ and the /etc/iscsi/send_targets/ folders and restart iscsid and the issue went away. Coming from the VMware world I think there should be a option to not auto discover all but just static entries but that is another topic. Also if you have unreachable target(s) then the syslog is just flooded with the unreachable target(s) entries.

 

[lwde]

The same bug has just occurred for me, but with an ESXi Import as Storage and I can reproduce it if I enable/disable the ESXi Import Path for the PVE node again.

 

[fweber]

The same bug has just occurred for me, but with an ESXi Import as Storage and I can reproduce it if I enable/disable the ESXi Import Path for the PVE node again.

This is likely a regression that should be fixed in libpve-storage-perl=8.2.3, see [1] for more information.

[1] https://forum.proxmox.com/threads/149172/page-2#post-677563

 

[daniel.b]

Resolved without disabling IPv6 on Synology

I experienced this problem after upgrading my 3 node cluster from 7.4 to 8.2.4. I also have a Synology storage device connected via iSCSI and was experiencing symptoms exactly as described by others, with intermittent (mostly non-functional) graphs in the UI and iSCSI connection failure logs in the syslog.

I have iSCSI multipath configured, because I'm running a Synology UC3200 with dual controllers, so I need to have multiple paths to my storage working. See here for more details about multipath. All was working fine before the upgrade.

I didn't want to disable IPv6 support as others have done, so here is how I solved it. The connection failures were occurring to IPv6 link local addresses. However, these addresses were not on any of my active Synology ports. I think what happened is that the Synology was connected via the 2nd LAN interface previously, but is now connected via the 1st LAN interface. At any rate, these IPv6 link local addresses were not reachable by my Proxmox nodes.

I performed the following on each of my Proxmox nodes, which restored graph operation without modifying my Synology config.

Steps to Resolve

Check the active sessions the node is using to communicate with the Synology storage.

iscsiadm -m session

Check the configured nodes on the system to which connections are being attempted.

iscsiadm -m node

Remove all nodes that don't have an active session.

iscsiadm -m node -T <target> -p <ip:port> -o delete

Example (the square brackets around an IPv6 address are important):

iscsiadm -m node -T iqn.2000-01.com.synology:proxmox-san.raid6.412a2ab768 -p [fe80::9209:d0ff:fe0a:7325]:3260 -o delete

If you need to remove all the entries for the same target, simply leave off the -p and all entries for that target will be removed.

iscsiadm -m node -T <target> -o delete

Verify that all the undesirable entries have been removed.

iscsiadm -m node

You should now only see the targets that match the active sessions displayed earlier.

Restart the iscsid service.

systemctl restart iscsid

Conclusion
These exact steps restored graph functionality for me without requiring any configuration changes on my Synology.