ISCSI connection causing error "pbs: error fetching datastores - 500 Can't connect to ipaddress:8007 (Connection refused) (500)"

[harunk]

Hello Everyone,

i have a PBS server (version 4.1) running as a VM in a PVE (8.4) with a synology ISCSI connected (have installed open-iscisi in PBS via debian repo) as a datastore. The issue i am facing is that, after a few minutes PVE reports the following error

pbs: error fetching datastores - 500 Can't connect to ipaddress:8007 (Connection refused) (500)

Looking at the journal logs in PBS itself, shows the following messages

pbs kernel:  connection1:0: ping timeout of 30 secs expired, recv timeout 30, last rx 4295286053, last ping 4295316480, now 4295346688
pbs kernel:  connection1:0: detected conn error (1022)
pbs iscsid[640]: Kernel reported iSCSI connection 1:0 error (1022 - ISCSI_ERR_NOP_TIMEDOUT: A NOP has timed out) state (3)
pbs iscsid[640]: connection1:0 is operational after recovery (5 attempts)
pbs kernel:  connection1:0: detected conn error (1020)
pbs iscsid[640]: Kernel reported iSCSI connection 1:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
pbs iscsid[640]: connection1:0 is operational after recovery (1 attempts)
pbs kernel:  connection1:0: ping timeout of 30 secs expired, recv timeout 30, last rx 4296108173, last ping 4296138240, now 4296168448
pbs kernel:  connection1:0: detected conn error (1022)
pbs iscsid[640]: Kernel reported iSCSI connection 1:0 error (1022 - ISCSI_ERR_NOP_TIMEDOUT: A NOP has timed out) state (3)
pbs iscsid[640]: connection1:0 is operational after recovery (1 attempts)
pbs kernel: hrtimer: interrupt took 4371530 ns
pbs kernel:  connection1:0: detected conn error (1020)
pbs iscsid[640]: Kernel reported iSCSI connection 1:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
pbs iscsid[640]: connection1:0 is operational after recovery (1 attempts)

Intrestingly, there are a couple fo SMB shares from the same NAS being used within PVE but they have no connection issues. This is only happening with the ISCSI connection.

What i have tried so far,

- Replaced the network cable (no changes)
- Checked for duplicate iscisi connector ids (there were none and i have also made sure that the only connection is between PBS virtual machine and the NAS)
- Everytime i restart the networking.service and iscsid.service inside PBS , the connection is restored but only for a few minutes and after that the same error as above.

Any suggestions on this error?

Thanks!

 

[bbgeek17]

Hi @harunk , welcome to the forum.

These are Linux kernel–level messages; they are not PBS or PVE specific. They indicate a network-layer issue in the communication path between the Debian OS and your NAS.

I would start by checking MTU, running long duration pings, and testing with iperf (or a similar tool) if it is available on the NAS.

When the issue occurs, before restarting any network services, verify connectivity using ping and other basic network troubleshooting tools. Keep in mind that ping uses ICMP, while iSCSI relies on TCP/IP, so a successful ping does not guarantee healthy iSCSI traffic.

Another useful troubleshooting tool is tcpdump. You can filter for iSCSI traffic and, during the failure window, determine whether requests are leaving the hypervisor and whether replies are returning from the NAS and reaching the VM.

Good luck.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox