iSCSI Disk as LXC Mount Point does not reconnect after outage

[jonnewman]

Hello all. I've been tinkering with Proxmox for a few months now, and most of the time I can find solutions to my config 'panics' on these forums (thanks all!), however there is an issue I'm having with the particular setup I have for PBS which I am running on an LXC with iSCSI host for it's backup store.
Brief setup overview:

• I have two non-clustered PVE nodes on the same network (primary and secondary), one on a NUC, the other a MacMini, both in the same rack.
• I have two PBS instances.
  • One PBS instance is installed as a VM on a Synology NAS (RS1221+) with it's datastore being a normal VM disk also on the NAS, managed by Synology VMM. This PBS instance is working well and both nodes backup various VMs and CTs to this datastore under their own namespaces.
  • The second PBS instance (#110) is an Unprivilaged LXC on the secondary PVE node, and is set up to simply 'sync' the first PBS datastore contents to it's own datastore, which I have configured as a Mount Point disk in the second PVE node UI. This MP disk is stored on an iSCSI LUN on another Synology NAS (DS918+) in another room.

This setup is usually working well, however the UPS attached to the DS918+ is failing, and until I can get a replacement battery (based in East Africa, so not easy to get genuine APC parts) every time we have a power cut for more than 5mins, the DS918+ powers down unsafely. All other network gear is in the main rack and has a 200ah backup UPS, so typically the main rack, both PVE nodes and the primary PBS instance will survive the power cut.

The issue is that the second PBS (in LXC) loses the connection to the iSCSI LUN and then refuses to reconnect or restore the connection once the NAS/LUN is back up.

It can be resolved easily by rebooting the LXC, however I'd like this to be automated if possible, similar to how Samba shares can be reconnected via FSTAB.

Example error when I try to do a GC task on the second PBS instance after the iSCSI LUN connection was lost but after the server is back up:

2024-05-29T13:35:42+03:00: starting garbage collection on store backups
2024-05-29T13:35:42+03:00: Start GC phase1 (mark used chunks)
2024-05-29T13:35:42+03:00: TASK ERROR: can't open index /mnt/backups/ns/macmini/ct/104/2024-04-20T23:02:26Z/catalog.pcat1.didx - Input/output error (os error 5)

And when I try to CD to the mount point within the PBS LXC Shell:

root@pbs-macmini:~# cd /mnt/backups/
-bash: cd: /mnt/backups/: Input/output error

Some other config:

• Everything is on the same subnet.
• LXC is debian based, using the Proxmox-Helpers script.

Content of /etc/pve/lxc/110.conf: (PBS LXC)
mp0: Syno_918_LUN1:vm-110-disk-0,mp=/mnt/backups,size=1000G

Extract from /etc/pve/storage.cfg showing the iSCSI connection: (IP and targets obscured)

iscsi: Syno_918_iSCSI
        portal 10.0.x.x
        target iqn.2000-01.com.synology:Synology-918.default-target.xxxxxx
        content none

lvm: Syno_918_LUN1
        vgname Synology_918
        base Syno_918_iSCSI:0.0.1.scsi-xxxxxx
        content images,rootdir
        saferemove 0
        shared 0

EDIT: Should note that I am on PVE 8.2.2 with all updates installed, PBS instances are also fully up to date. All network connections are wired between PVE and PBS.

I'm really hoping this is a simple config problem, otherwise I will have to re-think how I store the secondary backups outside of the current LUN.

Thanks in advance!