[SOLVED] Backup Server Storage status unknown, connection lost

[seteq]

Hi,

I'm testing the Backup Server at the moment and somehow 2 of 3 PVE test nodes lost the connection - the datastore just seems to be offline (but can be pinged from the PVE nodes, so it's not a network issue).

The backups view of the backup datastore shows this error:
error fetching datastores - 500 Can't connect to [PBS-IP]:8007 (Connection refused) (500)

Does anybody know how to reconnect the backup datastore without rebooting the PVE node?

EDIT: I just tried to reboot the PBS-Server and now it's saying status unknown on all PVE-nodes (even on those which worked before)...

Thank you

 

[fabian]

can you connect to the PBS GUI from you admin work station? are you using the same IP for that? if yes, then it is a networking/firewall issue. if no, then connect via SSH and verify that the services are running / correct IP(s) are assigned / ...

 

[seteq]

There are separate networks for management and storage traffic which are not being routed.
On each of the physical nodes (PVE and PBS) I have two 1GbE interfaces with LACP bonding for management and two 10GbE interfaces for storage and backup traffic.

PBS:


What I did is adding the 10GbE-IP-Address of the PBS server (10.250.0.111) as a cluster backup storage, this has been working great for several weeks now.
I found these connection problems after I had some unexpected power outages on some PVE nodes and the PBS node.
After a reboot of the PBS server, all of the PVE cluster nodes show this error message:



I just tried a reboot of one of the PVE nodes, it's now working again for this particular node.
I don't think this is a network issue at all, it seems more like a PVE bug on reconnecting to the PBS after it has gone away for some reason.

Btw. I'm on pve-no-subscription with all updates installed on these nodes.

Thank you

 

[fabian]

PVE does not have a permanent connection to PBS - it calls the client for every pvestatd status update run (and of course also for actions like listing backups, creating/restoring from a backup, ...). this is some kind of network issue for sure - whether it's a config issue on the PVE side, or one the PBS side, or some issue with the switches/routers in-between is something you need to find out (e.g., by looking at tcpdump at each side and see whether you get any traffic at all).

 

[seteq]

Wow, I am very sorry, I am officially dumb af.
I assigned the same IP address twice in my storage network (one of the switches happened to have the same IP as the PBS server).

It's all good now