PBS External Backup: error fetching datastores - 500

[slize26]

I just setup a second Backup Server on a second site of the office. The locations are connected via an ipsec site to site VPN. One end is running a UDM Pro and the other one a pfsense box. If i trust the logs the connection is successful and stable (also RDP is working from both sides - as a test). The PVE is on site A and the PBS is on site B.

The strange part is, that the connection to the PBS on site B only works sporadically. in 90% of the time i get this error message:

PBS1-EXT: error fetching datastores - 500 Can't connect to 172.16.100.12:8007 (500)

But in 10% the connections (and backups) are working just fine.

I am able to ping and SSH the PBS all the time. There are no drops when pinging the server (tested for ~6 hours) and the SSH connectionwas stable over an hour.

What is not working when the error occurs is to open the webui

https://10.0.20.12:8006/

of the PBS from any host on site A. It does only respond to requests from site B. (Shell log from PVE on site A below)

Linux srvpve1 5.11.22-4-pve #1 SMP PVE 5.11.22-8 (Fri, 27 Aug 2021 11:51:34 +0200) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sun Sep 12 16:14:28 CEST 2021 on pts/0
root@srvpve1:~# ping 172.16.100.12
PING 172.16.100.12 (172.16.100.12) 56(84) bytes of data.
64 bytes from 172.16.100.12: icmp_seq=1 ttl=62 time=20.0 ms
64 bytes from 172.16.100.12: icmp_seq=2 ttl=62 time=23.1 ms
^C
--- 172.16.100.12 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 19.963/21.534/23.105/1.571 ms
root@srvpve1:~# curl -I -k https://172.16.100.12:8007
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to 172.16.100.12:8007 
root@srvpve1:~# ssh root@172.16.100.12
root@172.16.100.12's password: 
Linux wg-pbs1-ext 5.11.22-4-pve #1 SMP PVE 5.11.22-8 (Fri, 27 Aug 2021 11:51:34 +0200) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sun Sep 12 14:40:57 2021
root@wg-pbs1-ext:~# exit
logout
Connection to 172.16.100.12 closed.
root@srvpve1:~#

Does anyone know how I can make the connection stable?

 

[dcsapak]

maybe a faulty mtu setting somewhere? deep packet inspection gone wrong?
does it work in the local network? if yes, i'd check the firewall/vpn/network settings....

 

[slize26]

I dont think that its related to the MTU/firewall/vpn because it does work from time to time - i just dont know why. But i might be wrong.

I tested curl again with tcp dump running on both sites. The ruslts are shown in the screenshots. It seems like The PVE can talk just fine to the PBS but only parts of the responses are coming back to the PBS. That is super weird since the connection does was up and running a few hours ago and i made a full backup of the PVE to the PBS (i did not change any firewall settings or any network related stuff).

 

[dcsapak]

still sounds like a network/firewall issue...
can you post the network configs of the 2 servers ?

 

[slize26]

still sounds like a network/firewall issue...
can you post the network configs of the 2 servers ?

Alright, thank you very much for your help.

Thats the config of the PBS server

root@wg-pbs1-ext:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto enp2s0
iface enp2s0 inet static
        address 172.16.100.12/24
        gateway 172.16.100.1



root@wg-pbs1-ext:~# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
172.16.100.12 wg-pbs1-ext.main wg-pbs1-ext

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

Thats the config of the PVE server:

root@srvpve1:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface enp35s0 inet manual

iface enp36s0 inet manual

iface enx4e1527300f44 inet manual

iface enxc68738aa944a inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.0.20.12/24
        gateway 10.0.20.1
        bridge-ports enp35s0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094



root@srvpve1:~# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
10.0.20.12 srvpve1.wg srvpve1

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

The pfsense is running 99% of the default config. The only things i changed are the ipsec VPN settings and the required firewall rule:


The UDM Pro has multiple networks and a few firewall rules to block inter V-LAN communication. But no rules targeting the IPSec subnet.

 

[dcsapak]

mhmm... i'd try to track down where exactly the network packages are lost (on the pfsense/firewall/host etc) and double check all firewall rules/network settings on all particpating machines
this might be tedious, but sadly there is not much to go on

 

[slize26]

Well i just changed the VPN type from IPSec to OpenVPN and now everything is working fine. I dont know whats wrong with the IPSec configuration but i am happy that its now working as expected. Thank you very much for your help and sorry for the annoyence.