proxmox-backup-client failed

D

diaolin

Renowned Member
Jul 7, 2008
332
8
83
Trentino Italy
Hi,
in many case i've this problem on one of three proxmox nodes, not always but it happens
: Error: connection error: Connection reset by peer (os error 104) (500)

(see Screenshot)

This appears when browsing from proxmox

I don't know how to debug this situation.

I've other two nodes in cluster and they have never a similar problem.

Anyone?

Tx


pveversion on the node
pve-manager/6.2-15/48bd51b6 (running kernel: 5.4.65-1-pve)


and on the pbs itself

proxmox-backup-manager versions
proxmox-backup-server 1.0.5-1 running version: 1.0.5
 

Attachments

  • Schermata del 2020-12-01 10-50-32.png
    Schermata del 2020-12-01 10-50-32.png
    209.1 KB · Views: 123
hi,

diaolin said:
I don't know how to debug this situation.
Click to expand...

it could be a networking misconfiguration between the backup server, but it's hard to tell without journal and system logs.

from the pve nodes please post:

* journalctl output
* /etc/pve/storage.cfg (to see pbs config in pve)
* ip a && ip r (networking info)
* cat /etc/network/interfaces

diaolin said:
pveversion on the node
pve-manager/6.2-15/48bd51b6 (running kernel: 5.4.65-1-pve)
Click to expand...
maybe you should upgrade to 6.3 as well (apt update && apt dist-upgrade on all nodes followed by reboots because of kernel upgrade)
 
can I jump onto this thread, today I saw a very similar error. I have installed PVE 6.3-2 AND PBS 1.0-5 on the SAME machine. This is my local homelab. So PBS is addresses as localhost and backups to the PBS are working but when browsing to the PBS via PVE I get the rror:

Code: 
pbs-local: error fetching datastores - 500 500 Status read failed: Connection reset by peer (500)


1606998556243.png
 
oguz said:
hi,



it could be a networking misconfiguration between the backup server, but it's hard to tell without journal and system logs.

from the pve nodes please post:

* journalctl output
* /etc/pve/storage.cfg (to see pbs config in pve)
* ip a && ip r (networking info)
* cat /etc/network/interfaces


maybe you should upgrade to 6.3 as well (apt update && apt dist-upgrade on all nodes followed by reboots because of kernel upgrade)
Click to expand...
 
Thanks, I saw that but don't see how networking issues can apply since PVE and PBS are installed on the same host. Nevertheless I will try a reboot and report back.
 
Reboot didn't help, any hints based upon this error message:
Code: 
pbs-local: error fetching datastores - 500 500 Status read failed: Connection reset by peer (500)
 
Ovidiu said:
Reboot didn't help, any hints based upon this error message:
Click to expand...

please, please send some output like i asked multiple times so me or someone else can help you...
 
I apologize, I thought you only hinted at "networking misconfiguration".

looking at journalctl right after the restart:

Code: 
Dec 03 15:30:39 nas proxmox-backup-proxy[3531]: read disk_usage on "/sixer/backups/pbs" failed - ENOENT: No such file or directory
Dec 03 15:30:39 nas proxmox-backup-proxy[3531]: find_mounted_device failed - ENOENT: No such file or directory
Dec 03 15:30:43 nas pvestatd[5005]: unable to activate storage 'local-enc' - directory is expected to be a mount point but is not mounted: '/sixer/backup
Dec 03 15:30:43 nas pvestatd[5005]: pbs-local: error fetching datastores - 500 500 Status read failed: Connection reset by peer

seems OK to me as /sixer/backups is an encrypted dataset which needs me to manually enter its password and mount it so at this moment it was not mounted.

this dataset is used as a directory storage type in PVE as local-enc and assigned for VZDump Backup files => /sixer/backups/dump is where the backups are found.

Obviously reading disk_usage of the /sixer/backups/pbs folder also failed since it was not mounted.

then this happens a couple of times:

Code: 
Dec 03 15:30:58 nas zvol_wait[3127]: Still waiting on 5 zvol links ...
Dec 03 15:30:58 nas zvol_wait[3127]: No progress since last loop.
Dec 03 15:30:58 nas zvol_wait[3127]: Checking if any zvols were deleted.

then I see:

Code: 
Dec 03 15:31:59 nas proxmox-backup-proxy[3531]: read disk_usage on "/sixer/backups/pbs" failed - ENOENT: No such file or directory
Dec 03 15:31:59 nas proxmox-backup-proxy[3531]: find_mounted_device failed - ENOENT: No such file or directory
Dec 03 15:32:00 nas proxmox-backup-proxy[3531]: lookup_datastore failed - unable to open chunk store 'pbs-local-enc' at "/sixer/backups/pbs/.chunks" - No

after this, I had entered the encryption password and mounted stuff so I am now left with this error:

Code: 
Dec 03 16:02:08 nas pvedaemon[5024]: pbs-local: error fetching datastores - 500 500 Status read failed: Connection reset by peer
Dec 03 16:02:11 nas pvestatd[5005]: pbs-local: error fetching datastores - 500 500 Status read failed: Connection reset by peer
Dec 03 16:02:21 nas pvestatd[5005]: pbs-local: error fetching datastores - 500 500 Status read failed: Connection reset by peer
Dec 03 16:02:31 nas pvestatd[5005]: pbs-local: error fetching datastores - 500 500 Status read failed: Connection reset by peer
Dec 03 16:02:41 nas pvestatd[5005]: pbs-local: error fetching datastores - 500 500 Status read failed: Connection reset by peer
Dec 03 16:02:51 nas pvestatd[5005]: pbs-local: error fetching datastores - 500 500 Status read failed: Connection reset by peer

cat /etc/pve/storage.cfg

Code: 
dir: local
        path /var/lib/vz
        content vztmpl,iso
        maxfiles 7
        shared 0

zfspool: local-zfs
        pool rpool/data
        content rootdir,images
        sparse 1

dir: local-enc
        path /sixer/backups/
        content backup
        is_mountpoint yes
        maxfiles 3
        shared 0

pbs: pbs-local
        datastore pbs-local-enc
        server localhost
        content backup
        encryption-key 1
        fingerprint 54:1c_xxx_:f1:9e:e9:68:03
        maxfiles 0
        username root@pam


ip a

(I have cleaned out all the docker interfaces)

Code: 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp35s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether d0:50:99:d6:ec:59 brd ff:ff:ff:ff:ff:ff
3: enp36s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether d0:50:99:d6:ec:5a brd ff:ff:ff:ff:ff:ff
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether d0:50:99:d6:ec:59 brd ff:ff:ff:ff:ff:ff
    inet 10.10.10.10/24 brd 10.10.10.255 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::d250:99ff:fed6:ec59/64 scope link
       valid_lft forever preferred_lft forever
5: ztfszfgoy6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2800 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether ba:92:7d:87:3a:45 brd ff:ff:ff:ff:ff:ff
    inet 10.124.135.1/24 brd 10.124.135.255 scope global ztfszfgoy6
       valid_lft forever preferred_lft forever
    inet6 fe80::8c67:7cff:fe65:18c/64 scope link
       valid_lft forever preferred_lft forever
6: br-6dec20fd6d2c: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:1e:c7:d0:4f brd ff:ff:ff:ff:ff:ff
    inet 192.168.32.1/20 brd 192.168.47.255 scope global br-6dec20fd6d2c
       valid_lft forever preferred_lft forever


ip r

Code: 
ip r
default via 10.10.10.1 dev vmbr0 onlink
10.10.10.0/24 dev vmbr0 proto kernel scope link src 10.10.10.10
10.124.135.0/24 dev ztfszfgoy6 proto kernel scope link src 10.124.135.1
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
172.19.0.0/16 dev br-c214d2d1a4e5 proto kernel scope link src 172.19.0.1
172.28.0.0/16 dev br-eb92f1151861 proto kernel scope link src 172.28.0.1
172.29.0.0/16 dev br-d4f0ebefe772 proto kernel scope link src 172.29.0.1
172.30.0.0/16 dev br-cc9c265191ab proto kernel scope link src 172.30.0.1
172.31.0.0/16 dev br-bb6e243c9c97 proto kernel scope link src 172.31.0.1 linkdown
192.168.32.0/20 dev br-6dec20fd6d2c proto kernel scope link src 192.168.32.1 linkdown

cat /etc/network/interfaces

Code: 
auto lo
iface lo inet loopback

iface enp35s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.10.10.10
        netmask 255.255.255.0
        gateway 10.10.10.1
        bridge_ports enp35s0
        bridge_stp off
        bridge_fd 0

iface enp36s0 inet manual

iface enp3s0f0u14u2c2 inet manual
 
Ovidiu said:
I apologize, I thought you only hinted at "networking misconfiguration".

looking at journalctl right after the restart:

Code: 
Dec 03 15:30:39 nas proxmox-backup-proxy[3531]: read disk_usage on "/sixer/backups/pbs" failed - ENOENT: No such file or directory
Dec 03 15:30:39 nas proxmox-backup-proxy[3531]: find_mounted_device failed - ENOENT: No such file or directory
Dec 03 15:30:43 nas pvestatd[5005]: unable to activate storage 'local-enc' - directory is expected to be a mount point but is not mounted: '/sixer/backup
Dec 03 15:30:43 nas pvestatd[5005]: pbs-local: error fetching datastores - 500 500 Status read failed: Connection reset by peer

seems OK to me as /sixer/backups is an encrypted dataset which needs me to manually enter its password and mount it so at this moment it was not mounted.

this dataset is used as a directory storage type in PVE as local-enc and assigned for VZDump Backup files => /sixer/backups/dump is where the backups are found.

Obviously reading disk_usage of the /sixer/backups/pbs folder also failed since it was not mounted.

then this happens a couple of times:

Code: 
Dec 03 15:30:58 nas zvol_wait[3127]: Still waiting on 5 zvol links ...
Dec 03 15:30:58 nas zvol_wait[3127]: No progress since last loop.
Dec 03 15:30:58 nas zvol_wait[3127]: Checking if any zvols were deleted.

then I see:

Code: 
Dec 03 15:31:59 nas proxmox-backup-proxy[3531]: read disk_usage on "/sixer/backups/pbs" failed - ENOENT: No such file or directory
Dec 03 15:31:59 nas proxmox-backup-proxy[3531]: find_mounted_device failed - ENOENT: No such file or directory
Dec 03 15:32:00 nas proxmox-backup-proxy[3531]: lookup_datastore failed - unable to open chunk store 'pbs-local-enc' at "/sixer/backups/pbs/.chunks" - No

after this, I had entered the encryption password and mounted stuff so I am now left with this error:

Code: 
Dec 03 16:02:08 nas pvedaemon[5024]: pbs-local: error fetching datastores - 500 500 Status read failed: Connection reset by peer
Dec 03 16:02:11 nas pvestatd[5005]: pbs-local: error fetching datastores - 500 500 Status read failed: Connection reset by peer
Dec 03 16:02:21 nas pvestatd[5005]: pbs-local: error fetching datastores - 500 500 Status read failed: Connection reset by peer
Dec 03 16:02:31 nas pvestatd[5005]: pbs-local: error fetching datastores - 500 500 Status read failed: Connection reset by peer
Dec 03 16:02:41 nas pvestatd[5005]: pbs-local: error fetching datastores - 500 500 Status read failed: Connection reset by peer
Dec 03 16:02:51 nas pvestatd[5005]: pbs-local: error fetching datastores - 500 500 Status read failed: Connection reset by peer

cat /etc/pve/storage.cfg

Code: 
dir: local
        path /var/lib/vz
        content vztmpl,iso
        maxfiles 7
        shared 0

zfspool: local-zfs
        pool rpool/data
        content rootdir,images
        sparse 1

dir: local-enc
        path /sixer/backups/
        content backup
        is_mountpoint yes
        maxfiles 3
        shared 0

pbs: pbs-local
        datastore pbs-local-enc
        server localhost
        content backup
        encryption-key 1
        fingerprint 54:1c_xxx_:f1:9e:e9:68:03
        maxfiles 0
        username root@pam


ip a

(I have cleaned out all the docker interfaces)

Code: 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp35s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether d0:50:99:d6:ec:59 brd ff:ff:ff:ff:ff:ff
3: enp36s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether d0:50:99:d6:ec:5a brd ff:ff:ff:ff:ff:ff
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether d0:50:99:d6:ec:59 brd ff:ff:ff:ff:ff:ff
    inet 10.10.10.10/24 brd 10.10.10.255 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::d250:99ff:fed6:ec59/64 scope link
       valid_lft forever preferred_lft forever
5: ztfszfgoy6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2800 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether ba:92:7d:87:3a:45 brd ff:ff:ff:ff:ff:ff
    inet 10.124.135.1/24 brd 10.124.135.255 scope global ztfszfgoy6
       valid_lft forever preferred_lft forever
    inet6 fe80::8c67:7cff:fe65:18c/64 scope link
       valid_lft forever preferred_lft forever
6: br-6dec20fd6d2c: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:1e:c7:d0:4f brd ff:ff:ff:ff:ff:ff
    inet 192.168.32.1/20 brd 192.168.47.255 scope global br-6dec20fd6d2c
       valid_lft forever preferred_lft forever


ip r

Code: 
ip r
default via 10.10.10.1 dev vmbr0 onlink
10.10.10.0/24 dev vmbr0 proto kernel scope link src 10.10.10.10
10.124.135.0/24 dev ztfszfgoy6 proto kernel scope link src 10.124.135.1
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
172.19.0.0/16 dev br-c214d2d1a4e5 proto kernel scope link src 172.19.0.1
172.28.0.0/16 dev br-eb92f1151861 proto kernel scope link src 172.28.0.1
172.29.0.0/16 dev br-d4f0ebefe772 proto kernel scope link src 172.29.0.1
172.30.0.0/16 dev br-cc9c265191ab proto kernel scope link src 172.30.0.1
172.31.0.0/16 dev br-bb6e243c9c97 proto kernel scope link src 172.31.0.1 linkdown
192.168.32.0/20 dev br-6dec20fd6d2c proto kernel scope link src 192.168.32.1 linkdown

cat /etc/network/interfaces

Code: 
auto lo
iface lo inet loopback

iface enp35s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.10.10.10
        netmask 255.255.255.0
        gateway 10.10.10.1
        bridge_ports enp35s0
        bridge_stp off
        bridge_fd 0

iface enp36s0 inet manual

iface enp3s0f0u14u2c2 inet manual
Click to expand...
Hi, try to restart proxmox-backup-proxy and proxmox-backup services by systemctl restart proxmox-backup proxmox-backup-proxy after unlocking the ZFS dataset containing the chunkstore. Further, check the status by systemctl status proxmox-backup proxmox-backup-proxy. Also check that the fingerprint in the pve storage config is okay.
 
Okay, so I was able to reproduce this. In order to get pvestatd to work as expected I had to change the /etc/pve/storage.cfg
Code: 
server localhost
username root
to
Code: 
server IP
username root@pam

Edit: Hostname instead of localhost works as well for me. Just localhost will not work...
Edit: 127.0.0.1 will work as well, just localhost will not (dig localhost returns 127.0.0.1 as expected and /etc/hosts is fine)
 
Last edited:
  • Like
Reactions: James Crook
Code: 
systemctl restart proxmox-backup proxmox-backup-proxy

did not help

Code: 
systemctl status proxmox-backup proxmox-backup-proxy

show all green and working

I have now used 127.0.0.1 instead of localhost and its up and working.
This is weird though as when I initially connected PVE to PBS I used localhost and its been running for about 10 days and I have had 10 daily successful backups.


Thanks everyone for their help.
 
oh, btw. looking at

cat /etc/pve/storage.cfg

Code: 
dir: local-enc
        path /sixer/backups/
        content backup
        is_mountpoint yes
        maxfiles 3
        shared 0

I had to add the argument: "is_mountpoint" yes so that PVE does not create a directory instead of waiting for the mount point to becoem available, is there such an argument to use for PBS or will PBS be fine when booting and waiting for me to unlock the encrypted zfs dataset it is configured to use as a datastore?
 
che said:
Okay, so I was able to reproduce this. In order to get pvestatd to work as expected I had to change the /etc/pve/storage.cfg
Code: 
server localhost
username root
to
Code: 
server IP
username root@pam

Edit: Hostname instead of localhost works as well for me. Just localhost will not work...
Edit: 127.0.0.1 will work as well, just localhost will not (dig localhost returns 127.0.0.1 as expected and /etc/hosts is fine)
Click to expand...
Had this error on my test unit and this fixed my issue of this !
 
Hello, same problem, but on external PBS and only on node1
1609705443093.png


On three other nodes everything ok.

Code: 
root@node1:~# pvesm list PBS-1
proxmox-backup-client failed: Error: connection error: Connection reset by peer (os error 104)

Lastest PVE 4-node cluster 6.3-3, PBS 1.0-6


Code: 
root@node1:~# pvesm status --storage PBS-1
Name         Type     Status           Total            Used       Available        %
PBS-1         pbs     active     32981255424      1312916864     31668338560    3.98%
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom