PBS with NFS

A

amalaklabi

New Member
Jul 27, 2025
4
0
1
Hi everyone,

I have installed PBS on a VM and have NFS mount with the NAS storage, and I run the backup job for all VMs , the backup moving smoothly with the VMs with Max disk 600 Gib, but the VMs with 1 Tib or more it getting failed with error timeout, command error write_data upload error pipelined request failed.


kindly your support.
 
Hi,
please share some more information:
  • output of proxmox-backup-manager version --verbose on PBS and pveversion -v on PVE
  • the task log of the backup job from both, the PVE and the PBS host.
  • the mount parameters used to mount your NFS storage
  • the systemd journal from the PBS for the timespan of the backup, which you can dump to a file journalctl --since <DATETIME> --until <DATETIME> > journal.txt specifying the time-range.
Is the NFS server still reachable from the PBS host while the timeout happens? Any errors on the NFS server side?
 
  • Like
Reactions: amalaklabi
amalaklabi said:
I have installed PBS on a VM and have NFS mount with the NAS storage
Click to expand...
amalaklabi said:
but the VMs with 1 Tib or more it getting failed with error timeout
Click to expand...

The physical storage in the NAS is solid state or rotating rust? Topology and filesystem?

Keep in mind that PBS needs "some" IOPS. My very first PBS was classic HDD only. It worked. But after some time, after storing some amount of data, it got slow. It was not even possible to list those backups in the Web Gui. The solution in my specific case was to add some fast metadata storage.

If small backups do work for you and large ones do not your software stack might fight with fast write-caches being full..., and now slowing down the whole system down to the actual slow speed.
 
  • Like
Reactions: news, amalaklabi and Johannes S
Chris said:
Hi,
please share some more information:
  • output of proxmox-backup-manager version --verbose on PBS and pveversion -v on PVE
  • the task log of the backup job from both, the PVE and the PBS host.
  • the mount parameters used to mount your NFS storage
  • the systemd journal from the PBS for the timespan of the backup, which you can dump to a file journalctl --since <DATETIME> --until <DATETIME> > journal.txt specifying the time-range.
Is the NFS server still reachable from the PBS host while the timeout happens? Any errors on the NFS server side?
Click to expand...
Hi Chris,

thank you for your replay, i have attached the required logs and answers.


- Output of proxmox-backup-manager versions --verbose on PBS:
proxmox-backup 3.4.0 running kernel: 6.8.12-9-pve
proxmox-backup-server 3.4.0-1 running version: 3.4.0
proxmox-kernel-helper 8.1.1
proxmox-kernel-6.8 6.8.12-11
proxmox-kernel-6.8.12-9-pve-signed 6.8.12-9
ifupdown2 3.2.0-1+pmx11
libjs-extjs 7.0.0-5
proxmox-backup-docs 3.4.0-1
proxmox-backup-client 3.4.1-1
proxmox-mail-forward 0.3.2
proxmox-mini-journalreader 1.4.0
proxmox-offline-mirror-helper 0.6.7
proxmox-widget-toolkit 4.3.11
pve-xtermjs 5.5.0-2
smartmontools 7.3-pve1
zfsutils-linux 2.2.7-pve2
----------------------------------------------------
- Output pveversion -v on PVE:
proxmox-ve: 8.4.0 (running kernel: 6.8.12-11-pve)
pve-manager: 8.4.1 (running version: 8.4.1/2a5fa54a8503f96d)
proxmox-kernel-helper: 8.1.1
proxmox-kernel-6.8.12-11-pve-signed: 6.8.12-11
proxmox-kernel-6.8: 6.8.12-11
proxmox-kernel-6.8.12-10-pve: 6.8.12-10
proxmox-kernel-6.8.12-9-pve-signed: 6.8.12-9
ceph: 19.2.2-pve1~bpo12+1
ceph-fuse: 19.2.2-pve1~bpo12+1
corosync: 3.1.9-pve1
criu: 3.17.1-2+deb12u1
frr-pythontools: 10.2.2-1+pve1
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.30-pve2
libproxmox-acme-perl: 1.6.0
libproxmox-backup-qemu0: 1.5.1
libproxmox-rs-perl: 0.3.5
libpve-access-control: 8.2.2
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.1.0
libpve-cluster-perl: 8.1.0
libpve-common-perl: 8.3.1
libpve-guest-common-perl: 5.2.2
libpve-http-server-perl: 5.2.2
libpve-network-perl: 0.11.2
libpve-rs-perl: 0.9.4
libpve-storage-perl: 8.3.6
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.6.0-2
proxmox-backup-client: 3.4.1-1
proxmox-backup-file-restore: 3.4.1-1
proxmox-firewall: 0.7.1
proxmox-kernel-helper: 8.1.1
proxmox-mail-forward: 0.3.2
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.11
pve-cluster: 8.1.0
pve-container: 5.2.6
pve-docs: 8.4.0
pve-edk2-firmware: 4.2025.02-3
pve-esxi-import-tools: 0.7.4
pve-firewall: 5.1.1
pve-firmware: 3.15-4
pve-ha-manager: 4.0.7
pve-i18n: 3.4.4
pve-qemu-kvm: 9.2.0-5
pve-xtermjs: 5.5.0-2
qemu-server: 8.3.12
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.7-pve2
-----------------------------------------------------
- The task log of the backup jop from both, PVE and PBS:

- PVE & PBS:
INFO: starting new backup job: vzdump 111 --storage PBS-Backup --notes-template '{{guestname}}' --remove 0 --mode snapshot --notification-mode auto --node proxmox-ve-04
INFO: Starting Backup of VM 111 (qemu)
INFO: Backup started at 2025-07-29 14:22:14
INFO: status = running
INFO: VM Name: server01
INFO: include disk 'scsi0' 'Ceph-Main-Pool:vm-111-disk-0' 100G
INFO: include disk 'scsi1' 'Ceph-Main-Pool:vm-111-disk-1' 1000G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/111/2025-07-29T11:22:14Z'
INFO: started backup task '38be2656-5824-4bc4-a23e-9b7150739735'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: scsi1: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: 0% (520.0 MiB of 1.1 TiB) in 3s, read: 173.3 MiB/s, write: 173.3 MiB/s
INFO: 1% (11.1 GiB of 1.1 TiB) in 1m 6s, read: 172.1 MiB/s, write: 172.1 MiB/s
INFO: 2% (22.1 GiB of 1.1 TiB) in 2m 10s, read: 176.1 MiB/s, write: 176.1 MiB/s
INFO: 3% (33.2 GiB of 1.1 TiB) in 3m 15s, read: 174.2 MiB/s, write: 174.2 MiB/s
INFO: 4% (44.1 GiB of 1.1 TiB) in 4m 21s, read: 169.5 MiB/s, write: 169.5 MiB/s
INFO: 4% (49.1 GiB of 1.1 TiB) in 20m 41s, read: 5.2 MiB/s, write: 5.2 MiB/s
ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: timed out
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 111 failed - backup write data failed: command error: write_data upload error: pipelined request failed: timed out
INFO: Failed at 2025-07-29 14:42:55
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
TASK ERROR: job errors




---------------------------------------------------
- The mount parameters used to mount NFS storage:


nfs-storage.local:/Proxmox-Backup /mnt/proxmox-backup nfs rw,async,atime,nolock,hard,vers=4.1,rsize=1048576,wsize=1048576,timeo=900,nconnect=4,retrans=5 0 0


---------------------------------------------------
- The systemd journal from the PBS VM for the timespan: journalctl --since DATETIME --until DATETIME > journal.txt

Jul 29 14:22:14 proxmox-backup-server proxmox-backup-proxy[1000]: starting new backup on datastore 'vms-backup' from ::ffff:10.70.22.68: "vm/111/2025-07-29T11:22:14Z"
Jul 29 14:22:41 proxmox-backup-server postfix/qmgr[1341]: 8D70E40A826B: from=<root@proxmox-backup-server>, size=4048, nrcpt=1 (queue active)
Jul 29 14:22:41 proxmox-backup-server postfix/qmgr[1341]: CA7F84000FE9: from=<root@proxmox-backup-server>, size=1487, nrcpt=1 (queue active)
Jul 29 14:22:41 proxmox-backup-server postfix/qmgr[1341]: 3BA4440A0FC8: from=<root@proxmox-backup-server>, size=4048, nrcpt=1 (queue active)
Jul 29 14:22:41 proxmox-backup-server postfix/qmgr[1341]: D5E3B40C9EA7: from=<root@proxmox-backup-server>, size=4048, nrcpt=1 (queue active)
Jul 29 14:44:55 proxmox-backup-server proxmox-backup-proxy[1000]: TASK ERROR: removing backup snapshot "/mnt/proxmox-backup/vm/111/2025-07-29T11:22:14Z" failed - Directory not empty (os error 39)


------------------------------
Is the nfs server still reachable from the PBS VM ? yes via port 111 and 2049
Any error from NFS server? No
 

Attachments

amalaklabi said:
proxmox-backup 3.4.0 running kernel: 6.8.12-9-pve
proxmox-backup-server 3.4.0-1 running version: 3.4.0
Click to expand...
amalaklabi said:
proxmox-ve: 8.4.0 (running kernel: 6.8.12-11-pve)
pve-manager: 8.4.1 (running version: 8.4.1/2a5fa54a8503f96d)
Click to expand...
Please upgrade both, PVE and PBS to the latest patch levels and see if the issue persists further.

Since you mentioned off list that the issue persists with a local datastore as well, please check the network connectivity throughout the whole backup runtime. How is the PVE host connected to the PBS? Are they in the same location? Are you using some VPN solution which might interfere with the traffic?
 
Upgrade the PVE and PBS to which version?

How is the PVE host connected to the PBS? PVE connected to PBS over port 8007 tcp
Are they in the same location? yes but different VLAN
Are you using some VPN solution which might interfere with the traffic? no VPN , only firewall and the traffic allowed between them.
 
amalaklabi said:
Upgrade the PVE and PBS to which version?
Click to expand...
To the latest versions available, you can upgrade via apt update && apt full-upgrade. Make sure to have the package repositories set up correctly, see https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_package_repositories for PVE and https://pbs.proxmox.com/docs/installation.html#debian-package-repositories for PBS.

amalaklabi said:
How is the PVE host connected to the PBS? PVE connected to PBS over port 8007 tcp
Click to expand...
Okay, but how does the network topology looks in-betwenn them?
amalaklabi said:
Are they in the same location? yes but different VLAN
Are you using some VPN solution which might interfere with the traffic? no VPN , only firewall and the traffic allowed between them.
Click to expand...
As this is most likely a networking issue, please verify that you can continuously ping the PBS host from the PVE host while the backup is running.
 
Chris said:
To the latest versions available, you can upgrade via apt update && apt full-upgrade. Make sure to have the package repositories set up correctly, see https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_package_repositories for PVE and https://pbs.proxmox.com/docs/installation.html#debian-package-repositories for PBS.


Okay, but how does the network topology looks in-betwenn them?

As this is most likely a networking issue, please verify that you can continuously ping the PBS host from the PVE host while the backup is running.
Click to expand...
dear,

The PVE can take a backup to PBS on the datastore for the VMs with disk 100 Gib till 999 Gib, but the VMs with more than 1 Tib no! is the issue with chunks?
 
I ran PBS in a VM on top of TrueNAS.
Until recently, I connected PBS to the NAS with an internal NFS share (through an internal bridge).
But I encountered also timeout error when synchronizing the backups with another server.

Now I use a 80TB zvol as a vDisk instead a NFS share.
Resizing is less flexible but the timeout error is gone.

I can backup VMs with 3TB disks. It is very long but it works.
 
  • Like
Reactions: Johannes S
You must log in or register to reply here.
Top Bottom