PBS Input/output error (os error 5) backup VMs

[Mikl01]

Hi

When creating a backup, an error appears on both PVE and on PBS , here are the job logs (ERROR: backup finish failed: command error: error during syncfs: Input/output error (os error 5))
PVE 7.1-11
PBS 3.2-2
Has anyone here encountered a similar error and knows how to solve it?

 

[Chris]

Hi

When creating a backup, an error appears on both PVE and on PBS , here are the job logs (ERROR: backup finish failed: command error: error during syncfs: Input/output error (os error 5))
PVE 7.1-11
PBS 3.2-2
Has anyone here encountered a similar error and knows how to solve it?

Hi,
what filesystem is used on the datastore's backing storage? Does it support syncfs? Please try to set a different synclevel for the datastore by navigating in the PBS WebUI to <datastore> > Options > Tuning options and setting e.g. None to see if the problem persists.

 

[Mikl01]

The settings are as follows:
/dev/vdb1 filesystem ext4
Tuning Options
Chunk Order: Default (lnode)
Sync Level: Default (Filesystem)

I will make changesand make a backup at night
I will be able to report the result tomorrow

Chris, thanks for your advice

 

[Chris]

The settings are as follows:
/dev/vdb1 filesystem ext4
Tuning Options
Chunk Order: Default (lnode)
Sync Level: Default (Filesystem)

I will make changesand make a backup at night
I will be able to report the result tomorrow

Chris, thanks for your advice

Double check also your systemd journal on the Proxmox Backup Server for errors and perform a disk health check on /dev/vdb1 (is this a virtual disk? maybe also check the underlying host storage if so).

 

[fiona]

Hi,

PVE 7.1-11

maybe not related to the issue at hand, but please not that you didn't get any (security) updates for over two years now. For how to upgrade, see:
https://pve.proxmox.com/wiki/Package_Repositories#_proxmox_ve_7_x_repositories
https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#system_software_updates
(and since Proxmox VE 7.4 will be end-of-life with the end of July: https://pve.proxmox.com/wiki/Upgrade_from_7_to_8)

 

[Mikl01]

Double check also your systemd journal on the Proxmox Backup Server for errors and perform a disk health check on /dev/vdb1 (is this a virtual disk? maybe also check the underlying host storage if so).

HI, friends

On host task completed successfully: Checking the virtual disk.

On PVE, the backup went without errors, but the archive did not pass verification on the PBS server

And the error in the СT backup message is alarming:

INFO: starting new backup job: vzdump 302 --remove 0 --mode suspend --storage pbs2 --node portal3
INFO: Starting Backup of VM 302 (lxc)
INFO: Backup started at 2024-06-05 21:48:35
INFO: status = running
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: DB1
INFO: including mount point rootfs ('/') in backup
INFO: starting first sync /proc/3890958/root/ to /var/tmp/vzdumptmp2752455_302
ERROR: Backup of VM 302 failed - command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/3890958/root//./ /var/tmp/vzdumptmp2752455_302' failed: exit code 10
INFO: Failed at 2024-06-05 22:02:45
INFO: Backup job finished with errors
TASK ERROR: job errors

 

[fiona]

ERROR: Backup of VM 302 failed - command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/3890958/root//./ /var/tmp/vzdumptmp2752455_302' failed: exit code 10

According to the rsync man page, this means:

o      10 - Error in socket I/O

But this is even before the backup is sent to PBS, just syncing to to /var/tmp. Do you have enough free space in /var/tmp for a full copy of the container?

Can you check/share the system logs/journal for both PVE and PBS? Maybe they contain additional messages about what's going wrong.

 

[Mikl01]

Can you tell me where there should be free space /var/tmp for a full copy of the container on PBS or PVE?

 

[Chris]

Can you tell me where there should be free space /var/tmp for a full copy of the container on PBS or PVE?

On the PVE side, since that is where the state is synced to for non stop mode snapshots, see https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_backup_modes

You might want to set a dedicated tmpdir in the /etc/vzdump.conf as described here https://pve.proxmox.com/pve-docs/pve-admin-guide.html#vzdump_configuration

This is however unrelated to your initial issue with the syncfs, which you should investigate further on the Proxmox Backup Server side.

 

[Mikl01]

On the PVE side, since that is where the state is synced to for non stop mode snapshots, see https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_backup_modes

You might want to set a dedicated tmpdir in the /etc/vzdump.conf as described here https://pve.proxmox.com/pve-docs/pve-admin-guide.html#vzdump_configuration

This is however unrelated to your initial issue with the syncfs, which you should investigate further on the Proxmox Backup Server side.

Hi,
Thank you for your help.
I have launched backup model for Containers - snapshot mode. The backup was completed successfully and has been verified.
Backup of virtual machines has not yet been possible to start, I will try to do it later.