ERROR: Backup of VM failed - command 'rsync .... ' failed: exit code 23

[Nass]

ERROR: Backup of VM failed - command 'rsync --stats -h --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/314139/root//./ /mnt/pve/nfs-truenas/dump/vzdump-lxc-116-2022_09_01-18_17_35.tmp' failed: exit code 23

With stop mode backup done, but with suspend mode i have this error.
while using -v --progress in shell of the host i can't found error because of thousends of line, is there any option to have the log on a file?
Note: I disable ACL on the disq of my container.

 

[Moayad]

Hello,

Please provide us with the output of pveversion -v and the container config pct config <VMID>.

Does another LXC have the same issue, or only 116 LXC?

 

[Nass]

Only this LXC all the reste are good, because it's the only LXC that have (features: fuse=1,mount=fuse,nesting=1) fuse activated.

pveversion -v

proxmox-ve: 7.2-1 (running kernel: 5.15.30-2-pve)
pve-manager: 7.2-3 (running version: 7.2-3/c743d6c1)
pve-kernel-helper: 7.2-2
pve-kernel-5.15: 7.2-1
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph-fuse: 15.2.16-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-8
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-6
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.2-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.12-1
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
openvswitch-switch: 2.15.0+ds1-2+deb11u1
proxmox-backup-client: 2.1.8-1
proxmox-backup-file-restore: 2.1.8-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-10
pve-cluster: 7.2-1
pve-container: 4.2-1
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.4-1
pve-ha-manager: 3.3-4
pve-i18n: 2.7-1
pve-qemu-kvm: 6.2.0-5
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-2
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.4-pve1

pct config <VMID>

arch: amd64
cores: 2
features: fuse=1,mount=fuse,nesting=1
hostname: nginx
memory: 8192
nameserver: 8.8.8.8
net0: name=...,bridge=...,firewall=1,gw=...,hwaddr=...,ip=...,tag=..,type=veth
ostype: ubuntu
rootfs: nfs-truenas:116/vm-116-disk-1.raw,acl=0,size=80G
swap: 4096
unprivileged: 1

 

[Moayad]

Hello,

Thank you for the output!

Do any database work on this container, such as "MariaDB"? If yes, might to be like this issue [0]?

Can you also provide us with full output of the backup task log for the 116 container?


[0] https://jira.mariadb.org/browse/MDEV-27196

 

[Neobin]

5.15.30-2-pve

Not going to say, that it has anything to do with your problem, but generally:
https://forum.proxmox.com/threads/q...king-with-gpu-passthrough.117550/#post-509094 (Mainly the first paragraph in that post.)

 

[Nass]

Hello,

Thank you for the output!

Do any database work on this container, such as "MariaDB"? If yes, might to be like this issue [0]?

Can you also provide us with full output of the backup task log for the 116 container?


[0] https://jira.mariadb.org/browse/MDEV-27196

Yes, it contain MariaDB.
Full output

2022-11-07 10:56:18 INFO: Starting Backup of VM 116 (lxc)
2022-11-07 10:56:18 INFO: status = running
2022-11-07 10:56:18 INFO: CT Name: nginx
2022-11-07 10:56:18 INFO: including mount point rootfs ('/') in backup
2022-11-07 10:56:18 INFO: mode failure - some volumes do not support snapshots
2022-11-07 10:56:18 INFO: trying 'suspend' mode instead
2022-11-07 10:56:18 INFO: backup mode: suspend
2022-11-07 10:56:18 INFO: ionice priority: 7
2022-11-07 10:56:18 INFO: CT Name: nginx
2022-11-07 10:56:18 INFO: including mount point rootfs ('/') in backup
2022-11-07 10:56:18 INFO: temporary directory is on NFS, disabling xattr and acl support, consider configuring a local tmpdir via /etc/vzdump.conf
2022-11-07 10:56:18 INFO: starting first sync /proc/2732972/root/ to /mnt/pve/nfs-truenas/dump/vzdump-lxc-116-2022_11_07-10_56_18.tmp
2022-11-07 11:06:26 ERROR: Backup of VM 116 failed - command 'rsync --stats -h --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/2732972/root//./ /mnt/pve/nfs-truenas/dump/vzdump-lxc-116-2022_11_07-10_56_18.tmp' failed: exit code 23

 

[Moayad]

Hello,

Thank you for the output.

Yes, it contain MariaDB.

A kernel with the fix (5.15.64-1-pve) should fix the issue. You can try that by installing the pve-kerenel and reboot your PVE server.

 

[DC-CA1]

Hello,

Thank you for the output.


A kernel with the fix (5.15.64-1-pve) should fix the issue. You can try that by installing the pve-kerenel and reboot your PVE server.

it didnt for me..

 

[DC-CA1]

INFO: including mount point rootfs ('/') in backup
INFO: starting first sync /proc/5996/root/ to /mnt/proxmoxdb1/tmp-backup/vzdumptmp9116_105/
ERROR: Backup of VM 105 failed - command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/var/lib/php/sessions/' '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/5996/root//./ /mnt/proxmoxdb1/tmp-backup/vzdumptmp9116_105/' failed: exit code 23

 

[Moayad]

it didnt for me..

Do the PVE is running the same pve-kernel mentioned above or higher?

 

[fabian]

could you try running the same command directly after the backup has failed but add "--verbose" to the rsync parameters?

 

[DC-CA1]

could you try running the same command directly after the backup has failed but add "--verbose" to the rsync parameters?

can you please copy the required lines from my post as im not sure where the command start

Do the PVE is running the same pve-kernel mentioned above or higher?

yes i was running the last version after your post , and after rebooting

 

[DC-CA1]

rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative --verbose '--exclude=/var/lib/php/sessions/' '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/5996/root//./ /mnt/proxmoxdb1/tmp-backup/vzdumptmp9116_105

?

 

[DC-CA1]

could you try running the same command directly after the backup has failed but add "--verbose" to the rsync parameters?

after doing it , i saw that it was saying permision not permited in folder /var/log

i excluded the folder in /etc/vzdump.conf

and it fixed the issue

 

[jensie]

Dear all,

I'm trying to do a backup of my LXC and get the following error message :

ERROR: Backup of VM 2000 failed - command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/1086712/root//./ /var/tmp/vzdumptmp1496453_2000' failed: exit code 23
@DC-CA1 I used the command listed in your post

rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative --verbose '--exclude=/var/lib/php/sessions/' '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/1086712/root//./ /var/tmp/vzdumptmp1458788_2000

but got the following response :

rsync: [generator] set_acl: sys_acl_set_file(var/log/journal, ACL_TYPE_ACCESS): Operation not supported (95) rsync: [generator] set_acl: sys_acl_set_file(var/log/journal/92a1899cf71b421195d2bace02687f72, ACL_TYPE_ACCESS): Operation not supported (95) rsync: [generator] set_acl: sys_acl_set_file(var/log/journal/92a1899cf71b421195d2bace02687f72/system.journal, ACL_TYPE_ACCESS): Operation not supported (95) rsync: [generator] set_acl: sys_acl_set_file(var/log/journal/92a1899cf71b421195d2bace02687f72/system@0006037b01295803-44461770961afd06.journal~, ACL_TYPE_ACCESS): Operation not supported (95)

Can you please advise how you got out of this ?

Please note that when I shutdown the LXC, the snapshot is done without any problems. Not sure why I can't i do when the LXC is up ?

 

[glitchwitch]

@jensie I ran into the same error with some recently deployed PVE systems using ZFS. I found your post when googling the same.

Turned out this is a long known issue, I found this post which resulted in an update to the wiki.

Following the instructions in the wiki ended up resolving this for me.

Snapshot of LXC on ZFS​

If you can't create a snapshot of an LXC container on ZFS and you get following message:

INFO: rsync: set_acl: sys_acl_set_file(archiv, ACL_TYPE_DEFAULT): Operation not supported (95)

you can run following commands

zfs create -o mountpoint=/mnt/vztmp rpool/vztmp
zfs set acltype=posixacl rpool/vztmp

Now set /mnt/vztmp in your /etc/vzdump.conf for tmp

 

[jensie]

@glitchwitch this method worked also for me ! Thank you so much.
For future reference, the last step is effectively uncommenting and adjusting the following line

tmpdir: /mnt/vztmp

in /etc/vzdump.conf