Issues creating backup

[Jori]

Hey there,

I am having some issues regarding backing up my LXC containers.
The situation is as followed:

I have a TrueNAS VM running that hands out 2 NFS shares. One for backups, and the other one for containers.
Both of those shares have the "open" preset from the ACL manager in TrueNAS and have been added to the datacenter in Proxmox.


Now when I want to backup a container that is on the "Containers" share, I get the following output:

INFO: starting new backup job: vzdump 101 --storage Backups --node proxmox --mode suspend --compress zstd --remove 0
INFO: Starting Backup of VM 101 (lxc)
INFO: Backup started at 2021-11-01 21:07:05
INFO: status = running
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: PiHole
INFO: including mount point rootfs ('/') in backup
INFO: starting first sync /proc/9680/root/ to /var/tmp/vzdumptmp1057873_101/
ERROR: Backup of VM 101 failed - command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/9680/root//./ /var/tmp/vzdumptmp1057873_101/' failed: exit code 23
INFO: Failed at 2021-11-01 21:07:33
INFO: Backup job finished with errors
TASK ERROR: job errors

Here is the LXC config:

root@proxmox:~# cat /etc/pve/lxc/101.conf 
arch: amd64
cores: 1
features: nesting=1
hostname: PiHole
memory: 512
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=EE:24:5B:EC:AC:03,ip=dhcp,ip6=dhcp,type=veth
onboot: 1
ostype: ubuntu
rootfs: Containers:101/vm-101-disk-0.raw,size=8G
startup: order=2
swap: 512
unprivileged: 1

The vzdump config:

root@proxmox:~# cat /etc/vzdump.conf 
# vzdump default settings

tmpdir: /var/tmp
#dumpdir: DIR
#storage: STORAGE_ID
#mode: snapshot|suspend|stop
#bwlimit: KBPS
#ionice: PRI
#lockwait: MINUTES
#stopwait: MINUTES
#stdexcludes: BOOLEAN
#mailto: ADDRESSLIST
#prune-backups: keep-INTERVAL=N[,...]
#script: FILENAME
#exclude-path: PATHLIST
#pigz: N

Code 23 refers to a permission issue according to the docs, but I cannot seem to figure it out.

Here is a list of things that I have tried:

• Changing the permissions in TrueNAS
• Changing the tmpdir in the vzdump config
• Different backup modes
• Backing up a container that is NOT on a NSF share works fine

I hope this is enough information for now.

Thanks in advance
- Jori

 

[fiona]

Hi,

Hey there,

I am having some issues regarding backing up my LXC containers.
The situation is as followed:

I have a TrueNAS VM running that hands out 2 NFS shares. One for backups, and the other one for containers.
Both of those shares have the "open" preset from the ACL manager in TrueNAS and have been added to the datacenter in Proxmox.
View attachment 31021

Now when I want to backup a container that is on the "Containers" share, I get the following output:

INFO: starting new backup job: vzdump 101 --storage Backups --node proxmox --mode suspend --compress zstd --remove 0
INFO: Starting Backup of VM 101 (lxc)
INFO: Backup started at 2021-11-01 21:07:05
INFO: status = running
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: PiHole
INFO: including mount point rootfs ('/') in backup
INFO: starting first sync /proc/9680/root/ to /var/tmp/vzdumptmp1057873_101/
ERROR: Backup of VM 101 failed - command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/9680/root//./ /var/tmp/vzdumptmp1057873_101/' failed: exit code 23
INFO: Failed at 2021-11-01 21:07:33
INFO: Backup job finished with errors
TASK ERROR: job errors

Do you have enough free space for the first copy in /var/tmp? Otherwise you need to change the tmpdir setting or not use suspend mode.

Here is the LXC config:

root@proxmox:~# cat /etc/pve/lxc/101.conf
arch: amd64
cores: 1
features: nesting=1
hostname: PiHole
memory: 512
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=EE:24:5B:EC:AC:03,ip=dhcp,ip6=dhcp,type=veth
onboot: 1
ostype: ubuntu
rootfs: Containers:101/vm-101-disk-0.raw,size=8G
startup: order=2
swap: 512
unprivileged: 1

The vzdump config:

root@proxmox:~# cat /etc/vzdump.conf
# vzdump default settings

tmpdir: /var/tmp
#dumpdir: DIR
#storage: STORAGE_ID
#mode: snapshot|suspend|stop
#bwlimit: KBPS
#ionice: PRI
#lockwait: MINUTES
#stopwait: MINUTES
#stdexcludes: BOOLEAN
#mailto: ADDRESSLIST
#prune-backups: keep-INTERVAL=N[,...]
#script: FILENAME
#exclude-path: PATHLIST
#pigz: N

Code 23 refers to a permission issue according to the docs, but I cannot seem to figure it out.

Do you mean errno 13? For rsync, exit code 23 means (from man rsync):

23     Partial transfer due to error

Here is a list of things that I have tried:

• Changing the permissions in TrueNAS
• Changing the tmpdir in the vzdump config
• Different backup modes
• Backing up a container that is NOT on a NSF share works fine

I hope this is enough information for now.

Thanks in advance
- Jori

 

[Jori]

Hey Fabian,

First of all, thanks for the reply.

Do you have enough free space for the first copy in /var/tmp? Otherwise you need to change the tmpdir setting or not use suspend mode.

I assume I do:

root@proxmox:~# df -H /var/tmp
Filesystem        Size  Used Avail Use% Mounted on
rpool/ROOT/pve-1  114G  2.7G  112G   3% /

Do you mean errno 13? For rsync, exit code 23 means (from man rsync):

I do mean 23 (since that's what in the logs), but you are right about that is means Partial transfer due to error. On the side note, searching that error code online does show that a lot of people get that error when they have permission errors, check here for an example.

I also did some more digging myself. What I've done is run the exact rsync command and added the -v to it and piped all errors to a file. It did finish with some errors and it does look like I have a permission error somewhere...

root@proxmox:~# cat errors
rsync: [generator] set_acl: sys_acl_set_file(var/log/journal, ACL_TYPE_ACCESS): Operation not supported (95)
rsync: [generator] set_acl: sys_acl_set_file(var/log/journal/173107477e7a4fb2bae4a013bdcfb46e, ACL_TYPE_ACCESS): Operation not supported (95)
rsync: [receiver] set_acl: sys_acl_set_file(var/log/journal/173107477e7a4fb2bae4a013bdcfb46e/.system.journal.7lZnJw, ACL_TYPE_ACCESS): Operation not supported (95)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1333) [sender=3.2.3]

 

[fiona]

Indeed, ZFS (on Linux) doesn't have acl on by default. Try setting zfs set acltype=posixacl rpool. If you don't want to enable it globally, I suggest creating a dedicated dataset for /var/tmp or whatever you choose the tmpdir to be.

 

[Jori]

zfs set acltype=posixacl rpool

Seemed to do the trick. Still pretty strange since I removed the ACL's and it still didn't work. Anyways, thanks a lot!