[SOLVED] LXC fails to backup if running

[biohead]

I'm having a few issues when it comes to backing up a specific LXC, if it's running.
All other LXCs and VMs backup fine, but this one is problematic. If I shut it down, it backs up fine - the failure is only when it is running.

I'm backing up the PVE to a local USB drive and also a remote PBS instance - the issue occurs on both destinations. Both targets are EXT4 if that matters, the PVE is ZFS.

The failure I get is an "error 23" from rsync:

Backup command/email:

Details
VMID    Name    Status    Time    Size    Filename
202    jellyfin    err    45s    0 B    null
Total running time: 45s
Logs
vzdump 202 --remove 0 --storage Backups --notes-template '{{guestname}}' --notification-mode auto --mode snapshot --compress zstd --node pve1


202: 2024-01-01 20:19:44 INFO: Starting Backup of VM 202 (lxc)
202: 2024-01-01 20:19:44 INFO: status = running
202: 2024-01-01 20:19:44 INFO: CT Name: jellyfin
202: 2024-01-01 20:19:44 INFO: including mount point rootfs ('/') in backup
202: 2024-01-01 20:19:44 INFO: excluding bind mount point mp0 ('/mnt/film') from backup (not a volume)
202: 2024-01-01 20:19:44 INFO: excluding bind mount point mp1 ('/mnt/tv') from backup (not a volume)
202: 2024-01-01 20:19:44 INFO: excluding bind mount point mp2 ('/mnt/music') from backup (not a volume)
202: 2024-01-01 20:19:44 INFO: excluding bind mount point mp3 ('/mnt/books') from backup (not a volume)
202: 2024-01-01 20:19:44 INFO: mode failure - some volumes do not support snapshots
202: 2024-01-01 20:19:44 INFO: trying 'suspend' mode instead
202: 2024-01-01 20:19:44 INFO: backup mode: suspend
202: 2024-01-01 20:19:44 INFO: ionice priority: 7
202: 2024-01-01 20:19:44 INFO: CT Name: jellyfin
202: 2024-01-01 20:19:44 INFO: including mount point rootfs ('/') in backup
202: 2024-01-01 20:19:44 INFO: excluding bind mount point mp0 ('/mnt/film') from backup (not a volume)
202: 2024-01-01 20:19:44 INFO: excluding bind mount point mp1 ('/mnt/tv') from backup (not a volume)
202: 2024-01-01 20:19:44 INFO: excluding bind mount point mp2 ('/mnt/music') from backup (not a volume)
202: 2024-01-01 20:19:44 INFO: excluding bind mount point mp3 ('/mnt/books') from backup (not a volume)
202: 2024-01-01 20:19:44 INFO: starting first sync /proc/22454/root/ to /mnt/vztmp/vzdumptmp23735_202/
202: 2024-01-01 20:20:29 ERROR: Backup of VM 202 failed - command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' '--exclude=/mnt/film' '--exclude=/mnt/tv' '--exclude=/mnt/music' '--exclude=/mnt/books' /proc/22454/root//./ /mnt/vztmp/vzdumptmp23735_202/' failed: exit code 23

If I run that exact command in the shell (and add -v and --progress) I get this output:

Output from running backup command: var/spool/postfix/lib/x86_64-linux-gnu/
var/spool/postfix/lib/x86_64-linux-gnu/libgcc_s.so.1
        125.31K 100%  258.18kB/s    0:00:00 (xfr#48123, to-chk=41/82202)
var/spool/postfix/lib/x86_64-linux-gnu/libnss_compat.so.2
rsync: [receiver] set_acl: sys_acl_set_file(var/log/journal/f62ae75316624224a73a7d73de82ca38/.system.journal.uTyoTO, ACL_TYPE_ACCESS): Operation not supported (95)
         39.90K 100%   81.68kB/s    0:00:00 (xfr#48124, to-chk=40/82202)
var/spool/postfix/lib/x86_64-linux-gnu/libnss_dns.so.2
         14.40K 100%   29.48kB/s    0:00:00 (xfr#48125, to-chk=39/82202)
var/spool/postfix/lib/x86_64-linux-gnu/libnss_files.so.2
         14.40K 100%   29.36kB/s    0:00:00 (xfr#48126, to-chk=38/82202)
var/spool/postfix/lib/x86_64-linux-gnu/libnss_hesiod.so.2
         27.14K 100%   55.32kB/s    0:00:00 (xfr#48127, to-chk=37/82202)
var/spool/postfix/lib/x86_64-linux-gnu/libnss_systemd.so.2
        325.90K 100%  656.20kB/s    0:00:00 (xfr#48128, to-chk=36/82202)
var/spool/postfix/lib/x86_64-linux-gnu/libresolv.so.2
         60.33K 100%  121.47kB/s    0:00:00 (xfr#48129, to-chk=35/82202)
var/spool/postfix/maildrop/
var/spool/postfix/pid/
var/spool/postfix/pid/master.pid
             33 100%    0.07kB/s    0:00:00 (xfr#48130, to-chk=34/82202)
var/spool/postfix/private/
var/spool/postfix/private/anvil
var/spool/postfix/private/bounce
var/spool/postfix/private/bsmtp
var/spool/postfix/private/defer
var/spool/postfix/private/discard
var/spool/postfix/private/error
var/spool/postfix/private/ifmail
var/spool/postfix/private/lmtp
var/spool/postfix/private/local
var/spool/postfix/private/maildrop
var/spool/postfix/private/mailman
var/spool/postfix/private/proxymap
var/spool/postfix/private/proxywrite
var/spool/postfix/private/relay
var/spool/postfix/private/retry
var/spool/postfix/private/rewrite
var/spool/postfix/private/scache
var/spool/postfix/private/scalemail-backend
var/spool/postfix/private/smtp
var/spool/postfix/private/tlsmgr
var/spool/postfix/private/trace
var/spool/postfix/private/uucp
var/spool/postfix/private/verify
var/spool/postfix/private/virtual
var/spool/postfix/public/
var/spool/postfix/public/cleanup
var/spool/postfix/public/flush
var/spool/postfix/public/pickup
var/spool/postfix/public/postlog
var/spool/postfix/public/qmgr
var/spool/postfix/public/showq
var/spool/postfix/saved/
var/spool/postfix/trace/
var/spool/postfix/usr/
var/spool/postfix/usr/lib/
var/spool/postfix/usr/lib/sasl2/
var/spool/postfix/usr/lib/zoneinfo/
var/spool/postfix/usr/lib/zoneinfo/localtime -> /etc/localtime
var/tmp/

Number of files: 82,202 (reg: 48,142, dir: 30,671, link: 3,357, dev: 2, special: 30)
Number of created files: 82,202 (reg: 48,142, dir: 30,671, link: 3,357, dev: 2, special: 30)
Number of deleted files: 0
Number of regular files transferred: 48,130
Total file size: 4.93G bytes
Total transferred file size: 4.88G bytes
Literal data: 4.88G bytes
Matched data: 0 bytes
File list size: 4.46M
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 4.89G
Total bytes received: 1.13M

sent 4.89G bytes  received 1.13M bytes  76.95M bytes/sec
total size is 4.93G  speedup is 1.01
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.2.7]
root@pve1:~#

I believe the issue is this line:
rsync: [receiver] set_acl: sys_acl_set_file(var/log/journal/f62ae75316624224a73a7d73de82ca38/.system.journal.uTyoTO, ACL_TYPE_ACCESS): Operation not supported (95)
It's the only failure I can see - looks like journals and ACLs - something I'm not really au fait with at all.
I've tried a very quick restore from backup, but that had no effect.

It has been backing up weekly to USB fine.
I had a spare Kimsufi KS1 instance I've installed PBS to and the issue was limited to that. But now my USB backups are also failing for that LXC.
I'm not sure if it's related, unfortunate or a sign something is wrong with that LXC.

Does anyone have any hints or tips I could try, or where to investigate further?

 

[fschauer]

When doing a suspend mode backup, the container data is copied to a temporary directory before being written to the target storage. If the container contains ACLs, then the temporary directory also has to support ACLs for the suspend mode backup to work.
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_backup_modes

You are using PVE with ZFS which has ACLs disabled by default. You can check with the following command:

zfs get acltype rpool/ROOT/pve-1

To enable ACLs use the following command:

zfs set acltype=posixacl rpool/ROOT/pve-1

Even better might be to create a mountpoint for the temporary directory and enable ACLs only for that:
https://pve.proxmox.com/wiki/ZFS:_Tips_and_Tricks#Snapshot_of_LXC_on_ZFS

 

[biohead]

Oh that's perfect! Worked straight away.

I was so concerned that it was the target storage with the issue I didn't even consider it could be the PVE's storage.