LXC Container Backup Suspend Mode exits with rsync error 23 after upgrade to PVE 7.1

[Exzellius]

@bennetgallein, my issue seems to be exactly the same as yours. I have an LXC with Docker and in it quite a few containers that are running nginx (nextcloud, swag proxy, heimdall, ...). Now I'd like to move the disk of the container to a different drive, however, the rsync command (initiated from Proxmox frontend using "Move Storage") fails with the same error as yours. The LXC is properly shut down.

Did you manage to solve this somehow?

...

rsync: [sender] get_xattr_data: lgetxattr("/var/lib/lxc/101/.copy-volume-2/var/lib/docker/overlay2/5ae2064b28b95eef52bc3ddd5a7d1a6330d4772a6087217464c250ceebe17a57/diff/var/lib/nginx/logs","user.overlay.origin",0) failed: No data available (61)
rsync: [sender] get_xattr_data: lgetxattr("/var/lib/lxc/101/.copy-volume-2/var/lib/docker/overlay2/5ae2064b28b95eef52bc3ddd5a7d1a6330d4772a6087217464c250ceebe17a57/diff/var/lib/nginx/modules","user.overlay.origin",0) failed: No data available (61)
rsync: [sender] get_xattr_data: lgetxattr("/var/lib/lxc/101/.copy-volume-2/var/lib/docker/overlay2/5ae2064b28b95eef52bc3ddd5a7d1a6330d4772a6087217464c250ceebe17a57/diff/var/lib/nginx/run","user.overlay.origin",0) failed: No data available (61)
rsync: [sender] get_xattr_data: lgetxattr("/var/lib/lxc/101/.copy-volume-2/var/lib/docker/overlay2/7dc98d61aff14288dac18b0de477379fcd475250e32b1edf1d9924de2c9e594b/diff/var/lib/nginx/logs","user.overlay.origin",0) failed: No data available (61)
rsync: [sender] get_xattr_data: lgetxattr("/var/lib/lxc/101/.copy-volume-2/var/lib/docker/overlay2/7dc98d61aff14288dac18b0de477379fcd475250e32b1edf1d9924de2c9e594b/diff/var/lib/nginx/modules","user.overlay.origin",0) failed: No data available (61)
rsync: [sender] get_xattr_data: lgetxattr("/var/lib/lxc/101/.copy-volume-2/var/lib/docker/overlay2/7dc98d61aff14288dac18b0de477379fcd475250e32b1edf1d9924de2c9e594b/diff/var/lib/nginx/run","user.overlay.origin",0) failed: No data available (61)
rsync: [sender] get_xattr_data: lgetxattr("/var/lib/lxc/101/.copy-volume-2/var/lib/docker/overlay2/eae1510a65c8b9c73832f0788ff78ed34144c10a3de95f521731e3f5ad695403/diff/var/lib/nginx/logs","user.overlay.origin",0) failed: No data available (61)
rsync: [sender] get_xattr_data: lgetxattr("/var/lib/lxc/101/.copy-volume-2/var/lib/docker/overlay2/eae1510a65c8b9c73832f0788ff78ed34144c10a3de95f521731e3f5ad695403/diff/var/lib/nginx/modules","user.overlay.origin",0) failed: No data available (61)
rsync: [sender] get_xattr_data: lgetxattr("/var/lib/lxc/101/.copy-volume-2/var/lib/docker/overlay2/eae1510a65c8b9c73832f0788ff78ed34144c10a3de95f521731e3f5ad695403/diff/var/lib/nginx/run","user.overlay.origin",0) failed: No data available (61)

...

TASK ERROR: command 'rsync --stats -X -A --numeric-ids -aH --whole-file --sparse --one-file-system '--bwlimit=0' /var/lib/lxc/101/.copy-volume-2/ /var/lib/lxc/101/.copy-volume-1' failed: exit code 23

Edit: Please note that these are symlinks. When accessing them from within the LXC, but outside the docker container, the symlinks are broken, but when accessing them from within the docker container all is fine and they can be resolved.

Interesting revelation I just got. All "files" that are problematic here are symlinks in the container. I tested it with a Valheim docker container and mapped the directory with the symlinks as a docker volume out of the container and now I have no issues anymore. Maybe this helps someone struggeling with the same issue.

 

[Exzellius]

I tracked the error down even more. The issue is specifically symlinks in the docker image.
rsync cannot get the extended attributes (Parameter -X) for the symlinks. If I omit the -X, the rsync runs flawlessly.

Where do I get the proxmox devs to look into this issue as I believe this is the extend of what I can do here?

 

[Exzellius]

Steps to reproduce the issue:
Install OpenSuSe 15.4 Container from official template
Do the following:

zypper mr -d 10
zypper in -y openssh vim man wget
systemctl enable sshd
systemctl disable apparmor
reboot
zypper --releasever=15.5 refresh
zypper --releasever=15.5 dup
reboot
zypper up -y
reboot
zypper in -y docker docker-compose
systemctl enable docker
systemctl restart docker
docker version
docker run --rm hello-world
docker images
docker rmi -f 9c7a54a9a43c
mkdir -p /docker/valheim-server
cd /docker/valheim-server
vi docker-compose.yaml
version: "3"

services:
  valheim:
    image: ghcr.io/lloesche/valheim-server
    cap_add:
      - sys_nice
    volumes:
      - /docker/valheim-server/config:/config
      - /docker/valheim-server/data:/opt/valheim
    ports:
      - "2456-2458:2456-2458/udp"
      - "9001:9001/tcp"
    env_file:
      - /docker/valheim-server/valheim.env
    restart: unless-stopped
    stop_grace_period: 2m

vi valheim.env
SERVER_NAME=<Server-IP>
WORLD_NAME=<Your Worldname>
SERVER_PASS=<Your Password>
SERVER_PUBLIC=false
SUPERVISOR_HTTP=true

docker-compose up -d

Also on the Host running the backup, we want to change the following settings:

vi /etc/vzdump.conf
tmpdir: /tmp/backup

Then run a suspend backup for this LXC and it will fail with the following error message:

Header
Proxmox
Virtual Environment 7.4-16
Container 205 (dockertest) on node 'pve01'
No Tags
Filter VMID
Logs
()
INFO: starting new backup job: vzdump 205 --notes-template '{{guestname}}' --storage pbs --node pve01 --mode suspend --remove 0
INFO: Starting Backup of VM 205 (lxc)
INFO: Backup started at 2023-08-22 09:44:32
INFO: status = running
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: dockertest
INFO: including mount point rootfs ('/') in backup
INFO: starting first sync /proc/866141/root/ to /tmp/backup/vzdumptmp924090_205/
ERROR: Backup of VM 205 failed - command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/866141/root//./ /tmp/backup/vzdumptmp924090_205/' failed: exit code 23
INFO: Failed at 2023-08-22 09:44:48
INFO: Backup job finished with errors
TASK ERROR: job errors

Running the rsync manually will produce more readable error output:

root@pve01 ~ # rsync --stats -X -h -A --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/866141/root//./ /tmp/backup/vzdumptmp880002_205
rsync: [sender] get_xattr_data: lgetxattr("/proc/866141/root/var/lib/docker/overlay2/e4c45afda3c20fc9e7b971910e6479c5ee5c759d09896d2ebeb59c83f2d36368/diff/home/valheim/.config/unity3d/IronGate/Valheim","user.overlay.origin",0) failed: No data available (61)
IO error encountered -- skipping file deletion

Number of files: 39,328 (reg: 30,784, dir: 5,547, link: 2,990, dev: 7)
Number of created files: 39,328 (reg: 30,784, dir: 5,547, link: 2,990, dev: 7)
Number of deleted files: 0
Number of regular files transferred: 26,460
Total file size: 2.71G bytes
Total transferred file size: 1.96G bytes
Literal data: 1.96G bytes
Matched data: 0 bytes
File list size: 982.99K
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 1.96G
Total bytes received: 710.80K

sent 1.96G bytes  received 710.80K bytes  301.32M bytes/sec
total size is 2.71G  speedup is 1.39
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1333) [sender=3.2.3]
root@pve01 ~ #

The file it is complaining about is a symlink in the container itself.

If we omit the -X parameter for rsync, it is running flawlessly:

root@pve01 ~ # rsync --stats -h -A --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/866141/root//./ /tmp/backup/vzdumptmp880002_205

Number of files: 39,329 (reg: 30,784, dir: 5,547, link: 2,991, dev: 7)
Number of created files: 39,329 (reg: 30,784, dir: 5,547, link: 2,991, dev: 7)
Number of deleted files: 0
Number of regular files transferred: 26,460
Total file size: 2.71G bytes
Total transferred file size: 1.96G bytes
Literal data: 1.96G bytes
Matched data: 0 bytes
File list size: 983.00K
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 1.96G
Total bytes received: 710.19K

sent 1.96G bytes  received 710.19K bytes  301.32M bytes/sec
total size is 2.71G  speedup is 1.39

 

[Exzellius]

I opened a bugreport on bugzilla:
https://bugzilla.proxmox.com/show_bug.cgi?id=4925

 

[Exzellius]

Bugreport got closed with the following information:

we don't support docker nested in LXC and actively warn about using it, it does break all the time in various ways. if you want to run docker, please do so inside a VM, not an LXC guest.

so I guess I am keeping my docker containers in a VM and case closed for me

 

[fiveangle]

Added my $0.02 on the ticket too. Bummer.