Nested container backup failure

[wrobelda]

Hi there,

I have a bunch of containers which run podman inside, effectively nesting containers. I got it to work fine, except Proxmox fails to backup such containers.

For example, I am getting:

INFO: starting new backup job: vzdump 115 --compress zstd --notes-template '{{guestname}}' --node proxmox --mode snapshot --storage external-hdd --remove 0
INFO: Starting Backup of VM 115 (lxc)
INFO: Backup started at 2023-08-28 21:09:54
INFO: status = running
INFO: CT Name: ctr-rssbridge
INFO: including mount point rootfs ('/') in backup
INFO: mode failure - some volumes do not support snapshots
INFO: trying 'suspend' mode instead
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: ctr-rssbridge
INFO: including mount point rootfs ('/') in backup
INFO: starting first sync /proc/4084/root/ to /mnt/hdd/vz/dump/vzdump-lxc-115-2023_08_28-21_09_54.tmp
ERROR: Backup of VM 115 failed - command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/4084/root//./ /mnt/hdd/vz/dump/vzdump-lxc-115-2023_08_28-21_09_54.tmp' failed: exit code 23
INFO: Failed at 2023-08-28 21:11:49
INFO: Backup job finished with errors
TASK ERROR: job errors

If I run the rsync command by hand, I am getting:

rsync: [sender] readlink_stat("/proc/4084/root/var/lib/containers/storage/overlay/315fcc90e1ccf389555de4fe5c853c6442f7dba4f056c1dd03d8db7a000059af/merged") failed: Permission denied (13)
IO error encountered -- skipping file deletion

And the the file at fault:

root@proxmox:~# ls -al /proc/4084/root/var/lib/containers/storage/overlay/315fcc90e1ccf389555de4fe5c853c6442f7dba4f056c1dd03d8db7a000059af/
ls: cannot access '/proc/4084/root/var/lib/containers/storage/overlay/315fcc90e1ccf389555de4fe5c853c6442f7dba4f056c1dd03d8db7a000059af/merged': Permission denied
total 32
drwx------  5 100000 100000  4096 Aug 22 15:15 .
drwx------ 25 100000 100000 12288 Aug 25 16:41 ..
dr-xr-xr-x  8 100000 100000  4096 Aug 22 15:15 diff
-rw-r--r--  1 100000 100000    26 Aug 22 15:15 link
-rw-r--r--  1 100000 100000   405 Aug 22 15:15 lower
d?????????  ? ?      ?          ?            ? merged
drwx------  3 100000 100000  4096 Aug 22 15:15 work

I am a bit puzzled as to what to do next, chmodding that file with x or X yields "Permission denied" every time, which I assume makes sense, since this is /proc filesystem.

 

[Chris]

Hi,
looks like your containers root filesystem got corrupted. Please see https://forum.proxmox.com/threads/backup-issue.43043/

 

[wrobelda]

That would be a rather odd, because all of the containers I have have the same issue. I checked fsck, and indeed they are no errors. That having said, turns out that the backup does succeed if the container is shut down — It only fails to back up when started. Proxmox resorts to suspending the container if it was previously running, so I assume these files causing issues belong to inner container managed by podman.

 

[Chris]

That would be a writer odd, because all of the containers I have had the same issue. I checked fsck, and in these they are no errors that however, it turns out that the backup does succeed when done, while the container is shut down, completely dead as soon, however, as I am start it, it will fail to back up. The backup in this case resorts to suspending the container, so I assume these files causing issues belong to inner container managed by podman.

If that is the case, the recommendation is to either perform the backups in shutdown mode or migrate from nested containers to a VM. In general, it is known that Docker ecc in LXC will cause troubles in the long run, so using a VM instead might be the better option.