[SOLVED] LXC container backup hanging

[polythermic]

Hi, I am trying to backup my LXC containers but the job just hangs after printing:

INFO: including mount point rootfs ('/') in backup

I have waited about two hours, but the job has not progressed. The container itself is about ~8gb, so I don't think size is an issue.

How do I go about troubleshooting this issue/get more info about what is causing it to hang?

 

[fiona]

Hi,
please share the output of pveversion -v, pct config <ID> --current and the full backup task log. What kind of storages are involved in the backup? Do you have any FUSE mounts within the container?

 

[polythermic]

Thanks for the reply!

For storages, I do not have any FUSE mounts in this container, but I am using NFS mounts.

Below are the requested outputs:

pveversion -v

root@pve:~# pveversion -v
proxmox-ve: 7.4-1 (running kernel: 5.15.104-1-pve)
pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a)
pve-kernel-5.15: 7.4-1
pve-kernel-5.15.104-1-pve: 5.15.104-2
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4.1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.4-2
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.7
libpve-storage-perl: 7.4-3
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.2-1
proxmox-backup-file-restore: 2.4.2-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.7.3
pve-cluster: 7.3-3
pve-container: 4.4-6
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-4~bpo11+1
pve-firewall: 4.3-4
pve-firmware: 3.6-5
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-2
qemu-server: 7.4-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.11-pve1

pct config 101 --current

root@pve:~# pct config 101 --current
arch: amd64
cores: 4
features: nesting=1
hostname: p
memory: 12288
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=B2:63:41:DA:E8:F1,ip=dhcp,type=veth
onboot: 1
ostype: debian
rootfs: local-lvm:vm-101-disk-0,size=16G
swap: 8192

Backup task log

INFO: starting new backup job: vzdump 101 102 106 105 --mailnotification always --node pve --storage local --compress zstd --mode snapshot --prune-backups 'keep-monthly=1' --all 0 --mailto ### --notes-template '{{guestname}}'
INFO: Starting Backup of VM 101 (lxc)
INFO: Backup started at 2023-08-07 14:26:36
INFO: status = running
INFO: CT Name: p
INFO: including mount point rootfs ('/') in backup

Here is a screenshot of the status as well, I manually stopped this one.

 

[fiona]

For storages, I do not have any FUSE mounts in this container, but I am using NFS mounts.

pct config 101 --current

root@pve:~# pct config 101 --current
arch: amd64
cores: 4
features: nesting=1
hostname: p
memory: 12288
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=B2:63:41:DA:E8:F1,ip=dhcp,type=veth
onboot: 1
ostype: debian
rootfs: local-lvm:vm-101-disk-0,size=16G
swap: 8192

Hmm, I was under the impression you would need to enable the mount=nfs feature for that. How exactly are you mounting the NFS in the container? Can you try if the backup works without the mount?

 

[polythermic]

Hmm, I was under the impression you would need to enable the mount=nfs feature for that. How exactly are you mounting the NFS in the container? Can you try if the backup works without the mount?

I am using nfs-utils and /etc/fstab in the container itself, I think it's working because it is privileged? I tried umount-ing and backing up again with no success, still just hanging.

I experimented with creating a brand new lxc container with no mounts and attempting a backup, surprisingly it also hangs. Could I have messed something up with my pve config?

 

[fiona]

I experimented with creating a brand new lxc container with no mounts and attempting a backup, surprisingly it also hangs. Could I have messed something up with my pve config?

Was that one also on the local-lvm storage? Can you try with a fresh container on the local storage? Does it hang for both privileged and unprivileged ones?

Likely not related to the issue, but it seems you are not running the very latest updates for 7.x, maybe worth giving upgrading a shot too.

 

[polythermic]

Was that one also on the local-lvm storage? Can you try with a fresh container on the local storage? Does it hang for both privileged and unprivileged ones?

Likely not related to the issue, but it seems you are not running the very latest updates for 7.x, maybe worth giving upgrading a shot too.

Yes, it was on [ICODE}local-lvm[/ICODE], it looks like I can only create containers on local-lvm:


I upgraded proxmox to 7.4-16 and tried backing up fresh privileged and unprivileged containers on local-lvm and they both have the same hanging issue.

Still at a loss on what else I can do to troubleshoot since the task log does not show any errors.

 

[fiona]

Yes, it was on [ICODE}local-lvm[/ICODE], it looks like I can only create containers on local-lvm:

You need to allow the content type in the storage configuration Datacenter > Storage > Local > Edit. Is there anything interesting in the system logs around the time the backup is done?

 

[polythermic]

You need to allow the content type in the storage configuration Datacenter > Storage > Local > Edit. Is there anything interesting in the system logs around the time the backup is done?

I found the issue after troubleshooting like you suggested: new containers on local backup up perfectly fine, and in the syslogs I saw a mount failing to connect. I had completely forgotten that I tried setting up a CIFS mount a while ago but ended up going a different way. The CIFS config was still present in pve's /etc/fstab, causing the hanging issues.

Now all of my existing containers are backing up properly.

Thank you for all of your help!

 

[fiona]

I found the issue after troubleshooting like you suggested: new containers on local backup up perfectly fine, and in the syslogs I saw a mount failing to connect. I had completely forgotten that I tried setting up a CIFS mount a while ago but ended up going a different way. The CIFS config was still present in pve's /etc/fstab, causing the hanging issues.

Now all of my existing containers are backing up properly.

Thank you for all of your help!

Thank you for sharing the solution! To help other users find solutions more quickly, please edit the thread (button on the right above the first post) and select the [SOLVED] prefix.