Backup failure for containers

[ibigbug]

INFO: Starting Backup of VM 100 (lxc)
INFO: Backup started at 2023-10-01 06:00:01
INFO: status = running
INFO: CT Name: unifi-controller
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
  Logical volume "snap_vm-100-disk-0_vzdump" created.
INFO: creating Proxmox Backup Server archive 'ct/100/2023-09-30T19:00:01Z'
INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp378309_100/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 100 --backup-time 1696100401 --repository runner@pbs@10.0.0.200:dsm-nfs
INFO: Starting backup: ct/100/2023-09-30T19:00:01Z
INFO: Client name: pve
INFO: Starting backup protocol: Sun Oct  1 06:00:02 2023
INFO: Downloading previous manifest (Thu Sep 28 06:00:07 2023)
INFO: Upload config file '/var/tmp/vzdumptmp378309_100/etc/vzdump/pct.conf' to 'runner@pbs@10.0.0.200:8007:dsm-nfs' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'runner@pbs@10.0.0.200:8007:dsm-nfs' as root.pxar.didx
INFO: Error: error at "usr/bin/mongod": Input/output error (os error 5)
INFO: cleanup temporary 'vzdump' snapshot
  Logical volume "snap_vm-100-disk-0_vzdump" successfully removed.
ERROR: Backup of VM 100 failed - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp378309_100/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 100 --backup-time 1696100401 --repository runner@pbs@10.0.0.200:dsm-nfs' failed: exit code 255
INFO: Failed at 2023-10-01 06:00:04

It seems the process inside the container "mongod" is the thing, but should the backup be affected by what's running insdie the containder?

 

[fiona]

Hi,
the error is when the backup is handling the file usr/bin/mongod, not the binary itself causing the issue. Please check the physical disks with e.g. smartctl and for any related messages in your host's system logs/journal.

 

[ibigbug]

Thank you. it's indeed the file which is suprising that a broken file can behave like this where I can move it but not cp. i reinstalled the pkg and it seem to be able to cp now. will see how the backup goes tomorrow.

 

[fiona]

Thank you. it's indeed the file which is suprising that a broken file can behave like this where I can move it but not cp. i reinstalled the pkg and it seem to be able to cp now. will see how the backup goes tomorrow.

If moving a file doesn't go across a filesystem boundary, I think it just changes the directories/metadata, but there's no need to read the actual file then.

 

[ibigbug]

very interesting though, the file(/usr/bin/mongod) that was working fine right after fresh installation was broken again. I can't cp. and it's fainling the backup with same error. any thoughts it would be the installation of the pkg or something wrong with the CT?

 

[fiona]

Can you check the system log/journal on the host at the time the issue happens? Everything okay with the underlying storage/physical disk? You can use e.g. smartctl to check.