Hello experts!
I have a CT that got caught in Config Locked (Backup) which I fixed by unlocking it (hoping it was a fluke) but it still does it. I have no idea why since the storage doesn't seem to be the problem (all other CT/VM run backup without getting locked). I do see an error in the log though but it is for the CT before the one crashing.
Log (note 103 finishes backup, then 104 starts and then 103 gets erorrs, 104 never stops):
I don't know if the above error is releated to the failing backup but there are no other errors at all in the log. The backup storage is a local drive so no network, etc. (backups are copied externally later).
Any idea how to find the fault?
EDIT: I forgot to write that there are no log for this backup in /var/lib/vz/dump/ only empty .tmp dir. The last successful log have no errors:
I have a CT that got caught in Config Locked (Backup) which I fixed by unlocking it (hoping it was a fluke) but it still does it. I have no idea why since the storage doesn't seem to be the problem (all other CT/VM run backup without getting locked). I do see an error in the log though but it is for the CT before the one crashing.
Log (note 103 finishes backup, then 104 starts and then 103 gets erorrs, 104 never stops):
Code:
Aug 09 04:07:13 pve vzdump[31420]: INFO: Finished Backup of VM 103 (00:01:46)
Aug 09 04:07:13 pve pvestatd[1350]: modified cpu set for lxc/102: 0-1,4-6,9-10,14-15,19-21
Aug 09 04:07:13 pve vzdump[31420]: INFO: Starting Backup of VM 104 (lxc)
Aug 09 04:07:18 pve kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Aug 09 04:07:18 pve kernel: fwbr103i0: port 2(veth103i0) entered blocking state
Aug 09 04:07:18 pve kernel: fwbr103i0: port 2(veth103i0) entered forwarding state
Aug 09 04:07:18 pve audit[3628]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-103_</var/lib/lxc>" name="/dev/" pid=3628 comm="(install)" flags="ro, nosuid, noexec, remount, strictatime"
Aug 09 04:07:18 pve kernel: audit: type=1400 audit(1628474838.865:52): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-103_</var/lib/lxc>" name="/dev/" pid=3628 comm="(install)" flags="ro, nosuid, noexec, remount, strictatime"
Aug 09 04:07:19 pve audit[3638]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-103_</var/lib/lxc>" name="/dev/" pid=3638 comm="(sh)" flags="ro, nosuid, noexec, remount, strictatime"
Aug 09 04:07:19 pve kernel: audit: type=1400 audit(1628474839.057:53): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-103_</var/lib/lxc>" name="/dev/" pid=3638 comm="(sh)" flags="ro, nosuid, noexec, remount, strictatime"
Aug 09 04:07:19 pve audit[3654]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-103_</var/lib/lxc>" name="/dev/" pid=3654 comm="(sh)" flags="ro, nosuid, noexec, remount, strictatime"
Aug 09 04:07:19 pve kernel: audit: type=1400 audit(1628474839.161:54): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-103_</var/lib/lxc>" name="/dev/" pid=3654 comm="(sh)" flags="ro, nosuid, noexec, remount, strictatime"
Aug 09 04:07:20 pve audit[3817]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-103_</var/lib/lxc>" name="/dev/" pid=3817 comm="(mysqld)" flags="ro, nosuid, noexec, remount, strictatime"
Aug 09 04:07:20 pve kernel: audit: type=1400 audit(1628474840.017:55): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-103_</var/lib/lxc>" name="/dev/" pid=3817 comm="(mysqld)" flags="ro, nosuid, noexec, remount, strictatime"
Aug 09 04:07:31 pve kernel: audit: type=1400 audit(1628474851.109:56): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-103_</var/lib/lxc>" name="/dev/" pid=4111 comm="(an-start)" flags="ro, nosuid, noexec, remount, strictatime"
Aug 09 04:07:31 pve audit[4111]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-103_</var/lib/lxc>" name="/dev/" pid=4111 comm="(an-start)" flags="ro, nosuid, noexec, remount, strictatime"
Aug 09 04:07:31 pve audit[4118]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-103_</var/lib/lxc>" name="/dev/" pid=4118 comm="(sh)" flags="ro, nosuid, noexec, remount, strictatime"
Aug 09 04:07:31 pve kernel: audit: type=1400 audit(1628474851.289:57): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-103_</var/lib/lxc>" name="/dev/" pid=4118 comm="(sh)" flags="ro, nosuid, noexec, remount, strictatime"
Aug 09 04:08:00 pve systemd[1]: Starting Proxmox VE replication runner...
Aug 09 04:08:01 pve systemd[1]: pvesr.service: Succeeded.I don't know if the above error is releated to the failing backup but there are no other errors at all in the log. The backup storage is a local drive so no network, etc. (backups are copied externally later).
Any idea how to find the fault?
EDIT: I forgot to write that there are no log for this backup in /var/lib/vz/dump/ only empty .tmp dir. The last successful log have no errors:
Code:
2021-08-01 17:10:56 INFO: Starting Backup of VM 104 (lxc)
2021-08-01 17:10:56 INFO: status = running
2021-08-01 17:10:56 INFO: CT Name: elk
2021-08-01 17:10:56 INFO: including mount point rootfs ('/') in backup
2021-08-01 17:10:56 INFO: backup mode: snapshot
2021-08-01 17:10:56 INFO: ionice priority: 7
2021-08-01 17:10:56 INFO: create storage snapshot 'vzdump'
2021-08-01 17:10:57 INFO: creating vzdump archive '/var/lib/vz/dump/vzdump-lxc-104-2021_08_01-17_10_56.tar.zst'
2021-08-01 17:13:03 INFO: Total bytes written: 3224299520 (3.1GiB, 25MiB/s)
2021-08-01 17:13:03 INFO: archive file size: 1.22GB
2021-08-01 17:13:03 INFO: cleanup temporary 'vzdump' snapshot
2021-08-01 17:13:04 INFO: Finished Backup of VM 104 (00:02:08)
Last edited:
(or with 'pvenode task list' / 'pvenode task log ..', but that is likely more cumbersome).