No access to /etc/pve/nodes/p4/lcx/ during backup - several backups fail.

[le_top]

I have daily backups for most of the virtual machines.
This night several backups failed for a new reason: Backup of VM XXX failed - unable to open file /etc/pve/nodes/p4/lxc/XXX.conf.tmp.PPPP' where XXX is the CT number and PPPP the process number.
A few backups worked and all following had this same error.
I checked if root could write to /etc/pve/nodes/p4/lcx/ and it can.

This is very strange - at this time I do not know where to look - I checked "pct list", "ha-manager status".

103: 2017-12-02 05:50:39 INFO: Starting Backup of VM 103 (lxc)
103: 2017-12-02 05:50:39 INFO: status = running
103: 2017-12-02 05:50:39 INFO: CT Name: va3
103: 2017-12-02 05:50:40 INFO: backup mode: snapshot
103: 2017-12-02 05:50:40 INFO: ionice priority: 7
103: 2017-12-02 05:50:40 INFO: create storage snapshot 'vzdump'
103: 2017-12-02 05:50:41 INFO: creating archive
'/mnt/pve/p4BackupFtp/dump/vzdump-lxc-103-2017_12_02-05_50_39.tar.gz'
103: 2017-12-02 05:51:59 INFO: Total bytes written: 2015293440 (1.9GiB,
25MiB/s)
103: 2017-12-02 05:52:01 INFO: archive file size: 672MB
103: 2017-12-02 05:52:01 INFO: delete old backup
'/mnt/pve/p4BackupFtp/dump/vzdump-lxc-103-2017_11_29-06_58_44.tar.gz'
103: 2017-12-02 05:52:02 INFO: remove vzdump snapshot
103: 2017-12-02 05:52:02 INFO: Finished Backup of VM 103 (00:01:23)

104: 2017-12-02 05:52:02 INFO: Starting Backup of VM 104 (lxc)
104: 2017-12-02 05:52:02 INFO: status = running
104: 2017-12-02 05:52:02 ERROR: Backup of VM 104 failed - unable to open file
'/etc/pve/nodes/p4/lxc/104.conf.tmp.39084' - Permission denied

105: 2017-12-02 05:52:02 INFO: Starting Backup of VM 105 (lxc)
105: 2017-12-02 05:52:02 INFO: status = running
105: 2017-12-02 05:52:02 ERROR: Backup of VM 105 failed - unable to open file
'/etc/pve/nodes/p4/lxc/105.conf.tmp.39084' - Permission denied​

 

[fabian]

check the system logs - maybe you lost quorum around that time and the cluster file system was read-only?

 

[le_top]

Thanks for the suggestion.
I forgot to come back to this thread - one of the nodes "silently" rebooted - silently because no VM was running on it and there is no mail after a reboot. It was at that moment that the backup progresse was "interrupted".

I think that backup on a functionnal node should not be impacted by this.

 

[fabian]

Thanks for the suggestion.
I forgot to come back to this thread - one of the nodes "silently" rebooted - silently because no VM was running on it and there is no mail after a reboot. It was at that moment that the backup progresse was "interrupted".

I think that backup on a functionnal node should not be impacted by this.

if you lose quorum, you lose quorum and are not allowed to make modifications which might affect other nodes (this includes locking or unlocking guest configs, which is needed for making backups). if you want to avoid this, you need enough stable nodes to not easily lose quorum.