[SOLVED] backup and cloning fail and reboot

[Giacomo Vallini]

Got a subscribed proxmox 5.4-6 with all packages updated.

Since 19 of april we're unable to do backups or cloning.
Tried changing compression and making all kind of backups (stop/suspend/snapshot) but the system reboots and lock the machine with the error:

Backup of VM 100 failed - command 'set -o pipefail && tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/var/tmp/vzdumptmp28583' ./etc/vzdump/pct.conf '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | lzop >/var/lib/vz/dump/vzdump-lxc-100-2019_04_24-18_36_41.tar.dat' failed: interrupted by signal

I noticed that in caso of backup, when the job reach:
"creating archive '/var/lib/vz/dump/vzdump-lxc-100-2019_05_28-15_50_34.tar.lzo'", the file stops growing fast after about 20GB (looking with ls -l)... and after a few time the entire system crash, donnow if it can help.

Any suggestions?

Thanks a lot,

Giacomo.

 

[oguz]

hi.

* how much space do you have on your backup storage/server?
* can you post a full backup log?

 

[Giacomo Vallini]

hi.

* how much space do you have on your backup storage/server?
* can you post a full backup log?

Got 310.05 GB used of 1.66TiB

This is the backup log:

2019-05-28 15:35:16 INFO: Starting Backup of VM 100 (lxc)
2019-05-28 15:35:16 INFO: status = running
2019-05-28 15:35:16 INFO: CT Name: <REDACTED>
2019-05-28 15:35:16 INFO: excluding bind mount point mp1 ('/mnt/ext-usb') from backup
2019-05-28 15:35:16 INFO: found old vzdump snapshot (force removal)
2019-05-28 15:35:21 INFO: backup mode: snapshot
2019-05-28 15:35:21 INFO: ionice priority: 7
2019-05-28 15:35:21 INFO: create storage snapshot 'vzdump'
2019-05-28 15:35:22 INFO: creating archive '/var/lib/vz/dump/vzdump-lxc-100-2019_05_28-15_35_16.tar.lzo'
2019-05-28 15:47:39 INFO: remove vzdump snapshot
2019-05-28 15:47:40 ERROR: Backup of VM 100 failed - command 'set -o pipefail && tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/var/tmp/vzdumptmp11950' ./etc/vzdump/pct.conf '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | lzop >/var/lib/vz/dump/vzdump-lxc-100-2019_05_28-15_35_16.tar.dat' failed: interrupted by signal

 

[oguz]

* what is the command you're running?
* what kind of storage are you using for backup?
* does this happen on other containers or other hosts or other storages?

 

[Giacomo Vallini]

* what is the command you're running?
* what kind of storage are you using for backup?
* does this happen on other containers or other hosts or other storages?

*For backup i'm using the web interface, tried all kind of backup with all kink of compressions, think the command is:

'set -o pipefail && tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/var/tmp/vzdumptmp28583' ./etc/vzdump/pct.conf '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | lzop >/var/lib/vz/dump/vzdump-lxc-100-2019_04_24-18_36_41.tar.dat'

Also when i try to create a new container it crashes.

From zpool status, Scrub says everything is ok...

I have the same system (hardware, container so etc) in another office and it got no problems.

 

[oguz]

can you check if you have snapshots on the container? they might be taking up space.

did you try without compression?

which storage are you backing up on? if it's nfs or similar network storage, it might be a network issue.

 

[Giacomo Vallini]

I got 4 of them, I'll try to move them in a external drive and delete.

Donnow how to backup without compression actually...

It's a local zfs-raid10.

Thanks a lot for your help.

 

[Giacomo Vallini]

Issue was solved after we cleaned the server's fans.

It was geting too warm and shutdown for protection.

Thanks a lot for your help.

 

[oguz]

oh that''s interesting. i'm glad the problem is solved.