Unable to backup VM to remote storage

[oliwel]

Hi All,

I have a remote storage mounted via sshfs to /mnt/backup, backup works fine for Containers but some backups jobs for VMs end with vma_queue_write: write error - Broken pipe

Below is one of the logs (5 in total from 2 nodes) - the VMs are between 10G and 250G in size, one with 12G is working while another one with 10G is not. Based on the progress and size the backup seems to break after 6-8G beeing transfered.

The remote storage has plenty of space left:
/dev/mapper/data-backup--pve 500G 151G 350G 31% /backup/pve

Any ideas?

Oliver


4210: 2019-02-28 22:01:38 INFO: Starting Backup of VM 4210 (qemu)
4210: 2019-02-28 22:01:38 INFO: status = running
4210: 2019-02-28 22:01:38 INFO: update VM 4210: -lock backup
4210: 2019-02-28 22:01:38 INFO: VM Name: xxxxx
4210: 2019-02-28 22:01:38 INFO: include disk 'scsi0' 'san-slow-1:vm-4210-disk-1' 120G
4210: 2019-02-28 22:01:38 INFO: backup mode: snapshot
4210: 2019-02-28 22:01:38 INFO: ionice priority: 7
4210: 2019-02-28 22:01:38 INFO: creating archive '/mnt/backup/remote/dump/vzdump-qemu-4210-2019_02_28-22_01_38.vma.lzo'
4210: 2019-02-28 22:01:38 INFO: started backup task '78db298a-ea50-4382-8b69-0b174fb1ddcf'
4210: 2019-02-28 22:01:41 INFO: status: 0% (564199424/128849018880), sparse 0% (29454336), duration 3, read/write 188/178 MB/s
4210: 2019-02-28 22:01:46 INFO: status: 1% (1307574272/128849018880), sparse 0% (199274496), duration 8, read/write 148/114 MB/s
4210: 2019-02-28 22:01:56 INFO: status: 2% (2600927232/128849018880), sparse 0% (251670528), duration 18, read/write 129/124 MB/s
4210: 2019-02-28 22:02:08 INFO: status: 3% (3887071232/128849018880), sparse 0% (333725696), duration 30, read/write 107/100 MB/s
4210: 2019-02-28 22:02:22 INFO: status: 4% (5199888384/128849018880), sparse 0% (440930304), duration 44, read/write 93/86 MB/s
4210: 2019-02-28 22:02:31 INFO: status: 5% (6568804352/128849018880), sparse 0% (467300352), duration 53, read/write 152/149 MB/s
4210: 2019-02-28 22:02:47 INFO: status: 6% (7742816256/128849018880), sparse 0% (601751552), duration 69, read/write 73/64 MB/s
4210: 2019-02-28 22:02:56 INFO: status: 6% (8381399040/128849018880), sparse 0% (606035968), duration 78, read/write 70/70 MB/s
4210: 2019-02-28 22:02:56 ERROR: vma_queue_write: write error - Broken pipe
4210: 2019-02-28 22:02:56 INFO: aborting backup job
4210: 2019-02-28 22:02:58 ERROR: Backup of VM 4210 failed - vma_queue_write: write error - Broken pipe

 

[mira]

Is there enough space in /var/lib/vz/dump? By default the backup is put there temporarily before being copied to the target storage.

 

[oliwel]

root@olix:/var/lib/vz/dump# df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pve-root 29G 24G 3.7G 87% /

...and I am unable to enlarge this as the boxes run with a Disk on Module and a SAN.

Is there a chance to directly write to the remote storage? Or is there another "intended solution"? My largest VM has approx 800GB and I dont want to waste such an amount of "expensive" SAN storage just to have temporary files...

Oliver

 

[mira]

Yes, you can either specify a 'tmpdir' if you execute vzdump directly, or add it to /etc/vzdump.conf. (see https://pve.proxmox.com/pve-docs/chapter-vzdump.html#_configuration)