[SOLVED] Backup failed, vma_queue_write:write error - broken pipe

[jct]

Hi all,

I have a automatic backup every night which systematically fails with error:
vma_queue_write: write error - Broken pipe

If I try again the backup manually, it fails at the same point (21%);
It works if I delete an existing backup file.

I run PVE version 3.4-11
Storage is on a local disk.
The storage is not full : 737GB used, 906 GB available, 1.6TB total
I check the disks and the file system. All are ok.
I do not exceed the maximum number of backups on this storage.

Where should I look to solve this issue ?
Thanks for your help.

 

[sdinet]

check to see if you set the maximum number of existing backups.
I know I have seen your error before but the solution was rather simple. I would check to make sure you can ssh to and from every node when you see this message. Pipe means ssh tunnel, and netcat6.

 

[jct]

Thanks for your reply.

I can ssh to proxmox node and to the VM which is backup.

The maximum number of backups is set to 30 and there are only 5 backup files currently in /var/lib/vz/dump where I am storing the backup.
It fails when there are 4 backup files but it works with 3.

Do you know how proxmox computes the number of backups and the space available ?

Some backup files have been destroyed manually (rm ..), not by the remove command of the web interface.

note: it is a snapshot backup

 

[sdinet]

you can double-click on the error in the "tasks" stream at the bottom of proxmox webgui. See which command is failing. Post it here.

 

[jct]

Below the command:
It indicates "no space left" this time.
But there are 906GB available.
Is there any quota defined somewhere by Proxmox ?


INFO: starting new backup job: vzdump 105 --mailnotification always --quiet 1 --mode snapshot --compress lzo --storage local
INFO: Starting Backup of VM 105 (qemu)
INFO: status = running
INFO: update VM 105: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/var/lib/vz/dump/vzdump-qemu-105-2016_03_01-08_30_01.vma.lzo'
INFO: started backup task '9715986e-1661-422c-94e8-fcc03f3de464'
INFO: status: 0% (282853376/85899345920), sparse 0% (44978176), duration 3, 94/79 MB/s
INFO: status: 1% (935067648/85899345920), sparse 0% (163540992), duration 9, 108/88 MB/s
INFO: status: 2% (1828847616/85899345920), sparse 0% (210493440), duration 17, 111/105 MB/s
INFO: status: 3% (2626551808/85899345920), sparse 0% (353222656), duration 24, 113/93 MB/s
INFO: status: 4% (3527409664/85899345920), sparse 0% (401723392), duration 31, 128/121 MB/s
INFO: status: 5% (4337106944/85899345920), sparse 0% (411361280), duration 53, 36/36 MB/s
INFO: status: 6% (5159387136/85899345920), sparse 0% (511090688), duration 59, 137/120 MB/s
INFO: status: 7% (6077939712/85899345920), sparse 0% (519614464), duration 66, 131/130 MB/s
INFO: status: 8% (6982598656/85899345920), sparse 0% (645345280), duration 72, 150/129 MB/s
INFO: status: 9% (7746617344/85899345920), sparse 0% (645455872), duration 98, 29/29 MB/s
INFO: status: 10% (8707309568/85899345920), sparse 0% (645545984), duration 105, 137/137 MB/s
INFO: status: 11% (9575989248/85899345920), sparse 0% (760066048), duration 111, 144/125 MB/s
INFO: status: 12% (10310713344/85899345920), sparse 0% (760066048), duration 117, 122/122 MB/s
INFO: status: 13% (11287068672/85899345920), sparse 1% (879722496), duration 124, 139/122 MB/s
INFO: status: 14% (12055412736/85899345920), sparse 1% (879779840), duration 157, 23/23 MB/s
INFO: status: 15% (12909084672/85899345920), sparse 1% (879865856), duration 164, 121/121 MB/s
INFO: status: 16% (13839630336/85899345920), sparse 1% (1002958848), duration 170, 155/134 MB/s
INFO: status: 17% (14611251200/85899345920), sparse 1% (1003159552), duration 207, 20/20 MB/s
INFO: status: 18% (15516041216/85899345920), sparse 1% (1119883264), duration 214, 129/112 MB/s
INFO: status: 19% (16403857408/85899345920), sparse 1% (1127882752), duration 222, 110/109 MB/s
INFO: status: 20% (17211326464/85899345920), sparse 1% (1134170112), duration 245, 35/34 MB/s
INFO: status: 21% (18059034624/85899345920), sparse 1% (1258352640), duration 269, 35/30 MB/s
lzop: No space left on device: <stdout>
INFO: status: 21% (18089377792/85899345920), sparse 1% (1258352640), duration 270, 30/30 MB/s
ERROR: vma_queue_write: write error - Broken pipe
INFO: aborting backup job
ERROR: Backup of VM 105 failed - vma_queue_write: write error - Broken pipe
INFO: Backup job finished with errors
TASK ERROR: job errors

 

[jct]

I found an interesting point when I run df -hl

Sys. fich. Taille Util. Dispo Uti% Monté sur
udev 10M 0 10M 0% /dev
/dev/disk/by-uuid/9a26dbac-40c2-4b2a-b688-b4f72ac4d1e4 23G 3,2G 19G 15% /
tmpfs 5,0M 0 5,0M 0% /run/lock
tmpfs 9,4G 31M 9,4G 1% /run/shm
/dev/mapper/pve-lv1 1,7T 655G 907G 42% /var/lib/vz
/dev/mapper/pve-lv2 159G 148G 3,3G 98% /var/lib/vz/dump
/dev/fuse 30M 16K 30M 1% /etc/pve

I am trying to backup to /var/lib/vz/dump which seems limited to 159G.

Do you know where this size comes from ?
Can I extend this size ?

 

[sdinet]

It looks like /var/lib/vz/dump is limited to 159Gb, but /var/lib/vz size is 655Gb. You should easily be able to change the size. I don't know how.

 

[jct]

Thanks for your help.
I will store the backup in another place.

 

[MarcinTD]

Dzięki za pomoc.
Przechowam kopię zapasową w innym miejscu.

Jak zmienić ścieżkę przechowywania kopii zapasowych aby np. nie robiły się na local a na pve?