VZDUMP backup error

[Drax]

I have a cluster with two nodes. The backups have been running fine. Recently I now get this error on a couple VMs on ONE node. write write error broken pipe. What does that mean? How do I fix that? I have rebooted the Failed VMs hoping that might cause the broken pipes to reconnect, but no luck. What should I do next?

 

[manu]

you should post your entire backup log so we get the context
most backup errors are due to the backup storage being unavalable when the backup is run

 

[Drax]

136: May 12 22:43:47 INFO: status: 8% (62029955072/751619276800), sparse 0% (1878495232), duration 2624, 3/2 MB/s
136: May 12 22:43:47 ERROR: vma_co_write write error - Broken pipe
136: May 12 22:43:47 INFO: aborting backup job
136: May 12 22:44:01 ERROR: Backup of VM 136 failed - vma_co_write write error - Broken pipe

3145: May 12 23:54:00 INFO: status: 33% (73000026112/214748364800), sparse 1% (3015663616), duration 3805, 3/3 MB/s
3145: May 12 23:54:00 ERROR: vma_co_write write error - Broken pipe
3145: May 12 23:54:00 INFO: aborting backup job
3145: May 12 23:54:20 ERROR: Backup of VM 3145 failed - vma_co_write write error - Broken pipe

There are the three VMs that didn't backup

 

[Drax]

So what does this mean? How is it fixed?

 

[dcsapak]

where do you backup to? nfs, local directory? check if the target has connection errors/ is full

 

[Drax]

It is backed up to a USB drive attached to the server. It does seem to be full. When I try to delete some backups it either times out or I get this message. can't lock file '/var/lock/pve-manager/pve-storage-usb_backup' - got timeout (500) How can I remove selected backups if the built in management tool is not allowing me?

 

[Drax]

So I have sourced a new larger USB HDD. Can I simply swap the drives or do I need to format it a certain way or establish a specificed folder path on the new drive? Can I simply copy the old drive to the new drive and then remove the old drive from the system? What steps do I need to take to replace this usb drive?

 

[VDNKH]

If you have a larger HDD i recommend you to add this disk without remove the old disk and test the backups

The firts thing that you need now is secure the backups

 

[Drax]

Solved? No not yet. I need the steps to solve it. I am not sure if I can just swap the drives, if there is preparation to the new drive required so no it's not solved at this point.

 

[VDNKH]

1) Add the new disk and do backups of the VMs
the firts test

 

[Drax]

Unfortunately there isn't an open USB port on the machine. So I will have to remove the existing USB then plug in the new one. Is there anything I have to do to add this to Proxmox to get it to begin backing up on the new drive? Is it as simple as unplug one and plug in the new one? Will the existing Proxmox configuration recognize the drive and back up to it?

 

[VDNKH]

OK! You have to do in the CLI delete the old backups and test the backups task, i think that the problem is the space on the disk.


In the WEB GUI in DATACENTER > STORAGE > Yourdisk > Edit. You have to set how many backups will save that disk, so that they are automatically deleted the old backups

 

[VDNKH]

Max backups per VM

I already tried when you run out of space, the message is the same as the one you showed me

My log:
INFO: starting kvm to execute backup task
INFO: started backup task '9da35092-b3bd-48e0-8151-9d9c53d49b76'
INFO: status: 14% (3141271552/21474836480), sparse 13% (2937888768), duration 3, 1047/67 MB/s
INFO: status: 15% (3314548736/21474836480), sparse 13% (2941128704), duration 6, 57/56 MB/s
INFO: status: 16% (3477798912/21474836480), sparse 13% (2959704064), duration 14, 20/18 MB/s
INFO: status: 17% (3751673856/21474836480), sparse 14% (3009675264), duration 17, 91/74 MB/s
INFO: status: 18% (4051959808/21474836480), sparse 14% (3112747008), duration 20, 100/65 MB/s
INFO: status: 23% (5138743296/21474836480), sparse 19% (4119052288), duration 23, 362/26 MB/s
INFO: status: 24% (5218893824/21474836480), sparse 19% (4119687168), duration 26, 26/26 MB/s
lzop: No space left on device: <stdout>
INFO: status: 24% (5359534080/21474836480), sparse 19% (4121120768), duration 47, 6/6 MB/s
ERROR: vma_queue_write: write error - Broken pipe
INFO: aborting backup job
INFO: stopping kvm after backup task
ERROR: Backup of VM 102 failed - vma_queue_write: write error - Broken pipe
INFO: Backup job finished with errors
TASK ERROR: job errors

 

[Drax]

I already figured out that the drive had to be replaced. All I need to know is can I simply swap the new drive for the old one? Or is there other steps I need to take? End of story.

 

[VDNKH]

Of course if you change the disk, you must format it and choose what fs type you want (ext4,ext3..)
Mount in a folder...
Add in fstab for "automount".
And add in the Web Gui

Its explained here: https://www.linuxtutorial.co.uk/proxmox-add-a-second-hard-drive-to-node-for-backups/

I hope it works for you