[Help]Error with PVE host backup by using proxmox-backup-client --backup-type host

[bala.ba]

Newly installed PVE8 with kernal 6.2.16-3, PBS3.0-2 was installed in a LXC container in the PVE, just want to have a easy way to manage my entire homelab under the PVE node, including the VM/CT and the PVE host itself.

All the other backup tasks for VM/CT where running well, but now I was blocked with using proxmox-backup-client to backup my PVE host with below script

#!/bin/bash
lvremove -y pve/root_snapshot || true # remove previous pve/root snapshot if there has
lvcreate -pr -L 4.95G --monitor y --snapshot --name root_snapshot pve/root # create a new snapshot for pve/root

export PBS_PASSWORD_FILE=/etc/pve/priv/storage/Backup_Root.pw # define password for PBS
proxmox-backup-client backup root.img:/dev/pve/root_snapshot --backup-type host --repository backup@pbs@192.168.1.225:Backup --ns PVE_Root # backup snapshot and upload to PBS server

lvremove -y pve/root_snapshot # remove the snapshot after backup

After several tries, always get the same error in the shell

Upload image '/dev/pve/root_snapshot' to 'backup@pbs@192.168.1.225:8007:Backup' as root.img.fidx
Error: Input/output error (os error 5)

It won't be a hardware issue as all harddisks passed the check, all enterprise SSDs.

Then I checked the log on PBS side, all these tasks were failed after execution around 10mins, similar error like this

2023-08-21T18:09:53+08:00: successfully added chunk 9f866250c9e7b07d7e1c2178ccbae5ea2d95dcb2509f51df23b518ec99042d2e to fixed index 1 (offset 89468698624, size 4194304)
2023-08-21T18:09:53+08:00: successfully added chunk 1eb11c6f6b74df8c41936b287ce0d0c4e724a5c5195dd1586c3867abb53ca26f to fixed index 1 (offset 89472892928, size 4194304)
2023-08-21T18:09:53+08:00: backup failed: connection error: not connected
2023-08-21T18:09:53+08:00: removing failed backup

Since both PBS and PVE is in the same intranet, I don't this is a network issue, can somebody help me to figure out the root cause and possible solution?

Great thanks in advance.

 

[bala.ba]

Anybody can make a hint?

 

[dcsapak]

Error: Input/output error (os error 5)

i know you said

It won't be a hardware issue as all harddisks passed the check, all enterprise SSDs.

but input/output error is still most likely an issue with the disks/storage/controller/cable etc.

does it work when you make the snapshot for root and simply read it into /dev/null? so e.g.

cat /path/to/you/snapshot/blockdev > /dev/null

this should read the snapshot and discard the output, but still show read errors on the console

it can be a network issue too, or a disk problem on the pbs side

 

[bala.ba]

i know you said

but input/output error is still most likely an issue with the disks/storage/controller/cable etc.

does it work when you make the snapshot for root and simply read it into /dev/null? so e.g.

cat /path/to/you/snapshot/blockdev > /dev/null

this should read the snapshot and discard the output, but still show read errors on the console

it can be a network issue too, or a disk problem on the pbs side

Hi Dominik,

Really appreciate for your support, no issue with that snapshot, no error output from that cat command. I know that might be a little bit weird, but my PBS datastore was set in the same SSD as what my PVE host located. Which means we can first ignore the network issue, and yes it might be the disk issue, but I have executed the disk diagnostic, no issue at all...

Any other possible problems for this issue?

Thanks,

 

[bala.ba]

I still dont't believe there's a disk issue, so I did something like this:


dd if=/dev/pve/root_snapshot of=/var/local/root.img bs=1M

Now I have a img file located in the disk, and then

proxmox-backup-client backup /var/local/root.img --backup-type host --repository backup@pbs@192.168.1.225:Backup --ns PVE_Root

And it works properly! The image was uploaded into PBS with no error, what I suppose to understand why...

 

[EllyMae]

Just a thought out of the blue: could it be a disk space issue?

 

[bala.ba]

Just a thought out of the blue: could it be a disk space issue?

I'm also thinking about this, the unused size of disk is pretty much enough for all these actions, the only thing is not so certain is the LV for the snapshot, I pre-allocated around 5G (slightly less than) disk space for this LV, that's why I used

lvcreate -pr -L 4.95G

instead of 5G, supposely it shall be enough for the rootfs.

But I just checked the file size of root.img which I manually created via dd, it's about 4.97G, don't know whether this is the root cause.