Proxmox backups failing for some VMs

[PyrexxSys]

Hi,

for some reason some VMs are failing to backup onto our Proxmox backup server. The error message seems a bit cryptic and googling for it brought some results which did not really help us. The error is

INFO: resuming VM again

INFO: scsi0: dirty-bitmap status: existing bitmap was invalid and has been cleared

INFO:   2% (2.0 GiB of 96.0 GiB) in 3s, read: 672.0 MiB/s, write: 154.7 MiB/s

INFO:   3% (2.9 GiB of 96.0 GiB) in 16s, read: 73.8 MiB/s, write: 61.5 MiB/s

INFO:   4% (3.9 GiB of 96.0 GiB) in 28s, read: 82.0 MiB/s, write: 60.7 MiB/s

INFO:   5% (4.9 GiB of 96.0 GiB) in 37s, read: 116.4 MiB/s, write: 116.4 MiB/s

INFO:   6% (5.8 GiB of 96.0 GiB) in 43s, read: 162.0 MiB/s, write: 162.0 MiB/s

INFO:   7% (6.8 GiB of 96.0 GiB) in 55s, read: 78.7 MiB/s, write: 74.0 MiB/s

INFO:   8% (7.7 GiB of 96.0 GiB) in 1m 10s, read: 62.9 MiB/s, write: 39.5 MiB/s

INFO:   9% (8.7 GiB of 96.0 GiB) in 1m 20s, read: 104.4 MiB/s, write: 76.0 MiB/s

INFO:  10% (9.7 GiB of 96.0 GiB) in 1m 37s, read: 60.2 MiB/s, write: 40.5 MiB/s

INFO:  11% (10.6 GiB of 96.0 GiB) in 1m 44s, read: 133.7 MiB/s, write: 125.1 MiB/s

INFO:  12% (11.6 GiB of 96.0 GiB) in 1m 53s, read: 112.0 MiB/s, write: 72.9 MiB/s

INFO:  13% (12.5 GiB of 96.0 GiB) in 2m 3s, read: 91.6 MiB/s, write: 58.8 MiB/s

INFO:  14% (13.5 GiB of 96.0 GiB) in 2m 17s, read: 71.4 MiB/s, write: 47.1 MiB/s

INFO:  15% (14.5 GiB of 96.0 GiB) in 2m 30s, read: 81.5 MiB/s, write: 64.9 MiB/s

INFO:  16% (15.4 GiB of 96.0 GiB) in 2m 35s, read: 181.6 MiB/s, write: 102.4 MiB/s

INFO:  16% (16.0 GiB of 96.0 GiB) in 2m 37s, read: 296.0 MiB/s, write: 130.0 MiB/s

ERROR: backup write data failed: command error: error:140943FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac:../ssl/record/rec_layer_s3.c:1543:SSL alert number 20

INFO: aborting backup job

INFO: resuming VM again

ERROR: Backup of VM 119 failed - backup write data failed: command error: error:140943FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac:../ssl/record/rec_layer_s3.c:1543:SSL alert number 20

INFO: Failed at 2022-09-06 10:07:48

INFO: Backup job finished with errors

TASK ERROR: job errors

The certs are generated over Let's Encrypt directly on the pve nodes running in a cluster and an external Proxmox backup server. This only happens on some VMs (always the same VMs) but not every time. It happened yesterday by manually triggering a backup but triggering it again worked and the backup finished on the second try.

Thanks and Regards,
PyrexxSys

 

[cowonmars]

I have had this same issue over the past couple of days. It is only the Ubuntu 22.04 VM with docker containers set up in it.

 

[PyrexxSys]

Yeah, for us it fails for different VMs using different storages, so there is no pattern we can see here. this is pretty urgent though as there are some VMs which are really important so right now we do a local backup on some HDDs. Would really love to get this fixed.

 

[cowonmars]

I ended up moving my home PBS instance (was having this issue) from an LXC to a bare metal server and the error is gone.

 

[PyrexxSys]

Good to hear, but in our case the PBS is running bare metal in the data center of another provider so that is sadly not an option for us.