IO Delay high during backups

[jolu1705]

Hi,

I am running 2 Dell T.40 Servers.
1x PVE 32GB RAM



1x PBS 16GB RAM

Both on the same switch connected with 1GB.

The PVE is used at home only and 99% doing nothing.

As soon as I start a backup to the PBS the performance is going extremly down and the IO Delays are extreme.


IOTOP on PVE is showing


IOSTAT ON PVE


On PVE I am using ext4 single disks.
On PBS a single disk using ZFS.

Backup is started in "STOPPED" mode to make sure there is no other action or access.

How can I fix the io delays and increase the backup speed?
At the moment they are terrible low



Thanks in advance
jolu

 

[jolu1705]

Ok what I have done so far:

I have installed an additional 2 port network card to PVE and PBS and enabled LCAP 082.3ad on both of them.
I have reconnected the PBS to the PVE using the new IP

The bond is working and using Layer 3+4 which was giving me good performance with 'iperf'.

The backup itself is still slow and the io delays extremly high so my above approach was not a solution or the right way to go.

I still need help here.

 

[itobin]

Hi,

did you get anywhere with this? we are also experiencing this issue. We have tried using PBS and also to the NAS itself on a seperate 2 disk stripe. the backups work ok, then all of a sudden IO delay jumps up and all VMs become unresponsive.

We never had this issue on V6, it only seems to have manifested recently.

thanks

 

[LnxBil]

You're using one disk and expect wonders with regard to IO performance?

If you have ONE program that does random read on ONE disk, you'll get on a very, very good enterprise disk around 200 operations per second with a blocksize of 512 bytes. This yields 200 ops * 512 bytes = 100 KB/s ... that is the worst case you can encounter. Those super-duper disk throughput of 200-250 MB/sec are only for sequential read and are seldomly possible and depend on correct circumstances. In the real world, you have anything in between and your ballpark numbers of 20 MB/sec are totally feasible.

 

[itobin]

its not one disk? The PBS server is on a 3 disk array.

The issue is that the backup will seemingly run ok, then suddenly the writes drop to 0kb and the IO wait time and load increases significantly on the pve server.

Originally we had a 2 disk stipe on the NAS server, we never had one issue with backups until recently, now we cannot backup to either without the load jumping and the VMs on the PVE server becoming unresponsive.

Its cant be hardware as the NAS hasnt changed, only the PVE versions. The PBS server is new, we wanted to move backups to a better solution.

 

[mvs]

I would recommend you to use snapshot mode for PBS backups. This will activate fast incremental backup based on qemu dirty-bitmaps.

 

[itobin]

i have been using snapshot mode anyway, but sadly this isnt working out.

So as a test i have been using nas nfs backup using gzip, so far no issues. So it seem when using ZSTD the issue starts.

Also with PBS, we cannot select a compression so i guess it is also using ZSTD.

 

[Dunuin]

PBS should always use ZSTD compression.

 

[itobin]

I don't think it can be the pbs server as when checking during backup its hardly doing anything.

Is there a way to slow the backup down on the pve server?

 

[Datajock]

I don't think it can be the pbs server as when checking during backup its hardly doing anything.

Is there a way to slow the backup down on the pve server?

You can go to the "Datacenter" under "Options/Bandwidth Limits" and set the "Backup Restore:" to some value.

This is not a fix for the issue, but will allow you to slow down the backups to the point that they will just take longer and not cause the host to get into a very high IO Delay.