doing a Restore make error on running VM

[vikozo]

hello
i do run a restore of a 500gig Backup.

on the same Node are some VM and there a got a error like this

Info: task cron:11937 blocked for more than 120 seconds.
Not tainted 4.19.0.8-amd64 #1 Debian 4.19.98-1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message

what's the meaning i how to resolve?
the error is that bad, the webserver in the VM do not run anymore, but ping still possible

4 x Intel(R) Core(TM) i3-9100F CPU @ 3.60GHz (1 Socket)
Kernel Version Linux 5.3.18-2-pve #1 SMP PVE 5.3.18-2 (Sat, 15 Feb 2020 15:11:52 +0100)

 

[Chris]

hello
i do run a restore of a 500gig Backup.

on the same Node are some VM and there a got a error like this

Info: task cron:11937 blocked for more than 120 seconds.
Not tainted 4.19.0.8-amd64 #1 Debian 4.19.98-1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message

what's the meaning i how to resolve?
the error is that bad, the webserver in the VM do not run anymore, but ping still possible

4 x Intel(R) Core(TM) i3-9100F CPU @ 3.60GHz (1 Socket)
Kernel Version Linux 5.3.18-2-pve #1 SMP PVE 5.3.18-2 (Sat, 15 Feb 2020 15:11:52 +0100)

Hi,
it seems like the backup restore chokes IO for the VMs. Try to rate limit the backup! You can set a rate limit in the restore dialog.

 

[vikozo]

@Chris thanks for your feedback, what would be a good number to start?

 

[Chris]

@Chris thanks for your feedback, what would be a good number to start?

This depends on what your storage is able to handle. You could for example have a look on the rate you achieved during the previous restore and go with 75% of that.

 

[vikozo]

This would be a backup (not restore) - local on zfs
but still it slow down other VM's too

INFO: status: 73% (352773668864/483183820800), sparse 1% (5946437632), duration 5977, read/write 63/62 MB/s
INFO: status: 74% (357652168704/483183820800), sparse 1% (6047719424), duration 6051, read/write 65/64 MB/s
INFO: status: 75% (362452615168/483183820800), sparse 1% (6123286528), duration 6133, read/write 58/57 MB/s
INFO: status: 76% (367291727872/483183820800), sparse 1% (6194470912), duration 6209, read/write 63/62 MB/s

 

[Chris]

You could try to run some performance test using fio in order to understand the storage IO limitations. See https://fio.readthedocs.io/en/latest/fio_doc.html
What disks are you using? Are you using RAIDZ1 or RAIDZ2? Are you using a ZIL?

 

[vikozo]

RAIDZ2
4 x 6TB but one is out (had to send it back)
TOSHIBA X300 (SATA-III, 6 TB)
TOSHIBA S300 (SATA-III, 6 TB)
SEAGATE Exos 7E8, 512e SATA, 6.0TB

and will try fio tonight
@Chris
fio is installed what would be a usefull command to find out?