Backup very slow - and restore to

[issa2020]

Hello there, so i got a PROXMOX 6.3.2 so the last version and update today

So i got a NAS QNAP with 1GIGAbites network :


so now on my PROXMOX i got a 1gb network card to :

On proxmox i don't know how found the speed of my network card even on command line :/


And when i backup my VM look the speed of the backup :



On graphics i got this :


Look the speed i got a max to 75 Mb ??

i got a 1 gb link what is wrong please ?

Do u know how to speed my netwokr with PROXMOX on my local network with 1 Giga lan please

thanks

 

[Dunuin]

First you are using zstd compression. So you need to make sure that your CPUs isn't the bottleneck. Possible that your CPU is too slow and can't compress the data fast enough. You could try lzo compression that isn't that CPU intense and look if it gets faster.
Second you need to make sure that your drives are fast enough. And that on both sides, your server and your QNAP. Its not unusual that a NAS can't handle more than 60MB/s of continuous writes.

You can try iperf and fio test benchmark your NIC and drives.

 

[issa2020]

another exemple :



is so long

 

[issa2020]

Thanks for u answer

my serveur don't like SUrload :

 

[Dunuin]

Your IO delay is way to high. So looks like your NAS or drives on the proxmox host are too slow and not the ethernet connection.

 

[issa2020]

Hello thanks how debug this and found the probléme please ?

 

[Dunuin]

What does your storage configuration look like?
What drives models?
Any raid?

 

[tuxevil]

Hi everyone, looking around the forum I found many post kind of similar but I couldnt find a solution and Im desperate and need some help.
Im running an HP Proliant DL380 Gen7 with 6 SAS 10K 300GB disks in HW RAID5 for system and VM images using LVM-thin and 2 SATA 5200 1TB in HW RAID1 for backups.
I have 11 VMs, including voip PBX, nextcloud, ubiquiti controller, eset admin console, mail server, web server and a few more.
Since I installed a GT710 graphic card I started to have a lot of reboots when the server tried to make the dayly backups so I decided to reinstall proxmox because I already tryed restore the VMs and it worked as expected.

So yesterday I reinstalled Proxmox 6.4 and mounted the backup disk, I started the restore process and notice it was fast at beginning but then it started to take around 40 minutes per every 1% on small VMs.
I found on 1 post that someone said it was proxmox 6.4 problem and he solved with 6.3, so I reinstalled everything today with 6.3 and started the restore process again but the speed hasent improved.

Can you give me a hint of what to do, or at least tell me "this is normal, just be patience", Im just watching the slow progress of the voip PBX that i need urgently and is just 100GB vHD, I need to find a way to improve this because nextcloud vHD are 500GB and I cant be 1 week restoring all 10 vms


Please help

 

[Dunuin]

Shouldn't be that slow. But your IO delay and CPU utilization are looking fine so the storage and compression shouldn't be bottlenecking...
So it looks like a bug to me.

 

[tuxevil]

Shouldn't be that slow. But your IO delay and CPU utilization are looking fine so the storage and compression shouldn't be bottlenecking...
So it looks like a bug to me.

Thanks for replying, you think it might be a HW bug? I mean something with the RAID controller?
I read something about updating RAID controller firmware solved the issue with a newer DL380 G10 but after that the same user says "restore are always very slow"
https://forum.proxmox.com/threads/restore-vm-very-slow.87259/post-382904

 

[Dunuin]

I also thought first the disks are too slow. Because it is fast first (writing to page file cache in RAM or to HW raids cache) and after the cache is full it would drop and be very slow. But 6x 10K RPM CMR disks in an array normally shouldn't be that slow.
Did you check that the drives are healthy? If one disk got continous SMART errors it can slowdown the complete array. And if your array is rebuilding it also would be super slow while doing that.

 

[tuxevil]

I also thought first the disks are too slow. Because it is fast first (writing to page file cache in RAM or to HW raids cache) and after the cache is full it would drop and be very slow. But 6x 10K RPM CMR disks in an array normally shouldn't be that slow.
Did you check that the drives are healthy? If one disk got continous SMART errors it can slowdown the complete array. And if your array is rebuilding it also would be super slow while doing that.

SMART of HDDs

 

[tuxevil]

Well, it seems that finally VM1 will finish (hasnt yet), after 57% the 43% left was fast (I guess because is free space)


iLO reports HHD drives as OK



I would like to do something before start the next small VM to check if it improves something before I restore my biggest VM

Do you suggest anything?

 

[tuxevil]

SMART of HDDs

View attachment 27324
View attachment 27325
View attachment 27326
View attachment 27327
View attachment 27328
View attachment 27329
View attachment 27330
View attachment 27331
View attachment 27332
View attachment 27333

Im researching about the results, and it looks like "Elements in grown defect list" and "Total uncorrected errors" are pretty bad for DISK0 and kind of bad for DISK2
DISK5 wich looks perfect is configured as spare drive, I have never done this before, but if I remove DISK0 I guess my RAID controller will use it to rebuild the array and might improve something? If everything went fine, then I can replace DISK0 with a better one and repeat the process with DISK2?

What about if i reinstall everything and instead of using HW RAID I use Proxmox ZFS? It might be better? At the moment, everything even reinstalling proxmox once again from scratch (as I did yesterday and today) is better than waiting 5 days for my nextcloud backup be restored

 

[Dunuin]

Im researching about the results, and it looks like "Elements in grown defect list" and "Total uncorrected errors" are pretty bad for DISK0 and kind of bad for DISK2
DISK5 wich looks perfect is configured as spare drive, I have never done this before, but if I remove DISK0 I guess my RAID controller will use it to rebuild the array and might improve something? If everything went fine, then I can replace DISK0 with a better one and repeat the process with DISK2?

What about if i reinstall everything and instead of using HW RAID I use Proxmox ZFS? It might be better? At the moment, everything even reinstalling proxmox once again from scratch (as I did yesterday and today) is better than waiting 5 days for my nextcloud backup be restored

You cant use ZFS if you got a raid controller. You need a HBA or atleast a raid controller that can be flashed to IT mode so it looses all raid functionality.
And I'm not sure how the spare in your HW raid works With ZFS for example, the spare ist just a temporary drive that isn't meant to permanently replace a failed drive. It will temporarily take the place of the failed drive but as soon as you replace the failed drive it will copy all data from the spare to the new one and will become a spare again.
And rebuilding a raid 5 can take days or weeks too where the disks are under heavy load and therefore are even slower.

 

[tuxevil]

You cant use ZFS if you got a raid controller. You need a HBA or atleast a raid controller that can be flashed to IT mode so it looses all raid functionality.
And I'm not sure how the spare in your HW raid works With ZFS for example, the spare ist just a temporary drive that isn't meant to permanently replace a failed drive. It will temporarily take the place of the failed drive but as soon as you replace the failed drive it will copy all data from the spare to the new one and will become a spare again.
And rebuilding a raid 5 can take days or weeks too where the disks are under heavy load and therefore are even slower.

Ok, so ZFS is discarted, because I dont want to mess it up with the RAID controller.
Looking at my SMART reports you think HDDs are needed to be replaced, even if they report as "OK"?

For a temporal solution Im copying the backup files to my test server (another proxmox in the cluster), it is just an i7 with 32GB ram and 2 RED 1TB HDD in ZFS mirror so Im hoping the restore procedure will be faster so I can get the most important VMs up and running tomorrow and the business can keep going while I find a solution for main server (probably replacing the failing drives)

Thanks again for replying you have being very helpfull