Backups in Proxmox7 - Slow restore & High IO on SSD

[bekax5]

Hi everyone,
Recently I made a clean install on my server, and I am trying to restore the VM Backups for a few days.

I cannot seem to find the reason why backups are this slow...
/dev/sdc with ext4 is the consumer HDD where backups are stored and /dev/sdb is the consumer SSD with Thin LVM where backups are being restored.
HDD is being utilized mostly at 1% and SSD usage is always at 80-100% with speeds around 700-300KBps.

What could be causing this? Shouldn't a backup restore like a simple file copy ?
The backups are not compressed either..
The Server has no other VM/CT running affecting IO to any of the disks and is a clean install.

 

[Dunuin]

What SSD model do you use? Consumer SSDs can be slower than HDDs when using QLC SSDs when writing a lot of data at once which might completely fill up the RAM cache and SLC cache.

 

[bekax5]

SSD is a MX500 2TB drive and HDD is a WD Black 2TB.

Do you think it makes sense to crawl to 300KBps at 50iops ? Even a consumer SSD should be able to restore a backup at normal speeds.
I wouldnt ask more than 50/60 MBps, but this is on another level.

Even reading some reviews of its performance, after filling SLC it goes from 500MBps to 250MBps. And 4k Random performance is expected to offer 40MBps+

Testing with fio, I got these results, which is what I'd expect from such a drive.


Does vzdump do something weird when restoring ?
Is there anyway to tune it ?

 

[bekax5]

I'm still trying to restore my VM backup...
Power failure led my 6 days restore to fail...
Then I used that to format the entire proxmox and reinstall.
Issue persists...
This is not something any kind of ssd/hdd should struggle with I think...

- Can anyone explain me why 8 r/s become 2500 w/s ?



- Is vzdump restore a restore like a sequential file? Or because of some reason it becomes random ?
- And I believe aqu-sz is some kind of queue... What makes the VM disk being restored have 1.5M queued requests?



Also, does this mean that the vzdump first goes to the system disk, and then to the final destination ?
I suppose /var/tmp is on the system disk?



And finaly.. How can I see the restoring speed of the backup??
According to this, the speed is different from what is projected in iostat...

 

[LnxBil]

The only explanation I can up with are the used SSDs. Those are cheap consumer SSDs that will lead to a bunch of strange behaviour if their cache is filled. Just use enteprise SSDs and you will not have those problems anymore. The forum is filled with people having problems with consumer and prosumer SSDs running on PVE, but it is not filled with people having problems with enterprise SSDs.

 

[bekax5]

I understand what you mean..
And yet after several fio tests with different configurations, the hardest it can come up to is still better throughput than a backup ?!?

I don't understand how a backup can create 2500 writes per second at 10Mbps and read from an hdd at 8 reads per second at the same speed, and then build up a 1.5M requests queue .

Shouldn't this be a sequential write ?
What kind of configuration could be used to improve ?

 

[LnxBil]

What kind of configuration could be used to improve ?

I don't think there is one. This is just "simple block level I/O".

 

[bekax5]

And any tip about the other questions ?

Why hdd has 8 reads, and that becomes 2500 writes for the same speed?

And is proxmox backup equivalent to a sequential write ?

 

[bekax5]

Resuming my issue...

I was trying to restore Backups from a HDD into Crucial MX500 2TB.
With every fio test the SSD showed "OK" results and I/O speed which is expected from a low cost consumer SSD.
Restoring Backups, the SSD would crawl at 10MBps maximum, and sometimes a few KBps.
I tried to add it into OMV and same results when creating a partition.. Average 10MBps speed.

At last, I tried to use it as LVM and not LVM-Thin!
Added it to OMV and jusmped from 10 MBs into 250 MBs+. This is what I would expect from sequential writes and reads...
Seems like this SSD doesn't handle well the extra load from Thin provisioning, and whatever extra work pve does with Thin partition.



My issue is then solved..

tldr: Crucial MX500 2TB very slow and High IO delay with LVM-Thin. Good with LVM.

 

[LnxBil]

Seems like this SSD doesn't handle well the extra load from Thin provisioning, and whatever extra work pve does with Thin partition.

Sounds reasonable. You know how thin provisioning works? In the worst case it scatters the data all over the disk and generates a lot of fragmentation (so a random I/O pattern). So if you cheap SSD is not going to handle this (I assume it's the huge write amplification due to the scattering), you're bound to use an older and dumber storage backend that sequentializes everything and therefore the write amplification is not so prevalent.

 

[bekax5]

Sounds reasonable. You know how thin provisioning works? In the worst case it scatters the data all over the disk and generates a lot of fragmentation (so a random I/O pattern). So if you cheap SSD is not going to handle this (I assume it's the huge write amplification due to the scattering), you're bound to use an older and dumber storage backend that sequentializes everything and therefore the write amplification is not so prevalent.

I didn't, and thanks for the explanation!
I will read more about it though...

And yes, seems like simple LVM solved my issues...
It did restore over 400GB in under 1 hour, between a container and VM over NFS, so it is more than my needs... and I'm happy.

I will have to study about how the thin system works, and about backups too, since I understand now it will not be as easy to do snapshots and backups compared to thin!