Abysmally slow restore from backup

[kaliszad]

Yes, full default settings. Install package, do the restore from webUI. This is the full log, which shows it used 4 restore threads, 16 parallel chunks:

new volume ID is 'Ceph_VMs:vm-5002-disk-0'
restore proxmox backup image: [REDACTED]
connecting to repository '[REDACTED]'
using up to 4 threads
open block backend for target 'rbd:Ceph_VMs/vm-5002-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/Ceph_VMs.keyring'
starting to restore snapshot 'vm/5000/2025-07-29T13:18:46Z'
download and verify backup index
fetching up to 16 chunks in parallel
progress 1% (read 805306368 bytes, zeroes = 3% (29360128 bytes), duration 0 sec)
progress 2% (read 1610612736 bytes, zeroes = 1% (29360128 bytes), duration 1 sec)
progress 3% (read 2415919104 bytes, zeroes = 1% (37748736 bytes), duration 1 sec)
progress 4% (read 3221225472 bytes, zeroes = 1% (37748736 bytes), duration 2 sec)
progress 5% (read 4026531840 bytes, zeroes = 0% (37748736 bytes), duration 3 sec)
progress 6% (read 4831838208 bytes, zeroes = 1% (50331648 bytes), duration 4 sec)
progress 7% (read 5637144576 bytes, zeroes = 0% (50331648 bytes), duration 4 sec)
progress 8% (read 6442450944 bytes, zeroes = 0% (58720256 bytes), duration 5 sec)
progress 9% (read 7247757312 bytes, zeroes = 0% (58720256 bytes), duration 5 sec)
progress 10% (read 8053063680 bytes, zeroes = 0% (58720256 bytes), duration 6 sec)
progress 11% (read 8858370048 bytes, zeroes = 0% (75497472 bytes), duration 7 sec)
progress 12% (read 9663676416 bytes, zeroes = 0% (75497472 bytes), duration 8 sec)
progress 13% (read 10468982784 bytes, zeroes = 0% (75497472 bytes), duration 8 sec)
progress 14% (read 11274289152 bytes, zeroes = 0% (92274688 bytes), duration 9 sec)
progress 15% (read 12079595520 bytes, zeroes = 0% (92274688 bytes), duration 10 sec)
progress 16% (read 12884901888 bytes, zeroes = 0% (117440512 bytes), duration 10 sec)
progress 17% (read 13690208256 bytes, zeroes = 0% (121634816 bytes), duration 11 sec)
progress 18% (read 14495514624 bytes, zeroes = 0% (121634816 bytes), duration 11 sec)
progress 19% (read 15300820992 bytes, zeroes = 0% (146800640 bytes), duration 12 sec)
progress 20% (read 16106127360 bytes, zeroes = 0% (146800640 bytes), duration 13 sec)
progress 21% (read 16911433728 bytes, zeroes = 0% (146800640 bytes), duration 14 sec)
progress 22% (read 17716740096 bytes, zeroes = 0% (176160768 bytes), duration 14 sec)
progress 23% (read 18522046464 bytes, zeroes = 0% (176160768 bytes), duration 15 sec)
progress 24% (read 19327352832 bytes, zeroes = 1% (218103808 bytes), duration 15 sec)
progress 25% (read 20132659200 bytes, zeroes = 1% (222298112 bytes), duration 16 sec)
progress 26% (read 20937965568 bytes, zeroes = 1% (222298112 bytes), duration 17 sec)
progress 27% (read 21743271936 bytes, zeroes = 1% (255852544 bytes), duration 18 sec)
progress 28% (read 22548578304 bytes, zeroes = 1% (255852544 bytes), duration 18 sec)
progress 29% (read 23353884672 bytes, zeroes = 1% (255852544 bytes), duration 19 sec)
progress 30% (read 24159191040 bytes, zeroes = 1% (289406976 bytes), duration 20 sec)
progress 31% (read 24964497408 bytes, zeroes = 1% (318767104 bytes), duration 20 sec)
progress 32% (read 25769803776 bytes, zeroes = 1% (327155712 bytes), duration 21 sec)
progress 33% (read 26575110144 bytes, zeroes = 1% (331350016 bytes), duration 22 sec)
progress 34% (read 27380416512 bytes, zeroes = 1% (331350016 bytes), duration 22 sec)
progress 35% (read 28185722880 bytes, zeroes = 1% (343932928 bytes), duration 23 sec)
progress 36% (read 28991029248 bytes, zeroes = 1% (343932928 bytes), duration 24 sec)
progress 37% (read 29796335616 bytes, zeroes = 1% (343932928 bytes), duration 24 sec)
progress 38% (read 30601641984 bytes, zeroes = 1% (364904448 bytes), duration 25 sec)
progress 39% (read 31406948352 bytes, zeroes = 1% (364904448 bytes), duration 26 sec)
progress 40% (read 32212254720 bytes, zeroes = 1% (377487360 bytes), duration 26 sec)
progress 41% (read 33017561088 bytes, zeroes = 1% (377487360 bytes), duration 27 sec)
progress 42% (read 33822867456 bytes, zeroes = 1% (377487360 bytes), duration 28 sec)
progress 43% (read 34628173824 bytes, zeroes = 1% (394264576 bytes), duration 29 sec)
progress 44% (read 35433480192 bytes, zeroes = 1% (394264576 bytes), duration 29 sec)
progress 45% (read 36238786560 bytes, zeroes = 1% (394264576 bytes), duration 30 sec)
progress 46% (read 37044092928 bytes, zeroes = 1% (436207616 bytes), duration 30 sec)
progress 47% (read 37849399296 bytes, zeroes = 1% (436207616 bytes), duration 31 sec)
progress 48% (read 38654705664 bytes, zeroes = 1% (469762048 bytes), duration 32 sec)
progress 49% (read 39460012032 bytes, zeroes = 1% (469762048 bytes), duration 32 sec)
progress 50% (read 40265318400 bytes, zeroes = 1% (503316480 bytes), duration 33 sec)
progress 51% (read 41070624768 bytes, zeroes = 1% (553648128 bytes), duration 33 sec)
progress 52% (read 41875931136 bytes, zeroes = 1% (553648128 bytes), duration 34 sec)
progress 53% (read 42681237504 bytes, zeroes = 1% (553648128 bytes), duration 34 sec)
progress 54% (read 43486543872 bytes, zeroes = 1% (633339904 bytes), duration 35 sec)
progress 55% (read 44291850240 bytes, zeroes = 1% (633339904 bytes), duration 36 sec)
progress 56% (read 45097156608 bytes, zeroes = 1% (683671552 bytes), duration 36 sec)
progress 57% (read 45902462976 bytes, zeroes = 1% (763363328 bytes), duration 37 sec)
progress 58% (read 46707769344 bytes, zeroes = 1% (763363328 bytes), duration 37 sec)
progress 59% (read 47513075712 bytes, zeroes = 1% (893386752 bytes), duration 38 sec)
progress 60% (read 48318382080 bytes, zeroes = 2% (981467136 bytes), duration 38 sec)
progress 61% (read 49123688448 bytes, zeroes = 1% (981467136 bytes), duration 39 sec)
progress 62% (read 49928994816 bytes, zeroes = 2% (1111490560 bytes), duration 39 sec)
progress 63% (read 50734301184 bytes, zeroes = 2% (1111490560 bytes), duration 40 sec)
progress 64% (read 51539607552 bytes, zeroes = 2% (1166016512 bytes), duration 40 sec)
progress 65% (read 52344913920 bytes, zeroes = 2% (1275068416 bytes), duration 41 sec)
progress 66% (read 53150220288 bytes, zeroes = 2% (1367343104 bytes), duration 41 sec)
progress 67% (read 53955526656 bytes, zeroes = 3% (1824522240 bytes), duration 41 sec)
progress 68% (read 54760833024 bytes, zeroes = 4% (2445279232 bytes), duration 42 sec)
progress 69% (read 55566139392 bytes, zeroes = 5% (3183476736 bytes), duration 42 sec)
progress 70% (read 56371445760 bytes, zeroes = 6% (3531603968 bytes), duration 42 sec)
progress 71% (read 57176752128 bytes, zeroes = 6% (3917479936 bytes), duration 42 sec)
progress 72% (read 57982058496 bytes, zeroes = 7% (4617928704 bytes), duration 42 sec)
progress 73% (read 58787364864 bytes, zeroes = 9% (5385486336 bytes), duration 42 sec)
progress 74% (read 59592671232 bytes, zeroes = 10% (6111100928 bytes), duration 43 sec)
progress 75% (read 60397977600 bytes, zeroes = 11% (6916407296 bytes), duration 43 sec)
progress 76% (read 61203283968 bytes, zeroes = 12% (7562330112 bytes), duration 43 sec)
progress 77% (read 62008590336 bytes, zeroes = 13% (8350859264 bytes), duration 43 sec)
progress 78% (read 62813896704 bytes, zeroes = 14% (9151971328 bytes), duration 43 sec)
progress 79% (read 63619203072 bytes, zeroes = 15% (9944694784 bytes), duration 43 sec)
progress 80% (read 64424509440 bytes, zeroes = 16% (10611589120 bytes), duration 43 sec)
progress 81% (read 65229815808 bytes, zeroes = 16% (11085545472 bytes), duration 43 sec)
progress 82% (read 66035122176 bytes, zeroes = 17% (11312037888 bytes), duration 43 sec)
progress 83% (read 66840428544 bytes, zeroes = 17% (11576279040 bytes), duration 44 sec)
progress 84% (read 67645734912 bytes, zeroes = 18% (12222201856 bytes), duration 44 sec)
progress 85% (read 68451041280 bytes, zeroes = 18% (12603883520 bytes), duration 44 sec)
progress 86% (read 69256347648 bytes, zeroes = 18% (12872318976 bytes), duration 45 sec)
progress 87% (read 70061654016 bytes, zeroes = 19% (13602127872 bytes), duration 45 sec)
progress 88% (read 70866960384 bytes, zeroes = 20% (14403239936 bytes), duration 45 sec)
progress 89% (read 71672266752 bytes, zeroes = 21% (15208546304 bytes), duration 45 sec)
progress 90% (read 72477573120 bytes, zeroes = 22% (16013852672 bytes), duration 45 sec)
progress 91% (read 73282879488 bytes, zeroes = 22% (16810770432 bytes), duration 45 sec)
progress 92% (read 74088185856 bytes, zeroes = 23% (17616076800 bytes), duration 45 sec)
progress 93% (read 74893492224 bytes, zeroes = 24% (18421383168 bytes), duration 45 sec)
progress 94% (read 75698798592 bytes, zeroes = 25% (19226689536 bytes), duration 45 sec)
progress 95% (read 76504104960 bytes, zeroes = 26% (20031995904 bytes), duration 45 sec)
progress 96% (read 77309411328 bytes, zeroes = 26% (20833107968 bytes), duration 45 sec)
progress 97% (read 78114717696 bytes, zeroes = 27% (21575499776 bytes), duration 45 sec)
progress 98% (read 78920024064 bytes, zeroes = 28% (22380806144 bytes), duration 45 sec)
progress 99% (read 79725330432 bytes, zeroes = 29% (23186112512 bytes), duration 45 sec)
progress 100% (read 80530636800 bytes, zeroes = 29% (23983030272 bytes), duration 45 sec)
restore image complete (bytes=80530636800, duration=45.22s, speed=1698.21MB/s)
rescan volumes...
TASK OK

Some people suggested how you can change the systemd unit files so you can have even better performance even in the WebUI. That's just a hint if you need that extra push... ;-)

 

[VictorSTS]

Some people suggested how you can change the systemd unit files so you can have even better performance even in the WebUI. That's just a hint if you need that extra push... ;-)

I know, I was involved in that conversation. I did not for two reasons:

- Had no time to implement a proper test methodology.
- Modifying each host systemd's files is a no go as that becomes unmanageable and hard to trace over time, so I'll just stick to defaults unless absolutely necessary and be happy with the improvements it provides.

 

[PwrBank]

I know, I was involved in that conversation. I did not for two reasons:

- Had no time to implement a proper test methodology.
- Modifying each host systemd's files is a no go as that becomes unmanageable and hard to trace over time, so I'll just stick to defaults unless absolutely necessary and be happy with the improvements it provides.

It'd be really nice to just have the options in the GUI under Datacenter > Backup

Or I guess an even more universal way of doing it would be to have a somewhat hidden Advanced option to list all system variables and have them editable. Similar to "about:config" in Firefox.

 

[kaliszad]

This backup restore speedups got some detailed publicity. Jan Sedlák of Lupa.cz asked me about all the little details and it got into the article almost without editing. Please excuse the title, it's not my doing, this was a global effort with people helping with testing even as far as the USA.

https://www.lupa.cz/clanky/cesi-vyr...nkurence-vmwaru-aktualizaci-dali-zdarma-vsem/

Here is a link to the Google Translate translation into English:

https://www-lupa-cz.translate.goog/...=cs&_x_tr_tl=en&_x_tr_hl=en-US&_x_tr_pto=wapp

Thank you Dominik for writing the patch series for inclusion, PwrBank, VictorSTS (your benchmark results are in the article!) and many others for testing and the work on this. Thank you to the Proxmox team for developing Proxmox into a platform, where even smaller changes like this have a large impact.

 

[lucius_the]

I would like to thank everyone on their effort in work around the speed issue !

What bothers a bit though, is these bottlenecks were attributed to software quite some time ago... read some earlier posts in this thread. Pretty much anyone could conclude it most likely wan't about hardware... except Proxmox staff. Their response was really odd. Some combination of denial and very low interest... with suggestions about all of us needing better hardware, futher spiced with dubious arguments against customizable knobs (the TLS argument in particular), etc - instead of actually giving it a look at code (AFAIK they wrote the code anyway) once it became indicative enough that the next logical thing to do was to look at code. That's a bit worying. I hope they consider changing the attitude a little - I know I was a bit hard in communication (forgotting for a moment this is an OSS project - sorry again) but techinally I was right and wasn't the only one, but that was all pretty much ignored. So that's not very great, considering they offer paid support but then reading through this thread... the team's response should have been better. You really don't have to dig into it immediately, but at least agree there's a problem than can (and/or will at some future point) be put on stack. Instead of being so quick to blame it on hardware (in spite of presented evidence, that strongly indicates this could be solved with better code, some of it possibly "low haning friut"...).

Luckily this is open source and some nice people (or... knowledgeable people in need, who were also nice to follow the OSS spirit of giving back) chimed in and did something about it, to move this issue from a dead spot. It was much needed and is much appriciated. Thank you.

I hope it doesn't stop here, there's more to be improved. I hope Proxmox team could now ackowledge there is room for improvement around performance in PBS (and/or components linked to it) and dedicate a bit work into it. With PVE being already great, a bit more performance & feature improved PBS (TLS-off option, improved verify speeds, some more konbs) - that could become a killer combo. Really, who'd need vSphere/Hyper-V + a good but way too expensive backup solution (not to name it directly) any more ? Less and less shops for sure.

Cheers to all and thanks again to great people jumping in and making restore speeds better for everone.

 

[kaliszad]

I would like to thank everyone on their effort in work around the speed issue !

What bothers a bit though, is these bottlenecks were attributed to software quite some time ago... read some earlier posts in this thread. Pretty much anyone could conclude it most likely wan't about hardware... except Proxmox staff. Their response was really odd. Some combination of denial and very low interest... with suggestions about all of us needing better hardware, futher spiced with dubious arguments against customizable knobs (the TLS argument in particular), etc - instead of actually giving it a look at code (AFAIK they wrote the code anyway) once it became indicative enough that the next logical thing to do was to look at code. That's a bit worying. I hope they consider changing the attitude a little - I know I was a bit hard in communication (forgotting for a moment this is an OSS project - sorry again) but techinally I was right and wasn't the only one, but that was all pretty much ignored. So that's not very great, considering they offer paid support but then reading through this thread... the team's response should have been better. You really don't have to dig into it immediately, but at least agree there's a problem than can (and/or will at some future point) be put on stack. Instead of being so quick to blame it on hardware (in spite of presented evidence, that strongly indicates this could be solved with better code, some of it possibly "low haning friut"...).

Luckily this is open source and some nice people (or... knowledgeable people in need, who were also nice to follow the OSS spirit of giving back) chimed in and did something about it, to move this issue from a dead spot. It was much needed and is much appriciated. Thank you.

I hope it doesn't stop here, there's more to be improved. I hope Proxmox team could now ackowledge there is room for improvement around performance in PBS (and/or components linked to it) and dedicate a bit work into it. With PVE being already great, a bit more performance & feature improved PBS (TLS-off option, improved verify speeds, some more konbs) - that could become a killer combo. Really, who'd need vSphere/Hyper-V + a good but way too expensive backup solution (not to name it directly) any more ? Less and less shops for sure.

Cheers to all and thanks again to great people jumping in and making restore speeds better for everone.

Yes, there will be people looking into some of the patterns in the code that look like they could be optimized for better performance. It's not the top most priority but it is something that they are going to do in the upcoming weeks I have been told.

 

[robertlukan]

I don't want to hijack this thread, but is there a way to speed up verification job ? Maybe the same approach can be taken ? I have Datastore with 70TB and it takes almost a day to verify it. I can provide more details, but (again) I don't want to hijack this thread.

 

[PwrBank]

I don't want to hijack this thread, but is there a way to speed up verification job ? Maybe the same approach can be taken ? I have Datastore with 70TB and it takes almost a day to verify it. I can provide more details, but (again) I don't want to hijack this thread.

Same. 128 cores and NVMe all around, less than 1% CPU overall while doing verification.

 

[Johannes S]

I don't want to hijack this thread, but is there a way to speed up verification job ? Maybe the same approach can be taken ? I have Datastore with 70TB and it takes almost a day to verify it. I can provide more details, but (again) I don't want to hijack this thread.

Please open a new thread with Details of your setup ( network, storage Media etc). Tag me so I don't overlook it

 

[lucius_the]

I don't want to hijack this thread, but is there a way to speed up verification job ? Maybe the same approach can be taken ? I have Datastore with 70TB and it takes almost a day to verify it. I can provide more details, but (again) I don't want to hijack this thread.

This has been mentioned (by me at least) several times in this thread so you're not hijacking

I believe the same kind of optimizations (parallel fetching & processing) could be made to make verify jobs run faster. It would possibly be even easier than for restores, because verifying chunks can run out-of-order. That's a lot less things to care for when doing parallel processing.

But in verifies there's other less obvious things that could be done, like, keeping track of chunks that have been verified in the current verify job run, so when you're checking a long chain or backups of similar VMs, you could skip verifying the chunks that you've already verified in this run, making the whole job run a lot faster. It's a more advanced optimization and you'd have to somewhere keep track of chunks you've just verified. So it's more work - but also more gain. Just an idea for @kaliszad (or whoever might be looking into verifies).

In case you open another thread for verify speeds, please post a link here.

 

[Johannes S]

I believe the same kind of optimizations (parallel fetching & processing) could be made to make verify jobs run faster. It would possibly be even easier than for restores, because verifying chunks can run out-of-order. That's a lot less things to care for when doing parallel processing.

@robertlukan said that he has a datastore of 70TB. I might be wrong but this sounds like a HDD-based datastore to me. That's expected to be slow for most operations. There are ways to get more out of it depending on the layout of a ZFS vpool and tricks like ZFS special device but all of them are off topic in this thread. For that reason I suggested that this is discussed in it's own thread.

 

[robertlukan]

@robertlukan said that he has a datastore of 70TB. I might be wrong but this sounds like a HDD-based datastore to me. That's expected to be slow for most operations. There are ways to get more out of it depending on the layout of a ZFS vpool and tricks like ZFS special device but all of them are off topic in this thread. For that reason I suggested that this is discussed in it's own thread.

I have two servers, very recent servers, 2 Xeon CPUs, disks are based on NVMe. Dedicated Servers are just for PBS(bare metal), with 40/100Gpbs uplink. Backups are very good, copy jobs are decent. I still need to test a new binary (library) that improves restore speed. I will open a new thread in a few days, with more details.

 

[PwrBank]

I have two servers, very recent servers, 2 Xeon CPUs, disks are based on NVMe. Dedicated Servers are just for PBS(bare metal), with 40/100Gpbs uplink. Backups are very good, copy jobs are decent. I still need to test a new binary (library) that improves restore speed. I will open a new thread in a few days, with more details.

Tag me when you get this opened if you would please. These are the specs of our backup servers and are seeing slow verification speeds as well. In a raw benchmark against the disks in a RAIDz1 we are seeing 30GB/s reads and 18GB/s writes, so 0 reasons it shouldn't be at least a few GB/s during verification.

CPU: 2x AMD EPYC 9354 (128 total cores)
RAM: 192GB DDR5 ECC
Storage: 5x 61.44TB NVMe in ZFS RAIDz1

We have 4 of these servers, all hooked up with a minimum of 2x25GbE and have the capability of 100GbE.

 

[t.lamprecht]

I hope it doesn't stop here, there's more to be improved. I hope Proxmox team could now ackowledge there is room for improvement around performance in PBS (and/or components linked to it) and dedicate a bit work into it.

We never said there was no room for improvement, let's please not misinterpret things.

Fabian replied very early in a nuanced way with lots of patience for many suggestions for "fixes" that were made without any technical background and were indeed not really helpful, and yes, we very often have users where the main cause is slow HW or high latency networks, and there throwing more threads in the mix just does not help at all there, rather the contrary.

And note that nothing was done inefficiently, just not parallel, and that was rather by design due to PBS having to cater for a wide range of HWs and use cases, just blindly employing high parallelism can cause significant IO pressure and thus result in less important things (verification, like asked here, or GC, ...) blocking important things like an ongoing restore, or a single restore starving ongoing backups, which can be more than problematic if then no recent backups are available just due to a restore, potentially of another user (we also know of multi-tenancy PBS setups, where many users access a PBS concurrently)
So doing smart parallelism is yet to be done, as already mentioned by fabian:

Post in thread 'Abysmally slow restore from backup'

Mar 31, 2025

the problem is we currently don't have the option to limit tasks like this (it is on our TODO list!). so enabling higher concurrency within tasks without at the same time offering a way to limit the number of active tasks while queuing the rest, makes this really unpredictable.

• fabian

• 

If such a generic scheduling was easy, or even just "normal" hard, it would have been done already. But resource scheduling is far from just a bit hard, and definitively not a solved problem in many areas (why else is there yet another CPU, IO, ... scheduler/rewrite in the kernel every other few years).

And do not get me wrong, this post is certainly not defending anything, I, as any other dev here, are certain that we have lots of things to improve, not only performance wise – just like any slightly more complex project. What I'm taking offense with is suggesting that we stated otherwise or basically implying that our developers and staff did not try to help, not only users but also the contributor on bringing the work over the finish line, which is most of the time a very significant amount of work (more often than not more work than assembling the initial proof-of-concept).

In the end performance is a complex beast with many factors that play into it, from use case, to hardware to naturally also software, and we constantly invest work in improving this, but there is no single "one size fits all", and having a thousand knobs that admins need to tune (and can easily mistune to go horrible wrong in critical situations) is also not a real solution, that's why we keep some things simpler at the start and favor less intra-job parallelism to reduce the chance to starve other tasks. Critical ones, or those where a user actually invests time indeed get some more special attention, like the nice work here done by kaliszad, but again, we also invested time here to get this work in, and that in a good shape.

In short, we never said that nothing can be improved in our projects, and will certainly not stop evaluating that, even if it might need a bit more time to catch some specific issues for specific use cases (be it very fast or rather slow HW), and naturally continue to always try to give external contributors that actually put in work to improve things productively a bit more attention from early on.