Abysmally slow restore from backup

[VictorSTS]

Using environment variables probably means running the restore command in shell, right ?

That would be a let down. This should be easily adjusted i.e. at datacenter.cfg so a normal restore from the web UI applies them (i.e. non root user(s) with enough privileges to restore their VMs). I would be ok if you had to use cli to set those values, although IMO they should be available in the webUI too.
I would love to see this backported to PVE8.4.x

 

[PwrBank]

Using environment variables probably means running the restore command in shell, right ?

Not if you set the variable system wide, then when the actual command is ran by the system it'll just use those values.

Basically saying, $threadcount=14

and then itstead of having a hard coded number, it references the $threadcount in the system.

 

[lucius_the]

Not if you set the variable system wide, then when the actual command is ran by the system it'll just use those values.

Basically saying, $threadcount=14

and then itstead of having a hard coded number, it references the $threadcount in the system.

Right. That's because of my (somewhat limited) Linux knowledge... I thought that env vars are per-user, but forgot about the existence of system wide env variables so that's should do. Thanks !

So basically I put them in /etc/environment.

 

[lucius_the]

Nope, I put this in /etc/environment:
PBS_RESTORE_FETCH_CONCURRENCY=8
PBS_RESTORE_MAX_THREADS=2

I did that both on PBS side and on PVE side, just to be sure (although I think this should only be set on PVE side). But it's still using 16 and 4 respectively, when I try restore via GUI... I must be missing something.

Edit: after some digging: it seems the restore session, where the restore command runs, doesn't take this system env var into account. I guess this can only be set when running restore via command line for now. Unless I'm completely missing something.

 

[lucius_the]

Oh, found a way that works. In case anyone is interested, on PVE host doing the restore:
> systemctl edit pvedaemon

Add the lines:

[Service]
Environment="PBS_RESTORE_FETCH_CONCURRENCY=8"
Environment="PBS_RESTORE_MAX_THREADS=2"

(or whatever numbers you want)

Save & exit, then:
> systemctl daemon-reexec
> systemctl restart pvedaemon

Now, when doing a restore via GUI, it takes these variables into account.

Explanation (as far as i understand it): pvedaemon service is responsible for running the restore commands when doing restores from GUI. So it needs to have these env vars set. This is creating a systemd unit override, that adds the required env vars on top of existing unit definition. Why it's not working when env vars are just put in /etc/environment I'm not sure, perhaps that file is not consulted by systemd - I don't know really. But when env vars are set to this service it works.

 

[VictorSTS]

Glad you found it. Had a half written post suggesting you to check pvedaemon systemd unit, as any process started by a systemd service inherits the environment of the service itself + any other envars set when starting the process. Here is where pvedaemon could read a setting from datacenter.cfg and pass the variables along to proxmox-backup-client to process the recovery.

> systemctl daemon-reexec

Use systemctl daemon-reload instead, you don't need to restart initd, just reload the unit's files.

Why it's not working when env vars are just put in /etc/environment I'm not sure

Because /etc/environment is used only for login shells, which a systemd service, by default, doesn't use.

 

[lucius_the]

Thanks for clarification @VictorSTS - new things learned

So... great progress in PBS, around VM restore speed. What remains:

• verify speed (important in case you have sync jobs, that may even be cascaded below the main PBS to other PBS-es -> if verifies could complete more quickly, sync jobs could use already verified chucks and complete the whole backup dance within a backup window time - without the need to verify chunks later, removing redundant processing from other PBS instances that are pulling chunks from main PBS)
• LXC restore speed (not affected with this - and I use of CT-s a lot)
• someone in this thread also mentioned host-based backup restore speed

We can hopefully now agree that code can actually be improved - and that CPU, slow storage and whatnot - was not the first candidate to blame.
I am wandering if PBS maintainers could now take a look to see if this code can be adopted to these other points of interest.

I would really be interested in making an effort on my side and producing a pull request, problem is I have zero experience with the language this is written in and such effort would likely prove either very unproductive, or produce some substandard code at best.

 

[dcsapak]

fyi, i'll probably look into doing similar changes to other parts of the code.

e.g. i already sent a patch to try to improve verify speed: https://lore.proxmox.com/pbs-devel/20250707132706.2854973-1-d.csapak@proxmox.com/
it's not configurable yet (the biggest hurdle here is that we don't simply want to have a large number of knobs the admin must use, so we'd like to have a better system for that) but i wanted to start sending something that shows we can improve the perf by increasing the number of work done in parallel (e.g. here the loading of chunks from disk)

 

[lucius_the]

This is great news !
I've read through the lore, this looks very promising and shows double throughput in your setup (I assume it could be more on different storage). Looking forward to trying it out when it's ready to be released.

Btw, don't run away from knobs... Most people will never change defaults anyway and those who have a need to will cherish the possibility.