Abysmally slow restore from backup

[VictorSTS]

Using environment variables probably means running the restore command in shell, right ?

That would be a let down. This should be easily adjusted i.e. at datacenter.cfg so a normal restore from the web UI applies them (i.e. non root user(s) with enough privileges to restore their VMs). I would be ok if you had to use cli to set those values, although IMO they should be available in the webUI too.
I would love to see this backported to PVE8.4.x

 

[PwrBank]

Using environment variables probably means running the restore command in shell, right ?

Not if you set the variable system wide, then when the actual command is ran by the system it'll just use those values.

Basically saying, $threadcount=14

and then itstead of having a hard coded number, it references the $threadcount in the system.

 

[lucius_the]

Not if you set the variable system wide, then when the actual command is ran by the system it'll just use those values.

Basically saying, $threadcount=14

and then itstead of having a hard coded number, it references the $threadcount in the system.

Right. That's because of my (somewhat limited) Linux knowledge... I thought that env vars are per-user, but forgot about the existence of system wide env variables so that's should do. Thanks !

So basically I put them in /etc/environment.

 

[lucius_the]

Nope, I put this in /etc/environment:
PBS_RESTORE_FETCH_CONCURRENCY=8
PBS_RESTORE_MAX_THREADS=2

I did that both on PBS side and on PVE side, just to be sure (although I think this should only be set on PVE side). But it's still using 16 and 4 respectively, when I try restore via GUI... I must be missing something.

Edit: after some digging: it seems the restore session, where the restore command runs, doesn't take this system env var into account. I guess this can only be set when running restore via command line for now. Unless I'm completely missing something.

 

[lucius_the]

Oh, found a way that works. In case anyone is interested, on PVE host doing the restore:
> systemctl edit pvedaemon

Add the lines:

[Service]
Environment="PBS_RESTORE_FETCH_CONCURRENCY=8"
Environment="PBS_RESTORE_MAX_THREADS=2"

(or whatever numbers you want)

Save & exit, then:
> systemctl daemon-reexec
> systemctl restart pvedaemon

Now, when doing a restore via GUI, it takes these variables into account.

Explanation (as far as i understand it): pvedaemon service is responsible for running the restore commands when doing restores from GUI. So it needs to have these env vars set. This is creating a systemd unit override, that adds the required env vars on top of existing unit definition. Why it's not working when env vars are just put in /etc/environment I'm not sure, perhaps that file is not consulted by systemd - I don't know really. But when env vars are set to this service it works.

 

[VictorSTS]

Glad you found it. Had a half written post suggesting you to check pvedaemon systemd unit, as any process started by a systemd service inherits the environment of the service itself + any other envars set when starting the process. Here is where pvedaemon could read a setting from datacenter.cfg and pass the variables along to proxmox-backup-client to process the recovery.

> systemctl daemon-reexec

Use systemctl daemon-reload instead, you don't need to restart initd, just reload the unit's files.

Why it's not working when env vars are just put in /etc/environment I'm not sure

Because /etc/environment is used only for login shells, which a systemd service, by default, doesn't use.

 

[lucius_the]

Thanks for clarification @VictorSTS - new things learned

So... great progress in PBS, around VM restore speed. What remains:

• verify speed (important in case you have sync jobs, that may even be cascaded below the main PBS to other PBS-es -> if verifies could complete more quickly, sync jobs could use already verified chucks and complete the whole backup dance within a backup window time - without the need to verify chunks later, removing redundant processing from other PBS instances that are pulling chunks from main PBS)
• LXC restore speed (not affected with this - and I use of CT-s a lot)
• someone in this thread also mentioned host-based backup restore speed

We can hopefully now agree that code can actually be improved - and that CPU, slow storage and whatnot - was not the first candidate to blame.
I am wandering if PBS maintainers could now take a look to see if this code can be adopted to these other points of interest.

I would really be interested in making an effort on my side and producing a pull request, problem is I have zero experience with the language this is written in and such effort would likely prove either very unproductive, or produce some substandard code at best.

 

[dcsapak]

fyi, i'll probably look into doing similar changes to other parts of the code.

e.g. i already sent a patch to try to improve verify speed: https://lore.proxmox.com/pbs-devel/20250707132706.2854973-1-d.csapak@proxmox.com/
it's not configurable yet (the biggest hurdle here is that we don't simply want to have a large number of knobs the admin must use, so we'd like to have a better system for that) but i wanted to start sending something that shows we can improve the perf by increasing the number of work done in parallel (e.g. here the loading of chunks from disk)