Spinners on slow RAID configs will never get very fast, their benchmark are just seldom a IO pattern that can be achievable by real world access. And increasing parallelization on spinner backed disks, which have a big penalty for random IO, most often makes things worse, we tested quite a bit on this. And different VM restore tasks run in parallel, so for your storage, where random IO gets penalized, it could be even better to reduce parallelism. Or what did you mean exactly here?
Note also that the target storage and the network between PBS and PVE has also some say, as that can slow down the whole restore, especially if multiple VMs are restored the IO queues will never get filled as deep, especially not with ordered requests, compared to your FIO case; because your 128 deep queue can be sorted by the IO scheduler of the controller or HDD itself and is not so random anymore, especially if laid out on a file that's just 64 GB big, so not the best comparison with achievable performance of a multi TB restore.
FWIW, you could try switching the "chunk-order" tuning option to "none", while sorting by inode should make it a bit more performant on spinning disks, as the data then should be more local meaning less movements required from the magnetic needle, some systems, i.e. especially proprietary RAID controllers, which are a black box that's not really predictable, and storage where metadata access is much slower than reading the data, the ordering might make it worse.
https://pbs3.work.tlmp.it:8007/docs/storage.html#datastore-tuning-options
Note that modern enterprise SSDs are not only much more reliable and long-living compared to spinners, meaning less redundancy is required to get the same statistical safety, they are also available in 30 TB form for 3200 to 4000 € (excl. VAT) here, i.e., bigger than current HDDs, and the trend is still pointing upward. Sure, that clocks in about 8 times the raw €/TB price as spinning HDDs, but they are also hardly comparable in terms of performance and, as mentioned, reliability. Thus, they can save lots of operating costs, both in less maintenance and wait times required by admins, and more importantly,
much less restore time – which, depending on the business, can often recoup the higher initial investment cost by a high margin. My point is, there's more than just the bucks-per-TB cost to look at when setting up a backup server. And FWIW, there are always combinations possible, like a smaller SSD backed datastore for incoming and latest backup and a bigger one HDD or even tape backed one to provide long term archival.
I mention that because we spent a significant amount of time with looking into improving throughput on spinners, and while it's certainly not impossible that there are some changes possible that could cause a dramatic improvement, it's rather likely that, if they exist, they are quite elaborate, which normally means high maintenance cost and regression potential for other things.
That said, we occasionally look into this, especially if we get a case in the enterprise support, and if we find anything we naturally are going to improve on it; but it's unlikely that the storage system you describe will get highly performant anytime soon.
I wonder how the CPU load is on this host when doing restores, it would not surprise me if we're hitting some sort of performance ceiling simply from a single restore thread being maxed out.
That would be rather surprising, the CPU doesn't need to do that much work here, especially not so much to be the bigger bottleneck as IO served from spinning storage.
But that would be easy to check through looking at top to see if one thread is maxed out and also the Pressure Stall Information, e.g.:
head /proc/pressure/*
The latter shows the time some or all (full) processes had to wait due to being starved by IO, CPU or Memory, ideally all numbers would be zero. If IO is non-zero during a restore it's very unlikely that increasing parallelism would help, rather the access pattern would require changing, and that's far from easy with the PBS chunk-store content addressable storage design (which brings many benefits, so IMO still a good trade-off).
