High I/O delay, fluctuating transfer rates

[AlpsView]

I'm testing PBS atm, to find out if it will be the solution for my backup needs.

• PBS is running in a VM on PVE.
• The VMs are stored in local NVMes.
• An external USB 3.0 harddisk is passed through to PBS. Filessystem is ext4.
• Backups are running using snapshots.
• Backups are encrypted.

During the backup process on PBS and on PVE i can see a high to very high iowait. Most oft the time 40-50% up to almost 100% at peak.




On PVE in the running Backup Job logs, I can see the read/write rates going up and down in waves with write rates from > 300 MiB/s to <30 MiB/s.




I'm trying to understand where the problem comes from. My conclusions so far:

• Encryption needs CPU power. The CPU never comes near 100%, so it doesn't seem to be the bottleneck.
• However the iowait indicates the CPU is waiting.
• So the question is, is it waiting to read more data or is it waiting to write data?
• Since read rates constantly are bigger than write rates, I conclude the CPU is reading at highest possible speed, is processing what ever it gets and then tries to write it to the target drive. And this write transfer does not seem to be fast enough. This is why we see the iowait.
• USB 3.0 (Super Speed) should be able to do 5 GBps = 400 MB/s net. At the very begining of each transfer write rates start somewhere near 400 MB/s. I see this as a "proof" the USB drive is operating on Superspeed indeed. However, rates drop fast to a fraction of the inital rate and move in waves after this, never coming near the 400 MB/s again.
• These waves to me looks like a faster cache in front of the acutally harddisk being filled faster than the disk itself is able to write. So each time the cache is full, the cpu goes into iowait while the disk tries to write the cache. I wondering if this is a cache on the disk or the USB controller or ....

Is my conclusion correct?
Anything i can improve in this case? Are there any potential BIOS settings to improve the transfer/cache handling? Anything else other than getting a faster harddisk (I'm aware these external USB drives are rather low end performance-wise - so maybe this is just what it is)? Other Filesystem?

For comparisoni had been doing backups to a second internal NVMe disk where i had much faster rates and didn't see these waves. I don't remember if I have seen iowait as i did not had an focus on this then, but i guess there wasn't as I might have noticed it.

 

[AlpsView]

This is how the backup process looked on PVE (showing only the last part of the backup run but it looked the same the whole time):

 

[UdoB]

An external USB 3.0 harddisk is passed through to PBS.

That's a brave attempt. It is not clear from your post but I assume rotating rust.

PBS needs IOPS, IOPS and also IOPS. Earlier or later you'll encounter timeouts and need to fight to get the data read back successfully. My very first PBS was so slow that listing the backups was impossible...

Background in a nutshell: PBS stores each and every backup in chunks of 2-4 MiB in size. (4 MiB minus compression.) For each of them the head has to move physically, possibly multiple times. Plus movements for metadata. That's just... slooooow.

If you really need to use a classic HDD go for "classic" vzdump backups instead of PBS. That would create one single large file, which will work without this drama.


For Reference, the requirements: https://pbs.proxmox.com/docs/installation.html#recommended-server-system-requirements

 

[AlpsView]

Thanks a lot for your your input and the insights UdoB.

Yes, it's plausible the bottleneck is the harddisk. I know that sata harddrives are at the low end of performance. I just didn't expect to see these waves, specially these "valleys". Might just be because I have never seen this before and I wasn't aware of how much the write pattern influences the amount and time for the head movements.

From the amount of data transfered divided by the total time the backup took, it averages at around 43 MB/s (initial backup run, so a full run without dedup). Given the fact that this is my homelab, I could live with this if it's not going to be much worse over time, specially since the amount of data being changed between two runs is rather small. This is why I wanted to try PBS instead of vzdump, as with the latter only full backups are possible while PBS works with deduplication, consuming only a small fraction of additional storage in subsequent runs and as lesser data has to be written, also completing faster. So even with a rusty harddrive, I thought PBS might be the better solution in terms of duration and storage needed.

I just completed a second run (like 5h after the first one with very little changed - some logs, some wiki entries, some emails, ...). This one only took around 8 minutes and iowait was minimal. However, I'm a bit alerted you write that in your experience in such a setup even listing the backup meta data failed after a while. Is this because you were holding backups for ever, not pruning old ones?

 

[UdoB]

However, I'm a bit alerted you write that in your experience in such a setup even listing the backup meta data failed after a while. Is this because you were holding backups for ever, not pruning old ones?

The age is irrelevant. The culprit lies in the pure number of chunks.

It was a separate setup with a Raid6 (no ZFS), constructed only with HDDs. That's of course the worst case as IOPS are basically identical to one single drive. Nevertheless I wanted to give it a try - while already expecting problems from the beginning. But I wanted to learn the actual behavior on my hardware.

It worked. With the amount of backups stored the number of chunks increased. After several weeks(!) and a few (still single-digit, iirc) Terabytes stored I had the effect that the WebGUI could not list the backups. The first click just got a timeout, showing an empty table as the result. When I clicked a second time the most of the backup's listing was already in cache and now this second click worked. This started slowly to happen, depending on cache+buffers. After another two weeks this effect was "stable" reproducible. It was clear that this was not acceptable

I do not remember if an actual restore worked at that time. Probably it was not necessary, so I did not test it in that situation...

So my solution (better: workaround) was to introduce fast meta data storage. This approach is possible for ZFS (with a fast "Special Device" added before filling in much data) and also (for example) for btrfs on a Synology. But not for your single HDD.

Meanwhile I've setup other instances of PBS, all in the lower two-digit TB range. The most of them use ZFS with mirrored vdevs plus that mentioned "Special Device". I never had another problem after the above story.

Restore is still slow with this construct - data is still stored on HDD - but in my setups it works reliably. That said... I think I should run some more restore-tests to verify that statement...

 

[tcabernoch]

That's a brave attempt.

LOL. I have one of these!

It started as a joke. I grabbed a couple large USB external drives from one of our colos. Took them back to my office, where we did not have a NAS.
(I think I was going to do passthrough w/esxi and a vm to make a nas. Then broadcom screwed us all.)

I stood up Rocky Linux on a mini Dell desktop. Plugged in the drives. Sorted out all the mounting BS so they always work regardless of port. It was fun.
Built out NFS and SMB shares. Got all the perms and stuff sorted. More fun.
Installed KVM. Learned to work with it directly without PVE. Setup huge datastore on one of the USB mounts. Lotta fun.
Installed PBS as a guest. (Several times. I was new.) Gave it a big disk on the big USB datastore.

So ya, it runs like cr@p. What would you expect?
It was a test ground for several technologies and still serves a purpose in my office today.
And still runs like cr@p. Keeps workin.

....
BTW ... OP seems both really on point but also like they don't want to believe themselves.
This is true. "faster cache in front of the acutally harddisk being filled faster than the disk itself is able to write"
The rest of this post is just discussion. It was all resolved at that sentence.

 

[UdoB]

LOL. I have one of these!

Sure. A lot of "special" things do work in a homelab. Until they don't ;-)

The most important aspect from my story is that one should know that some use cases are not "officially supported" although they seem to work at a first glance.

 

[tcabernoch]

I should go back and join my two USB drives into a ZFS stripe. For maximum fragility. And probably zero speed gain.
... Except I don't have much to learn about basic ZFS anymore. Don't need to do that.

I like building ridiculous things. Some days I can't believe they pay me to do this. Test it in Prod!

 

[AlpsView]

I'm acutally trying to differentiate between "is slow", "is not reliable" and "does not work". From a home lab (sorry, I forgot to point this out from the very begining) point of view, "is slow" might be acceptable. "Is not reliable" when it comes to data consistency is a red line also in the home lab. And "is not working" we don't need to discuss. In the end, for private use the price tag surely plays a role (I'm not working in a DC where enterprise grade industrial SSDs are available on a self-service buffet à discrétion ;-)) and would justify some trade off regarding transfer rates. That's why I was asking @UdbB about his scenario, experience and the "root cause" of the problem he observed. Since if i.e. his backup volume is 20 times mine and the number of chunks is the driver, I might conclude that when his problems started after let's say 2 months, mine would start after 40 months? Which might be an acceptable mid term solution. And if there were some tweaks which could improve the situation, also a couple of little improvements sum up.

So, my take-aways from all of the above for my situation:
- Withou t a high I/O storage system use full backup on PVE, don't use PBS
- Better use SSDs instead of HDDs. If SSD not possible, go for faster HDD
- USB3.x as interface should be fine as with 5 Gbps max it's in the same league as SATAIII SSDs.

SSD makes me fear the recurring costs. From what I read about the wear, enterprise disks are needed. And when using full backups, quite a bunch of enterprise disks are needed.

Rain and backup problems - not my happiest sunday

 

[UdoB]

From what I read about the wear, enterprise disks are needed.

For PBS? I doubt that. PBS as a backups system is writing new data only and reading it once in a blue moon (or for scheduled verification). Note that HDD + Metadata on SSD is a special case though. "Enterprise Class" is...

• highly recommended for running PVE with multiple active VMs hammering their virtual drives
• always recommended for all use cases which require high IOPS and/or write performance
• usually recommended when using ZFS; it is not a technical requirement; ZFS just writes more volume and more often --> higher wearout and slow on cheap SSDs w/o PLP

Said differently: "Enterprise Class" is nearly always "better" - but it is usually not required.

Disclaimer: just from my own experience with no hard evidence --> my opinion; ymmv.

(( In my Homelab I use "Second Hand Enterprise Class, SATA" - because of the price. Though several NVMe do not have PLP here and are also not redundant = MiniPC, only single slot. This is not recommended and I do expect trouble earlier or later - but I am prepared for that. ))