LXC Containers Backing Up Incredibly Slow

@fabian If You got >10 milions files rsync doesn't work well (with expected performance - any file-based-tool won't).

Default storage agnostic procedure is OK - but for ZFS we got much more better options.

If we use ZFS on both sides:
- Proxmox->PBS for backup,
- PBS->Proxmox for backup restore,
- Proxmox->Proxmox for Move Storage.

We could use ZFS native tools like 'zfs send' for backup and storage migration between pools/hosts.
Performance profits are not 5% or 10% but 300-500% in many cases so it's worth implementing.
 
@yarii dude, I gave up on advising zfs specific tools to proxmox chaps. "You can't make somebody change the opinion that they have convinced them self into". ATM only answer given is "give it more hardcore nvme hyper-high iop storage and cpu to boot".


as a side note, I honestly still can't understand what is so complex with this work flow:

1. pve asks pbs to start backup session
2. pve tells pbs that it's container / vm is on storage X / Y / Z
- pbs has same storage X - continues with X optimised backup snapshot
- pbs has same storage Y - continues with Y optimised backup snapshot
- pbs has same storage Z - continues with Z optimised backup snapshot
- pbs and pve storage are not matching - falls back to default "agnostic backup strategy"
3 ....
4. profit.
 
I clearly understand the concept of being agnostic and agree this approach.
But from other side: Proxmox recommends ZFS - so IMO work with ZFS should be done with native ZFS tools.
 
I will join the discussion

Consider that I'm making backup for LXC which was already backed up few minutes ago
99% of the files already live in PBS

Now check screenshot below.

1784990538597.png
On the left it's proxmox making LXC backup (you can see how slow it it's. I'm on 0% iowait, I have plenty of ram and CPU available)
On the right it's ZFS doing scrub
As you can see SSD can issue reads much more faster
Is it somehow possible to tell proxmox lxc backup to do more than one read thread?
That would solve this thread completely

Maybe something like this:
max-workers=<integer> (1 - 256) (default = 16)
Applies to VMs. Allow up to this many IO workers at the same time.

But for LXC?

More details:

Code: 
proxmox-backup-client benchmark
SHA256 speed: 2261.36 MB/s  
Compression speed: 573.18 MB/s  
Decompress speed: 720.88 MB/s  
AES256/GCM speed: 4739.51 MB/s  
Verify speed: 546.48 MB/s  
┌───────────────────────────────────┬─────────────────────┐
│ Name                              │ Value               │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ not tested          │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum computation speed │ 2261.36 MB/s (112%) │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed    │ 573.18 MB/s (76%)   │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed  │ 720.88 MB/s (60%)   │
├───────────────────────────────────┼─────────────────────┤
│ Chunk verification speed          │ 546.48 MB/s (72%)   │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed       │ 4739.51 MB/s (130%) │
└───────────────────────────────────┴─────────────────────┘
P.S. I've tried 10x --pbs-entries-max and that didn't changed anything
 
Last edited:
Did you try to change the CT change detection mode to Metadata?
This should speeds up container backups
 
Last edited:
Yes I'm aware that metadata detection method would be faster. However it could have disadvantages in some cases. Hence not default yet
My suggestion is for default change detection method.

I wonder if there's any difficulties making lxc backup to be done in more than one thread ? Thoughts?
 
Last edited:
You must log in or register to reply here.
Top Bottom
Back