[SOLVED] How are raw volumes on btrfs backed up?

[A.R.]

I see very slow backups of one of my vm's. I has 3 volumes totaling 750GB. All raw in BTRFS. Two volumes are on nvme's in btrfs raid-1 config and 1 on btrfs raid 0
The backup starts off with quite some read performance but then slows to a few MB/s sometimes 20 MB/s. The vm is often unresponsive or slows down so much that nothing happens.

Proxmox backup server is installed in an lxc container. It has direct access to the backup storage via bind mounts.

I have been looking if this is something in my config etc, switched from proxmox backup server in a vm to within an lxc.

Example log:

INFO:  17% (132.9 GiB of 781.0 GiB) in 4h 28m 58s, read: 4.8 MiB/s, write: 2.2 MiB/s
INFO:  18% (140.6 GiB of 781.0 GiB) in 5h 38m 52s, read: 1.9 MiB/s, write: 18.6 KiB/s
INFO:  19% (148.5 GiB of 781.0 GiB) in 6h 23m 52s, read: 3.0 MiB/s, write: 13.7 KiB/s
INFO:  20% (156.3 GiB of 781.0 GiB) in 6h 38m 31s, read: 9.1 MiB/s, write: 1.4 MiB/s
INFO:  21% (164.1 GiB of 781.0 GiB) in 7h 29m 58s, read: 2.6 MiB/s, write: 6.6 KiB/s
INFO:  22% (171.9 GiB of 781.0 GiB) in 8h 13m 35s, read: 3.1 MiB/s, write: 23.5 KiB/s
INFO:  23% (179.6 GiB of 781.0 GiB) in 8h 40m 41s, read: 4.9 MiB/s, write: 55.4 KiB/s
INFO:  24% (187.5 GiB of 781.0 GiB) in 8h 59m 5s, read: 7.3 MiB/s, write: 530.5 KiB/s

Image backups are mucht faster.

On a hunch I listed all btrfs subvolumes on the storage volumes. I only see the regular subvolumes for the raw images but no shapshots for the backup to read from.

So I am wondering, is the backup being made from the active raw image and not from a readonly snapshot?

(EDIT: updated log snippet)

 

[A.R.]

Found my answer in the man page of vzdump. A reference to this document: https://git.proxmox.com/?p=pve-qemu.git;a=blob_plain;f=backup.txt

tl;dr
Raw file is read while writes are intercepted.
When a write is intercepted, the original data is written to backup and then the write is committed.
So the backup is not sequential.
Downside: slows down the vm and is as slow as the backup backend.

So the proxmox backup server is much slower than an NFS storage?

 

[cheiss]

Two volumes are on nvme's in btrfs raid-1 config and 1 on btrfs raid 0

What brand and model of NVMe SSDs have you got? Are these consumer ones?

Proxmox backup server is installed in an lxc container. It has direct access to the backup storage via bind mounts.

Just FYI, this is a discouraged configuration as it isn't really a good idea to let the backup server run on the same server.
What the (hardware) configuration of the backup storage? What's the filesystem used there? Etc ..

So the proxmox backup server is much slower than an NFS storage?

There is something in the works called "fleecing backups", which should alleviate most of these problems (see e.g. this post for a bit more information).

Also, what the workload in the affected VMs? Is it very I/O-intensive?