strange issues with snapshot backups

K-P4ul

Member
Jan 28, 2021
14
1
23
Hi,
I have a strange issue with the Proxmox Backup function. I have 4 Clusters (2+1) running with DRBD as storage backend.
On a other location (in a data center) runs a Proxmox Backup server. Both locations are connected via 1GBit/s.
Now I want to backup all of my VMs to the PBS over the 1GBit internet link. I know the initial run will last several hours, maybe days. The backup mode on the PVEs is set to snapshot and i have one job per Cluster (seems to run 2 jobs in parallel).
When the Backup runs, after a few hours i got I/O errors on the VMs where the backup is running. It seems the blockdevice stalls for a longer time (over 600 seconds). Then the vm kernel remounts the filesystems ro.

This leads me to my first question:

Is a longer runtime a Problem when using Snapshot mode?

For better understanding whats going on, I want to see where the snapshots are created when backup is runnning.
When i perform a snapshot in proxmox ui i can see that drbd is creating a snapshot and after succsefull creation i can see the snapshot in the linstor controller. But when proxmox backup is running i can not see anything about snapshots in drbd or linstor.


This brings me to my second question:

Does proxmox backup in snapshot mode really create snapshots? Or does it skip the task and just copy the mounted drbd volume? Maybe there is some hidden proxmox magic going on I don't know about?
 
A "snapshot backup" creates a dirty bitmap at QEMU level (not storage) of the blocks that are being changed during the backup.
If the block is changed during the backup but prior to this block actually placed in backup media two things can happen:
a) the change (write) is held until the block is successfully pushed to PBS
b) if fleecing-storage is enabled the block is copied to fleecing storage and the write is acknowledged.

If your backup flow gets bogged down for any reason the (a) can caused IO timeouts and errors. This is why (b) was created.

There is no integration/interaction with DRBD or any other storage type.



Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: leesteken