Proxmox is backing up my entire physical disk rather than just used space

[tailsofautonomy]

I am new to Proxmox but so far am very impressed with how intuitive it is and how much of it just works without additional configuration.

I am however running into an issue with my Proxmox Backup Server and my OMV7 VM running on PVE. I am trying to back the VM (64GB) up alongside it's 1TB data drive (LUKS Encrypted) that has been passed through to the VM. The issue is PBS is trying to backup the entire 1TB drive despite the fact only 600GB is in use. As my backup drive is only 1TB in size, this leads to the backup failing. I have tried to following but still running into the same issue:

• Enabled Discard and SSD Emulation
• Enabled fstrim support within OMV7 and confirmed it is working via fstrim -av and made it persidant via cryptsetup --allow-discards --persistent refresh /dev/mapper/sdb-crypt
• Tried backing up just the pass through 1TB data drive without the VM being backed up.
• Tried changing my backup mode from Snapshot to Suspend.

Here is an example log of what I mean:

INFO: starting new backup job: vzdump 100 --remove 0 --node redacted --mode suspend --notes-template '{{guestname}}' --notification-mode auto --storage proxmox-pbs
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2025-04-14 19:12:45
INFO: status = running
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: VM Name: OMV
INFO: include disk 'scsi0' 'local-lvm:vm-100-disk-0' 64G
INFO: include disk 'scsi5' '/dev/disk/by-id/ata-Samsung_SSD_860_QVO_1TB_S4CZNF0N592583L' 976762584K
INFO: suspending guest
INFO: skip unused drive 'local-lvm:vm-100-disk-1' (not included into backup)
INFO: pending configuration changes found (not included into backup)
INFO: creating Proxmox Backup Server archive 'vm/100/2025-04-14T18:12:45Z'
INFO: enabling encryption
INFO: started backup task '443d0a84-8972-4c4f-9088-af6d612d916e'
INFO: resuming VM again after 1 seconds
INFO: scsi0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: scsi5: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: 0% (324.0 MiB of 995.5 GiB) in 3s, read: 108.0 MiB/s, write: 108.0 MiB/s

It then proceeds to fill up around 900GB's or so and error out. Is there a way with my configuration to get PBS to only backup the used space rather than the empty space of the drive?

Edit: I should add I am using EXT4 rather than ZFS as my filesystem.

 

[jlauro]

I didn't think you could backup pass through disks and could only backup virtual disks? Wonder if that's new?

Is vm100 your PBS server? If so, it can't backup itself.

 

[UdoB]

data drive (LUKS Encrypted)

Are you sure that LUKS does "fstrim" the underlying physical device for empty blocks?

In my (possibly wrong!) understanding the whole disk (or partition, whatever you gave to LUKS) should look like randomized(!) data. This means a) it is not clear which sector is used at all and b) it is not compressible. I do expect a backup of a LUKS device to be exactly nearly as large as the source, independent of the backup mechanism.

Just my generic two €¢...

 

[tailsofautonomy]

I didn't think you could backup pass through disks and could only backup virtual disks? Wonder if that's new?

Is vm100 your PBS server? If so, it can't backup itself.

VM100 is my OMV VM, my Proxmox Backup Server is running on bare metal and exists on a separate machine.

I used the command qm set 100 -scsi5 /dev/disk/by-id/ata-Samsung_SSD_860_QVO_1TB_S4CZNF0N592583L to get my SSD passed through to the VM and then just ticked the 'backup' box under the Hardware section and as can be seen from my log, it appears to be backing up the drive.

 

[tailsofautonomy]

Are you sure that LUKS does "fstrim" the underlying physical device for empty blocks?

In my (possibly wrong!) understanding the whole disk (or partition, whatever you gave to LUKS) should look like randomized(!) data. This means a) it is not clear which sector is used at all and b) it is not compressible. I do expect a backup of a LUKS device to be exactly nearly as large as the source, independent of the backup mechanism.

Just my generic two €¢...

That was also my initial suspicion as well but googling the issue it appeared to suggest that while LUKS disables fstrim support by default as it weakens drive encryption, it can be enabled and run if set with the correct flags (--allow discard). I tested this by running fstrim and it removed several hundred GB's from my SSD within my VM. So it appears to be working within my OMV. It just seems that Proxmox does not see this and treats the drive as if it is still full.

I also decided to confirm this issue by passing through an unencrypted hard drive I had in use on another system, running fstrim on it within OMV and running a PBS job again and it still attempted to backup the entire disk. Here is that output of that job:

INFO: starting new backup job: vzdump 100 --storage proxmox-pbs --notification-mode auto --node max --mode suspend --notes-template '{{guestname}}' --remove 0
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2025-04-15 12:33:07
INFO: status = running
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: VM Name: OMV
INFO: include disk 'scsi0' 'local-lvm:vm-100-disk-0' 64G
INFO: exclude disk 'scsi1' 'local-lvm:vm-100-disk-1' (backup=no)
INFO: exclude disk 'scsi5' '/dev/disk/by-id/ata-Samsung_SSD_860_QVO_1TB_S4CZNF0N592583L' (backup=no)
INFO: include disk 'scsi6' '/dev/disk/by-id/ata-TOSHIBA_MQ04ABF100_99I4P3Q2T' 976762584K
INFO: suspending guest
INFO: creating Proxmox Backup Server archive 'vm/100/2025-04-15T11:33:07Z'
INFO: enabling encryption
INFO: started backup task 'ccaedbb0-2e48-4993-bbf2-50635af057d2'
INFO: resuming VM again after 1 seconds
INFO: scsi0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: scsi6: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: 0% (404.0 MiB of 995.5 GiB) in 3s, read: 134.7 MiB/s, write: 125.3 MiB/s
INFO: 0% (592.0 MiB of 995.5 GiB) in 5s, read: 94.0 MiB/s, write: 76.0 MiB/s

My vm-100-disk-0, which is my OMV VM I have my drives attached to is thin provisioned, I wonder if this is causing a conflict or changing the way PBS treats my physical drive's (EXT4 formatted) even if they are not virtual drives as in the case of my VM.

 

[Taomyn]

As I understand the backups on PBS are basically disk images, and the log will always show backing up the entire disk e.g. 404.0 MiB of 995 GiB, as all my backups are the same:

INFO: 0% (680.0 MiB of 250.0 GiB) in 3s, read: 226.7 MiB/s, write: 192.0 MiB/s
INFO: 1% (2.6 GiB of 250.0 GiB) in 24s, read: 94.1 MiB/s, write: 83.2 MiB/s
INFO: 2% (5.0 GiB of 250.0 GiB) in 55s, read: 80.8 MiB/s, write: 75.0 MiB/s
INFO: 3% (7.6 GiB of 250.0 GiB) in 1m 25s, read: 88.7 MiB/s, write: 78.7 MiB/s
INFO: 4% (10.2 GiB of 250.0 GiB) in 1m 51s, read: 99.1 MiB/s, write: 86.2 MiB/s
INFO: 5% (12.6 GiB of 250.0 GiB) in 2m 22s, read: 80.1 MiB/s, write: 64.0 MiB/s
INFO: 6% (15.1 GiB of 250.0 GiB) in 2m 57s, read: 73.1 MiB/s, write: 56.0 MiB/s
INFO: 7% (17.6 GiB of 250.0 GiB) in 3m 20s, read: 113.4 MiB/s, write: 74.4 MiB/s
INFO: 8% (20.1 GiB of 250.0 GiB) in 3m 40s, read: 125.0 MiB/s, write: 79.8 MiB/s
INFO: 9% (23.0 GiB of 250.0 GiB) in 4m 3s, read: 131.8 MiB/s, write: 59.0 MiB/s
INFO: 10% (25.3 GiB of 250.0 GiB) in 4m 10s, read: 326.9 MiB/s, write: 110.3 MiB/s
INFO: 11% (27.6 GiB of 250.0 GiB) in 4m 24s, read: 167.7 MiB/s, write: 66.6 MiB/s
INFO: 12% (30.5 GiB of 250.0 GiB) in 4m 39s, read: 203.2 MiB/s, write: 94.4 MiB/s
INFO: 13% (32.5 GiB of 250.0 GiB) in 4m 56s, read: 118.8 MiB/s, write: 63.1 MiB/s
INFO: 14% (35.1 GiB of 250.0 GiB) in 5m 11s, read: 180.3 MiB/s, write: 80.3 MiB/s
INFO: 15% (38.2 GiB of 250.0 GiB) in 5m 24s, read: 239.7 MiB/s, write: 68.3 MiB/s
INFO: 16% (40.4 GiB of 250.0 GiB) in 5m 39s, read: 150.4 MiB/s, write: 58.1 MiB/s
INFO: 17% (43.3 GiB of 250.0 GiB) in 5m 49s, read: 298.0 MiB/s, write: 51.2 MiB/s
INFO: 18% (45.5 GiB of 250.0 GiB) in 5m 58s, read: 253.8 MiB/s, write: 66.2 MiB/s
INFO: 19% (47.7 GiB of 250.0 GiB) in 6m 6s, read: 283.0 MiB/s, write: 74.0 MiB/s
INFO: 20% (50.7 GiB of 250.0 GiB) in 6m 15s, read: 340.0 MiB/s, write: 50.7 MiB/s
INFO: 21% (52.7 GiB of 250.0 GiB) in 6m 21s, read: 336.0 MiB/s, write: 84.0 MiB/s
INFO: 22% (55.5 GiB of 250.0 GiB) in 6m 31s, read: 285.2 MiB/s, write: 55.6 MiB/s
INFO: 23% (57.9 GiB of 250.0 GiB) in 6m 47s, read: 151.5 MiB/s, write: 45.5 MiB/s
INFO: 24% (60.2 GiB of 250.0 GiB) in 6m 58s, read: 217.5 MiB/s, write: 68.0 MiB/s
INFO: 25% (63.0 GiB of 250.0 GiB) in 7m 9s, read: 258.9 MiB/s, write: 15.3 MiB/s
INFO: 26% (65.0 GiB of 250.0 GiB) in 7m 12s, read: 701.3 MiB/s, write: 0 B/s
INFO: 27% (68.0 GiB of 250.0 GiB) in 7m 15s, read: 1022.7 MiB/s, write: 172.0 MiB/s
INFO: 28% (70.0 GiB of 250.0 GiB) in 7m 27s, read: 169.3 MiB/s, write: 53.3 MiB/s
INFO: 29% (73.1 GiB of 250.0 GiB) in 7m 33s, read: 526.7 MiB/s, write: 67.3 MiB/s
INFO: 30% (75.1 GiB of 250.0 GiB) in 7m 41s, read: 262.0 MiB/s, write: 35.0 MiB/s
INFO: 31% (77.6 GiB of 250.0 GiB) in 7m 56s, read: 166.9 MiB/s, write: 44.8 MiB/s
INFO: 32% (80.8 GiB of 250.0 GiB) in 7m 59s, read: 1.1 GiB/s, write: 80.0 MiB/s
INFO: 33% (82.6 GiB of 250.0 GiB) in 8m 8s, read: 208.9 MiB/s, write: 31.6 MiB/s
INFO: 34% (85.0 GiB of 250.0 GiB) in 8m 16s, read: 309.0 MiB/s, write: 83.0 MiB/s
INFO: 35% (87.7 GiB of 250.0 GiB) in 8m 40s, read: 114.7 MiB/s, write: 68.8 MiB/s
INFO: 36% (90.2 GiB of 250.0 GiB) in 8m 51s, read: 236.0 MiB/s, write: 18.5 MiB/s
INFO: 37% (92.6 GiB of 250.0 GiB) in 9m 5s, read: 171.4 MiB/s, write: 50.6 MiB/s
INFO: 38% (95.6 GiB of 250.0 GiB) in 9m 39s, read: 90.8 MiB/s, write: 49.5 MiB/s
INFO: 39% (97.5 GiB of 250.0 GiB) in 10m, read: 93.7 MiB/s, write: 38.7 MiB/s
INFO: 40% (100.1 GiB of 250.0 GiB) in 10m 11s, read: 237.5 MiB/s, write: 83.3 MiB/s
INFO: 41% (102.6 GiB of 250.0 GiB) in 10m 23s, read: 213.7 MiB/s, write: 121.7 MiB/s
INFO: 44% (111.7 GiB of 250.0 GiB) in 10m 26s, read: 3.1 GiB/s, write: 0 B/s
INFO: 49% (122.7 GiB of 250.0 GiB) in 10m 29s, read: 3.7 GiB/s, write: 0 B/s
INFO: 53% (133.4 GiB of 250.0 GiB) in 10m 32s, read: 3.6 GiB/s, write: 0 B/s
INFO: 57% (144.6 GiB of 250.0 GiB) in 10m 35s, read: 3.7 GiB/s, write: 0 B/s
INFO: 61% (154.5 GiB of 250.0 GiB) in 10m 38s, read: 3.3 GiB/s, write: 0 B/s
INFO: 66% (165.7 GiB of 250.0 GiB) in 10m 41s, read: 3.7 GiB/s, write: 0 B/s
INFO: 70% (176.5 GiB of 250.0 GiB) in 10m 44s, read: 3.6 GiB/s, write: 0 B/s
INFO: 74% (187.3 GiB of 250.0 GiB) in 10m 47s, read: 3.6 GiB/s, write: 0 B/s
INFO: 79% (198.6 GiB of 250.0 GiB) in 10m 50s, read: 3.8 GiB/s, write: 0 B/s
INFO: 83% (209.4 GiB of 250.0 GiB) in 10m 53s, read: 3.6 GiB/s, write: 0 B/s
INFO: 88% (220.6 GiB of 250.0 GiB) in 10m 56s, read: 3.7 GiB/s, write: 0 B/s
INFO: 92% (231.7 GiB of 250.0 GiB) in 10m 59s, read: 3.7 GiB/s, write: 0 B/s
INFO: 97% (242.6 GiB of 250.0 GiB) in 11m 2s, read: 3.6 GiB/s, write: 0 B/s
INFO: 100% (250.0 GiB of 250.0 GiB) in 11m 5s, read: 2.5 GiB/s, write: 5.5 MiB/s

But I then look at the end of each line and see that the write value is what is actually being backed up. I'm pretty sure if it was always writing the entire disk I'd have issues with my PBS storage being unable to hold all the backups. It's difficult to tell if you see the same as you have truncated your example logs.

When I've restored disks I see a similar log and I some lines take time as the data is read then written, but many where they go much quicker presumably because there is no actual data to restore for that part of the disk.

Of course, I could be completely wrong, but your thread peaked my interest.

 

[SteveITS]

@tailsofautonomy
Just to clarify how (I understand) PBS works it will see and show the entire disk but the empty chunks are deduplicated down to one. Every backup is a full backup.

Despite having “several TB” of disk in our cluster, our PBS is using under 350 GB so far.

So if we ignore the disk size shown what is the actual problem? PBS is out of space? Can you show that failure?

 

[UdoB]

It just seems that Proxmox does not see this and treats the drive as if it is still full.

Okay.

Probably my post regarding "fstrim" was completely stupid - that's one level lower than backup handles data. Sorry for the confusion.

@Taomyn 's #6 makes more sense