huge dirty-bitmap after sunday

chengkinhung

Active Member
Feb 11, 2016
39
6
33
23
Hi, every one,

I was using PBS to backup VM created on ZFS, it was running well, however sometimes I found one VM got very huge dirty-bitmap, even bigger than the real data usage insider that VM. This VMs was installed CloudLinux and cPanel, it was using XFS filesystem, I also found it almost always happened after sunday.

Here is the disk usage insider this VM:
Code:
# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        8.8G     0  8.8G   0% /dev
tmpfs           8.8G  4.0K  8.8G   1% /dev/shm
tmpfs           8.8G  858M  8.0G  10% /run
tmpfs           8.8G     0  8.8G   0% /sys/fs/cgroup
/dev/sda1       1.5T  380G  1.2T  26% /
/dev/loop0      3.9G  146M  3.5G   4% /tmp
tmpfs           1.7G     0  1.7G   0% /run/user/0

Here is the ZFS disk information:
Code:
# zfs get all pool1/vm-103-disk-0
NAME                 PROPERTY              VALUE                  SOURCE
pool1/vm-103-disk-0  type                  volume                 -
pool1/vm-103-disk-0  creation              Fri Apr  8 13:15 2022  -
pool1/vm-103-disk-0  used                  389G                   -
pool1/vm-103-disk-0  available             990G                   -
pool1/vm-103-disk-0  referenced            389G                   -
pool1/vm-103-disk-0  compressratio         1.00x                  -
pool1/vm-103-disk-0  reservation           none                   default
pool1/vm-103-disk-0  volsize               1.47T                  local
pool1/vm-103-disk-0  volblocksize          8K                     default
pool1/vm-103-disk-0  checksum              on                     default
pool1/vm-103-disk-0  compression           off                    default
pool1/vm-103-disk-0  readonly              off                    default
pool1/vm-103-disk-0  createtxg             669645                 -
pool1/vm-103-disk-0  copies                1                      default
pool1/vm-103-disk-0  refreservation        none                   default
pool1/vm-103-disk-0  guid                  10756677095636455937   -
pool1/vm-103-disk-0  primarycache          all                    default
pool1/vm-103-disk-0  secondarycache        all                    default
pool1/vm-103-disk-0  usedbysnapshots       275M                   -
pool1/vm-103-disk-0  usedbydataset         389G                   -
pool1/vm-103-disk-0  usedbychildren        0B                     -
pool1/vm-103-disk-0  usedbyrefreservation  0B                     -
pool1/vm-103-disk-0  logbias               latency                default
pool1/vm-103-disk-0  objsetid              16288                  -
pool1/vm-103-disk-0  dedup                 off                    default
pool1/vm-103-disk-0  mlslabel              none                   default
pool1/vm-103-disk-0  sync                  standard               default
pool1/vm-103-disk-0  refcompressratio      1.00x                  -
pool1/vm-103-disk-0  written               298M                   -
pool1/vm-103-disk-0  logicalused           387G                   -
pool1/vm-103-disk-0  logicalreferenced     386G                   -
pool1/vm-103-disk-0  volmode               default                default
pool1/vm-103-disk-0  snapshot_limit        none                   default
pool1/vm-103-disk-0  snapshot_count        none                   default
pool1/vm-103-disk-0  snapdev               hidden                 default
pool1/vm-103-disk-0  context               none                   default
pool1/vm-103-disk-0  fscontext             none                   default
pool1/vm-103-disk-0  defcontext            none                   default
pool1/vm-103-disk-0  rootcontext           none                   default
pool1/vm-103-disk-0  redundant_metadata    all                    default
pool1/vm-103-disk-0  encryption            off                    default
pool1/vm-103-disk-0  keylocation           none                   default
pool1/vm-103-disk-0  keyformat             none                   default
pool1/vm-103-disk-0  pbkdf2iters           0                      default

Here is the backup log:
Code:
INFO: Starting Backup of VM 103 (qemu)
INFO: Backup started at 2022-05-30 02:15:05
INFO: status = running
INFO: VM Name: vps.xxxxx.com
INFO: include disk 'scsi0' 'pool1:vm-103-disk-0' 1510G
INFO: include disk 'scsi1' 'pool1:vm-103-disk-1' 64G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: pending configuration changes found (not included into backup)
INFO: creating Proxmox Backup Server archive 'vm/103/2022-05-29T18:15:05Z'
INFO: started backup task 'a81f30e9-0f3b-441a-b666-cf9b1938b113'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: OK (1.2 TiB of 1.5 TiB dirty)
INFO: scsi1: dirty-bitmap status: OK (drive clean)
INFO: using fast incremental mode (dirty-bitmap), 1.2 TiB dirty of 1.5 TiB total
INFO:   0% (56.0 MiB of 1.2 TiB) in 3s, read: 18.7 MiB/s, write: 17.3 MiB/s
INFO:   1% (12.7 GiB of 1.2 TiB) in 8m 1s, read: 27.1 MiB/s, write: 13.8 MiB/s
INFO:   2% (25.6 GiB of 1.2 TiB) in 11m 28s, read: 63.7 MiB/s, write: 9.6 MiB/s
INFO:   3% (38.1 GiB of 1.2 TiB) in 11m 57s, read: 443.7 MiB/s, write: 26.1 MiB/s
INFO:   4% (50.7 GiB of 1.2 TiB) in 14m 53s, read: 73.0 MiB/s, write: 20.7 MiB/s
INFO:   5% (63.5 GiB of 1.2 TiB) in 15m 32s, read: 336.7 MiB/s, write: 23.9 MiB/s
INFO:   6% (77.3 GiB of 1.2 TiB) in 15m 57s, read: 566.1 MiB/s, write: 28.0 MiB/s
INFO:   7% (89.8 GiB of 1.2 TiB) in 16m 23s, read: 491.7 MiB/s, write: 26.6 MiB/s
INFO:   8% (102.9 GiB of 1.2 TiB) in 17m 20s, read: 236.1 MiB/s, write: 24.1 MiB/s
INFO:   9% (114.3 GiB of 1.2 TiB) in 17m 57s, read: 314.6 MiB/s, write: 25.5 MiB/s
INFO:  10% (126.8 GiB of 1.2 TiB) in 18m 34s, read: 346.6 MiB/s, write: 25.1 MiB/s
INFO:  11% (139.3 GiB of 1.2 TiB) in 18m 51s, read: 751.1 MiB/s, write: 27.8 MiB/s
INFO:  12% (152.5 GiB of 1.2 TiB) in 19m 23s, read: 421.4 MiB/s, write: 18.8 MiB/s
INFO:  13% (165.1 GiB of 1.2 TiB) in 19m 54s, read: 418.1 MiB/s, write: 29.4 MiB/s
INFO:  14% (177.7 GiB of 1.2 TiB) in 20m 59s, read: 197.5 MiB/s, write: 39.4 MiB/s
INFO:  15% (190.2 GiB of 1.2 TiB) in 21m 54s, read: 233.6 MiB/s, write: 24.6 MiB/s
INFO:  16% (202.7 GiB of 1.2 TiB) in 22m 55s, read: 210.2 MiB/s, write: 16.3 MiB/s
INFO:  17% (215.7 GiB of 1.2 TiB) in 23m 18s, read: 576.5 MiB/s, write: 21.0 MiB/s
INFO:  18% (228.0 GiB of 1.2 TiB) in 24m 3s, read: 280.4 MiB/s, write: 22.0 MiB/s
INFO:  19% (241.4 GiB of 1.2 TiB) in 25m 15s, read: 190.2 MiB/s, write: 18.8 MiB/s
INFO:  20% (254.1 GiB of 1.2 TiB) in 25m 45s, read: 433.2 MiB/s, write: 16.4 MiB/s
INFO:  21% (266.6 GiB of 1.2 TiB) in 26m 31s, read: 278.3 MiB/s, write: 20.3 MiB/s
INFO:  22% (278.5 GiB of 1.2 TiB) in 26m 53s, read: 556.2 MiB/s, write: 23.8 MiB/s
INFO:  23% (291.7 GiB of 1.2 TiB) in 28m 13s, read: 169.2 MiB/s, write: 16.3 MiB/s
INFO:  24% (304.4 GiB of 1.2 TiB) in 29m 9s, read: 231.3 MiB/s, write: 17.4 MiB/s
INFO:  25% (316.6 GiB of 1.2 TiB) in 30m 19s, read: 177.9 MiB/s, write: 21.9 MiB/s
INFO:  26% (330.1 GiB of 1.2 TiB) in 31m 18s, read: 235.8 MiB/s, write: 16.0 MiB/s
INFO:  27% (342.1 GiB of 1.2 TiB) in 31m 51s, read: 371.5 MiB/s, write: 23.2 MiB/s
INFO:  28% (355.4 GiB of 1.2 TiB) in 32m 49s, read: 234.3 MiB/s, write: 17.9 MiB/s
INFO:  29% (367.4 GiB of 1.2 TiB) in 33m 10s, read: 588.4 MiB/s, write: 29.7 MiB/s
INFO:  30% (380.7 GiB of 1.2 TiB) in 33m 36s, read: 520.3 MiB/s, write: 30.2 MiB/s
INFO:  31% (393.1 GiB of 1.2 TiB) in 34m 3s, read: 473.0 MiB/s, write: 28.3 MiB/s
INFO:  32% (405.2 GiB of 1.2 TiB) in 34m 39s, read: 342.9 MiB/s, write: 20.9 MiB/s
INFO:  33% (418.2 GiB of 1.2 TiB) in 35m 14s, read: 381.0 MiB/s, write: 21.0 MiB/s
INFO:  34% (430.9 GiB of 1.2 TiB) in 35m 35s, read: 619.0 MiB/s, write: 28.0 MiB/s
INFO:  35% (444.6 GiB of 1.2 TiB) in 36m 25s, read: 280.9 MiB/s, write: 18.7 MiB/s
INFO:  36% (455.9 GiB of 1.2 TiB) in 37m 11s, read: 250.7 MiB/s, write: 19.3 MiB/s
INFO:  37% (468.8 GiB of 1.2 TiB) in 37m 41s, read: 440.9 MiB/s, write: 18.7 MiB/s
INFO:  38% (481.3 GiB of 1.2 TiB) in 37m 59s, read: 710.9 MiB/s, write: 24.2 MiB/s
INFO:  39% (493.8 GiB of 1.2 TiB) in 38m 24s, read: 510.1 MiB/s, write: 24.3 MiB/s
INFO:  40% (506.4 GiB of 1.2 TiB) in 39m 36s, read: 180.2 MiB/s, write: 26.7 MiB/s
INFO:  41% (519.8 GiB of 1.2 TiB) in 40m 54s, read: 176.3 MiB/s, write: 17.0 MiB/s
INFO:  42% (531.8 GiB of 1.2 TiB) in 41m 24s, read: 408.1 MiB/s, write: 19.1 MiB/s
INFO:  43% (545.6 GiB of 1.2 TiB) in 41m 41s, read: 832.2 MiB/s, write: 26.1 MiB/s
INFO:  44% (557.2 GiB of 1.2 TiB) in 41m 59s, read: 656.4 MiB/s, write: 26.7 MiB/s
INFO:  45% (570.3 GiB of 1.2 TiB) in 42m 26s, read: 499.3 MiB/s, write: 27.0 MiB/s
INFO:  46% (582.4 GiB of 1.2 TiB) in 42m 45s, read: 650.9 MiB/s, write: 24.0 MiB/s
INFO:  47% (595.4 GiB of 1.2 TiB) in 43m 15s, read: 444.4 MiB/s, write: 33.6 MiB/s
INFO:  48% (609.2 GiB of 1.2 TiB) in 43m 55s, read: 352.3 MiB/s, write: 27.9 MiB/s
INFO:  49% (620.6 GiB of 1.2 TiB) in 44m 23s, read: 416.4 MiB/s, write: 22.0 MiB/s
INFO:  50% (633.2 GiB of 1.2 TiB) in 45m 5s, read: 307.0 MiB/s, write: 34.7 MiB/s
INFO:  51% (646.4 GiB of 1.2 TiB) in 45m 43s, read: 357.2 MiB/s, write: 19.1 MiB/s
INFO:  52% (659.3 GiB of 1.2 TiB) in 46m 12s, read: 455.3 MiB/s, write: 25.1 MiB/s
INFO:  53% (671.0 GiB of 1.2 TiB) in 47m 19s, read: 178.4 MiB/s, write: 20.1 MiB/s
INFO:  54% (684.4 GiB of 1.2 TiB) in 47m 50s, read: 442.5 MiB/s, write: 32.3 MiB/s
INFO:  55% (696.3 GiB of 1.2 TiB) in 48m 11s, read: 579.2 MiB/s, write: 25.5 MiB/s
INFO:  56% (709.1 GiB of 1.2 TiB) in 48m 49s, read: 345.7 MiB/s, write: 20.4 MiB/s
INFO:  57% (722.2 GiB of 1.2 TiB) in 49m 36s, read: 285.7 MiB/s, write: 21.0 MiB/s
INFO:  58% (734.2 GiB of 1.2 TiB) in 50m 15s, read: 316.2 MiB/s, write: 19.5 MiB/s
INFO:  59% (746.9 GiB of 1.2 TiB) in 50m 55s, read: 324.1 MiB/s, write: 25.5 MiB/s
INFO:  60% (760.2 GiB of 1.2 TiB) in 52m 4s, read: 197.3 MiB/s, write: 11.1 MiB/s
INFO:  61% (772.5 GiB of 1.2 TiB) in 53m 13s, read: 183.2 MiB/s, write: 7.0 MiB/s
INFO:  62% (785.7 GiB of 1.2 TiB) in 54m 27s, read: 182.4 MiB/s, write: 8.6 MiB/s
INFO:  63% (798.0 GiB of 1.2 TiB) in 55m 49s, read: 153.7 MiB/s, write: 8.8 MiB/s
INFO:  64% (810.9 GiB of 1.2 TiB) in 56m 38s, read: 269.4 MiB/s, write: 12.7 MiB/s
INFO:  65% (823.9 GiB of 1.2 TiB) in 56m 53s, read: 888.0 MiB/s, write: 27.7 MiB/s
INFO:  66% (835.6 GiB of 1.2 TiB) in 57m 38s, read: 266.6 MiB/s, write: 12.0 MiB/s
INFO:  67% (849.2 GiB of 1.2 TiB) in 1h 11s, read: 91.0 MiB/s, write: 13.8 MiB/s
INFO:  68% (861.1 GiB of 1.2 TiB) in 1h 1m 29s, read: 155.2 MiB/s, write: 9.8 MiB/s
INFO:  69% (874.3 GiB of 1.2 TiB) in 1h 2m 45s, read: 178.3 MiB/s, write: 12.1 MiB/s
INFO:  70% (886.6 GiB of 1.2 TiB) in 1h 3m 26s, read: 306.7 MiB/s, write: 11.9 MiB/s
INFO:  71% (899.0 GiB of 1.2 TiB) in 1h 4m 25s, read: 215.5 MiB/s, write: 14.4 MiB/s
INFO:  72% (911.9 GiB of 1.2 TiB) in 1h 5m 7s, read: 315.7 MiB/s, write: 13.4 MiB/s
INFO:  73% (925.0 GiB of 1.2 TiB) in 1h 8m 36s, read: 64.0 MiB/s, write: 15.7 MiB/s
INFO:  74% (937.3 GiB of 1.2 TiB) in 1h 9m 15s, read: 322.9 MiB/s, write: 9.5 MiB/s
INFO:  75% (949.5 GiB of 1.2 TiB) in 1h 10m 18s, read: 198.0 MiB/s, write: 12.3 MiB/s
INFO:  76% (962.7 GiB of 1.2 TiB) in 1h 11m 35s, read: 175.8 MiB/s, write: 9.8 MiB/s
INFO:  77% (976.2 GiB of 1.2 TiB) in 1h 11m 54s, read: 725.7 MiB/s, write: 16.2 MiB/s
INFO:  78% (987.4 GiB of 1.2 TiB) in 1h 13m 3s, read: 167.2 MiB/s, write: 8.5 MiB/s
INFO:  79% (1000.3 GiB of 1.2 TiB) in 1h 13m 59s, read: 236.3 MiB/s, write: 11.6 MiB/s
INFO:  80% (1013.2 GiB of 1.2 TiB) in 1h 14m 20s, read: 627.0 MiB/s, write: 24.8 MiB/s
INFO:  81% (1.0 TiB of 1.2 TiB) in 1h 14m 52s, read: 390.6 MiB/s, write: 25.2 MiB/s
INFO:  82% (1.0 TiB of 1.2 TiB) in 1h 15m 30s, read: 348.6 MiB/s, write: 8.1 MiB/s
INFO:  83% (1.0 TiB of 1.2 TiB) in 1h 16m 13s, read: 306.9 MiB/s, write: 10.2 MiB/s
INFO:  84% (1.0 TiB of 1.2 TiB) in 1h 16m 38s, read: 526.1 MiB/s, write: 20.2 MiB/s
INFO:  85% (1.1 TiB of 1.2 TiB) in 1h 17m 9s, read: 440.5 MiB/s, write: 16.0 MiB/s
INFO:  86% (1.1 TiB of 1.2 TiB) in 1h 18m 36s, read: 135.2 MiB/s, write: 16.3 MiB/s
INFO:  87% (1.1 TiB of 1.2 TiB) in 1h 19m 15s, read: 351.5 MiB/s, write: 11.5 MiB/s
INFO:  88% (1.1 TiB of 1.2 TiB) in 1h 19m 46s, read: 424.3 MiB/s, write: 10.5 MiB/s
INFO:  89% (1.1 TiB of 1.2 TiB) in 1h 20m 28s, read: 287.2 MiB/s, write: 16.9 MiB/s
INFO:  90% (1.1 TiB of 1.2 TiB) in 1h 21m 3s, read: 387.3 MiB/s, write: 56.9 MiB/s
INFO:  91% (1.1 TiB of 1.2 TiB) in 1h 21m 26s, read: 557.0 MiB/s, write: 19.5 MiB/s
INFO:  92% (1.1 TiB of 1.2 TiB) in 1h 22m 16s, read: 247.9 MiB/s, write: 13.7 MiB/s
INFO:  93% (1.1 TiB of 1.2 TiB) in 1h 22m 35s, read: 683.4 MiB/s, write: 21.3 MiB/s
INFO:  94% (1.2 TiB of 1.2 TiB) in 1h 23m 35s, read: 216.1 MiB/s, write: 15.4 MiB/s
INFO:  95% (1.2 TiB of 1.2 TiB) in 1h 24m 38s, read: 203.5 MiB/s, write: 14.2 MiB/s
INFO:  96% (1.2 TiB of 1.2 TiB) in 1h 25m 36s, read: 232.8 MiB/s, write: 11.7 MiB/s
INFO:  97% (1.2 TiB of 1.2 TiB) in 1h 26m 22s, read: 277.7 MiB/s, write: 10.2 MiB/s
INFO:  98% (1.2 TiB of 1.2 TiB) in 1h 27m 49s, read: 161.6 MiB/s, write: 13.3 MiB/s
INFO:  99% (1.2 TiB of 1.2 TiB) in 1h 30m 23s, read: 75.0 MiB/s, write: 3.9 MiB/s
INFO: 100% (1.2 TiB of 1.2 TiB) in 1h 32m 41s, read: 93.7 MiB/s, write: 2.6 MiB/s
INFO: backup is sparse: 1.03 TiB (83%) total zero data     
INFO: backup was done incrementally, reused 1.45 TiB (94%)
INFO: transferred 1.24 TiB in 5562 seconds (233.1 MiB/s)   
INFO: Finished Backup of VM 103 (01:32:44)
INFO: Backup finished at 2022-05-30 03:47:49
 

danielb

Well-Known Member
Jun 1, 2018
212
54
48
36
Bordeaux (france)
This is most likely because you have a weekly fstrim job running, which will mark all the trimmed sectors as dirty. It shouldn't affect backup speed that much as all those "dirty" sectors will not really be red from disk
 
  • Like
Reactions: Stoiko Ivanov

chengkinhung

Active Member
Feb 11, 2016
39
6
33
23
@danielb , yes, I just run fstrim to test, and confirm it will cause huge dirty-bitmap, thanks for your reply.
Code:
# fstrim -a -v
/var/tmp: 0 B (0 bytes) trimmed
/: 1.1 TiB (1212771516416 bytes) trimmed
Code:
INFO: starting new backup job: vzdump 103 --storage pb1_backup3 --notes-template '{{guestname}}' --mode snapshot --node XXX --remove 0
INFO: Starting Backup of VM 103 (qemu)
INFO: Backup started at 2022-05-30 18:56:58
INFO: status = running
INFO: VM Name: vps.xxxxx.com
INFO: include disk 'scsi0' 'pool1:vm-103-disk-0' 1510G
INFO: include disk 'scsi1' 'pool1:vm-103-disk-1' 64G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: pending configuration changes found (not included into backup)
INFO: creating Proxmox Backup Server archive 'vm/103/2022-05-30T10:56:58Z'
INFO: started backup task '8b30bf59-000b-4072-878d-8c25b3813aa9'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: OK (1.2 TiB of 1.5 TiB dirty)
INFO: scsi1: dirty-bitmap status: OK (drive clean)
INFO: using fast incremental mode (dirty-bitmap), 1.2 TiB dirty of 1.5 TiB total
INFO:   0% (340.0 MiB of 1.2 TiB) in 3s, read: 113.3 MiB/s, write: 49.3 MiB/s
...
EDIT. in my case, huge dirty-bitmap will cause backup task take almost 2 hours to complete. I do not really sure if guest VM need to run fstrim scheduly if enable the DISCARD in /etc/fstab, I will test this with other VM later ...
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!