Hi, every one,
I was using PBS to backup VM created on ZFS, it was running well, however sometimes I found one VM got very huge dirty-bitmap, even bigger than the real data usage insider that VM. This VMs was installed CloudLinux and cPanel, it was using XFS filesystem, I also found it almost always happened after sunday.
Here is the disk usage insider this VM:
Here is the ZFS disk information:
Here is the backup log:
I was using PBS to backup VM created on ZFS, it was running well, however sometimes I found one VM got very huge dirty-bitmap, even bigger than the real data usage insider that VM. This VMs was installed CloudLinux and cPanel, it was using XFS filesystem, I also found it almost always happened after sunday.
Here is the disk usage insider this VM:
Code:
# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 8.8G 0 8.8G 0% /dev
tmpfs 8.8G 4.0K 8.8G 1% /dev/shm
tmpfs 8.8G 858M 8.0G 10% /run
tmpfs 8.8G 0 8.8G 0% /sys/fs/cgroup
/dev/sda1 1.5T 380G 1.2T 26% /
/dev/loop0 3.9G 146M 3.5G 4% /tmp
tmpfs 1.7G 0 1.7G 0% /run/user/0Here is the ZFS disk information:
Code:
# zfs get all pool1/vm-103-disk-0
NAME PROPERTY VALUE SOURCE
pool1/vm-103-disk-0 type volume -
pool1/vm-103-disk-0 creation Fri Apr 8 13:15 2022 -
pool1/vm-103-disk-0 used 389G -
pool1/vm-103-disk-0 available 990G -
pool1/vm-103-disk-0 referenced 389G -
pool1/vm-103-disk-0 compressratio 1.00x -
pool1/vm-103-disk-0 reservation none default
pool1/vm-103-disk-0 volsize 1.47T local
pool1/vm-103-disk-0 volblocksize 8K default
pool1/vm-103-disk-0 checksum on default
pool1/vm-103-disk-0 compression off default
pool1/vm-103-disk-0 readonly off default
pool1/vm-103-disk-0 createtxg 669645 -
pool1/vm-103-disk-0 copies 1 default
pool1/vm-103-disk-0 refreservation none default
pool1/vm-103-disk-0 guid 10756677095636455937 -
pool1/vm-103-disk-0 primarycache all default
pool1/vm-103-disk-0 secondarycache all default
pool1/vm-103-disk-0 usedbysnapshots 275M -
pool1/vm-103-disk-0 usedbydataset 389G -
pool1/vm-103-disk-0 usedbychildren 0B -
pool1/vm-103-disk-0 usedbyrefreservation 0B -
pool1/vm-103-disk-0 logbias latency default
pool1/vm-103-disk-0 objsetid 16288 -
pool1/vm-103-disk-0 dedup off default
pool1/vm-103-disk-0 mlslabel none default
pool1/vm-103-disk-0 sync standard default
pool1/vm-103-disk-0 refcompressratio 1.00x -
pool1/vm-103-disk-0 written 298M -
pool1/vm-103-disk-0 logicalused 387G -
pool1/vm-103-disk-0 logicalreferenced 386G -
pool1/vm-103-disk-0 volmode default default
pool1/vm-103-disk-0 snapshot_limit none default
pool1/vm-103-disk-0 snapshot_count none default
pool1/vm-103-disk-0 snapdev hidden default
pool1/vm-103-disk-0 context none default
pool1/vm-103-disk-0 fscontext none default
pool1/vm-103-disk-0 defcontext none default
pool1/vm-103-disk-0 rootcontext none default
pool1/vm-103-disk-0 redundant_metadata all default
pool1/vm-103-disk-0 encryption off default
pool1/vm-103-disk-0 keylocation none default
pool1/vm-103-disk-0 keyformat none default
pool1/vm-103-disk-0 pbkdf2iters 0 defaultHere is the backup log:
Code:
INFO: Starting Backup of VM 103 (qemu)
INFO: Backup started at 2022-05-30 02:15:05
INFO: status = running
INFO: VM Name: vps.xxxxx.com
INFO: include disk 'scsi0' 'pool1:vm-103-disk-0' 1510G
INFO: include disk 'scsi1' 'pool1:vm-103-disk-1' 64G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: pending configuration changes found (not included into backup)
INFO: creating Proxmox Backup Server archive 'vm/103/2022-05-29T18:15:05Z'
INFO: started backup task 'a81f30e9-0f3b-441a-b666-cf9b1938b113'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: OK (1.2 TiB of 1.5 TiB dirty)
INFO: scsi1: dirty-bitmap status: OK (drive clean)
INFO: using fast incremental mode (dirty-bitmap), 1.2 TiB dirty of 1.5 TiB total
INFO: 0% (56.0 MiB of 1.2 TiB) in 3s, read: 18.7 MiB/s, write: 17.3 MiB/s
INFO: 1% (12.7 GiB of 1.2 TiB) in 8m 1s, read: 27.1 MiB/s, write: 13.8 MiB/s
INFO: 2% (25.6 GiB of 1.2 TiB) in 11m 28s, read: 63.7 MiB/s, write: 9.6 MiB/s
INFO: 3% (38.1 GiB of 1.2 TiB) in 11m 57s, read: 443.7 MiB/s, write: 26.1 MiB/s
INFO: 4% (50.7 GiB of 1.2 TiB) in 14m 53s, read: 73.0 MiB/s, write: 20.7 MiB/s
INFO: 5% (63.5 GiB of 1.2 TiB) in 15m 32s, read: 336.7 MiB/s, write: 23.9 MiB/s
INFO: 6% (77.3 GiB of 1.2 TiB) in 15m 57s, read: 566.1 MiB/s, write: 28.0 MiB/s
INFO: 7% (89.8 GiB of 1.2 TiB) in 16m 23s, read: 491.7 MiB/s, write: 26.6 MiB/s
INFO: 8% (102.9 GiB of 1.2 TiB) in 17m 20s, read: 236.1 MiB/s, write: 24.1 MiB/s
INFO: 9% (114.3 GiB of 1.2 TiB) in 17m 57s, read: 314.6 MiB/s, write: 25.5 MiB/s
INFO: 10% (126.8 GiB of 1.2 TiB) in 18m 34s, read: 346.6 MiB/s, write: 25.1 MiB/s
INFO: 11% (139.3 GiB of 1.2 TiB) in 18m 51s, read: 751.1 MiB/s, write: 27.8 MiB/s
INFO: 12% (152.5 GiB of 1.2 TiB) in 19m 23s, read: 421.4 MiB/s, write: 18.8 MiB/s
INFO: 13% (165.1 GiB of 1.2 TiB) in 19m 54s, read: 418.1 MiB/s, write: 29.4 MiB/s
INFO: 14% (177.7 GiB of 1.2 TiB) in 20m 59s, read: 197.5 MiB/s, write: 39.4 MiB/s
INFO: 15% (190.2 GiB of 1.2 TiB) in 21m 54s, read: 233.6 MiB/s, write: 24.6 MiB/s
INFO: 16% (202.7 GiB of 1.2 TiB) in 22m 55s, read: 210.2 MiB/s, write: 16.3 MiB/s
INFO: 17% (215.7 GiB of 1.2 TiB) in 23m 18s, read: 576.5 MiB/s, write: 21.0 MiB/s
INFO: 18% (228.0 GiB of 1.2 TiB) in 24m 3s, read: 280.4 MiB/s, write: 22.0 MiB/s
INFO: 19% (241.4 GiB of 1.2 TiB) in 25m 15s, read: 190.2 MiB/s, write: 18.8 MiB/s
INFO: 20% (254.1 GiB of 1.2 TiB) in 25m 45s, read: 433.2 MiB/s, write: 16.4 MiB/s
INFO: 21% (266.6 GiB of 1.2 TiB) in 26m 31s, read: 278.3 MiB/s, write: 20.3 MiB/s
INFO: 22% (278.5 GiB of 1.2 TiB) in 26m 53s, read: 556.2 MiB/s, write: 23.8 MiB/s
INFO: 23% (291.7 GiB of 1.2 TiB) in 28m 13s, read: 169.2 MiB/s, write: 16.3 MiB/s
INFO: 24% (304.4 GiB of 1.2 TiB) in 29m 9s, read: 231.3 MiB/s, write: 17.4 MiB/s
INFO: 25% (316.6 GiB of 1.2 TiB) in 30m 19s, read: 177.9 MiB/s, write: 21.9 MiB/s
INFO: 26% (330.1 GiB of 1.2 TiB) in 31m 18s, read: 235.8 MiB/s, write: 16.0 MiB/s
INFO: 27% (342.1 GiB of 1.2 TiB) in 31m 51s, read: 371.5 MiB/s, write: 23.2 MiB/s
INFO: 28% (355.4 GiB of 1.2 TiB) in 32m 49s, read: 234.3 MiB/s, write: 17.9 MiB/s
INFO: 29% (367.4 GiB of 1.2 TiB) in 33m 10s, read: 588.4 MiB/s, write: 29.7 MiB/s
INFO: 30% (380.7 GiB of 1.2 TiB) in 33m 36s, read: 520.3 MiB/s, write: 30.2 MiB/s
INFO: 31% (393.1 GiB of 1.2 TiB) in 34m 3s, read: 473.0 MiB/s, write: 28.3 MiB/s
INFO: 32% (405.2 GiB of 1.2 TiB) in 34m 39s, read: 342.9 MiB/s, write: 20.9 MiB/s
INFO: 33% (418.2 GiB of 1.2 TiB) in 35m 14s, read: 381.0 MiB/s, write: 21.0 MiB/s
INFO: 34% (430.9 GiB of 1.2 TiB) in 35m 35s, read: 619.0 MiB/s, write: 28.0 MiB/s
INFO: 35% (444.6 GiB of 1.2 TiB) in 36m 25s, read: 280.9 MiB/s, write: 18.7 MiB/s
INFO: 36% (455.9 GiB of 1.2 TiB) in 37m 11s, read: 250.7 MiB/s, write: 19.3 MiB/s
INFO: 37% (468.8 GiB of 1.2 TiB) in 37m 41s, read: 440.9 MiB/s, write: 18.7 MiB/s
INFO: 38% (481.3 GiB of 1.2 TiB) in 37m 59s, read: 710.9 MiB/s, write: 24.2 MiB/s
INFO: 39% (493.8 GiB of 1.2 TiB) in 38m 24s, read: 510.1 MiB/s, write: 24.3 MiB/s
INFO: 40% (506.4 GiB of 1.2 TiB) in 39m 36s, read: 180.2 MiB/s, write: 26.7 MiB/s
INFO: 41% (519.8 GiB of 1.2 TiB) in 40m 54s, read: 176.3 MiB/s, write: 17.0 MiB/s
INFO: 42% (531.8 GiB of 1.2 TiB) in 41m 24s, read: 408.1 MiB/s, write: 19.1 MiB/s
INFO: 43% (545.6 GiB of 1.2 TiB) in 41m 41s, read: 832.2 MiB/s, write: 26.1 MiB/s
INFO: 44% (557.2 GiB of 1.2 TiB) in 41m 59s, read: 656.4 MiB/s, write: 26.7 MiB/s
INFO: 45% (570.3 GiB of 1.2 TiB) in 42m 26s, read: 499.3 MiB/s, write: 27.0 MiB/s
INFO: 46% (582.4 GiB of 1.2 TiB) in 42m 45s, read: 650.9 MiB/s, write: 24.0 MiB/s
INFO: 47% (595.4 GiB of 1.2 TiB) in 43m 15s, read: 444.4 MiB/s, write: 33.6 MiB/s
INFO: 48% (609.2 GiB of 1.2 TiB) in 43m 55s, read: 352.3 MiB/s, write: 27.9 MiB/s
INFO: 49% (620.6 GiB of 1.2 TiB) in 44m 23s, read: 416.4 MiB/s, write: 22.0 MiB/s
INFO: 50% (633.2 GiB of 1.2 TiB) in 45m 5s, read: 307.0 MiB/s, write: 34.7 MiB/s
INFO: 51% (646.4 GiB of 1.2 TiB) in 45m 43s, read: 357.2 MiB/s, write: 19.1 MiB/s
INFO: 52% (659.3 GiB of 1.2 TiB) in 46m 12s, read: 455.3 MiB/s, write: 25.1 MiB/s
INFO: 53% (671.0 GiB of 1.2 TiB) in 47m 19s, read: 178.4 MiB/s, write: 20.1 MiB/s
INFO: 54% (684.4 GiB of 1.2 TiB) in 47m 50s, read: 442.5 MiB/s, write: 32.3 MiB/s
INFO: 55% (696.3 GiB of 1.2 TiB) in 48m 11s, read: 579.2 MiB/s, write: 25.5 MiB/s
INFO: 56% (709.1 GiB of 1.2 TiB) in 48m 49s, read: 345.7 MiB/s, write: 20.4 MiB/s
INFO: 57% (722.2 GiB of 1.2 TiB) in 49m 36s, read: 285.7 MiB/s, write: 21.0 MiB/s
INFO: 58% (734.2 GiB of 1.2 TiB) in 50m 15s, read: 316.2 MiB/s, write: 19.5 MiB/s
INFO: 59% (746.9 GiB of 1.2 TiB) in 50m 55s, read: 324.1 MiB/s, write: 25.5 MiB/s
INFO: 60% (760.2 GiB of 1.2 TiB) in 52m 4s, read: 197.3 MiB/s, write: 11.1 MiB/s
INFO: 61% (772.5 GiB of 1.2 TiB) in 53m 13s, read: 183.2 MiB/s, write: 7.0 MiB/s
INFO: 62% (785.7 GiB of 1.2 TiB) in 54m 27s, read: 182.4 MiB/s, write: 8.6 MiB/s
INFO: 63% (798.0 GiB of 1.2 TiB) in 55m 49s, read: 153.7 MiB/s, write: 8.8 MiB/s
INFO: 64% (810.9 GiB of 1.2 TiB) in 56m 38s, read: 269.4 MiB/s, write: 12.7 MiB/s
INFO: 65% (823.9 GiB of 1.2 TiB) in 56m 53s, read: 888.0 MiB/s, write: 27.7 MiB/s
INFO: 66% (835.6 GiB of 1.2 TiB) in 57m 38s, read: 266.6 MiB/s, write: 12.0 MiB/s
INFO: 67% (849.2 GiB of 1.2 TiB) in 1h 11s, read: 91.0 MiB/s, write: 13.8 MiB/s
INFO: 68% (861.1 GiB of 1.2 TiB) in 1h 1m 29s, read: 155.2 MiB/s, write: 9.8 MiB/s
INFO: 69% (874.3 GiB of 1.2 TiB) in 1h 2m 45s, read: 178.3 MiB/s, write: 12.1 MiB/s
INFO: 70% (886.6 GiB of 1.2 TiB) in 1h 3m 26s, read: 306.7 MiB/s, write: 11.9 MiB/s
INFO: 71% (899.0 GiB of 1.2 TiB) in 1h 4m 25s, read: 215.5 MiB/s, write: 14.4 MiB/s
INFO: 72% (911.9 GiB of 1.2 TiB) in 1h 5m 7s, read: 315.7 MiB/s, write: 13.4 MiB/s
INFO: 73% (925.0 GiB of 1.2 TiB) in 1h 8m 36s, read: 64.0 MiB/s, write: 15.7 MiB/s
INFO: 74% (937.3 GiB of 1.2 TiB) in 1h 9m 15s, read: 322.9 MiB/s, write: 9.5 MiB/s
INFO: 75% (949.5 GiB of 1.2 TiB) in 1h 10m 18s, read: 198.0 MiB/s, write: 12.3 MiB/s
INFO: 76% (962.7 GiB of 1.2 TiB) in 1h 11m 35s, read: 175.8 MiB/s, write: 9.8 MiB/s
INFO: 77% (976.2 GiB of 1.2 TiB) in 1h 11m 54s, read: 725.7 MiB/s, write: 16.2 MiB/s
INFO: 78% (987.4 GiB of 1.2 TiB) in 1h 13m 3s, read: 167.2 MiB/s, write: 8.5 MiB/s
INFO: 79% (1000.3 GiB of 1.2 TiB) in 1h 13m 59s, read: 236.3 MiB/s, write: 11.6 MiB/s
INFO: 80% (1013.2 GiB of 1.2 TiB) in 1h 14m 20s, read: 627.0 MiB/s, write: 24.8 MiB/s
INFO: 81% (1.0 TiB of 1.2 TiB) in 1h 14m 52s, read: 390.6 MiB/s, write: 25.2 MiB/s
INFO: 82% (1.0 TiB of 1.2 TiB) in 1h 15m 30s, read: 348.6 MiB/s, write: 8.1 MiB/s
INFO: 83% (1.0 TiB of 1.2 TiB) in 1h 16m 13s, read: 306.9 MiB/s, write: 10.2 MiB/s
INFO: 84% (1.0 TiB of 1.2 TiB) in 1h 16m 38s, read: 526.1 MiB/s, write: 20.2 MiB/s
INFO: 85% (1.1 TiB of 1.2 TiB) in 1h 17m 9s, read: 440.5 MiB/s, write: 16.0 MiB/s
INFO: 86% (1.1 TiB of 1.2 TiB) in 1h 18m 36s, read: 135.2 MiB/s, write: 16.3 MiB/s
INFO: 87% (1.1 TiB of 1.2 TiB) in 1h 19m 15s, read: 351.5 MiB/s, write: 11.5 MiB/s
INFO: 88% (1.1 TiB of 1.2 TiB) in 1h 19m 46s, read: 424.3 MiB/s, write: 10.5 MiB/s
INFO: 89% (1.1 TiB of 1.2 TiB) in 1h 20m 28s, read: 287.2 MiB/s, write: 16.9 MiB/s
INFO: 90% (1.1 TiB of 1.2 TiB) in 1h 21m 3s, read: 387.3 MiB/s, write: 56.9 MiB/s
INFO: 91% (1.1 TiB of 1.2 TiB) in 1h 21m 26s, read: 557.0 MiB/s, write: 19.5 MiB/s
INFO: 92% (1.1 TiB of 1.2 TiB) in 1h 22m 16s, read: 247.9 MiB/s, write: 13.7 MiB/s
INFO: 93% (1.1 TiB of 1.2 TiB) in 1h 22m 35s, read: 683.4 MiB/s, write: 21.3 MiB/s
INFO: 94% (1.2 TiB of 1.2 TiB) in 1h 23m 35s, read: 216.1 MiB/s, write: 15.4 MiB/s
INFO: 95% (1.2 TiB of 1.2 TiB) in 1h 24m 38s, read: 203.5 MiB/s, write: 14.2 MiB/s
INFO: 96% (1.2 TiB of 1.2 TiB) in 1h 25m 36s, read: 232.8 MiB/s, write: 11.7 MiB/s
INFO: 97% (1.2 TiB of 1.2 TiB) in 1h 26m 22s, read: 277.7 MiB/s, write: 10.2 MiB/s
INFO: 98% (1.2 TiB of 1.2 TiB) in 1h 27m 49s, read: 161.6 MiB/s, write: 13.3 MiB/s
INFO: 99% (1.2 TiB of 1.2 TiB) in 1h 30m 23s, read: 75.0 MiB/s, write: 3.9 MiB/s
INFO: 100% (1.2 TiB of 1.2 TiB) in 1h 32m 41s, read: 93.7 MiB/s, write: 2.6 MiB/s
INFO: backup is sparse: 1.03 TiB (83%) total zero data
INFO: backup was done incrementally, reused 1.45 TiB (94%)
INFO: transferred 1.24 TiB in 5562 seconds (233.1 MiB/s)
INFO: Finished Backup of VM 103 (01:32:44)
INFO: Backup finished at 2022-05-30 03:47:49