root@node1:~# pveversion
pve-manager/8.2.4/faa83925c9641325 (running kernel: 6.8.12-1-pve)
root@node1:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 63G 0 63G 0% /dev
tmpfs 13G 2.5M 13G 1% /run
/dev/mapper/pve-root 13G 4.6G 6.9G 40% /
tmpfs 63G 66M 63G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda1 15T 662G 14T 5% /mnt/pve/node1-datastore-1
/dev/fuse 128M 56K 128M 1% /etc/pve
//10.16.101.249/ISOs 35T 6.0T 29T 18% /mnt/pve/ISOs
10.16.101.249:/volume1/NFS-Storage 35T 6.0T 29T 18% /mnt/pve/NFS-Storage
tmpfs 13G 0 13G 0% /run/user/0
Log from Backup taking so long:
INFO: starting new backup job: vzdump 1000 --quiet 1 --prune-backups 'keep-last=30' --notes-template '{{guestname}}' --mode snapshot --fleecing 0 --storage PBS-Backup-S01
INFO: Starting Backup of VM 1000 (qemu)
INFO: Backup started at 2024-10-19 12:00:07
INFO: status = running
INFO: VM Name: XX-WEB01
INFO: include disk 'scsi0' 'node2-datastore-2:1000/vm-1000-disk-0.qcow2' 500G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/1000/2024-10-19T01:00:07Z'
INFO: issuing guest-agent 'fs-freeze' command
ERROR: VM 1000 qmp command 'guest-fsfreeze-freeze' failed - got timeout
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 1000 qmp command 'guest-fsfreeze-thaw' failed - got timeout
INFO: started backup task '8f88366e-b94c-4f03-a014-f90ae4e57532'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: created new
INFO: 6% (31.3 GiB of 500.0 GiB) in 3s, read: 10.4 GiB/s, write: 860.0 MiB/s
INFO: 7% (35.2 GiB of 500.0 GiB) in 24s, read: 188.4 MiB/s, write: 0 B/s
INFO: 8% (40.1 GiB of 500.0 GiB) in 51s, read: 187.7 MiB/s, write: 303.4 KiB/s
INFO: 9% (45.0 GiB of 500.0 GiB) in 1m 9s, read: 278.7 MiB/s, write: 6.9 MiB/s
INFO: 10% (51.5 GiB of 500.0 GiB) in 1m 36s, read: 245.6 MiB/s, write: 0 B/s
INFO: 11% (56.5 GiB of 500.0 GiB) in 1m 41s, read: 1020.0 MiB/s, write: 0 B/s
INFO: 12% (60.9 GiB of 500.0 GiB) in 1m 45s, read: 1.1 GiB/s, write: 0 B/s
INFO: 13% (65.6 GiB of 500.0 GiB) in 1m 49s, read: 1.2 GiB/s, write: 0 B/s
INFO: 14% (71.0 GiB of 500.0 GiB) in 1m 54s, read: 1.1 GiB/s, write: 0 B/s
INFO: 15% (75.0 GiB of 500.0 GiB) in 2m, read: 688.7 MiB/s, write: 3.3 MiB/s
INFO: 16% (81.0 GiB of 500.0 GiB) in 2m 5s, read: 1.2 GiB/s, write: 0 B/s
INFO: 17% (86.5 GiB of 500.0 GiB) in 2m 10s, read: 1.1 GiB/s, write: 0 B/s
INFO: 18% (90.5 GiB of 500.0 GiB) in 2m 14s, read: 1.0 GiB/s, write: 0 B/s
INFO: 19% (95.5 GiB of 500.0 GiB) in 2m 19s, read: 1017.6 MiB/s, write: 0 B/s
INFO: 20% (101.5 GiB of 500.0 GiB) in 2m 24s, read: 1.2 GiB/s, write: 0 B/s
INFO: 21% (106.0 GiB of 500.0 GiB) in 2m 28s, read: 1.1 GiB/s, write: 0 B/s
INFO: 22% (110.4 GiB of 500.0 GiB) in 2m 32s, read: 1.1 GiB/s, write: 0 B/s
INFO: 23% (115.5 GiB of 500.0 GiB) in 2m 37s, read: 1.0 GiB/s, write: 0 B/s
INFO: 24% (120.5 GiB of 500.0 GiB) in 2m 42s, read: 1.0 GiB/s, write: 0 B/s
INFO: 25% (126.5 GiB of 500.0 GiB) in 2m 47s, read: 1.2 GiB/s, write: 0 B/s
INFO: 26% (130.5 GiB of 500.0 GiB) in 2m 51s, read: 1023.0 MiB/s, write: 0 B/s
INFO: 27% (135.0 GiB of 500.0 GiB) in 3m 25s, read: 136.6 MiB/s, write: 87.4 MiB/s
INFO: 28% (140.1 GiB of 500.0 GiB) in 3m 57s, read: 161.8 MiB/s, write: 28.9 MiB/s
INFO: 29% (145.1 GiB of 500.0 GiB) in 4m 25s, read: 181.9 MiB/s, write: 0 B/s
INFO: 30% (150.1 GiB of 500.0 GiB) in 4m 52s, read: 189.8 MiB/s, write: 0 B/s
INFO: 31% (155.0 GiB of 500.0 GiB) in 5m 19s, read: 188.1 MiB/s, write: 0 B/s
INFO: 32% (160.0 GiB of 500.0 GiB) in 5m 46s, read: 189.5 MiB/s, write: 0 B/s
INFO: 33% (165.2 GiB of 500.0 GiB) in 6m 10s, read: 221.0 MiB/s, write: 3.3 MiB/s
INFO: 34% (170.2 GiB of 500.0 GiB) in 6m 35s, read: 203.2 MiB/s, write: 1.1 MiB/s
INFO: 35% (175.5 GiB of 500.0 GiB) in 6m 48s, read: 420.9 MiB/s, write: 0 B/s
INFO: 36% (180.0 GiB of 500.0 GiB) in 6m 53s, read: 922.4 MiB/s, write: 0 B/s
INFO: 37% (185.4 GiB of 500.0 GiB) in 6m 58s, read: 1.1 GiB/s, write: 0 B/s
INFO: 38% (190.5 GiB of 500.0 GiB) in 7m 3s, read: 1.0 GiB/s, write: 0 B/s
INFO: 39% (195.2 GiB of 500.0 GiB) in 7m 8s, read: 954.4 MiB/s, write: 0 B/s
INFO: 40% (200.0 GiB of 500.0 GiB) in 7m 15s, read: 708.6 MiB/s, write: 1.7 MiB/s
INFO: 41% (205.8 GiB of 500.0 GiB) in 7m 21s, read: 992.7 MiB/s, write: 1.3 MiB/s
INFO: 42% (211.5 GiB of 500.0 GiB) in 7m 26s, read: 1.1 GiB/s, write: 0 B/s
INFO: 43% (215.5 GiB of 500.0 GiB) in 7m 30s, read: 1021.0 MiB/s, write: 0 B/s
INFO: 44% (220.5 GiB of 500.0 GiB) in 7m 34s, read: 1.3 GiB/s, write: 0 B/s
INFO: 45% (225.0 GiB of 500.0 GiB) in 7m 39s, read: 924.0 MiB/s, write: 0 B/s
INFO: 46% (230.2 GiB of 500.0 GiB) in 7m 44s, read: 1.0 GiB/s, write: 0 B/s
INFO: 47% (236.5 GiB of 500.0 GiB) in 7m 50s, read: 1.0 GiB/s, write: 0 B/s
INFO: 48% (240.5 GiB of 500.0 GiB) in 7m 54s, read: 1.0 GiB/s, write: 0 B/s
INFO: 49% (245.7 GiB of 500.0 GiB) in 7m 59s, read: 1.0 GiB/s, write: 0 B/s
INFO: 50% (251.1 GiB of 500.0 GiB) in 8m 4s, read: 1.1 GiB/s, write: 0 B/s
INFO: 51% (255.0 GiB of 500.0 GiB) in 8m 16s, read: 338.3 MiB/s, write: 83.3 MiB/s
INFO: 52% (260.0 GiB of 500.0 GiB) in 8m 49s, read: 154.7 MiB/s, write: 40.0 MiB/s
INFO: 53% (265.0 GiB of 500.0 GiB) in 9m 16s, read: 189.3 MiB/s, write: 0 B/s
INFO: 54% (270.0 GiB of 500.0 GiB) in 9m 46s, read: 170.1 MiB/s, write: 32.4 MiB/s
INFO: 55% (275.1 GiB of 500.0 GiB) in 10m 14s, read: 186.6 MiB/s, write: 14.4 MiB/s
INFO: 56% (280.2 GiB of 500.0 GiB) in 10m 41s, read: 192.4 MiB/s, write: 1.6 MiB/s
INFO: 57% (285.1 GiB of 500.0 GiB) in 11m 7s, read: 195.1 MiB/s, write: 0 B/s
INFO: 58% (290.2 GiB of 500.0 GiB) in 11m 29s, read: 235.1 MiB/s, write: 6.4 MiB/s
INFO: 59% (295.1 GiB of 500.0 GiB) in 11m 55s, read: 192.6 MiB/s, write: 0 B/s
INFO: 60% (300.1 GiB of 500.0 GiB) in 12m 21s, read: 197.5 MiB/s, write: 157.5 KiB/s
INFO: 61% (305.0 GiB of 500.0 GiB) in 12m 44s, read: 218.8 MiB/s, write: 1.2 MiB/s
INFO: 62% (310.2 GiB of 500.0 GiB) in 13m 11s, read: 196.9 MiB/s, write: 0 B/s
INFO: 63% (316.5 GiB of 500.0 GiB) in 13m 32s, read: 307.0 MiB/s, write: 0 B/s
INFO: 64% (320.0 GiB of 500.0 GiB) in 13m 35s, read: 1.2 GiB/s, write: 1.3 MiB/s
INFO: 65% (325.1 GiB of 500.0 GiB) in 13m 55s, read: 258.4 MiB/s, write: 409.6 KiB/s
INFO: 66% (331.4 GiB of 500.0 GiB) in 14m 19s, read: 267.8 MiB/s, write: 0 B/s
INFO: 67% (335.6 GiB of 500.0 GiB) in 14m 24s, read: 876.8 MiB/s, write: 1.6 MiB/s
INFO: 68% (340.7 GiB of 500.0 GiB) in 14m 27s, read: 1.7 GiB/s, write: 0 B/s
INFO: 69% (346.0 GiB of 500.0 GiB) in 14m 30s, read: 1.8 GiB/s, write: 0 B/s
INFO: 70% (351.1 GiB of 500.0 GiB) in 14m 33s, read: 1.7 GiB/s, write: 0 B/s
INFO: 71% (356.0 GiB of 500.0 GiB) in 14m 36s, read: 1.6 GiB/s, write: 0 B/s
INFO: 72% (361.0 GiB of 500.0 GiB) in 14m 39s, read: 1.7 GiB/s, write: 0 B/s
INFO: 73% (365.9 GiB of 500.0 GiB) in 14m 42s, read: 1.6 GiB/s, write: 0 B/s
INFO: 74% (371.0 GiB of 500.0 GiB) in 14m 45s, read: 1.7 GiB/s, write: 0 B/s
INFO: 75% (376.1 GiB of 500.0 GiB) in 14m 48s, read: 1.7 GiB/s, write: 0 B/s
INFO: 76% (380.1 GiB of 500.0 GiB) in 15m 13s, read: 164.6 MiB/s, write: 64.5 MiB/s
INFO: 77% (385.1 GiB of 500.0 GiB) in 15m 45s, read: 159.0 MiB/s, write: 33.5 MiB/s
INFO: 78% (390.0 GiB of 500.0 GiB) in 16m 11s, read: 193.5 MiB/s, write: 4.2 MiB/s
INFO: 79% (395.2 GiB of 500.0 GiB) in 16m 38s, read: 195.1 MiB/s, write: 0 B/s
INFO: 80% (400.0 GiB of 500.0 GiB) in 17m 4s, read: 192.2 MiB/s, write: 0 B/s
INFO: 81% (406.1 GiB of 500.0 GiB) in 17m 18s, read: 444.0 MiB/s, write: 0 B/s
INFO: 82% (410.1 GiB of 500.0 GiB) in 17m 39s, read: 192.2 MiB/s, write: 7.6 MiB/s
INFO: 83% (415.2 GiB of 500.0 GiB) in 18m 5s, read: 202.0 MiB/s, write: 2.8 MiB/s
INFO: 84% (420.0 GiB of 500.0 GiB) in 18m 30s, read: 198.2 MiB/s, write: 0 B/s
INFO: 85% (425.0 GiB of 500.0 GiB) in 18m 56s, read: 196.2 MiB/s, write: 0 B/s
INFO: 86% (430.2 GiB of 500.0 GiB) in 19m 24s, read: 188.1 MiB/s, write: 0 B/s
INFO: 87% (435.1 GiB of 500.0 GiB) in 19m 50s, read: 195.7 MiB/s, write: 0 B/s
INFO: 88% (440.1 GiB of 500.0 GiB) in 20m 16s, read: 196.9 MiB/s, write: 0 B/s
INFO: 89% (445.8 GiB of 500.0 GiB) in 20m 34s, read: 320.4 MiB/s, write: 1.3 MiB/s
INFO: 90% (450.1 GiB of 500.0 GiB) in 20m 57s, read: 195.7 MiB/s, write: 0 B/s
INFO: 91% (455.8 GiB of 500.0 GiB) in 21m 5s, read: 720.5 MiB/s, write: 0 B/s
INFO: 92% (460.7 GiB of 500.0 GiB) in 21m 8s, read: 1.6 GiB/s, write: 0 B/s
INFO: 93% (465.5 GiB of 500.0 GiB) in 21m 11s, read: 1.6 GiB/s, write: 0 B/s
INFO: 94% (470.8 GiB of 500.0 GiB) in 21m 14s, read: 1.7 GiB/s, write: 0 B/s
INFO: 95% (475.7 GiB of 500.0 GiB) in 21m 17s, read: 1.7 GiB/s, write: 0 B/s
INFO: 96% (481.0 GiB of 500.0 GiB) in 21m 20s, read: 1.8 GiB/s, write: 0 B/s
INFO: 97% (486.1 GiB of 500.0 GiB) in 21m 23s, read: 1.7 GiB/s, write: 0 B/s
INFO: 98% (491.1 GiB of 500.0 GiB) in 21m 26s, read: 1.7 GiB/s, write: 0 B/s
INFO: 99% (496.0 GiB of 500.0 GiB) in 21m 29s, read: 1.6 GiB/s, write: 0 B/s
INFO: 100% (500.0 GiB of 500.0 GiB) in 21m 32s, read: 1.3 GiB/s, write: 0 B/s
INFO: backup is sparse: 276.89 GiB (55%) total zero data
INFO: backup was done incrementally, reused 486.59 GiB (97%)
INFO: transferred 500.00 GiB in 1292 seconds (396.3 MiB/s)
INFO: adding notes to backup
INFO: prune older backups with retention: keep-last=30
INFO: running 'proxmox-backup-client prune' for 'vm/1000'
INFO: pruned 0 backup(s)
INFO: Finished Backup of VM 1000 (01:24:34)
INFO: Backup finished at 2024-10-19 13:24:41
INFO: Backup job finished successfully
INFO: notified via target `mail-to-root`
TASK OK
** It is important to note that the CPU is spiking. cPanel also said that it seemed that the disk was being thrashed. I have checked the raid for disk failures as well as watched the physical disk I/o and its not solid... i have seen it alot worse when with vmware.
