Hi everyone, i have a problem with my backups. My speeds are slow, even when the backups are on local disk or NFS.
this is my info:
And the backups look like this:
I don't see usage on server when it works.. any ideea what can be wrong?
this is my info:
Code:
root@server:~# fio --size=9G --bs=64k --rw=write --direct=1 --runtime=60 --name=64kwrite --group_reporting
64kwrite: (g=0): rw=write, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
64kwrite: Laying out IO file (1 file / 9216MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=499MiB/s][w=7981 IOPS][eta 00m:00s]
64kwrite: (groupid=0, jobs=1): err= 0: pid=9450: Sat Mar 12 11:29:48 2022
write: IOPS=7936, BW=496MiB/s (520MB/s)(9216MiB/18580msec); 0 zone resets
clat (usec): min=77, max=173860, avg=122.59, stdev=482.04
lat (usec): min=77, max=173861, avg=123.75, stdev=482.04
clat percentiles (usec):
| 1.00th=[ 82], 5.00th=[ 86], 10.00th=[ 90], 20.00th=[ 102],
| 30.00th=[ 109], 40.00th=[ 114], 50.00th=[ 120], 60.00th=[ 124],
| 70.00th=[ 128], 80.00th=[ 135], 90.00th=[ 143], 95.00th=[ 151],
| 99.00th=[ 172], 99.50th=[ 190], 99.90th=[ 676], 99.95th=[ 717],
| 99.99th=[ 8291]
bw ( KiB/s): min=345344, max=609408, per=99.99%, avg=507869.41, stdev=38660.36, samples=37
iops : min= 5396, max= 9522, avg=7935.46, stdev=604.07, samples=37
lat (usec) : 100=17.66%, 250=81.97%, 500=0.20%, 750=0.13%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 250=0.01%
cpu : usr=4.25%, sys=24.31%, ctx=147465, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,147456,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=496MiB/s (520MB/s), 496MiB/s-496MiB/s (520MB/s-520MB/s), io=9216MiB (9664MB), run=18580-18580msec
Disk stats (read/write):
dm-1: ios=44/145633, merge=0/0, ticks=4472/13288, in_queue=17760, util=99.51%, aggrios=68/147623, aggrmerge=0/112, aggrticks=23896/14611, aggrin_queue=23996, aggrutil=99.45%
sda: ios=68/147623, merge=0/112, ticks=23896/14611, in_queue=23996, util=99.45%
Code:
root@server:~# pveperf
CPU BOGOMIPS: 136020.20
REGEX/SECOND: 1262223
HD SIZE: 93.99 GB (/dev/mapper/pve-root)
BUFFERED READS: 452.26 MB/sec
AVERAGE SEEK TIME: 7.06 ms
FSYNCS/SECOND: 3824.85
DNS EXT: 29.03 ms
Code:
INFO: starting new backup job: vzdump 103 --node server --storage NAS-1 --compress zstd --mode stop --remove 0
INFO: Starting Backup of VM 103 (qemu)
INFO: Backup started at 2022-03-12 11:35:45
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: ubuntuNagios
INFO: include disk 'sata0' 'local-lvm:vm-103-disk-0' 100G
INFO: creating vzdump archive '/mnt/pve/NAS-1/dump/vzdump-qemu-103-2022_03_12-11_35_45.vma.zst'
INFO: starting kvm to execute backup task
INFO: started backup task '347791da-510e-4018-ae09-f25185814d21'
INFO: 0% (337.1 MiB of 100.0 GiB) in 3s, read: 112.4 MiB/s, write: 37.8 MiB/s
INFO: 1% (1.2 GiB of 100.0 GiB) in 6s, read: 286.4 MiB/s, write: 38.9 MiB/s
INFO: 2% (2.1 GiB of 100.0 GiB) in 12s, read: 163.7 MiB/s, write: 11.5 MiB/s
INFO: 4% (4.5 GiB of 100.0 GiB) in 15s, read: 818.4 MiB/s, write: 57.3 KiB/s
INFO: 5% (5.9 GiB of 100.0 GiB) in 18s, read: 481.9 MiB/s, write: 22.9 MiB/s
INFO: 6% (6.3 GiB of 100.0 GiB) in 21s, read: 138.2 MiB/s, write: 31.9 MiB/s
INFO: 7% (7.1 GiB of 100.0 GiB) in 27s, read: 124.2 MiB/s, write: 57.5 MiB/s
INFO: 8% (8.7 GiB of 100.0 GiB) in 30s, read: 543.7 MiB/s, write: 13.6 MiB/s
INFO: 10% (10.2 GiB of 100.0 GiB) in 33s, read: 516.0 MiB/s, write: 16.9 MiB/s
INFO: 11% (12.0 GiB of 100.0 GiB) in 36s, read: 607.3 MiB/s, write: 13.1 MiB/s
INFO: 13% (13.6 GiB of 100.0 GiB) in 39s, read: 562.9 MiB/s, write: 12.1 MiB/s
INFO: 15% (15.5 GiB of 100.0 GiB) in 42s, read: 660.6 MiB/s, write: 4.4 MiB/s
INFO: 18% (18.0 GiB of 100.0 GiB) in 45s, read: 852.6 MiB/s, write: 2.9 MiB/s
INFO: 20% (20.1 GiB of 100.0 GiB) in 48s, read: 715.2 MiB/s, write: 3.6 MiB/s
INFO: 22% (22.7 GiB of 100.0 GiB) in 51s, read: 883.0 MiB/s, write: 1.7 MiB/s
INFO: 25% (25.4 GiB of 100.0 GiB) in 54s, read: 927.2 MiB/s, write: 561.3 KiB/s
INFO: 28% (28.0 GiB of 100.0 GiB) in 57s, read: 885.8 MiB/s, write: 1.3 MiB/s
INFO: 30% (30.8 GiB of 100.0 GiB) in 1m, read: 947.5 MiB/s, write: 642.7 KiB/s
INFO: 32% (32.7 GiB of 100.0 GiB) in 1m 3s, read: 645.5 MiB/s, write: 4.2 MiB/s
INFO: 35% (35.3 GiB of 100.0 GiB) in 1m 6s, read: 879.4 MiB/s, write: 453.3 KiB/s
INFO: 37% (37.9 GiB of 100.0 GiB) in 1m 9s, read: 896.8 MiB/s, write: 12.0 KiB/s
INFO: 40% (40.4 GiB of 100.0 GiB) in 1m 12s, read: 864.0 MiB/s, write: 3.8 MiB/s
INFO: 42% (42.9 GiB of 100.0 GiB) in 1m 15s, read: 838.1 MiB/s, write: 20.0 KiB/s
INFO: 45% (45.3 GiB of 100.0 GiB) in 1m 18s, read: 818.0 MiB/s, write: 553.3 KiB/s
INFO: 47% (47.5 GiB of 100.0 GiB) in 1m 21s, read: 760.2 MiB/s, write: 1.3 KiB/s
INFO: 48% (48.3 GiB of 100.0 GiB) in 1m 24s, read: 279.1 MiB/s, write: 68.8 MiB/s
INFO: 49% (49.0 GiB of 100.0 GiB) in 1m 28s, read: 182.7 MiB/s, write: 76.4 MiB/s
INFO: 51% (51.2 GiB of 100.0 GiB) in 1m 31s, read: 733.5 MiB/s, write: 5.3 KiB/s
INFO: 53% (53.6 GiB of 100.0 GiB) in 1m 34s, read: 819.5 MiB/s, write: 38.7 KiB/s
INFO: 56% (56.3 GiB of 100.0 GiB) in 1m 37s, read: 919.9 MiB/s, write: 3.7 MiB/s
INFO: 59% (59.3 GiB of 100.0 GiB) in 1m 40s, read: 1.0 GiB/s, write: 1.3 KiB/s
INFO: 62% (62.0 GiB of 100.0 GiB) in 1m 43s, read: 925.1 MiB/s, write: 10.7 KiB/s
INFO: 63% (63.2 GiB of 100.0 GiB) in 1m 50s, read: 181.8 MiB/s, write: 28.5 MiB/s
INFO: 64% (64.2 GiB of 100.0 GiB) in 1m 53s, read: 337.6 MiB/s, write: 27.0 MiB/s
INFO: 65% (65.3 GiB of 100.0 GiB) in 1m 56s, read: 360.4 MiB/s, write: 40.6 MiB/s
INFO: 66% (66.3 GiB of 100.0 GiB) in 1m 59s, read: 357.0 MiB/s, write: 19.9 MiB/s
INFO: 68% (68.2 GiB of 100.0 GiB) in 2m 2s, read: 620.2 MiB/s, write: 7.8 MiB/s
INFO: 69% (69.3 GiB of 100.0 GiB) in 2m 5s, read: 408.4 MiB/s, write: 25.4 MiB/s
INFO: 70% (70.3 GiB of 100.0 GiB) in 2m 8s, read: 317.5 MiB/s, write: 29.7 MiB/s
INFO: 72% (72.2 GiB of 100.0 GiB) in 2m 11s, read: 640.2 MiB/s, write: 9.0 MiB/s
INFO: 74% (74.0 GiB of 100.0 GiB) in 2m 14s, read: 633.6 MiB/s, write: 8.1 MiB/s
INFO: 75% (75.5 GiB of 100.0 GiB) in 2m 17s, read: 500.2 MiB/s, write: 18.0 MiB/s
INFO: 76% (76.2 GiB of 100.0 GiB) in 2m 20s, read: 239.3 MiB/s, write: 16.8 MiB/s
INFO: 77% (77.4 GiB of 100.0 GiB) in 2m 23s, read: 421.0 MiB/s, write: 23.6 MiB/s
INFO: 78% (78.2 GiB of 100.0 GiB) in 2m 26s, read: 271.6 MiB/s, write: 22.9 MiB/s
INFO: 80% (80.0 GiB of 100.0 GiB) in 2m 29s, read: 624.5 MiB/s, write: 15.6 MiB/s
INFO: 81% (81.9 GiB of 100.0 GiB) in 2m 32s, read: 626.2 MiB/s, write: 12.1 MiB/s
INFO: 82% (82.9 GiB of 100.0 GiB) in 2m 35s, read: 359.3 MiB/s, write: 21.8 MiB/s
INFO: 84% (84.9 GiB of 100.0 GiB) in 2m 38s, read: 681.8 MiB/s, write: 5.5 MiB/s
INFO: 86% (86.2 GiB of 100.0 GiB) in 2m 41s, read: 444.0 MiB/s, write: 10.8 MiB/s
INFO: 87% (87.0 GiB of 100.0 GiB) in 2m 48s, read: 115.3 MiB/s, write: 48.6 MiB/s
INFO: 88% (88.2 GiB of 100.0 GiB) in 2m 51s, read: 400.7 MiB/s, write: 25.1 MiB/s
INFO: 89% (89.7 GiB of 100.0 GiB) in 3m 2s, read: 145.0 MiB/s, write: 47.7 MiB/s
INFO: 90% (90.4 GiB of 100.0 GiB) in 3m 5s, read: 212.8 MiB/s, write: 17.7 MiB/s
INFO: 91% (91.1 GiB of 100.0 GiB) in 3m 10s, read: 151.8 MiB/s, write: 28.0 MiB/s
INFO: 92% (92.0 GiB of 100.0 GiB) in 3m 21s, read: 83.4 MiB/s, write: 55.9 MiB/s
INFO: 93% (93.0 GiB of 100.0 GiB) in 3m 36s, read: 70.6 MiB/s, write: 48.4 MiB/s
INFO: 94% (94.1 GiB of 100.0 GiB) in 3m 50s, read: 74.2 MiB/s, write: 57.6 MiB/s
INFO: 95% (95.0 GiB of 100.0 GiB) in 4m 5s, read: 64.9 MiB/s, write: 50.5 MiB/s
INFO: 96% (96.0 GiB of 100.0 GiB) in 4m 22s, read: 62.2 MiB/s, write: 52.8 MiB/s
INFO: 97% (97.0 GiB of 100.0 GiB) in 4m 33s, read: 91.2 MiB/s, write: 61.1 MiB/s
INFO: 98% (98.0 GiB of 100.0 GiB) in 4m 44s, read: 95.3 MiB/s, write: 64.6 MiB/s
INFO: 99% (99.2 GiB of 100.0 GiB) in 4m 50s, read: 198.9 MiB/s, write: 50.3 MiB/s
INFO: 100% (100.0 GiB of 100.0 GiB) in 4m 52s, read: 408.6 MiB/s, write: 4.0 KiB/s
INFO: backup is sparse: 90.88 GiB (90%) total zero data
INFO: transferred 100.00 GiB in 292 seconds (350.7 MiB/s)
INFO: stopping kvm after backup task
INFO: archive file size: 2.16GB
INFO: Finished Backup of VM 103 (00:04:56)
INFO: Backup finished at 2022-03-12 11:40:41
INFO: Backup job finished successfully
TASK OK
I don't see usage on server when it works.. any ideea what can be wrong?
Last edited: