[PROBLEM] Backup from nvme to SATA hdd

O

onepamopa

Active Member
Dec 1, 2019
94
13
28
39
Hey guys,

I'm experiencing some issues backing up from NVME to a SATA hdd. The SATA drive just can't keep up.

Is it possible to implement a backup read/write speed limiter for similar situations?

Here's the log:
INFO: starting new backup job: vzdump 102 --compress zstd --node proxmox --mode stop --storage BACKUP_2TB --remove 0
INFO: Starting Backup of VM 102 (qemu)
INFO: Backup started at 2020-07-18 00:19:44
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: WWW
INFO: include disk 'virtio0' 'NVME:vm-102-disk-0' 300G
INFO: include disk 'efidisk0' 'NVME:vm-102-disk-1' 4M
INFO: creating vzdump archive '/mnt/pve/BACKUP_2TB/dump/vzdump-qemu-102-2020_07_18-00_19_44.vma.zst'
INFO: starting kvm to execute backup task
INFO: started backup task '6226f461-6cdf-473a-b1be-026a5145f2a2'
INFO: status: 0% (1.3 GiB of 300.0 GiB), duration 3, read: 453.1 MiB/s, write: 250.2 MiB/s
INFO: status: 1% (3.1 GiB of 300.0 GiB), duration 10, read: 253.2 MiB/s, write: 232.7 MiB/s
INFO: status: 2% (6.0 GiB of 300.0 GiB), duration 21, read: 278.1 MiB/s, write: 252.8 MiB/s
INFO: status: 3% (9.1 GiB of 300.0 GiB), duration 31, read: 311.0 MiB/s, write: 278.7 MiB/s
INFO: status: 4% (12.0 GiB of 300.0 GiB), duration 44, read: 232.6 MiB/s, write: 220.7 MiB/s
INFO: status: 5% (15.1 GiB of 300.0 GiB), duration 56, read: 262.8 MiB/s, write: 243.8 MiB/s
INFO: status: 6% (18.2 GiB of 300.0 GiB), duration 71, read: 211.5 MiB/s, write: 195.9 MiB/s
INFO: status: 7% (21.1 GiB of 300.0 GiB), duration 83, read: 248.9 MiB/s, write: 226.4 MiB/s
INFO: status: 8% (24.1 GiB of 300.0 GiB), duration 95, read: 255.3 MiB/s, write: 237.1 MiB/s
INFO: status: 9% (27.3 GiB of 300.0 GiB), duration 105, read: 320.4 MiB/s, write: 290.2 MiB/s
INFO: status: 10% (30.0 GiB of 300.0 GiB), duration 118, read: 217.2 MiB/s, write: 205.4 MiB/s
INFO: status: 11% (33.0 GiB of 300.0 GiB), duration 131, read: 238.6 MiB/s, write: 210.3 MiB/s
INFO: status: 12% (36.1 GiB of 300.0 GiB), duration 143, read: 262.9 MiB/s, write: 232.4 MiB/s
INFO: status: 13% (39.2 GiB of 300.0 GiB), duration 154, read: 287.7 MiB/s, write: 243.6 MiB/s
INFO: status: 14% (42.1 GiB of 300.0 GiB), duration 164, read: 296.5 MiB/s, write: 251.8 MiB/s
INFO: status: 15% (45.2 GiB of 300.0 GiB), duration 174, read: 321.2 MiB/s, write: 274.3 MiB/s
INFO: status: 16% (48.2 GiB of 300.0 GiB), duration 185, read: 279.5 MiB/s, write: 233.6 MiB/s
INFO: status: 17% (51.3 GiB of 300.0 GiB), duration 197, read: 259.8 MiB/s, write: 214.7 MiB/s
INFO: status: 18% (54.1 GiB of 300.0 GiB), duration 207, read: 287.5 MiB/s, write: 253.8 MiB/s
INFO: status: 19% (57.1 GiB of 300.0 GiB), duration 219, read: 260.4 MiB/s, write: 241.7 MiB/s
INFO: status: 20% (60.2 GiB of 300.0 GiB), duration 231, read: 258.3 MiB/s, write: 244.4 MiB/s
INFO: status: 21% (63.1 GiB of 300.0 GiB), duration 241, read: 299.8 MiB/s, write: 277.9 MiB/s
INFO: status: 22% (66.2 GiB of 300.0 GiB), duration 253, read: 263.1 MiB/s, write: 242.2 MiB/s
INFO: status: 23% (69.2 GiB of 300.0 GiB), duration 264, read: 276.4 MiB/s, write: 238.3 MiB/s
INFO: status: 24% (73.0 GiB of 300.0 GiB), duration 274, read: 397.3 MiB/s, write: 216.6 MiB/s
INFO: status: 25% (77.4 GiB of 300.0 GiB), duration 277, read: 1.5 GiB/s, write: 45.6 MiB/s
INFO: status: 27% (81.5 GiB of 300.0 GiB), duration 280, read: 1.4 GiB/s, write: 55.5 MiB/s
INFO: status: 28% (85.5 GiB of 300.0 GiB), duration 283, read: 1.3 GiB/s, write: 62.0 MiB/s
INFO: status: 29% (89.8 GiB of 300.0 GiB), duration 286, read: 1.4 GiB/s, write: 54.9 MiB/s
INFO: status: 31% (94.5 GiB of 300.0 GiB), duration 289, read: 1.6 GiB/s, write: 15.9 MiB/s
INFO: status: 32% (98.8 GiB of 300.0 GiB), duration 292, read: 1.4 GiB/s, write: 52.2 MiB/s
INFO: status: 34% (102.7 GiB of 300.0 GiB), duration 295, read: 1.3 GiB/s, write: 68.3 MiB/s
INFO: status: 35% (107.2 GiB of 300.0 GiB), duration 298, read: 1.5 GiB/s, write: 22.0 MiB/s
INFO: status: 37% (112.0 GiB of 300.0 GiB), duration 301, read: 1.6 GiB/s, write: 2.7 KiB/s
INFO: status: 38% (116.9 GiB of 300.0 GiB), duration 304, read: 1.6 GiB/s, write: 4.0 KiB/s
INFO: status: 40% (121.8 GiB of 300.0 GiB), duration 307, read: 1.6 GiB/s, write: 2.7 KiB/s
INFO: status: 42% (126.6 GiB of 300.0 GiB), duration 310, read: 1.6 GiB/s, write: 2.7 KiB/s
INFO: status: 43% (130.9 GiB of 300.0 GiB), duration 313, read: 1.4 GiB/s, write: 48.9 MiB/s
INFO: status: 44% (134.8 GiB of 300.0 GiB), duration 316, read: 1.3 GiB/s, write: 73.2 MiB/s
INFO: status: 46% (138.9 GiB of 300.0 GiB), duration 319, read: 1.4 GiB/s, write: 63.2 MiB/s
INFO: status: 47% (142.9 GiB of 300.0 GiB), duration 322, read: 1.3 GiB/s, write: 65.6 MiB/s
INFO: status: 49% (147.1 GiB of 300.0 GiB), duration 325, read: 1.4 GiB/s, write: 65.3 MiB/s
INFO: status: 50% (150.7 GiB of 300.0 GiB), duration 329, read: 923.3 MiB/s, write: 207.1 MiB/s
INFO: status: 51% (155.5 GiB of 300.0 GiB), duration 332, read: 1.6 GiB/s, write: 4.0 KiB/s
INFO: status: 53% (160.2 GiB of 300.0 GiB), duration 335, read: 1.6 GiB/s, write: 14.1 MiB/s
INFO: status: 55% (165.0 GiB of 300.0 GiB), duration 338, read: 1.6 GiB/s, write: 4.0 KiB/s
INFO: status: 56% (169.9 GiB of 300.0 GiB), duration 341, read: 1.6 GiB/s, write: 2.7 KiB/s
INFO: status: 58% (174.7 GiB of 300.0 GiB), duration 344, read: 1.6 GiB/s, write: 4.0 KiB/s
INFO: status: 59% (179.6 GiB of 300.0 GiB), duration 347, read: 1.6 GiB/s, write: 2.7 KiB/s
INFO: status: 61% (184.4 GiB of 300.0 GiB), duration 350, read: 1.6 GiB/s, write: 2.7 KiB/s
INFO: status: 63% (189.3 GiB of 300.0 GiB), duration 353, read: 1.6 GiB/s, write: 4.0 KiB/s
INFO: status: 64% (194.1 GiB of 300.0 GiB), duration 356, read: 1.6 GiB/s, write: 2.7 KiB/s
INFO: status: 66% (199.0 GiB of 300.0 GiB), duration 359, read: 1.6 GiB/s, write: 4.0 KiB/s
INFO: status: 67% (203.8 GiB of 300.0 GiB), duration 362, read: 1.6 GiB/s, write: 6.7 KiB/s
INFO: status: 69% (208.5 GiB of 300.0 GiB), duration 365, read: 1.6 GiB/s, write: 2.7 KiB/s
INFO: status: 71% (213.4 GiB of 300.0 GiB), duration 368, read: 1.6 GiB/s, write: 4.0 KiB/s
INFO: status: 72% (218.2 GiB of 300.0 GiB), duration 371, read: 1.6 GiB/s, write: 9.3 KiB/s
INFO: status: 74% (223.0 GiB of 300.0 GiB), duration 374, read: 1.6 GiB/s, write: 4.0 KiB/s
INFO: status: 75% (227.9 GiB of 300.0 GiB), duration 377, read: 1.6 GiB/s, write: 2.7 KiB/s
INFO: status: 77% (232.8 GiB of 300.0 GiB), duration 380, read: 1.6 GiB/s, write: 8.0 KiB/s
INFO: status: 79% (237.6 GiB of 300.0 GiB), duration 383, read: 1.6 GiB/s, write: 2.7 KiB/s
INFO: status: 80% (240.6 GiB of 300.0 GiB), duration 386, read: 1003.3 MiB/s, write: 137.0 MiB/s
INFO: status: 81% (243.3 GiB of 300.0 GiB), duration 389, read: 932.6 MiB/s, write: 159.0 MiB/s
INFO: status: 82% (246.9 GiB of 300.0 GiB), duration 395, read: 618.3 MiB/s, write: 199.6 MiB/s
INFO: status: 83% (249.1 GiB of 300.0 GiB), duration 399, read: 562.4 MiB/s, write: 212.4 MiB/s
INFO: status: 84% (252.9 GiB of 300.0 GiB), duration 404, read: 770.7 MiB/s, write: 142.6 MiB/s
INFO: status: 85% (255.3 GiB of 300.0 GiB), duration 407, read: 833.8 MiB/s, write: 165.8 MiB/s
INFO: status: 86% (258.4 GiB of 300.0 GiB), duration 410, read: 1.0 GiB/s, write: 107.2 MiB/s
INFO: status: 87% (262.4 GiB of 300.0 GiB), duration 413, read: 1.3 GiB/s, write: 66.3 MiB/s
INFO: status: 88% (265.2 GiB of 300.0 GiB), duration 416, read: 951.1 MiB/s, write: 148.5 MiB/s
INFO: status: 89% (268.5 GiB of 300.0 GiB), duration 419, read: 1.1 GiB/s, write: 82.7 MiB/s
INFO: status: 90% (272.1 GiB of 300.0 GiB), duration 422, read: 1.2 GiB/s, write: 97.5 MiB/s
INFO: status: 92% (276.3 GiB of 300.0 GiB), duration 425, read: 1.4 GiB/s, write: 55.9 MiB/s
INFO: status: 93% (280.7 GiB of 300.0 GiB), duration 428, read: 1.5 GiB/s, write: 43.4 MiB/s
INFO: status: 94% (284.0 GiB of 300.0 GiB), duration 431, read: 1.1 GiB/s, write: 103.1 MiB/s
INFO: status: 95% (286.9 GiB of 300.0 GiB), duration 434, read: 1005.6 MiB/s, write: 125.0 MiB/s
INFO: status: 96% (290.6 GiB of 300.0 GiB), duration 437, read: 1.2 GiB/s, write: 110.9 MiB/s
INFO: status: 98% (294.9 GiB of 300.0 GiB), duration 440, read: 1.4 GiB/s, write: 43.1 MiB/s
INFO: status: 99% (299.7 GiB of 300.0 GiB), duration 443, read: 1.6 GiB/s, write: 1.3 KiB/s
INFO: status: 100% (300.0 GiB of 300.0 GiB), duration 444, read: 310.2 MiB/s, write: 12.0 KiB/s
INFO: transferred 300.00 GiB in 444 seconds (691.9 MiB/s)
INFO: Backup is sparse: 75% (226.46 GiB) zero data
INFO: stopping kvm after backup task
INFO: archive file size: 39.49GB
trying to acquire lock...
OK
INFO: Finished Backup of VM 102 (00:07:28)
INFO: Backup finished at 2020-07-18 00:27:12
INFO: Backup job finished successfully
TASK OK
Click to expand...
 
H4R0 said:
Speed limit is already implemented see "bwlimit" https://pve.proxmox.com/pve-docs/chapter-vzdump.html

But your log looks fine, i dont see a problem. So what do you mean by "The SATA drive just can't keep up."
Click to expand...

Where do you see speed limit on backing up? I don't see such option @ the UI, there is only a limit on restore.

And where do I see a problem in the log:
- Normal - INFO: status: 23% (69.2 GiB of 300.0 GiB), duration 264, read: 276.4 MiB/s, write: 238.3 MiB/s
- A problem - INFO: status: 63% (189.3 GiB of 300.0 GiB), duration 353, read: 1.6 GiB/s, write: 4.0 KiB/s

Do you see it? :) And it is not actually a problem with writing, but maybe reading too fast from the nvme?
The experts would tell..

Actually, I found a "global" limit on "Backup restore" @ "Datacenter" -> Options. Testing now..
Aaaand no... it's for restore only.

So, as soon as read speed jumps... write speed goes down to a crawl.
 
Last edited:
Read the docs i linked, its not a ui option, you have to edit the vzdump conf file.

And there is nothing bad with your backup job, if you read lots of zeros and do compression you will read really fast but dont write at all.

This has to do with thin-provisioning.
 
  • Like
Reactions: onepamopa
H4R0 said:
Read the docs i linked, its not a ui option, you have to edit the vzdump conf file.

And there is nothing bad with your backup job, if you read lots of zeros and do compression you will read really fast but dont write at all.

This has to do with thin-provisioning.
Click to expand...

I'll check it tomorrow, it's 04:33 here @ the moment..
 
H4R0 said:
Read the docs i linked, its not a ui option, you have to edit the vzdump conf file.
Click to expand...

You can set the limits via GUI, see "Datacenter/Options/Bandwith Limits"
 
tom said:
You can set the limits via GUI, see "Datacenter/Options/Bandwith Limits"
Click to expand...

Setting any limits in "Datacenter/Options/Bandwith Limits" ---- does nothing, currently have:
Bandwidth Limits clone: 50.00 MiB/s,default: 50.00 MiB/s,move: 50.00 MiB/s,restore: 50.00 MiB/s

And this is during a backup (NOT RESTORE), doesn't seem the limits did anything:
INFO: status: 0% (1.3 GiB of 300.0 GiB), duration 3, read: 460.0 MiB/s, write: 257.2 MiB/s
INFO: status: 1% (3.0 GiB of 300.0 GiB), duration 10, read: 248.2 MiB/s, write: 228.0 MiB/s
INFO: status: 2% (6.1 GiB of 300.0 GiB), duration 21, read: 283.1 MiB/s, write: 257.7 MiB/s
INFO: status: 3% (9.3 GiB of 300.0 GiB), duration 31, read: 324.8 MiB/s, write: 292.2 MiB/s
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back