[SOLVED] Ionice priority

[Mikepop]

Hello, I know this can be configured cause I see it in logs, default is 7 but I cannot find where can it be changed.

Regards

 

[Stoiko Ivanov]

Where do you see it in the logs and what do you want to configure?

I'm not sure I get the question completely

 

[Mikepop]

Well, I know people wants backups done as soon as possible, but we have some serious iowait at the beginning of the backup process and we really do not mind to nice it higher and check if it's possible to wipe or reduce that iowait:

INFO: Starting Backup of VM 905 (qemu)
INFO: Backup started at 2021-06-25 02:09:30
INFO: status = running
INFO: VM Name: Cloud13
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/905/2021-06-25T00:09:30Z'
INFO: started backup task '51f0cbe3-5e2b-47d4-860d-33f23276e9cc'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: OK (77.5 GiB of 300.0 GiB dirty)
INFO: using fast incremental mode (dirty-bitmap), 77.5 GiB dirty of 300.0 GiB total
INFO: 0% (144.0 MiB of 77.5 GiB) in 3s, read: 48.0 MiB/s, write: 48.0 MiB/s
INFO: 1% (812.0 MiB of 77.5 GiB) in 1m 15s, read: 9.3 MiB/s, write: 9.3 MiB/s
INFO: 2% (1.6 GiB of 77.5 GiB) in 2m 27s, read: 11.3 MiB/s, write: 11.3 MiB/s
INFO: 3% (2.4 GiB of 77.5 GiB) in 4m, read: 8.5 MiB/s, write: 8.5 MiB/s
INFO: 4% (3.1 GiB of 77.5 GiB) in 5m 11s, read: 11.2 MiB/s, write: 11.2 MiB/s
INFO: 5% (3.9 GiB of 77.5 GiB) in 6m 41s, read: 9.1 MiB/s, write: 9.1 MiB/s
INFO: 6% (4.7 GiB of 77.5 GiB) in 7m 44s, read: 12.3 MiB/s, write: 12.3 MiB/s
INFO: 7% (5.4 GiB of 77.5 GiB) in 8m 47s, read: 12.3 MiB/s, write: 12.3 MiB/s
INFO: 8% (6.2 GiB of 77.5 GiB) in 10m 12s, read: 9.7 MiB/s, write: 9.7 MiB/s
INFO: 9% (7.0 GiB of 77.5 GiB) in 11m 43s, read: 8.7 MiB/s, write: 8.7 MiB/s
INFO: 10% (7.8 GiB of 77.5 GiB) in 12m 51s, read: 11.8 MiB/s, write: 11.8 MiB/s
INFO: 11% (8.6 GiB of 77.5 GiB) in 14m 37s, read: 7.5 MiB/s, write: 7.5 MiB/s
INFO: 12% (9.4 GiB of 77.5 GiB) in 15m 50s, read: 10.8 MiB/s, write: 10.8 MiB/s
INFO: 13% (10.1 GiB of 77.5 GiB) in 17m 22s, read: 8.4 MiB/s, write: 8.4 MiB/s
INFO: 14% (10.9 GiB of 77.5 GiB) in 18m 22s, read: 13.3 MiB/s, write: 13.3 MiB/s
INFO: 15% (11.6 GiB of 77.5 GiB) in 19m 34s, read: 10.8 MiB/s, write: 10.8 MiB/s
INFO: 16% (12.4 GiB of 77.5 GiB) in 20m 42s, read: 11.6 MiB/s, write: 11.6 MiB/s
INFO: 17% (13.2 GiB of 77.5 GiB) in 21m 53s, read: 11.2 MiB/s, write: 11.2 MiB/s
INFO: 18% (14.0 GiB of 77.5 GiB) in 23m 3s, read: 11.5 MiB/s, write: 11.5 MiB/s
INFO: 19% (14.8 GiB of 77.5 GiB) in 24m 18s, read: 10.7 MiB/s, write: 10.7 MiB/s
INFO: 20% (15.5 GiB of 77.5 GiB) in 25m 27s, read: 11.2 MiB/s, write: 11.2 MiB/s
INFO: 21% (16.3 GiB of 77.5 GiB) in 26m 30s, read: 13.0 MiB/s, write: 13.0 MiB/s
INFO: 22% (17.1 GiB of 77.5 GiB) in 27m 12s, read: 18.5 MiB/s, write: 18.5 MiB/s
INFO: 23% (17.9 GiB of 77.5 GiB) in 28m 43s, read: 9.8 MiB/s, write: 9.8 MiB/s
INFO: 24% (18.6 GiB of 77.5 GiB) in 29m 55s, read: 9.8 MiB/s, write: 9.8 MiB/s
INFO: 25% (19.4 GiB of 77.5 GiB) in 31m 19s, read: 9.4 MiB/s, write: 9.4 MiB/s
INFO: 26% (20.2 GiB of 77.5 GiB) in 32m 39s, read: 9.8 MiB/s, write: 9.8 MiB/s
INFO: 27% (21.0 GiB of 77.5 GiB) in 34m 3s, read: 9.9 MiB/s, write: 9.9 MiB/s
INFO: 28% (21.7 GiB of 77.5 GiB) in 34m 41s, read: 19.9 MiB/s, write: 19.9 MiB/s
INFO: 29% (22.5 GiB of 77.5 GiB) in 35m 44s, read: 12.8 MiB/s, write: 12.8 MiB/s
INFO: 30% (23.3 GiB of 77.5 GiB) in 36m 44s, read: 13.0 MiB/s, write: 12.9 MiB/s
INFO: 31% (24.1 GiB of 77.5 GiB) in 37m 33s, read: 16.6 MiB/s, write: 16.5 MiB/s
INFO: 32% (24.8 GiB of 77.5 GiB) in 38m 48s, read: 10.6 MiB/s, write: 10.6 MiB/s
INFO: 33% (25.6 GiB of 77.5 GiB) in 39m 57s, read: 11.4 MiB/s, write: 11.4 MiB/s
INFO: 34% (26.4 GiB of 77.5 GiB) in 40m 59s, read: 13.7 MiB/s, write: 13.7 MiB/s
INFO: 35% (27.2 GiB of 77.5 GiB) in 41m 59s, read: 12.8 MiB/s, write: 12.8 MiB/s
INFO: 36% (27.9 GiB of 77.5 GiB) in 42m 56s, read: 13.6 MiB/s, write: 13.5 MiB/s
INFO: 37% (28.7 GiB of 77.5 GiB) in 44m, read: 12.2 MiB/s, write: 12.0 MiB/s
INFO: 38% (29.5 GiB of 77.5 GiB) in 45m 22s, read: 10.4 MiB/s, write: 10.4 MiB/s
INFO: 39% (30.3 GiB of 77.5 GiB) in 46m 40s, read: 10.5 MiB/s, write: 10.5 MiB/s
INFO: 40% (31.0 GiB of 77.5 GiB) in 47m 32s, read: 13.8 MiB/s, write: 13.8 MiB/s
INFO: 41% (31.8 GiB of 77.5 GiB) in 48m 30s, read: 13.4 MiB/s, write: 13.4 MiB/s
INFO: 42% (32.6 GiB of 77.5 GiB) in 49m 14s, read: 18.0 MiB/s, write: 18.0 MiB/s
INFO: 43% (33.4 GiB of 77.5 GiB) in 50m 36s, read: 10.1 MiB/s, write: 10.1 MiB/s
INFO: 44% (34.1 GiB of 77.5 GiB) in 51m 29s, read: 14.6 MiB/s, write: 14.6 MiB/s
INFO: 45% (34.9 GiB of 77.5 GiB) in 52m 33s, read: 12.1 MiB/s, write: 12.1 MiB/s
INFO: 46% (35.7 GiB of 77.5 GiB) in 53m 50s, read: 10.8 MiB/s, write: 10.8 MiB/s
INFO: 47% (36.5 GiB of 77.5 GiB) in 54m 52s, read: 13.1 MiB/s, write: 13.0 MiB/s
INFO: 48% (37.2 GiB of 77.5 GiB) in 55m 40s, read: 15.8 MiB/s, write: 15.8 MiB/s
INFO: 49% (38.1 GiB of 77.5 GiB) in 56m 50s, read: 12.2 MiB/s, write: 12.1 MiB/s
INFO: 50% (38.8 GiB of 77.5 GiB) in 57m 18s, read: 25.6 MiB/s, write: 25.0 MiB/s
INFO: 51% (39.6 GiB of 77.5 GiB) in 58m 15s, read: 14.4 MiB/s, write: 14.2 MiB/s
INFO: 52% (40.4 GiB of 77.5 GiB) in 59m 46s, read: 9.1 MiB/s, write: 9.0 MiB/s
INFO: 53% (41.1 GiB of 77.5 GiB) in 1h 40s, read: 13.6 MiB/s, write: 12.3 MiB/s
INFO: 54% (41.9 GiB of 77.5 GiB) in 1h 2m 2s, read: 9.7 MiB/s, write: 9.7 MiB/s
INFO: 55% (42.7 GiB of 77.5 GiB) in 1h 2m 59s, read: 14.2 MiB/s, write: 14.2 MiB/s
INFO: 56% (43.5 GiB of 77.5 GiB) in 1h 4m 1s, read: 13.1 MiB/s, write: 12.4 MiB/s
INFO: 57% (44.2 GiB of 77.5 GiB) in 1h 4m 51s, read: 15.3 MiB/s, write: 15.2 MiB/s
INFO: 58% (45.0 GiB of 77.5 GiB) in 1h 6m 28s, read: 8.6 MiB/s, write: 8.6 MiB/s
INFO: 59% (45.8 GiB of 77.5 GiB) in 1h 7m 25s, read: 13.4 MiB/s, write: 11.8 MiB/s
INFO: 60% (46.5 GiB of 77.5 GiB) in 1h 8m 23s, read: 13.4 MiB/s, write: 13.4 MiB/s
INFO: 61% (47.3 GiB of 77.5 GiB) in 1h 9m 27s, read: 12.6 MiB/s, write: 12.4 MiB/s
INFO: 62% (48.1 GiB of 77.5 GiB) in 1h 10m 55s, read: 9.4 MiB/s, write: 9.3 MiB/s
INFO: 63% (48.9 GiB of 77.5 GiB) in 1h 11m 47s, read: 15.2 MiB/s, write: 14.5 MiB/s
INFO: 64% (49.7 GiB of 77.5 GiB) in 1h 12m 50s, read: 12.8 MiB/s, write: 12.3 MiB/s
INFO: 65% (50.4 GiB of 77.5 GiB) in 1h 13m 32s, read: 18.5 MiB/s, write: 16.9 MiB/s
INFO: 66% (51.2 GiB of 77.5 GiB) in 1h 14m 25s, read: 14.5 MiB/s, write: 14.2 MiB/s
INFO: 67% (52.0 GiB of 77.5 GiB) in 1h 15m 16s, read: 15.5 MiB/s, write: 15.1 MiB/s
INFO: 68% (52.7 GiB of 77.5 GiB) in 1h 16m 21s, read: 12.2 MiB/s, write: 12.1 MiB/s
INFO: 69% (53.5 GiB of 77.5 GiB) in 1h 17m 8s, read: 17.1 MiB/s, write: 16.8 MiB/s
INFO: 70% (54.3 GiB of 77.5 GiB) in 1h 17m 55s, read: 17.9 MiB/s, write: 17.1 MiB/s
INFO: 71% (55.1 GiB of 77.5 GiB) in 1h 18m 33s, read: 19.6 MiB/s, write: 19.6 MiB/s
INFO: 72% (55.9 GiB of 77.5 GiB) in 1h 19m 23s, read: 16.7 MiB/s, write: 14.9 MiB/s
INFO: 73% (56.6 GiB of 77.5 GiB) in 1h 20m 13s, read: 15.0 MiB/s, write: 14.9 MiB/s
INFO: 74% (57.4 GiB of 77.5 GiB) in 1h 21m 32s, read: 10.0 MiB/s, write: 9.8 MiB/s
INFO: 75% (58.2 GiB of 77.5 GiB) in 1h 22m 16s, read: 19.1 MiB/s, write: 18.5 MiB/s
INFO: 76% (58.9 GiB of 77.5 GiB) in 1h 23m, read: 17.2 MiB/s, write: 16.3 MiB/s
INFO: 77% (59.7 GiB of 77.5 GiB) in 1h 23m 59s, read: 13.6 MiB/s, write: 13.3 MiB/s
INFO: 78% (60.5 GiB of 77.5 GiB) in 1h 24m 47s, read: 16.8 MiB/s, write: 14.1 MiB/s
INFO: 79% (61.3 GiB of 77.5 GiB) in 1h 25m 23s, read: 21.3 MiB/s, write: 17.9 MiB/s
INFO: 80% (62.1 GiB of 77.5 GiB) in 1h 26m 41s, read: 10.4 MiB/s, write: 9.3 MiB/s
INFO: 81% (62.8 GiB of 77.5 GiB) in 1h 27m 19s, read: 20.9 MiB/s, write: 20.5 MiB/s
INFO: 82% (63.6 GiB of 77.5 GiB) in 1h 28m 5s, read: 17.3 MiB/s, write: 15.0 MiB/s
INFO: 83% (64.4 GiB of 77.5 GiB) in 1h 28m 52s, read: 16.4 MiB/s, write: 16.2 MiB/s
INFO: 84% (65.2 GiB of 77.5 GiB) in 1h 29m 56s, read: 12.7 MiB/s, write: 12.6 MiB/s
INFO: 85% (66.0 GiB of 77.5 GiB) in 1h 30m 40s, read: 18.6 MiB/s, write: 18.6 MiB/s
INFO: 86% (66.7 GiB of 77.5 GiB) in 1h 31m 34s, read: 14.4 MiB/s, write: 14.4 MiB/s
INFO: 87% (67.5 GiB of 77.5 GiB) in 1h 32m 23s, read: 16.1 MiB/s, write: 15.9 MiB/s
INFO: 88% (68.3 GiB of 77.5 GiB) in 1h 33m 47s, read: 9.7 MiB/s, write: 9.7 MiB/s
INFO: 89% (69.0 GiB of 77.5 GiB) in 1h 34m 29s, read: 17.9 MiB/s, write: 17.8 MiB/s
INFO: 90% (69.8 GiB of 77.5 GiB) in 1h 35m 8s, read: 21.1 MiB/s, write: 20.9 MiB/s
INFO: 91% (70.6 GiB of 77.5 GiB) in 1h 35m 59s, read: 15.2 MiB/s, write: 14.8 MiB/s
INFO: 92% (71.4 GiB of 77.5 GiB) in 1h 36m 39s, read: 20.1 MiB/s, write: 19.6 MiB/s
INFO: 93% (72.1 GiB of 77.5 GiB) in 1h 37m 19s, read: 19.3 MiB/s, write: 19.3 MiB/s
INFO: 94% (72.9 GiB of 77.5 GiB) in 1h 38m 12s, read: 15.2 MiB/s, write: 15.2 MiB/s
INFO: 95% (73.7 GiB of 77.5 GiB) in 1h 38m 52s, read: 20.0 MiB/s, write: 19.8 MiB/s
INFO: 96% (74.5 GiB of 77.5 GiB) in 1h 39m 47s, read: 14.3 MiB/s, write: 14.3 MiB/s
INFO: 97% (75.2 GiB of 77.5 GiB) in 1h 40m 41s, read: 14.8 MiB/s, write: 14.7 MiB/s
INFO: 98% (76.0 GiB of 77.5 GiB) in 1h 41m 32s, read: 15.4 MiB/s, write: 14.9 MiB/s
INFO: 99% (76.8 GiB of 77.5 GiB) in 1h 42m 28s, read: 15.2 MiB/s, write: 14.5 MiB/s
INFO: 100% (77.5 GiB of 77.5 GiB) in 1h 43m 31s, read: 11.5 MiB/s, write: 11.4 MiB/s
INFO: backup was done incrementally, reused 223.79 GiB (74%)
INFO: transferred 77.54 GiB in 6232 seconds (12.7 MiB/s)
INFO: Finished Backup of VM 905 (01:44:25)
INFO: Backup finished at 2021-06-25 03:53:55


Another usefull feature will be to add a delay per node to start the backups or any other feature so not every node start backups at the same time.
We do the backups over a 10Gb link at a 2 or 3 hours rate per node to mechanical disks cabins. We'd preffer a longer backups and less impact on reading VM disks and less or none iowait in the VMs if possible. We use a dual 40 gb bonding for ceph and hba and pm863 disks.

Regards

 

[Stoiko Ivanov]

The ionice priority and class are only honored with the 'cfq' io-scheduler, which, to my knowledge, is not commonly used these days:
* you can check the io-scheduler of a block-device with `cat /sys/block/<blockdevice>/queue/scheduler`
* In theory you can configure the ionice priority in /etc/vzdump.conf - but as said - I doubt that it will help on a current installation

I would try to set a bandwidth limit on Backup/Restore (GUI->Datacenter->Options) - but keep in mind that this will make your backups take longer, which in turn can have negative effects on the VM performance.

I hope this helps!

 

[Mikepop]

Thanks Stoiko, I though vzdump.conf only affected to vzdump, not to pbs. I'd preffer do not limit backup restore process, we will end changing manually with every restore process, but I guess I can give it a try just to test it
The origin of the problem to my understanding it's the hole VM disk it's read to check for the different chunks and there is where all the iowait it's generated. I do not know if there is another way to address the problem different to nice or limit the backup process or use a hole flash storage PBS.

Regards

 

[Mikepop]

There is a new backup option, exclude selected VMs that solve this problem.

Regards

 

[SamTzu]

Where is this option?

 

[SamTzu]

This appears to be Intel firmware issue.
https://forum.proxmox.com/.../proxmox-random-reboots.../...