Wildly different read/write speeds on backup?

[C8Z06]

I have a 70GB VM disk that backs up nightly (suspend). I am having some odd issues (that I still am trying to wrap my head around) but I just noticed that the read and write speed is all over the place. This is writing to a mounted network share on a windows machine FYI.

What could cause this wild swing in read and write speed? How could I get to the bottom of it?

Craig










I noticed this in the syslog:

Dec 11 00:22:25 prox1 smartd[2510]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 64 to 67
Dec 11 00:22:25 prox1 smartd[2510]: Device: /dev/sdb [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 67 to 69

I have a pair of 1TB SSD's as boot drives in a RAID Z2, which is where this VM is stored.


Update: When the backup finally finished (it took a while after the copy was complete) I saw this happen in the syslog:

Dec 11 00:29:14 prox1 kernel: perf: interrupt took too long (2511 > 2500), lowering kernel.perf_event_max_sample_rate to 79500
Dec 11 00:31:00 prox1 kernel: kvm [4203]: ignored rdmsr: 0x10f data 0x0
Dec 11 00:31:00 prox1 kernel: kvm [4203]: ignored rdmsr: 0x123 data 0x0
Dec 11 00:31:00 prox1 kernel: kvm [4203]: ignored rdmsr: 0xc0011020 data 0x0
Dec 11 00:31:03 prox1 kernel: CIFS: VFS: cifs_invalidate_mapping: Could not invalidate inode 000000007de236e3
Dec 11 00:31:03 prox1 QEMU[4203]: kvm: terminating on signal 15 from pid 2506 (/usr/sbin/qmeventd)
Dec 11 00:31:03 prox1 pvedaemon[52139]: INFO: Finished Backup of VM 102 (00:22:45)

 

[rason]

It looks like there could be a few potential issues causing the wildly different read/write speeds during your VM backup. One possible cause could be the temperature of your SSDs. The messages in the syslog indicating a change in the airflow temperature of the drives could indicate that the drives are overheating, which can affect their performance.

Another potential cause could be the network share on the Windows machine. The messages in the syslog mentioning CIFS and VFS could indicate that there is some sort of issue with the network share, potentially leading to slower speeds.

I would recommend checking the temperature of your SSDs to ensure they are not overheating. You can also try backing up to a different location, such as a local drive, to see if that improves the performance. Additionally, checking the network connection and ensuring that it is stable and fast enough to handle the backup could also help.

Overall, it's important to identify the root cause of the issue in order to properly address it and ensure consistent and efficient backups for your VM.

 

[C8Z06]

Thanks a lot for the reply!

Yea this windows machine is the "old" system before I even got into ProxMox and it still has a big shared volume of disks. I have built a TrueNAS server with 100TB of storage and am using it for my Plex VM but have not yet moved the backups to write there. It is on my todo list, so I will see what happens when I do that. It did cross my mind to attempt to write to a different location to see what happened. I also wondered if I could / should write the backups locally and then copy them over to the NAS but I realized that makes no sense since that is what is happening as part of the backup job lol.

As for the hard drives, there are two drives in a raid z2 in the rear of a supermicro chassis. I am not sure what in the world I could do to cool them any further than they already are. It did just occur to me that I have ALL the hard drive caddies removed from the front of this machine. It seems odd that more free air flow would cause an issue but I just reinstalled all the caddies to see if it makes a difference. They needed to go in there for beautification anyway.