vzdump backup in 8.2.2 takes almost 3 times longer

A

askprox

Member
Proxmox Subscriber
Mar 16, 2023
9
1
8
Hi.
I upgrated Proxmox from 8.1 to 8.2.2 I was surprised when backup of two VMs took almost 3 times longer.
I did not notice any other change.
1715881702339.png

Version 8.2.2. :
1715881890065.png

I noticed in logs diffrent speed in read and writing.
version 8.1.
100: 2024-05-07 23:30:36 INFO: 1% (2.0 GiB of 200.0 GiB) in 3s, read: 695.2 MiB/s, write: 500.1 MiB/s
100: 2024-05-07 23:30:49 INFO: 2% (4.0 GiB of 200.0 GiB) in 16s, read: 158.6 MiB/s, write: 157.4 MiB/s
100: 2024-05-07 23:31:00 INFO: 3% (6.1 GiB of 200.0 GiB) in 27s, read: 191.5 MiB/s, write: 190.5 MiB/s
100: 2024-05-07 23:31:13 INFO: 4% (8.1 GiB of 200.0 GiB) in 40s, read: 158.1 MiB/s, write: 155.8 MiB/s
100: 2024-05-07 23:31:24 INFO: 5% (10.1 GiB of 200.0 GiB) in 51s, read: 183.1 MiB/s, write: 175.4 MiB/s
100: 2024-05-07 23:31:35 INFO: 6% (12.1 GiB of 200.0 GiB) in 1m 2s, read: 185.9 MiB/s, write: 175.1 MiB/s
100: 2024-05-07 23:31:46 INFO: 7% (14.2 GiB of 200.0 GiB) in 1m 13s, read: 195.5 MiB/s, write: 189.3 MiB/s
100: 2024-05-07 23:31:56 INFO: 8% (16.1 GiB of 200.0 GiB) in 1m 23s, read: 192.6 MiB/s, write: 191.4 MiB/s
100: 2024-05-07 23:32:11 INFO: 9% (18.0 GiB of 200.0 GiB) in 1m 38s, read: 135.1 MiB/s, write: 133.5 MiB/s
100: 2024-05-07 23:32:24 INFO: 10% (20.1 GiB of 200.0 GiB) in 1m 51s, read: 161.3 MiB/s, write: 160.2 MiB/s
100: 2024-05-07 23:32:34 INFO: 11% (22.1 GiB of 200.0 GiB) in 2m 1s, read: 203.2 MiB/s, write: 199.5 MiB/s
100: 2024-05-07 23:32:45 INFO: 12% (24.0 GiB of 200.0 GiB) in 2m 12s, read: 180.6 MiB/s, write: 178.6 MiB/s
100: 2024-05-07 23:33:00 INFO: 13% (26.1 GiB of 200.0 GiB) in 2m 27s, read: 145.0 MiB/s, write: 144.2 MiB/s
100: 2024-05-07 23:33:12 INFO: 14% (28.1 GiB of 200.0 GiB) in 2m 39s, read: 165.8 MiB/s, write: 164.4 MiB/s
100: 2024-05-07 23:33:23 INFO: 15% (30.2 GiB of 200.0 GiB) in 2m 50s, read: 194.1 MiB/s, write: 190.7 MiB/s

version8.2.2
100: 2024-05-08 23:30:36 INFO: 0% (2.0 GiB of 200.0 GiB) in 3s, read: 666.3 MiB/s, write: 471.3 MiB/s
100: 2024-05-08 23:30:39 INFO: 1% (2.4 GiB of 200.0 GiB) in 6s, read: 154.1 MiB/s, write: 153.2 MiB/s
100: 2024-05-08 23:30:50 INFO: 2% (4.1 GiB of 200.0 GiB) in 17s, read: 159.2 MiB/s, write: 157.8 MiB/s
100: 2024-05-08 23:31:37 INFO: 3% (6.0 GiB of 200.0 GiB) in 1m 4s, read: 41.4 MiB/s, write: 41.3 MiB/s
100: 2024-05-08 23:32:21 INFO: 4% (8.1 GiB of 200.0 GiB) in 1m 48s, read: 48.3 MiB/s, write: 47.6 MiB/s
100: 2024-05-08 23:33:02 INFO: 5% (10.2 GiB of 200.0 GiB) in 2m 29s, read: 51.7 MiB/s, write: 49.2 MiB/s
100: 2024-05-08 23:33:34 INFO: 6% (12.0 GiB of 200.0 GiB) in 3m 1s, read: 59.6 MiB/s, write: 56.5 MiB/s
100: 2024-05-08 23:34:12 INFO: 7% (14.0 GiB of 200.0 GiB) in 3m 39s, read: 53.5 MiB/s, write: 51.8 MiB/s
100: 2024-05-08 23:34:52 INFO: 8% (16.0 GiB of 200.0 GiB) in 4m 19s, read: 51.1 MiB/s, write: 50.8 MiB/s
100: 2024-05-08 23:35:28 INFO: 9% (18.0 GiB of 200.0 GiB) in 4m 55s, read: 57.9 MiB/s, write: 57.2 MiB/s
100: 2024-05-08 23:36:08 INFO: 10% (20.0 GiB of 200.0 GiB) in 5m 35s, read: 50.4 MiB/s, write: 50.0 MiB/s
100: 2024-05-08 23:36:47 INFO: 11% (22.0 GiB of 200.0 GiB) in 6m 14s, read: 52.6 MiB/s, write: 51.7 MiB/s
100: 2024-05-08 23:37:27 INFO: 12% (24.0 GiB of 200.0 GiB) in 6m 54s, read: 51.1 MiB/s, write: 50.4 MiB/s
100: 2024-05-08 23:38:04 INFO: 13% (26.0 GiB of 200.0 GiB) in 7m 31s, read: 56.3 MiB/s, write: 56.0 MiB/s
100: 2024-05-08 23:38:35 INFO: 14% (28.0 GiB of 200.0 GiB) in 8m 2s, read: 65.4 MiB/s, write: 65.2 MiB/s
100: 2024-05-08 23:39:15 INFO: 15% (30.1 GiB of 200.0 GiB) in 8m 42s, read: 51.9 MiB/s, write: 50.8 MiB/s

HW configuration is same , there was no change.
Any idea what to check it?
Thank you for help.
Regards
Stanislav
 
Last edited:
Hi,
you could try booting into an older kernel to see if it's related to the kernel upgrade. What kind of storage are you using for the VM disks and the backup target? What type/file system and what physical models?

How does the load on the node look like during backup? Is there anything interesting in the system logs/journal around the time the issue happened?
 
Hi.

Thank you for your attention.

We are using LVM storage in Proxmox.
VMs are backuped to Synology – type SMB/CIFS
Proxmox is installed on HW - ProLiant ML350 Gen10
Proxmox system is installed on 300GB 12G SAS HDD

DATA(VM) are on 2TB 12G SAS HDD

1716115233414.png
1716115279889.png
The backup is running , just now.
I see that network traffic is high, but backup is running.
During the business as usual , no one from users complained.
1716115305672.png
1716115330064.png
1716115345328.png
Synology : DS415+

1716115493010.png
1716115503558.png
During the week , I will check all latest drivers, firmware’s and I will try to boot into an older kernel as a first.

Also I will upgrade another same server, but there is used ZFS. I can compare at least , the backup.

Thank you
Stanislav
 
  • Like
Reactions: Kingneutron
Hi.
I booted from Kernel 6.8.4.2. Backup of two VMs was same long.
Total running time: 8h 34min 20s
Total size: 803.092 GiB
Next week I will apply latest SP for HP server.
 

Attachments

  • kernel.png
    kernel.png
    469.3 KB · Views: 6
askprox said:
I booted from Kernel 6.8.4.2. Backup of two VMs was same long.
Click to expand...
Sorry, should have been more specific. It would've been interesting to see how it behaves on the latest 6.5 kernel, because 6.5 was the default kernel in Proxmox VE 8.1.
 
Hi,

I tried to boot with kernel 6.5.13.5. Proxmox booted, but the web GUI did not work, and pinging to Proxmox failed as well. I realized that when I updated to 8.2, I had to modify the interface file. You can see both versions. Could this be a problem with the backup? If I switch to 6.5.13.5 and change the interface file , should I expect any other issues?
Thank you, best regards. Stanislav
 

Attachments

Hi.
I booted with kernel 6.5.13.5, Before that I changed interace file as it was before update to 8.2.
I booted without problem. I tried to backup my VMs. Everything was running as before update.

Details​

VMID​
Name​
Status​
Time​
Size​
Filename​
100SERVER01ok22min 25s126.56 GiB/mnt/pve/BACKUPVMAS/dump/vzdump-qemu-100-2024_06_01-08_01_06.vma.zst
101SERVER02ok3h 8min 43s676.711 GiB/mnt/pve/BACKUPVMAS/dump/vzdump-qemu-101-2024_06_01-08_23_31.vma.zst

style='font-family:"Arial",sans-serif'>Total running time: 3h 31min 8s
style='font-family:"Arial",sans-serif'>Total size: 803.271 GiB

Than I changed interface file according to ling above and I created file in /etc/systemd/network/10-enwan0.link.
I booted in latest kernel 6.8.4.3, but but the web GUI did not work, and pinging to Proxmox failed as well.

I booted back to 6.5.13.5.
I see that new interface file is ''accepted'' .
1717238771815.png
I do not understand , why is not bootig in latest kernel.

Thank you for help.
 

Attachments

Hi.
Thank you for answer.
Before I tried in production server , I tested my solution posted above on my test server. There is only latest kernel.
It works. see attached files.
After I red your msg, I tried , respectively I wanted to try , bud I do not understand.

it should be like?

iface lo inet loopback
iface eno1 inet manual
iface eno2 inet manual
iface eno3 inet manual
iface eno4 inet manual

auto vmbr0
iface vmbr0 inet static
address 192.168.9.100/24
gateway 192.168.9.1
bridge-ports ???? enwan0
bridge-stp off
bridge-fd 0

source /etc/network/interfaces.d/*


Thank you for help.
 

Attachments

  • 8_2.jpg
    8_2.jpg
    71 KB · Views: 2
  • test.txt
    test.txt
    745 bytes · Views: 1
There is no problem to get back to previous configuration. Proxmox is already running.
But I have problem how to do correct setup. The interfaces file,what I atached in previous post, was according to you
correct?
 
i get similar backup speeds as your first example on ssd-zfs datastore i thought it might be because of compression or something?
 
Finally I fixed my problem.
I am using interface configuration which I used before upgrade to 8.2.2. Nothing changed in this file.
I created file in /etc/systemd/network/10-eno1.link with
[Match]
MACAddress=F6:9B:9F:21:4D:22

[Link]
Name=eno1

I made mistake in MAC addres. That's why I did not boot in 8.2.2 version.
After I corrected MAC, I booted and backup is running again like before upgrade.
Thanks for help.
 
You must log in or register to reply here.
Top Bottom