[SOLVED] Proxmox VE Clone and Backup Work Very Slow

[bedo]

Hello everyone,

I did a fresh install of Proxmox VE Version 7.1-7. Backup and clone functions do not work properly. I have a problem when I want to load a clone from a template or restore a backup file. The process is not responding and all virtual machines on the server are stopped. Even after waiting for 30 minutes, I can't get any results and the only way to solve this problem is to restart the server. Restarting the server also takes too long. A similar problem happens when I want to take snapshots.

Is this a known issue? My server is HP DL380 G6. You can see other information of my server below. Is there something I did wrong?

proxmox-ve: 7.1-1 (running kernel: 5.13.19-2-pve) pve-manager: 7.1-7 (running version: 7.1-7/df5740ad) pve-kernel-helper: 7.1-6 pve-kernel-5.13: 7.1-5 pve-kernel-5.13.19-2-pve: 5.13.19-4 ceph-fuse: 15.2.15-pve1 corosync: 3.1.5-pve2 criu: 3.15-1+pve-1 glusterfs-client: 9.2-1 ifupdown2: 3.1.0-1+pmx3 ksm-control-daemon: 1.4-1 libjs-extjs: 7.0.0-1 libknet1: 1.22-pve2 libproxmox-acme-perl: 1.4.0 libproxmox-backup-qemu0: 1.2.0-1 libpve-access-control: 7.1-5 libpve-apiclient-perl: 3.2-1 libpve-common-perl: 7.0-14 libpve-guest-common-perl: 4.0-3 libpve-http-server-perl: 4.0-4 libpve-storage-perl: 7.0-15 libspice-server1: 0.14.3-2.1 lvm2: 2.03.11-2.1 lxc-pve: 4.0.11-1 lxcfs: 4.0.11-pve1 novnc-pve: 1.2.0-3 proxmox-backup-client: 2.1.2-1 proxmox-backup-file-restore: 2.1.2-1 proxmox-mini-journalreader: 1.3-1 proxmox-widget-toolkit: 3.4-4 pve-cluster: 7.1-2 pve-container: 4.1-2 pve-docs: 7.1-2 pve-edk2-firmware: 3.20210831-2 pve-firewall: 4.2-5 pve-firmware: 3.3-3 pve-ha-manager: 3.3-1 pve-i18n: 2.6-2 pve-qemu-kvm: 6.1.0-3 pve-xtermjs: 4.12.0-1 qemu-server: 7.1-4 smartmontools: 7.2-1 spiceterm: 3.2-2 swtpm: 0.7.0~rc1+2 vncterm: 1.7-1 zfsutils-linux: 2.1.1-pve3

 

[mira]

That's a rather old machine.

Is there anything in the logs? Please provide the output of journalctl -b > journal.txt. It creates a file called journal.txt which you can then attach here.

Please run this command after the server has entered this strange state, meaning after a clone or backup.

 

[bedo]

Thank you very much for your reply. Will it be a problem if it's an old machine? I have attached the file you requested in my answer. Best regards..

 

[mira]

Please provide the VM config (qm config 100).

 

[bedo]

root@sanal:~# qm config 100
boot: order=ide0;net0
cores: 8
ide0: local-lvm:vm-100-disk-0,size=32G
memory: 16384
meta: creation-qemu=6.1.0,ctime=1639396808
name: FreeBSD-DB
net0: e1000=3A:CA:D6:D1:CD:91,bridge=vmbr0,firewall=1
numa: 0
ostype: other
scsihw: virtio-scsi-pci
smbios1: uuid=493ab454-9d01-41ea-aaf3-dc1797510a9c
sockets: 1
vmgenid: 9cbfb970-233a-4bf3-a5ee-5f1b5bfa039f

The clone process and backup restore process are still in progress. There are new error messages in the journal.txt file. I am sending the updated version. Best regards..

 

[mira]

Thank you for the updated journal.

Do you use a RAID controller? If so, is the firmware up-to-date?
is there an update for the BIOS available?

Please also provide the output of ps aux.
Maybe the processes are stuck on I/O.

You could try switching from `io_uring` to `threads` or `native` under `Advanced` when editing a disk.

 

[bedo]

Yes I am using RAID controller. The server is running with the HP Smart Array Raid Controller. I have a chance to reinstall the server. Do you recommend installing without RAID? Also, the firmware of the RAID controller is probably out of date.

 

[mira]

According to the `ps aux` output it seems multiple processes are stuck in `D` state -> uninterruptible sleep.
It seems only disk I/O processes are affected. Based on this I'd assume an issue with the RAID controller.
First I'd suggest switching from `io_uring` to `threads` and updating the firmware of the RAID controller, as well as the BIOS.

 

[bedo]

First I'd suggest switching from `io_uring` to `threads`

I'm very new to Proxmox VE. Can you provide more details for this process?

 

[mira]

As mentioned above, edit the disk in question (ide0 on VM 100), enable the advanced options at the bottom and then select `threads` for `Async IO`.

 

[bedo]

Thank you very much.. I did it..

 

[bedo]

I made the setting you said and took a backup of my virtual machine and restored it. Restore prosess was successful but took too long. The log output is attached. Best regards..

 

[bedo]

Hello everyone,
I noticed that this problem occurs if I install virtual machine disks on "local-lvm". So I set up a virtual machine on "local" to give it a try. When I took a backup of this virtual machine and then restored it, I saw that there was no problem. So I pulled my virtual machines onto "local" by deleting "local-lvm". Everything seems to be fine right now. I don't know what the reason is, but that's the case.
Best regards..