Search results

When target host doesn't have enough free memory, migration process doesn't stop. It tries to allocate memory on the target host, which result in OOM killing the largest VM, and another, and another. In earlier versions of Proxmox VE it wasn't like that. Migration was giving an error "cannot...

I've tried it and posted results here: https://forum.proxmox.com/threads/backup-speed-limited-to-1-gbps.74209/#post-331218 Backup without compression gives up to 3 Gbps.

root@adm61:/home/avn# dd if=/dev/vms4/vm-112-disk-0 of=/dev/null bs=1M status=progress 53406072832 bytes (53 GB, 50 GiB) copied, 146 s, 366 MB/s 51200+0 records in 51200+0 records out 53687091200 bytes (54 GB, 50 GiB) copied, 149.697 s, 359 MB/s root@adm61:/home/avn# dd...

I didn't said read speed is unlimited. But it's clearly higher than 1 Gbps. Without compression backup gives up to 3 Gbps (as you can see in my previous posts). Lzo and pigz also performing ~1.5 times better than zstd. Edit: Ok, let's check iperf...

So, since the last post I've enabled multithread for zstd (set 'zstd: 0' in /etc/vzdump.conf). Now zstd can use up to half available cores instead of one. But the setting hardly changed anything. Backup performance with zstd is still worse than any other compression algorithms. It's rarely...

You're right. Only zstd compression limited to 1 Gbps: 1 - lzo: max 1.5 Gbps 2 - no compression: max 2.85 Gbps 3 - pigz: max 1.7 Gbps 4 - zstd: max 0.98 Gbps No bandwidth limits set in the GUI or vzdump.conf.

NAS used only for backups. All VMs and hosts run from SAS storage. Between hosts and NAS - no. NAS is FreeNAS (based on FreeBSD). But there's no issue with NAS. As I said, write speed to NAS is not limited. Multiple hosts can write backups up to 3 Gbps, 1 Gbps each. Also, as you can see...

Hi, This is how backup and resore looks like in 10 Gbps network: As you can see, backup speed limited at exactly 1 Gbps. Question: Why backup speed limited and how to remove this limit? PVE version: 6.2-10; VMs on SAS storage, storage type: LVM; Read speed from VM storage not limited...

Ismael, good thought. Maybe I just need to set errors=continue. I have backups of system volumes anyway.

Multipath, dmsetup, pvs, vgs, lvs don't show anything out of ordinary. All seems ok. Dmesg and jornalctl show that multipath lost one path, then another. FS was remounted read-only, then both paths were restored. Here is cut from journal: янв 15 04:30:38 adm55 kernel: hpsa 0000:86:00.0: scsi...

I agree, apparently when HDD dies, there's some critical delay in disk IO. And it's not FS switched to RO by itself, but because underlying block device became write-protected. I don't know what precisely causing this - device mapper, multipathd, or LVM. Removing HDD will lead to another...

All PVE nodes connected to HPE storage via SAS. Two SAS links per node, one for each controller on the storage, for redundancy. Multipath configured on all nodes.

Hello, We have Proxmox cluster with shared storage HPE MSA 2050. Nodes don't have local disks, so root file system located on the storage. Not every time, but rarely when hard drive on the storage dies, root file system may become read-only on one of the nodes (it's different nodes, not the...

I found another way to restore backup speed. It's enough to restart just one node, even node without any VMs. Even node, connected to the storage through SAS switch (so storage doesn't know anything about restart). Speed increases on every node in the cluster exactly when PVE on restarted node...

So. I moved whole cluster to MSA2040. It didn't help. The problem still there. About caching - I've tested read speed by running command "dd if=/dev/vmsX/vm-XXX-disk-X of=/dev/null status=progress" from physical nodes for many virtual machines. The results always multiple times higher than...

Alwin, thank you for your suggestions. I've checked old backup logs and found out that the problem started when we installed MSA2050 instead of previous storage MSA2040. We will try to revert back to MSA2040 and see if the problem remains. It will take time, but I will post results here.

Overall transfer rate go down as well, because backup generate most of the data tranfer. But I can manually initiate copy of some huge file or block device to /dev/null inside VMs, and in that case even when backup already slowed down, copy run at fast speed, hundreds MB/s. Maybe, but we...

New thread: https://forum.proxmox.com/threads/backup-slowing-down.48375/

Hello, I have a similar (or same) problem as in this topic: https://forum.proxmox.com/threads/backup-slowing-down-to-a-crawl.44870. I was asked to create a new thread, because the setup is different. In my case the storage is HPE MSA 2050 (attached by SAS). There's 4 LUNs where VMs located as...

Hello, I have a similar (or same) problem. In my case the storage is HPE MSA 2050 (attached by SAS). There's 4 LUNs where VMs located as LVM, shared between multiple nodes. Backup (snapshot mode) started weekly, and at first performance is great, it's 3-digit numbers. Pure read of sparse data...