Hi Community,
Recently we have encountered some performance degradation during LXC snaphost. We are trying to understand what is the root of this problem and are there any workarounds.
Brief intro.
We are running proxmox 7.1-12/b3c09de3 (running kernel: 5.13.19-6-pve)
Storage: RAID10 (4x 4TB SSD SAS) with lvm-thin on top.
There is an LXC with mysql db with moderate load of 30-50MB/s data writes. Two volumes are attached:
Every weekend we are running LXC snapshot of this instance, and every time in the backup finish phase when temporary vzdump snaphots are being deleted, high spike of IO Delay hits us, and workloads running on this server unfortunately get very slow.
Backup of this particular LXC takes about 10.5 h
We have tried to understand the nature of LVM2 snapshots, but different sources are a bit different on details.
There is a nice post about snapshots with thin volumes https://www.ilsistemista.net/index.php/linux-a-unix/46-lvm-thin-volume-explained.html?start=3 , but unfortunately it does not give complete view of what happening after snapshot has finishes. What does happen with snapshot metadata, original data; are there any merges; etc..
Could anyone give us at least a direction where to search for an answer, or maybe someone has encountered something similar?
Thanks!
Best regards,
Matt
Recently we have encountered some performance degradation during LXC snaphost. We are trying to understand what is the root of this problem and are there any workarounds.
Brief intro.
We are running proxmox 7.1-12/b3c09de3 (running kernel: 5.13.19-6-pve)
Storage: RAID10 (4x 4TB SSD SAS) with lvm-thin on top.
There is an LXC with mysql db with moderate load of 30-50MB/s data writes. Two volumes are attached:
Code:
rootfs: local-lvm:vm-1001-disk-0,size=600G (~100GB used)
mp0: local-lvm:vm-1001-disk-1,mp=/var/lib/mysql,backup=1,size=4000G (~2TB used)Every weekend we are running LXC snapshot of this instance, and every time in the backup finish phase when temporary vzdump snaphots are being deleted, high spike of IO Delay hits us, and workloads running on this server unfortunately get very slow.
Code:
INFO: cleanup temporary 'vzdump' snapshot
Logical volume "snap_vm-1001-disk-0_vzdump" successfully removed
Logical volume "snap_vm-1001-disk-1_vzdump" successfully removedBackup of this particular LXC takes about 10.5 h
We have tried to understand the nature of LVM2 snapshots, but different sources are a bit different on details.
There is a nice post about snapshots with thin volumes https://www.ilsistemista.net/index.php/linux-a-unix/46-lvm-thin-volume-explained.html?start=3 , but unfortunately it does not give complete view of what happening after snapshot has finishes. What does happen with snapshot metadata, original data; are there any merges; etc..
Could anyone give us at least a direction where to search for an answer, or maybe someone has encountered something similar?
Thanks!
Best regards,
Matt