VM Move disk cleaup - Erase data slow

[harritz3257]

Hi.

Setup:

3 PVE (Running 8.3.3)
1 HP MSA 2040 FibreChannel storage.
- 2 volumes with SAS SSD's configures in RAID10

Running Multipath and LVM (Shared)

Performance is good and multipath is working as it should.

On to the issue..
When i Move Storage of a VMs Hard Disk ex. sata1 from MSA Volume 1 til Volume 2, PVE starts 2 jobs.

VM 100 - Move Disk - Output:
create full clone of drive sata1 (MSA-vg1:vm-100-disk-2)
Logical volume "vm-100-disk-1" created.
drive mirror is starting for drive-sata1
drive-sata1: transferred 0.0 B of 80.0 GiB (0.00%) in 0s
drive-sata1: transferred 602.0 MiB of 80.0 GiB (0.73%) in 1s
...
drive-sata1: transferred 80.0 GiB of 80.0 GiB (100.00%) in 1m 39s, ready
all 'mirror' jobs are ready
drive-sata1: Completing block job...
drive-sata1: Completed successfully.
drive-sata1: mirror-job finished
Renamed "vm-100-disk-2" to "del-vm-100-disk-2" in volume group "MSA_vg1"
TASK OK
This is done within 2 minutes, which is as expected.

Then PVE starts the job Erase data - Output:
zero-out data on image vm-101-disk-2 (/dev/MSA_vg1/del-vm-101-disk-2)
105906176 B 101.0 MB 10.00 s 10590534 B/s 10.10 MB/s
210763776 B 201.0 MB 20.00 s 10538079 B/s 10.05 MB/s
315621376 B 301.0 MB 30.00 s 10520609 B/s 10.03 MB/s
420478976 B 401.0 MB 40.00 s 10511874 B/s 10.02 MB/s
525336576 B 501.0 MB 50.00 s 10506641 B/s 10.02 MB/s
630919168 B 601.7 MB 60.00 s 10515212 B/s 10.03 MB/s
735776768 B 701.7 MB 70.00 s 10510990 B/s 10.02 MB/s
840634368 B 801.7 MB 80.00 s 10507826 B/s 10.02 MB/s
945491968 B 901.7 MB 90.00 s 10505365 B/s 10.02 MB/s
1050349568 B 1.0 GB 100.00 s 10503396 B/s 10.02 MB/s
1155207168 B 1.1 GB 110.00 s 10501785 B/s 10.02 MB/s
1260064768 B 1.2 GB 120.00 s 10500442 B/s 10.01 MB/s
1364922368 B 1.3 GB 130.00 s 10499307 B/s 10.01 MB/s
1469779968 B 1.4 GB 140.00 s 10498333 B/s 10.01 MB/s
1574637568 B 1.5 GB 150.00 s 10497492 B/s 10.01 MB/s
1679495168 B 1.6 GB 160.00 s 10496753 B/s 10.01 MB/s
1784352768 B 1.7 GB 170.00 s 10496101 B/s 10.01 MB/s
1889210368 B 1.8 GB 180.00 s 10495521 B/s 10.01 MB/s
1994067968 B 1.9 GB 190.00 s 10495003 B/s 10.01 MB/s
2098925568 B 2.0 GB 200.0 s (3:20 min) 10494537 B/s 10.01 MB/s

This i would guess, is intended to run as a background task, so it doesnt matter if it takes 2 minutes or 2 days..
Issue:
The issue is that the VM is locked while this is done, so while this is running for 2,5 hours, i cannot change settings, move storage or migrate the VM.
Example 1:
VM 100 - Move Storage - Output:
trying to acquire lock...
TASK ERROR: can't lock file '/var/lock/qemu-server/lock-100.conf' - got timeout

Example 2:
VM -> Hardware -> CD/DVD drive -> change to ISO
can't lock file '/var/lock/qemu-server/lock-100.conf' - got timeout (500)

Help:
Is there a solution to this?
A workaround could be to increase the 10MB/s zero-out data limit to 500MB/s, can this be done?

Regards Michael

 

[waltar]

msa2040fc: nr of disks and setup config ? Both volume in same raidset ?
Thought about doing 3 (1 active, 2 standby) nfs fileservers for your 3 pve config as rm 1 file is in msec and not 2,5h ... ?

 

[harritz3257]

>msa2040fc: nr of disks and setup config ? Both volume in same raidset ?
24 disks in total (3.84TB each)
2x 12 disk RAID 10. 1 LUN/Volume on each.

>Thought about doing 3 (1 active, 2 standby) nfs fileservers for your 3 pve config as rm 1 file is in msec and not 2,5h ... ?
Afraid not.. It's to advanced for us and I wouldnt trust it for production. NFS in a enterprise dual controller NAS would be okay, but not a homebrew.

 

[waltar]

Mmh, this 2040 is a >decade old and for a production cluster ... or you using it at home with 24x 4TB ssd's ... ? All 3 pve nodes direct connected or maybe 2 fc switche also ?

 

[harritz3257]

It is, but it's still under warranty and NBD replacement service from HPE.
3 nodes are going through a HP branded Brocade 16GB FC switch and the MSA is also being used for a VMware setup, which im moving away from to use Proxmox insted.

 

[WolfP]

Your system is set to not only delete files but to zero them out. Of course, if your write-rate for this job is set to 10MB/s, you will need roughly 2,5 hours to fill 80 GB with zeros.
I think zeroing deleted files - which is a questionable idea per se - should not be done at each delete, but by a background scrub process, as it otherwise unneccessarily impairs performance. And on a SSD you reduce lifetime to one half of the life without zeroing.

 

[waltar]

Your system is set to not only delete files but to zero them out.

Where could that setting be find ... in pve, in msa ??

 

[harritz3257]

Your system is set to not only delete files but to zero them out. Of course, if your write-rate for this job is set to 10MB/s, you will need roughly 2,5 hours to fill 80 GB with zeros.
I think zeroing deleted files - which is a questionable idea per se - should not be done at each delete, but by a background scrub process, as it otherwise unneccessarily impairs performance. And on a SSD you reduce lifetime to one half of the life without zeroing.

VMware has their own space reclaiming through VAAI UNMAP, which runs in the background.
As to reduced lifetime, i dont think it's an issue as the 3.84TB SSD's had 7.000TBW before failure, but Proxmox could insted prompt the user on thick provisioned hard disk creation to zero out of disk before usage.

 

[harritz3257]

Where could that setting be find ... in pve, in msa ??

I just did a quick test and PVE will not reclaim/free the space on the volume before Erase disk / Zero-out has been completed.

 

[waltar]

Just a further reason to mv from lvm to nfs-xfs (without doing any additional trim/discard and therefor life+speed degration) ...

 

[harritz3257]

i know, but that direction would require a big investment into blockbridge, pure or another unified storage.

 

[waltar]

That's your meaning as I just think about other ways of using the msa but anyway thinking about those future systems is again block type.

 

[harritz3257]

Just a quick update on this.

After a lot of seaching and testing i created a new LUN + LVM volume... and the new volume doesn't have the issue.. data is deleted within seconds..
So after a little investergating i found that there is a configuration difference between the old and new volume: "saferemove"

cat /etc/pve/storage.cfg
lvm: MSA-vg1
vgname MSA_vg1
content rootdir,images
saferemove 1
shared 1

lvm: MSA_test_lvm
vgname MSA-test
content rootdir,images
saferemove 0
shared 1

After a little googleing:
https://pve.proxmox.com/wiki/Storage:_LVM

saferemove
Called "Wipe Removed Volumes" in the web UI. Zero-out data when removing LVs. When removing a volume, this makes sure that all data gets erased and cannot be accessed by other LVs created later (which happen to be assigned the same physical extents). This is a costly operation, but may be required as a security measure in certain environments.

So my guess is that i selected that when i created the LVM volume and this is the reason why it takes so long.
When i delete a VM the space is reclaimed instantly on the test volume and there is no trace of the ID from Proxmox's web interface. My guess is that it would still be possible to recover the data from the lvm volume with a recovery tool.

I will remove the check in "Wipe Removed Volumes" and give it a test..