Deleted files on VM connected to local-storage directory not deleted - Help!!!

[orenr]

Hello all,
I have 2 vm's with LOCAL-Storage on 2 proxmox nodes there are in cluster of 5.
Proxmox ve-6.4.8

My issue is:

The mentioned VM's are Elasticsearch data-nodes. feeded with the same data
Local Storage volume is 7T and presented as Directory Storage on each node they are on.

When deleting indexes in vm's lets say around 500GB, is not get reflected on local-Directory storage
Means if I saw it is 90% percentage use and the size is around (6.5T) so after deletion im expected the storage to be with 6T and approximaly around 80%

But im still seeing the local-directory storage dedicated to those vm's are on the same Size and percentage while in the VM's themselves i clearly see the disk-size on operation system is decreased and with lower percentage

Proxmox-node directory storage reference:



Operation System reference




What should i do?

I have tried thin-provisioning before and got the same thing that why switched to Directory

Appreciate your kind and help since it is a Mega Production sensitive Data.

Regards.
Oren

 

[Dunuin]

Did you verify that...:
1.) your guest OS uses TRIM/discard
2.) that "discard" is also enabled for each disk in your VMs hardware tab
3.) that your virtual disk controller uses something like virtio-SCSI that actually supports TRIM command
4.) that your physical disk controller supports TRIM

When using VMs on a directory storage you are using qcow2 and that is thin-provisioned too. So you need to use TRIM/discard or deleted stuff won't be deleted. And the filesystem your directory storage points to also needs to support hole-punching for qcow2.

 

[orenr]

those VM's guest OS are Centos7 so it should be able to use Trim

Server controllers - for ceph using p440ar (uses trim), local storages and OS controller p440 (support and uses trim as well)
My vm is uses 2 disks - operating system disk space with 60 GB that located on CEPH
and Second disks which is the Local Disk Directory Storage with all data

The discard wasn't enable on disks till now. Now are enabled with discard option both disks



yet seeing it taking affect.
Should it takes affect immediately or i need to perform more actions?

Regards,
Oren

 

[leesteken]

those VM's guest OS are Centos7 so it should be able to use Trim

Are the filesystems inside the VM configured to use discard/trim? What filesystems are you using? Do they support discard/trim? Are they mounted with discard on? This is not always enabled automatically, especially on older distro's.

Should it takes affect immediately or i need to perform more actions?

Does Centos7 have the fstrim command? That can be use to trigger a discard of unused disk blocks.

 

[orenr]

found the issue:

Since former system Linux admin was mounting second disk directly on dev and not on partition.
discard option wasn't able to find dev directory via OS

I had to trim all FS on vm OS for it to take affect. Now its ok, and reflect vm the right data percentage disk on local directory storage




Since it is a legacy i canot perform much and will have to manually trim every once in a while

 

[orenr]

ISSUE SOLVED!!

Thanks to :

Dunuin​

leesteken​

 

[Dunuin]

I setup my VMs with a daily or hourly crontab entry running fstrim -a to trim all the disk once per day or hour.

 

[orenr]

im deleting elastic indexes once in a while - it detects when it grows but not when deleted
so when i come to delete ill just run fstrim manually on the same opportunity

Thanx