.raw disk files of containers keep it's size after deleting files inside the container

[limone]

Hi,
this is a (kind of a) continuation of this thread: https://forum.proxmox.com/threads/container-shows-wrong-disk-usage.37429/

But to keep things clear and because of a new problem I open a new topic.

So the old problem was, that whenever I delete files inside a container the raw disk file of the container itself does not shrink and that is a problem.

The last threads solution was to mount every container and do and fstrim (or if the container is not unprivileged you can run fstrim directly in the container), but to keep things clear I wrote myself this script:

#!/bin/bash
vmid=$(ls /etc/pve/nodes/*nodename*/lxc/)
list=$(echo "$vmid" | sed -r 's/.conf//g')

for i in $list
do
        pct mount $i
done

for j in $list
do
        fstrim /var/lib/lxc/$j/rootfs
done

for k in $list
do
        pct unmount $k
done

I think it's very simple but works good, mounts all containers, do fstrim and after that unmount.

BUT, I've ran out of disk space so I used a NFS share for some containers disk images.
So far so good, but as it's the same, just the .raw disk images saved on the NFS server, I get the same problem: delete a file in a container and the disk space will be still used.
Ah, this aint no problem, right? Just do fstrim like always!

root@pve:~# pct mount 102
root@pve:~# fstrim /var/lib/lxc/102/rootfs
fstrim: /var/lib/lxc/102/rootfs: the discard operation is not supported

So it looks like fstrim is not supported on nfs shares. My only option would be to mount the disk inside the nfs server and then do fstrim. But as the NFS server is no proxmox node I cant simply do pct mount.

So I'm really screwed now, any idea how to solve my problem?

 

[LnxBil]

raw images do not reclaim free space and it is the raw big space (only thin provisioned as a sparse file) - hence the name RAW. Please use a filesystem that supports TRIM in the way you want, e.g. ZFS, LVM (to some extend) and QCOW2 (to some extend).

 

[limone]

And how to do that? In the webinterface you cannot choose a filesystem.
And what does LVM has to do with this? I cannot add more space to the proxmox node itself, only space which gets applied over the network (like nfs)

 

[LnxBil]

You have to change the storage backed to e.g. ZFS to achieve this.
You can install PVE inside of PVE with various backend devices and test if everything works.

 

[limone]

I think I cannot run pve inside pve because I run proxmox inside a kvm, because I only need containers. That kvm has nested virtualization disabled.

Or can you install pve inside a container?
If so, I still can't follow your mind, what exactly do I have to do?

 

[LnxBil]

You're currently running your PVE inside of another KVM? If so, just create another kvm PVE instance with ZFS and play around if it fits your need with respect to reclaiming space with trimming - yet if you use ZFS for a container, you already have it, because you do not need to trim a ZFS. It only occupies the space it needs inside of your ZFS - outside of your KVM, it's another story, but normally you should run PVE as your main virtualisation platform.

 

[limone]

Ehm, did you read my start post?
I can't solve my problem with using another file system on the main node, because I am limited to 50GB disk space! There is nothing I can change without buying a different machine, which I don't like to do.

And I run proxmox inside a KVM because, as said, I don't need virtualization, I just think proxmox is a nice GUI for LXC.

 

[LnxBil]

Ehm, did you read my start post?
I can't solve my problem with using another file system on the main node, because I am limited to 50GB disk space! There is nothing I can change without buying a different machine, which I don't like to do.

And I run proxmox inside a KVM because, as said, I don't need virtualization, I just think proxmox is a nice GUI for LXC.

So to get it right, you've a paid KVM machine with 50 GB of space in which you run PVE? If so, just reinstall your PVE with ZFS and you're good to go - I do not see any other solution for your problem. Everything will just work without even having to perform TRIM with ZFS.

 

[limone]

Dude, third time now:

My disk space is full, not because of no trimming, but because the containers grew over time with data.
REINSTALL WONT SOLVE MY PROBLEM

Trim works fine on plain folder or lvm, no need for zfs.
But trim on NFS does not work, so i need any method to trim the files on the nfs node.
(for example a way to mount the raw disk files on the storage node and trim that then, like on the pve node)

Thats why I have a seconds kvm with 400GB disk space, which I want to use to expand the host nodes storage.

 

[LnxBil]

But trim on NFS does not work, so i need any method to trim the files on the nfs node.
(for example a way to mount the raw disk files on the storage node and trim that then, like on the pve node)

There is no TRIM in NFS, so it won't work - ever. What you can do is fill all empty space with zeros and

• run the image through qemu-img
• or backup&restore you VM

Both attempts will only work offline and manually.

 

[limone]

I wont trim through NFS.

All I do on the local pve host is to mount the raw disk image and then trim that mounted folder?

NFS just places exactly the same raw disk image file on the nfs server, so why shouldnt it be possible to mount this file on the storage server itself and trim that?

All I need for that would be the packages proxmox uses to mount raw image files. right?
normal mount does not work

mount vm-102-disk-1.raw /pve
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

 

[LnxBil]

You have to use kpartx to mount the partitions inside of the raw file and mount them. But be aware, raw files cannot be trimmed if the underlying filesystem does not support trim (as I already stated above)

 

[limone]

kpartx looks promising, but i cannot get it working.

kpartx -l /storage/images/102/vm-102-disk-1.raw
loop deleted : /dev/loop0

shouldnt this pop up with -d option?
anyway kpartx -a does not give any output, -d neither

any /dev/loopX is not mountable,

wrong fs type, bad option, bad superblock on /dev/loop4,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

I mean, proxmox uses "pct mount", is there no way to do this on another device than a proxmox node?
I don't think you can give me the answer to that, only a proxmox developer maybe.

 

[LnxBil]

I mean, proxmox uses "pct mount", is there no way to do this on another device than a proxmox node?

sure, look at the code. PVE is OpenSource.

Can you run this on the nfs server:

blkid /storage/images/102/vm-102-disk-1.raw

 

[limone]

blkid /storage/images/102/vm-102-disk-1.raw
/storage/images/102/vm-102-disk-1.raw: UUID="3672c88b-d628-4634-b9bf-88ed605607b8" TYPE="ext4"

 

[LnxBil]

And now a

mount -v /storage/images/102/vm-102-disk-1.raw /srv
dmesg | tail

 

[limone]

Finally!

[ 3206.096098] EXT4-fs warning (device loop0): ext4_multi_mount_protect:331: Device is already active on another node.

So just shutdown the container, then mount on the nfs server and run fstrim.

Thank you, I guess.