[SOLVED] Cannot fstrim VM's yet again

[djfreak]

Tremendous frustration over this,. I just spent an entire day trying to fix this - again. I had to switch my IP addresses and I get that I had to zero out over that because the cluster if a very complex thing but this restoration process is maddening.

I'm using up barely 11GB of a 64GB volume Debian VM. If I'm forced to restore 64GB that renders the backup absolutely useless. I cannot believe how difficult this is.

I have run fstrim. I have added discard to fstab in the guest Debian OS. I have made sure that discard is checked in Proxmox VM GUI for the VM. I have rebooted and run fstrim in the node and the guest so many times that I'm losing my mind.

Why is this very basic thing so tremendously difficult? Am I missing some basic step here? I'm on Proxmox 8.0.3 running a 3 node cluster storing VM's on a 3 disk ZFS pool - all of which is rendered useless if my backups aren't restorable.

When I ran fstrim verbosely in the guest Debian OS after adding discard to fstab it said that it trimmed 51GB. Why am I not getting credit for that when I move my VM onto my ZFS?

Thank you in advance for any help.

 

[Chris]

I'm using up barely 11GB of a 64GB volume Debian VM. If I'm forced to restore 64GB that renders the backup absolutely useless. I cannot believe how difficult this is.

Please share your storage configuration cat /etc/pve/storage.cfg and VM configuration qm config <VMID>. What makes you think that the restore is not sparse? Please share the restore task log. Also, what filesystem are you using within the VM?

As a first step I suggest to check that the qemu-guest-agent is running within the VM and that it is enable in the VM config, further detail for setup and possible pitfalls can be found in the docs https://pve.proxmox.com/pve-docs/chapter-qm.html#qm_qemu_agent

Why is this very basic thing so tremendously difficult? Am I missing some basic step here? I'm on Proxmox 8.0.3 running a 3 node cluster storing VM's on a 3 disk ZFS pool - all of which is rendered useless if my backups aren't restorable.

What do you mean by not restorable? Does the restore fail or do you mean that it is not sparse?

When I ran fstrim verbosely in the guest Debian OS after adding discard to fstab it said that it trimmed 51GB. Why am I not getting credit for that when I move my VM onto my ZFS?

Also not really clear what you mean by moving your VM onto ZFS. Are you talking about the restore, VM migration?

 

[djfreak]

Hi Chris,

Thank you so much for your help and sorry for the slow response. Had to focus on the money job.

I'm back to enjoyable development Proxmox work and I just want to report what I'm observing. Since it's so easy, I decided to just go ahead and download a brand new Debian, create a new VM and install the Portainer agent from scratch. It went fine but I looked at my zfs pool and I'm getting hit once again for every GB of the Volume capacity which is 64GB. Again this is not a restoration now. This is a brand new updated Debian with a brand new working Portainer agent on it.

(As an aside, this entire thing has forced me to admit that this Portainer agent could all be done - even with my persistent storage - in a Linux container which I'm going to work on this weekend. Again the Linux containers backup and restore perfectly as long as I restore them back onto the NFS they were backed up on and then do a Volume move in Proxmox. The Linux containers behave so much better in terms of storage. I would still however like to get this VM issue solved if I can.)

Here is my cat /etc/pve/storage.cfg

dir: local
path /var/lib/vz
content images,iso,snippets,vztmpl,backup,rootdir
shared 0

zfspool: local-zfs
pool rpool/data
content rootdir,images
sparse 1

nfs: bhnfsstorage1
export /export/PROXMOX-BACKUP
path /mnt/pve/bhnfsstorage1
server 10.27.112.6
content iso,snippets,vztmpl,backup,rootdir,images
prune-backups keep-all=1

zfspool: bhpmzfs1
pool bhpmzfs1
content images,rootdir
mountpoint /bhpmzfs1
nodes proxmox3,proxmox2,proxmox

dir: bhlocalbackup
path /mnt/pve/bhlocalbackup
content backup,rootdir,iso,snippets,vztmpl,images
is_mountpoint 1
nodes proxmox

Here is my qm config 106 (Brand new Debian - not backed up or restored. Hasn't even been migrated from Node 1. Been on ZFS its whole life)

agent: 1
boot: order=scsi0;ide2;net0
cores: 1
cpu: x86-64-v2-AES
ide2: local:iso/debian-12.0.0-amd64-netinst.iso,media=cdrom,size=738M
machine: q35
memory: 4096
meta: creation-qemu=8.0.2,ctime=1688533626
name: bhdebian
net0: virtio=FA:BF:0D:10:C2:FF,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: bhpmzfs1:vm-106-disk-0,discard=on,iothread=1,size=64G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=7ff08685-a513-43a1-87b5-490837c366de
sockets: 2
vmgenid: ac89da3c-9044-4aa5-8d14-c6e2c4972934


Here is df -kh on my brand new Debian:


Filesystem Size Used Avail Use% Mounted on


udev 1.9G 0 1.9G 0% /dev


tmpfs 392M 1.4M 391M 1% /run


/dev/sda1 62G 5.0G 54G 9% /


tmpfs 2.0G 0 2.0G 0% /dev/shm


tmpfs 5.0M 8.0K 5.0M 1% /run/lock


tmpfs 392M 1.3M 391M 1% /run/user/1000

If this is just the nature of pooled VM storage now I'll be forced to accept it but it definitely motivates me to build Linux and Docker containers whenever possible.

Thanks again Chris.

 

[Chris]

Please note that you did not set the sparse: 1 option for bhpmzfs1, so the volumes crated on that pool will reserve the full size.
You can set the property as documented [0], which will affect newly created volumes. For existing ones you will have to follow the steps as discussed in this thread [1].

[0] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#storage_zfspool
[1] https://forum.proxmox.com/threads/enable-sparse-on-existing-zfs-storage.74078/

 

[djfreak]

Thank you very much Chris. That was it. It didn’t even require the complex instructions. I simply forgot the thin checkbox in the ZFS storage listing in the GUI. I restored my Windows VM to my node 1 using my EXT4 formatted drive for temporary storage. Simply moved storage over to my ZFS volume and it worked perfectly. Less than 30gb used on my ZFS.

The side effect of this was it forced me to fix persistent NFS storage in my Debian containers so I don’t even need VMs as much now. Windows is all I need VMs for now. Containers can do everything.

Thanks again for your help Chris.

 

[Chris]

Glad it works as expected, please mark the thread as solved for others to find solutions more easily.

 

[djfreak]

Marked solved.