opnsense guest trim - reclaim host storage space?

[keeka]

I created an opnsense 24.1 guest. Backed up, this vm uses 1.2GB.
I upgraded the guest to opnsense 24.7. The backup now occupies 2.84GB.
I'd enabled the discard option on the vm. But it seems the opnsense installer, unable to determione the disk type, did not enable trim:

~ # camcontrol identify da0
camcontrol: ATA ATA_IDENTIFY via pass_16 failed
camcontrol: ATA ATAPI_IDENTIFY via pass_16 failed

~ # tunefs -p /
...
tunefs: trim: (-t)                                         disabled
...

I then booted the guest to single user mode and enabled trim:

~ # fsck -y
~ # tunefs -t enable /
~ # tunefs -p /
...
tunefs: trim: (-t)                                         enabled
...

My understanding is this enables trim for deletions going forward. But it will not trim previously deleted, now free, blocks (of which there are alot following the upgrade). And sure enough, the size of subsequent backups remains unchanged, at around 2.84GB.

How to issue a full/manual trim of the filesystem? Running fsck_ffs -E / in the guest does not seem to have had any effect on the size of new backups.

 

[sw-omit]

Can't help you with the cleanup-question, but just as a thought;
Why not install a new 24.7 opnsense-VM, set the trim-option now so you don't have to re-do this in the future, make an in-opnsense backup of the config, import the backup with reboot, during the reboot kill it and change the mac-addresses of the adaptors to the original ones too.
Don't change it before, you don't want two systems using the same MAC on the network, and you'll need access to it running to load in the backup if you do it from the GUI (instead of from a disk or iso) with the least downtime.

Would give you downtime not much longer then a single reboot and a clean install with a small backup-footprint.

 

[leesteken]

How to issue a full/manual trim of the filesystem? Running fsck_ffs -E / in the guest does not seem to have had any effect on the size of new backups.

What storage type are you using for this VM? It appears that some link in the trim/discard chain is not present ( https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_hard_disk ). Please also share the VM configuration file (qm config ),

 

[keeka]

Why not install a new 24.7 opnsense-VM, set the trim-option now so you don't have to re-do this in the future, make an in-opnsense backup of the config, import the backup with reboot, during the reboot kill it and change the mac-addresses of the adaptors to the original ones too.

I was considering something similar but would like to get to the bottom of it because I don't see these issues with a pfsense vm with identical config. That vm has been through several upgrades and the disk use (lvm-thin backed disk and PVE backups) has remained the same size (1.2GB) for several years.

 

[keeka]

What storage type are you using for this VM? It appears that some link in the trim/discard chain is not present ( https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_hard_disk ). Please also share the VM configuration file (qm config),

Storage is lvm-thin. The opnsense guest config is similar in most respects to the pfsense guest I mentioned in my reply to @sw-omit.

# qm config 101
affinity: 0,4
agent: 1
balloon: 0
boot: order=scsi0
cores: 2
cpu: host
hotplug: 0
memory: 2048
name: opn
net0: virtio=46:AB:CB:D4:A5:B3,bridge=vmbr0
net1: virtio=86:C6:BF:BE:3A:D9,bridge=vmbr1
net2: virtio=FE:1D:B8:13:0C:08,bridge=vmbr2
net3: virtio=A2:9D:9F:A4:AB:3D,bridge=vmbr3
net4: virtio=BE:FD:1C:D5:CF:7B,bridge=vmbr4
net5: virtio=B2:83:7E:DB:94:82,bridge=vmbr5
net6: virtio=BC:24:11:52:F2:59,bridge=vmbr6
numa: 0
ostype: l26
scsi0: vmdisks:vm-101-disk-0,discard=on,size=32G
scsihw: virtio-scsi-single
serial0: socket
smbios1: uuid=e245defa-9bb1-4c6c-8c46-ee8b98c024ae
sockets: 1
startup: order=20,up=30
tablet: 0
tags: dev;dmz
vmgenid: 66cb2bed-9a73-4413-bee9-5d570fe71271

# diff 100.conf 101.conf
24,25c4
< boot: c
< bootdisk: scsi0
---
> boot: order=scsi0
30c9
< name: fw
---
> name: opn
39d17
< onboot: 1
41c19
< scsi0: vmdisks:vm-100-disk-0,discard=on,size=32G
---
> scsi0: vmdisks:vm-101-disk-0,discard=on,size=32G
44c22
< smbios1: uuid=23119b27-800e-4b81-834b-f1349d91a223
---
> smbios1: uuid=e245defa-9bb1-4c6c-8c46-ee8b98c024ae
48,49c26,27
< tags: dmz
< vmgenid: 3a381ac2-eb28-4b9e-88d7-2590348d23e0
---
> tags: dev;dmz
> vmgenid: 66cb2bed-9a73-4413-bee9-5d570fe71271

 

[keeka]

OK, it does seem that trim is working in as far as: I create a large file in the guest, I see the lvm-thin volume on the host grow and then shrink back when I delete the file in the guest.
However, I don't seem to be able to reclaim the jump in storage needs following the opnsense guest upgrade. I'm assuming here that the upgraded guest OS doesn't actually use all that space or something is amiss with my opnsense upgrade.
Disk use in the guest is reported as:

~ # df -h
Filesystem                   Size    Used   Avail Capacity  Mounted on
/dev/gpt/rootfs               23G    2.7G     18G    13%    /
devfs                        1.0K      0B    1.0K     0%    /dev
/dev/gpt/efifs               256M    1.7M    254M     1%    /boot/efi
devfs                        1.0K      0B    1.0K     0%    /var/dhcpd/dev
devfs                        1.0K      0B    1.0K     0%    /var/unbound/dev
/usr/local/lib/python3.11     23G    2.7G     18G    13%    /var/unbound/usr/local/lib/python3.11
/lib                          23G    2.7G     18G    13%    /var/unbound/lib

There is an 8GB swap partiton. Could that consume untrimmed space?

~ # gpart show
=>      40  67108784  da0  GPT  (32G)
        40    532480    1  efi  (260M)
    532520      1024    2  freebsd-boot  (512K)
    533544  49798144    3  freebsd-ufs  (24G)
  50331688  16777136    4  freebsd-swap  (8.0G)

EDIT: I 'solved' this in the end by dd'ing several large files to almost fill the guest disk, then deleting them. Moments later, trim cleaned up and the underlying lvm-thin backed volume had reduced right down. All I can think is, fsck_ffs -E / in the freebsd guest was not doing what I expected. So only question remains how to do something like fstrim on freebsd. But that is for another day/forum.

 

[leesteken]

There is an 8GB swap partiton. Could that consume untrimmed space?

It definitely can. And so can the EFI partition. Better find out how to trim those in FreeBSD.

 

[keeka]

It definitely can. And so can the EFI partition. Better find out how to trim those in FreeBSD.

Thanks. Next time I will try the guest 'ssd emulation' option and see if the opn installer recognises it as such. The camcontrol output in post #1 is apparently why trim was not enabled. But I'd have expected it to display something more informative - the above appears to be a failure of some kind.
I also wonder if the opn installer, when presented with an ssd, enables trim on the swap partition too? Post-install, tunefs enable trim didn't work for the swap partition. I am having a hard time tracking down info for trim on FreeBSD, but that's compounded by lack of knowledge of the OS. It is somewhat different to Linux in many ways.
Next time I boot the pfsense vm, I will check camcontrol output and the root partiton's trim status.

 

[keeka]

Just a follow up. I checked the pfsense guest. camcontrol output is similar to that of the opnsense guest. However, trim is enabled for rootfs. I must have manually enabled it on the pfsense guest post-install too, but before applying a major upgrade. My memory fails me on that one.
Next time, I'll see what effect ssd emulation has on the guest install.