[SOLVED] TRIM SSDs

S

sshaikh

Well-Known Member
Apr 23, 2017
78
22
48
I have read all posts marked with trim but am still a little confused. Currently I have two PE nodes. Both have PE installed to an SSD. One has two further SSDs for VM disk images (using LVM-thin, discard always on); the other has HDDs which are passed through directly to the respective VMs.
  1. Should we enable trim in the host for the pve-root SSDs? If so, how? I can't see the fstab entry to add discard to.
  2. Should we enable trim in the host for the SSDs used as VM storage? If so, how? Again, I can't see the fstab entry to add discard to.
  3. Should we enable trim in the guests?
Is it just a matter of scheduling fstrim -a once a week on both PE nodes?
 
I would schedule fstrim on a weekly basis. Depending on the SSD trim can result in garbage collection and adding unwanted latencies. The general advise in the linux world is trimming on a weekly basis.

The systemd units are in the examples, so you can enable it this way:

cp /usr/share/doc/util-linux/examples/fstrim.{service,timer} /etc/systemd/system
systemctl enable fstrim.timer
systemctl start fstrim.timer

In the guests you can do discard also online, as it is decoupled from the SSD's, but to have the setup in an identical manner, I would do it also on a weekly basis. For example Windows 2016 trims on a regular basis. You have to use the Virtio Scsi driver with discard option set, all other drivers do not have discard support.

One warning: Windows up to 7 / 2008R2 has no unmap support in the SCSI layer, so discard is not working. It works only with 2012R2 / Windows 8.1 and up.
 
Thank you for the informative reply.

So a potential recipe can be:
  1. Schedule fstrim to run weekly on PE hosts
  2. Use the Virtio SCSI driver for VMs when possible, enabling the discard option each time
  3. Schedule trim (eg fstrim in Linux) to run weekly in guest VMs
  4. Schedule fstrim to run weekly in PE containers
 
  • Like
Reactions: raydel92
Last edited:
Checking discard option means that every time you mark a block of data as free due to deleting something a corresponding scsi unmap command will be sent to the disk controller (virtio-scsi) which passes this information through to the physical disk where your disk image hosted. If discard option is not checked the unmap command will not be passed through in which case the physical disk considers the newly freed data block still in use. To free all data blocks on the physical disk which is no longer used in your VM you either needs to check discard option or now and then run fstrim. Eg. the two options can substitute each other. Performance wise it is a bad thing to run discard/fstrim every time you free a block of data so it is advisable to have discard unchecked and then run fstrim on a regular basis. You can compare checking discard to disable disk cache.
 
I just did some tests, and with the VirtIO SCSI discard option on, deleting a (large) file from the guest DOES NOT reduce the usage on the host.

When fstrim is run in the guest, the usage is then reduced in the host.

As such, the recipe in #4 is the approach I'm taking.

EDIT: Running fstrim in unprivileged containers proves to be a bit of an issue so using the workaround here:

https://forum.proxmox.com/threads/d...-thin-doesnt-release-space.32179/#post-159371
 
Last edited:
sshaikh said:
I just did some tests, and with the VirtIO SCSI discard option on, deleting a (large) file from the guest DOES NOT reduce the usage on the host.
Click to expand...
It is not enough to activate the discard option in the host for this VM.
You also have to mount your filesystem (within the guest) with the discard option if you want it to issue discard commands to the storage device. If you don't, it won't do anything.
Or, you can use fstrim from time to time in the guest (probably the best option since it doesn't add overhead to deletion actions in your guest filesystem).
 
@Symbol that is what was understood by @Klaus Steinberger first reply, but a later posted suggested that having the discard option set on the Virtio adaptor was alone enough.

mir said:
To free all data blocks on the physical disk which is no longer used in your VM you either needs to check discard option or now and then run fstrim. Eg. the two options can substitute each other.
Click to expand...

Hence the testing. The confusion appears to be in the PE option and the fstab option both being using the word "discard".

Just for clarity I have used the recipe in post #4 and workaround linked for unprivileged containers and now reclaim all space on both VMs, containers and hosts on a weekly basis.

Thanks again for all the replies.
 
I have the host (fully up to date, on Debian 10.1) on HDD (so I do not enable fstrim.timer), and containers and VMs on NVME (LVM-thin, container storage is not in fstab)

Regarding step 4 in post #4, do you enable fstrim.timer inside the container? Because on an Ubuntu 18.04.3 guest), journalctl -u fstrim.service gives me

Code: 
fstrim: /: FITRIM ioctl failed: Operation not permitted

and systemctl status fstrim.timer

Code: 
Loaded: loaded (/lib/systemd/system/fstrim.timer; enabled; vendor preset: enabled)
   Loaded: loaded (/lib/systemd/system/fstrim.timer; enabled; vendor preset: enabled)
   Active: inactive (dead)
  Trigger: n/a
Condition: start condition failed at Sat 2019-11-09 22:14:34 UTC; 2 days ago
           └─ ConditionVirtualization=!container was not met

UPDATE: fstrim inside the guest only works on privileged containers. on unprivileged containers it has to be run from the host. here is a script to do it using pct.
 
Last edited:
Klaus Steinberger said:
One more note: If you use ZoL (ZFS on Linux), there is currently no Trim support in ZFS
Click to expand...

Just to confirm, looking at proxmox roadmap it seems that since Proxmox VE 6.0 it incorporates ZFS 0.8.1, which already provides full trim support, correct?

So from that version on, trim is supported on ZFS pool, ZVOL KVM/Qemu VM, ZFS LXC Container, correct?


mir said:
This is wrong. Having this option set means running discard every time you delete something rendering the scheduled trim unnecessary. Either using option discard or run trim on a regular basis.
Click to expand...

sshaikh said:
Thank you for the informative reply.

So a potential recipe can be:
  1. Schedule fstrim to run weekly on PE hosts
  2. Use the Virtio SCSI driver for VMs when possible, enabling the discard option each time
  3. Schedule trim (eg fstrim in Linux) to run weekly in guest VMs
  4. Schedule fstrim to run weekly in PE containers
Click to expand...

sshaikh said:
I just did some tests, and with the VirtIO SCSI discard option on, deleting a (large) file from the guest DOES NOT reduce the usage on the host.

When fstrim is run in the guest, the usage is then reduced in the host.

As such, the recipe in #4 is the approach I'm taking.

EDIT: Running fstrim in unprivileged containers proves to be a bit of an issue so using the workaround here:

https://forum.proxmox.com/threads/d...-thin-doesnt-release-space.32179/#post-159371
Click to expand...


I've been gathering info regarding this subject, I would like to confirm that your recipe is working as expected.

You put the check on discard in every guest VM disk, AND set fstrim on Proxmox VE host system?

In case of Windows VMs, they should be able to manage trim operations by themselves, is it also required the discard check on disks, right? Any other config on those VMs?

For instance, on a Windows VM (which will be migrated to the new server, once it is running), I have the following config:
- SCSI Controller: VirtIO SCSI
- Hard Disk (scsi0): local-lvm:vm-100-disk-0,size=40G
- Hard Disk (scsi1): local-lvm:vm-100-disk-1,size=10G
- Hard Disk (virtio2): /dev/disk/by-id/ata-HGST_HUS7...,size=...

I know I have installed virtio driver on windows guest system in order to properly work with the virtualized controller.

So this is the required VM config with VirtIO driver that will allow me to check "discard" option on each disk and have trim support?
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom