[SOLVED] TRIM SSDs

Discussion in 'Proxmox VE: Installation and configuration' started by sshaikh, Aug 21, 2018.

Tags:
  1. sshaikh

    sshaikh Member

    Joined:
    Apr 23, 2017
    Messages:
    40
    Likes Received:
    10
    I have read all posts marked with trim but am still a little confused. Currently I have two PE nodes. Both have PE installed to an SSD. One has two further SSDs for VM disk images (using LVM-thin, discard always on); the other has HDDs which are passed through directly to the respective VMs.
    1. Should we enable trim in the host for the pve-root SSDs? If so, how? I can't see the fstab entry to add discard to.
    2. Should we enable trim in the host for the SSDs used as VM storage? If so, how? Again, I can't see the fstab entry to add discard to.
    3. Should we enable trim in the guests?
    Is it just a matter of scheduling fstrim -a once a week on both PE nodes?
     
  2. Klaus Steinberger

    Klaus Steinberger Active Member
    Proxmox Subscriber

    Joined:
    Jan 16, 2018
    Messages:
    165
    Likes Received:
    27
    I would schedule fstrim on a weekly basis. Depending on the SSD trim can result in garbage collection and adding unwanted latencies. The general advise in the linux world is trimming on a weekly basis.

    The systemd units are in the examples, so you can enable it this way:

    cp /usr/share/doc/util-linux/examples/fstrim.{service,timer} /etc/systemd/system
    systemctl enable fstrim.timer
    systemctl start fstrim.timer

    In the guests you can do discard also online, as it is decoupled from the SSD's, but to have the setup in an identical manner, I would do it also on a weekly basis. For example Windows 2016 trims on a regular basis. You have to use the Virtio Scsi driver with discard option set, all other drivers do not have discard support.

    One warning: Windows up to 7 / 2008R2 has no unmap support in the SCSI layer, so discard is not working. It works only with 2012R2 / Windows 8.1 and up.
     
  3. Klaus Steinberger

    Klaus Steinberger Active Member
    Proxmox Subscriber

    Joined:
    Jan 16, 2018
    Messages:
    165
    Likes Received:
    27
    One more note: If you use ZoL (ZFS on Linux), there is currently no Trim support in ZFS
     
  4. sshaikh

    sshaikh Member

    Joined:
    Apr 23, 2017
    Messages:
    40
    Likes Received:
    10
    Thank you for the informative reply.

    So a potential recipe can be:
    1. Schedule fstrim to run weekly on PE hosts
    2. Use the Virtio SCSI driver for VMs when possible, enabling the discard option each time
    3. Schedule trim (eg fstrim in Linux) to run weekly in guest VMs
    4. Schedule fstrim to run weekly in PE containers
     
  5. efeu

    efeu Member
    Proxmox Subscriber

    Joined:
    Nov 6, 2015
    Messages:
    74
    Likes Received:
    6
  6. mir

    mir Well-Known Member
    Proxmox Subscriber

    Joined:
    Apr 14, 2012
    Messages:
    3,481
    Likes Received:
    96
    This is wrong. Having this option set means running discard every time you delete something rendering the scheduled trim unnecessary. Either using option discard or run trim on a regular basis.
     
  7. sshaikh

    sshaikh Member

    Joined:
    Apr 23, 2017
    Messages:
    40
    Likes Received:
    10
    #7 sshaikh, Aug 21, 2018
    Last edited: Aug 21, 2018
  8. mir

    mir Well-Known Member
    Proxmox Subscriber

    Joined:
    Apr 14, 2012
    Messages:
    3,481
    Likes Received:
    96
    Checking discard option means that every time you mark a block of data as free due to deleting something a corresponding scsi unmap command will be sent to the disk controller (virtio-scsi) which passes this information through to the physical disk where your disk image hosted. If discard option is not checked the unmap command will not be passed through in which case the physical disk considers the newly freed data block still in use. To free all data blocks on the physical disk which is no longer used in your VM you either needs to check discard option or now and then run fstrim. Eg. the two options can substitute each other. Performance wise it is a bad thing to run discard/fstrim every time you free a block of data so it is advisable to have discard unchecked and then run fstrim on a regular basis. You can compare checking discard to disable disk cache.
     
  9. sshaikh

    sshaikh Member

    Joined:
    Apr 23, 2017
    Messages:
    40
    Likes Received:
    10
    I just did some tests, and with the VirtIO SCSI discard option on, deleting a (large) file from the guest DOES NOT reduce the usage on the host.

    When fstrim is run in the guest, the usage is then reduced in the host.

    As such, the recipe in #4 is the approach I'm taking.

    EDIT: Running fstrim in unprivileged containers proves to be a bit of an issue so using the workaround here:

    https://forum.proxmox.com/threads/d...-thin-doesnt-release-space.32179/#post-159371
     
    #9 sshaikh, Aug 23, 2018
    Last edited: Aug 23, 2018
  10. Symbol

    Symbol Member
    Proxmox Subscriber

    Joined:
    Mar 1, 2017
    Messages:
    42
    Likes Received:
    4
    It is not enough to activate the discard option in the host for this VM.
    You also have to mount your filesystem (within the guest) with the discard option if you want it to issue discard commands to the storage device. If you don't, it won't do anything.
    Or, you can use fstrim from time to time in the guest (probably the best option since it doesn't add overhead to deletion actions in your guest filesystem).
     
  11. sshaikh

    sshaikh Member

    Joined:
    Apr 23, 2017
    Messages:
    40
    Likes Received:
    10
    @Symbol that is what was understood by @Klaus Steinberger first reply, but a later posted suggested that having the discard option set on the Virtio adaptor was alone enough.

    Hence the testing. The confusion appears to be in the PE option and the fstab option both being using the word "discard".

    Just for clarity I have used the recipe in post #4 and workaround linked for unprivileged containers and now reclaim all space on both VMs, containers and hosts on a weekly basis.

    Thanks again for all the replies.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice