fstrim doesn't work in containers (any OS) - workarounds?

[interstellar]

Hi again

I've been trying all day to get fstrim working inside unprivileged LXC containers with no luck. Initially I thought it was just a Debian/Ubuntu issue, but I've tried multiple OS templates and it always results in: FITRIM ioctl failed: Operation not permitted. FYI it works fine in privileged containers, but as people have pointed out they're not as secure.

Can anyone help me get this working, or tell me if there's a workaround? I've switched to lvm-thin provisioning so I need to trim container disks to free up deleted space.

Or is it just not supported in unprivileged containers at the moment? I found a similar issue at linuxcontainers.org. One of the maintainers said:

"Unfortunately I expect it’s simply the kernel refusing an unprivileged user requesting a TRIM operation on a block device.​

Unless you’re using a privileged container, I don’t expect you’ll get away from that…

One option would be to have root on the host run fstrim against all /dev/rbd* devices, effectively running TRIM against all running containers.

Otherwise, you’d need to wait until we have a way to catch such system calls in userspace (there’s ongoing kernel work to allow that), at which point we could have LXD catch that particular ioctl and replay it as real root."


If doing it inside containers won't work, is a way of triming disk images externally from the host?

Thanks!

 

[dietmar]

See "man pct"

# pct fstrim <vmid>

 

[interstellar]

From what I can tell, it can only be done from the host at this time... The script I'm using is below. Tested and it works. I'm running it as a weekly cronjob

Thanks to http://wiki.csnu.org/index.php/Proxmox_:_TRIM_sous_LXC and https://forum.proxmox.com/threads/lxc-lvm-thin-discard-fstrim.34393/#post-168569 for the scripts.

Mods, could we get this added to the lvmthin documentation page?

### Trim Proxmox host and all unprivileged LXCs on node ###

#!/bin/bash

### Path to FSTRIM
FSTRIM=/sbin/fstrim

### Static list of volumes to trim
# Proxmox host
TRIMVOLS="/"

### Trim volumes
# Trim static list
for i in $TRIMVOLS; do
  echo "Trimming $i"
  $FSTRIM -v $i 2>&1 | logger -t "fstrim [$$]"
done

# Trim all LXC containers
for i in $(pct list | awk '/^[0-9]/ {print $1}'); do
  echo "Trimming CT $i"
  $FSTRIM -v /proc/$(lxc-info -n $i -p | awk '{print $2}')/root 2>&1 | logger -t "fstrim [$$]"
done

 

[interstellar]

Thanks @dietmar ! You posted while I was in the middle of my reply so I only just saw this Makes things a lot eaiser. Here's a tidied up script using PCT in case anyone finds it useful

### Trim Proxmox host volumes and all LXCs on node ###
# Run as weekly cron job
# !/bin/bash

# Script colors
green=`tput setaf 2`
reset=`tput sgr0`

# Path to FSTRIM
FSTRIM=/sbin/fstrim

# List of host volumes to trim separated by spaces
# eg TRIMVOLS="/mnt/a /mnt/b /mnt/c"
TRIMVOLS="/"

## Trim all LXC containers ##
echo "${green}LXC CONTAINERS${reset}"
for i in $(pct list | awk '/^[0-9]/ {print $1}'); do
  echo "Trimming Container $i"
  pct fstrim $i 2>&1 | logger -t "pct fstrim [$$]"
done
echo ""

## Trim host volumes ##
echo "${green}HOST VOLUMES${reset}"
for i in $TRIMVOLS; do
  echo "Trimming $i"
  $FSTRIM -v $i 2>&1 | logger -t "fstrim [$$]"
done

 

[LnxBil]

The shebang in the third line is ignored. Shebang has to go in the first line.

Here is an example that it is not working:

root@proxmox ~ > ls -l test
-rwxr-xr-x 1 root root 104 Mai 19 09:14 test
root@proxmox ~ > cat test
#not working perl
#!/usr/bin/perl

use strict;
use warnings;

my $test = "Hello World\n";
print $test;

1;

root@proxmox ~ > ./test
./test: line 4: use: command not found
./test: line 5: use: command not found
./test: line 7: my: command not found
./test: line 10: 1: command not found

 

[mir]

Except if you specify command interpreter on the command line like:
perl ./test
bash some_bash_script

 

[LnxBil]

Except if you specify command interpreter on the command line like:

Yes, sure, but shebang is also ignored in that case :-D

 

[Proxygen]

I get 'tput: No value for $TERM and no -T specified' when I run the script via cron (nor when ran manually). pct was of course replaced by /usr/sbin/pct

Thanks @dietmar ! You posted while I was in the middle of my reply so I only just saw this Makes things a lot eaiser. Here's a tidied up script using PCT in case anyone finds it useful

### Trim Proxmox host volumes and all LXCs on node ###
# Run as weekly cron job
# !/bin/bash

# Script colors
green=`tput setaf 2`
reset=`tput sgr0`

# Path to FSTRIM
FSTRIM=/sbin/fstrim

# List of host volumes to trim separated by spaces
# eg TRIMVOLS="/mnt/a /mnt/b /mnt/c"
TRIMVOLS="/"

## Trim all LXC containers ##
echo "${green}LXC CONTAINERS${reset}"
for i in $(pct list | awk '/^[0-9]/ {print $1}'); do
  echo "Trimming Container $i"
  pct fstrim $i 2>&1 | logger -t "pct fstrim [$$]"
done
echo ""

## Trim host volumes ##
echo "${green}HOST VOLUMES${reset}"
for i in $TRIMVOLS; do
  echo "Trimming $i"
  $FSTRIM -v $i 2>&1 | logger -t "fstrim [$$]"
done

 

[oguz]

dirty one-liner to trim all containers:

pct list | awk '/^[0-9]/ {print $1}' | while read ct; do pct fstrim ${ct}; done

 

[legrems]

One-liner with xargs:

pct list | grep -Eo '^[0-9]+' | xargs -P X -n 1 pct fstrim

With `X` is the number of fstrim you want to do in parallel

 

[jure965]

Systemd service files, based on fstrim.service, fstrim.timer and a one liner with xargs from a previous post (kudos!)
Feel free to correct/modify/improve.

# /etc/systemd/system/pct-fstrim.service
[Unit]
Description=Discard unused blocks of all LXC volumes on current node

[Service]
Type=oneshot
ExecStart=/usr/bin/bash -c '/usr/sbin/pct list | /usr/bin/grep -Eo "\x5e\x5b0\x2d9\x5d\x2b" | /usr/bin/xargs -n 1 /usr/sbin/pct fstrim'

# /etc/systemd/system/pct-fstrim.timer
[Unit]
Description=Discard unused blocks of all LXC volumes on current node once a week

[Timer]
OnCalendar=weekly
AccuracySec=1h
Persistent=true

[Install]
WantedBy=timers.target

 

[Dunuin]

Thanks for the oneliners and systemd script.

Here it looks like my LXCs on LVM-thin get trimmed but not my LXCs on ZFS. Is there anything special that I need to do so the datasets used by ZFS to store the LXC disks get trimmed too? That ZFS pool is thin and only using SSDs so trimming should be needed there but pct fstrim returns that trim is not supported:

root@Hypervisor:~# pct list | awk '/^[0-9]/ {print $1}' | while read ct; do pct fstrim ${ct}; done
fstrim: /var/lib/lxc/100/rootfs/: the discard operation is not supported
command 'fstrim -v /var/lib/lxc/100/rootfs/' failed: exit code 1
fstrim: /var/lib/lxc/101/rootfs/: the discard operation is not supported
command 'fstrim -v /var/lib/lxc/101/rootfs/' failed: exit code 1
/var/lib/lxc/121/rootfs/: 248.9 MiB (261033984 bytes) trimmed
fstrim: /var/lib/lxc/126/rootfs/: the discard operation is not supported
command 'fstrim -v /var/lib/lxc/126/rootfs/' failed: exit code 1
fstrim: /var/lib/lxc/133/rootfs/: the discard operation is not supported
command 'fstrim -v /var/lib/lxc/133/rootfs/' failed: exit code 1

VMID 121 ist the one LXC stored on LVM-thin, all the other LXCs are stored on ZFS.

 

[LnxBil]

Here it looks like my LXCs on LVM-thin get trimmed but not my LXCs on ZFS. Is there anything special that I need to do so the datasets used by ZFS to store the LXC disks get trimmed too?

They are trimmed automatically. ZFS in LXC is already a filesystem, so if you delete a file, ZFS knows that you delete a file and frees the space (if there are no snapshots relying on it).

 

[Dunuin]

Ok, that makes sense. But only if autotrim is enabled right?

root@Hypervisor:~# zpool get autotrim
NAME    PROPERTY  VALUE     SOURCE
VMpool  autotrim  on        local

 

[LnxBil]

Ok, that makes sense. But only if autotrim is enabled right?

root@Hypervisor:~# zpool get autotrim
NAME    PROPERTY  VALUE     SOURCE
VMpool  autotrim  on        local

No, that is for the underlying zpool.

ZFS datasets (the filesystem) does this on its own even if the underlying block storage (e.g. harddisk) does not have trim capability. The autotrim property is for ssd-based block storage hat has trim itself for cleaning up stuff. This is relatively new and normally not necessary on enterprise SSDs.

 

[Twitch6004]

I used

pct list | grep -Eo '^[0-9]+' | xargs -P X -n 1 pct fstrim

" But the error

"WARNING: Thin volume pve/vm-103-disk-0 maps 2876309504 while the size is only 2147483648. LXC"

Still remains so im assuming that the command isnt doing anything?

 

[LnxBil]

I used

pct list | grep -Eo '^[0-9]+' | xargs -P X -n 1 pct fstrim

" But the error

"WARNING: Thin volume pve/vm-103-disk-0 maps 2876309504 while the size is only 2147483648. LXC"

Still remains so im assuming that the command isnt doing anything?

It's not an error, it's a warning and it warns that the thin volume maps more data than the size of the volume is .. which is odd.

 

[Twitch6004]

It's not an error, it's a warning and it warns that the thin volume maps more data than the size of the volume is .. which is odd.

i figured it out, I migrated to a new HDD and the old volumes still showed up on the old drive. Had to use CLI to delete the old stuff then i ran the command again and it worked