ext4 trim and Ceph

alchemyx

New Member
Apr 2, 2014
7
0
1
Hello, I have set up three node Ceph cluster to store VMs. It works very well. Altough I don't know whyfreeing spaces does not work. For example I create Linux VM with 500G disk. It only takes smallfraction of that space after installation. Then I create two big files let say 20G. With "ceph -w" I seethat "data usage" went up those 20G. Then I remove those files and execute in guest "fstrim -v /" andit says it trimmed that that data. But it is not freed in ceph. I am using IDE of course and ext4 insideof guest.What am I doing wrong?Thanks!
 
Hello, I have set up three node Ceph cluster to store VMs. It works very well. Altough I don't know whyfreeing spaces does not work. For example I create Linux VM with 500G disk. It only takes smallfraction of that space after installation. Then I create two big files let say 20G. With "ceph -w" I seethat "data usage" went up those 20G. Then I remove those files and execute in guest "fstrim -v /" andit says it trimmed that that data. But it is not freed in ceph. I am using IDE of course and ext4 insideof guest.What am I doing wrong?Thanks!

you need 3 more things:

use virtio-scsi (or scsi, but it's slower) for you guests disk controller.

and enable discard (you need to edit your vmid.conf file, and add "discard=on" to drive properties

and also,you need to enable discard on ceph-osd filesystem (xfs defaults,discard) for example.
 
you need 3 more things:use virtio-scsi (or scsi, but it's slower) for you guests disk controller.and enable discard (you need to edit your vmid.conf file, and add "discard=on" to drive propertiesand also,you need to enable discard on ceph-osd filesystem (xfs defaults,discard) for example.

Hello,

So I got somehow misleaded by this - http://ceph.com/docs/master/rbd/qemu-rbd/

"Note that this uses the IDE driver. The virtio driver does not support discard."

So adding discard=on to an IDE drive definition fixed it! Thanks!

And about last one - discard for OSD?
I don't know why would it be important in any way? I use xfs on spindle drive so discard seems to be unnecessary here.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!