CephFS trough Proxmox GUI - No VM storage possible?

Ceph is more than just a filesystem. Ceph provides three separate types of storage. Object storage like Amazon S3, block storage through RBD or iSCSI, and file storage through CephFS. For a VM you don't want it to be stored in CephFS, you want to use block level storage. I set up Ceph on my ProxMox cluster before the newer GUI was added so I'm not 100% sure what that adds by default but what you want is a pool for your VMs and then have that added as RBD storage to your nodes.
 
I did everything through the GUI.

The steps I did were:

1) Create CephFS first https://pve.proxmox.com/wiki/Manage_Ceph_Services_on_Proxmox_VE_Nodes#pveceph_fs. This will create two pools called cephfs_data and cephfs_data
2) Create RBD by clicking on Datacenter -> Add -> RBD. It should default to cephfs_data.

When you create your CephFS pools, you want to make sure you calculate the correct PGs (placement groups) otherwise performance will be bad. Calculate it manually https://stackoverflow.com/questions/40771273/ceph-too-many-pgs-per-osd
 
You don't want to add RBD storage on the cephfs_data pool. Ignore that suggestion. Your VMs should be in a separate pool for manageability reasons. It's easy to add a pool under the GUI if you don't already have a pool labeled vm. The reason to put it under a separate pool is so that later on if you wish to change the CRUSH rules for your VMs verses your CephFS storage you can do so. For example, due to the layout of my cluster I use size 3 min_size 3 for my VMs with the primary OSD and at least one other OSD on SSD storage but for my CephFS data which is just my backups I use size 2 min_size 2 and weight that storage towards my HDDs instead of SSDs. There's a reason why you can have multiple pools, don't just throw everything in one just because it's 2 seconds quicker.

As to sizing your PG count, Google it, there are plenty of easy to use calculators out there for picking a good size. One thing to watch out for, if you pick too small of a PG count that's just going to lead to the cluster being a bit unbalanced and filling up some OSDs faster than others. If it's too large that'll put more load on the OSDs and use more resources for less throughput. If it's too small it's easy to increase the number of PGs, if it's too large you can't combine them again without exporting all of your data out of the pool, deleting it, and recreating it with the new size and adding all of your data back in. It's a massive pain. Ceph Nautilus fixes this but Ceph Mimic and above do not support Debian (and by extension ProxMox) until Debian Buster is released. Once Buster is released, ProxMox will work on upgrading ProxMox to be based around Buster and eventually we'll be able to upgrade Ceph. The point here is that even with Debian Buster around the corner it's still going to be a little while before we can upgrade so make sure you don't have too many PGs, it's much better to have too few PGs than too many.
 
  • Like
Reactions: Alwin

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!