Removed OSDs without using Out first, now can't delete container that was using the pool as storage

[omber]

Hi there,

I am learning Proxmox and I got myself stuck. This a single node PVE 6.1-7 node.

I did not Out OSDs before Stopping them. This, I think, broke the storage my sole Linux container uses, as it locked up and could not be stopped. As per another thread I killed the monitor process and that stopped the container. Now I am trying to remove it but I receive following error:

TASK ERROR: error with cfs lock 'storage-ceph-lxc-storage': can't unmap rbd device /dev/rbd/ceph-lxc-storage/vm-301-disk-0: rbd: sysfs write failed

How can I destroy this LXC so I can start fresh? Or at this point should I just reinstall PVE and be done with it?

Background:

I am running a single node learning setup. When creating individual pools in Ceph > Pools, it is possible to specify size and minimum size to be 1 and 1 so that works. However such pools can be only used for Container and VM storage, but not for general storage, at least as far as I can tell.

I tried to create a CephFS for general storage but I ran into the issue where data and metadata pools created for CephFS are always made with minimum size of 2 and size of 3 values. This does not work for my single node setup. I don't see where this option can be adjusted during creation of CephFS from PVE Web UI.

I tried using terminal to first create the pools with desired min / size values, then creating CephFS using those pools, but it seems when Pools are created individually they are set to "application rbd" type and cannot be use with CephFS. I never found a way to adjust this, so I decided to give up on Ceph for now, until I can get more nodes and just use BTRFS RAID0.

These are off-topic questions:

* Is it possible to adjust pool size and minimum size when creating CephFS?
* If not, how does this scale on clusters of 5+ nodes?

 

[aaron]

Well, running Ceph on only one node with only one copy goes against everything it was designed for

If this is a machine to just play around and try out new things I would suggest the following to test out Ceph the way it is meant to be used.

Reinstall the machine and then create a nested Cluster of at least 3 PVE nodes and run Ceph on these. You can even simulate having multiple NICs in the Ceph nodes. Create a Linux bridge without any bridge port assigned. This will make it a virtual switch. Then you can add a NIC to the VM with that bridge.

Running PVE on top of PVE is how we test a lot of stuff here ourselves.

* Is it possible to adjust pool size and minimum size when creating CephFS?

Probably but I would have to look it up, in the end a CephFS is a regular pool with some meta data pools around it.

* If not, how does this scale on clusters of 5+ nodes?

What do you mean exactly? In the default setup (size 3, min_size 2) for a pool, Ceph will make sure that 3 nodes will have a copy of the data. In a 3 node cluster it is easy to grasp, each node will store one copy. Once you go above that, not every node will store a copy of each data block (Placement Group or PG) but 3 nodes in the cluster will.