Cannot create CEPH Pool with min_size=1 or min_size > 2

AKA · Oct 19, 2020

Hello everyone,

i'm running a 3 Node Proxmox Cluster on Version 6.2-4.
I'm having CEPH configured as my primary storage system.

One of the Nodes is a VM acting as a tiebreaker / CEPH Monitor restricted from running VMs.
I used a private 10GbE Network as Cluster_network for CEPH for syncing the OSDs.

Here's my config:

Code:

[global]
     auth_client_required = cephx
     auth_cluster_required = cephx
     auth_service_required = cephx
     cluster_network = 192.168.100.52/24
     fsid = 0b4b4108-0d40-44d7-916b-0546e915ddb5
     mon_allow_pool_delete = true
     mon_host = 10.23.0.52 10.23.0.53 10.23.0.56
     osd_pool_default_min_size = 2
     osd_pool_default_size = 2
     public_network = 10.23.0.52/17

[client]
     keyring = /etc/pve/priv/$cluster.$name.keyring

[mds]
     keyring = /var/lib/ceph/mds/ceph-$id/keyring

[mon.proxmox-01]
     public_addr = 10.23.0.52

[mon.proxmox-02]
     public_addr = 10.23.0.53

[mon.tiebreaker-proxmox-01]
     public_addr = 10.23.0.56

My Issue is, that i can not create a ceph_pool with min_size = 1.

I tried it over the web-UI and the command line with the same result.
The command was:
pveceph pool create vm_pool_new --add_storages 1 --min_size 1 --size 2
It does not throw any errors

But it creates a pool with min_size = 2 as it shows in the Web-UI.

Code:

root@proxmox-02:~# pveceph  pool create vm_pool_new --add_storages 1 --min_size 1 --size 2
root@proxmox-02:~# pveceph pool ls
Name                       size   min_size     pg_num     %-used                 used
vm_pool                       2          2        128       0.24        1799372792324
vm_pool_new                   2          2        128       0.00                    0

Any idea why this occurs or how i can fix this?

Thanks,
AKA

EDIT:
I just realised, that there are default values in the global section of the ceph configuration.
However, i thought those could be overwritten and if not, i'm at least expecting an error message.

Klaus Steinberger · Oct 19, 2020

every Documentation of CEPH tells you cleanly why min_size 1 is a very bad idea, keep your fingers from it.

min_size 2 and size 3 is fine!

RokaKen · Oct 19, 2020

@AKA I agree this should work -- it did in 5.4, but I get the same result as you in 6.2, plus doing so results in the following errors when the pool is 'destroyed':

Code:

Oct 19 09:20:38 ash01-rs-prmxt04 pveceph[1057791]: <root@pam> starting task UPID:ash01-rs-prmxt04:00102400:05B0665D:5F8D92A6:cephcreatepool:vm_pool_new:root@pam:
Oct 19 09:20:43 ash01-rs-prmxt04 pveceph[1057791]: <root@pam> end task UPID:ash01-rs-prmxt04:00102400:05B0665D:5F8D92A6:cephcreatepool:vm_pool_new:root@pam: OK
Oct 19 09:20:49 ash01-rs-prmxt04 pveproxy[1047606]: Clearing outdated entries from certificate cache
Oct 19 09:21:00 ash01-rs-prmxt04 systemd[1]: Starting Proxmox VE replication runner...
Oct 19 09:21:01 ash01-rs-prmxt04 systemd[1]: pvesr.service: Succeeded.
Oct 19 09:21:01 ash01-rs-prmxt04 systemd[1]: Started Proxmox VE replication runner.
Oct 19 09:21:23 ash01-rs-prmxt04 pveceph[1058111]: <root@pam> starting task UPID:ash01-rs-prmxt04:00102540:05B077E3:5F8D92D3:cephdestroypool:vm_pool_new:root@pam:
Oct 19 09:21:24 ash01-rs-prmxt04 pveceph[1058111]: <root@pam> end task UPID:ash01-rs-prmxt04:00102540:05B077E3:5F8D92D3:cephdestroypool:vm_pool_new:root@pam: OK
Oct 19 09:21:26 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $free in addition (+) at /usr/share/perl5/PVE/Storage/RBDPlugin.pm line 557.
Oct 19 09:21:26 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $used in addition (+) at /usr/share/perl5/PVE/Storage/RBDPlugin.pm line 557.
Oct 19 09:21:26 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $avail in int at /usr/share/perl5/PVE/Storage.pm line 1150.
Oct 19 09:21:26 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $used in int at /usr/share/perl5/PVE/Storage.pm line 1151.
Oct 19 09:21:36 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $free in addition (+) at /usr/share/perl5/PVE/Storage/RBDPlugin.pm line 557.
Oct 19 09:21:36 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $used in addition (+) at /usr/share/perl5/PVE/Storage/RBDPlugin.pm line 557.
Oct 19 09:21:36 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $avail in int at /usr/share/perl5/PVE/Storage.pm line 1150.
Oct 19 09:21:36 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $used in int at /usr/share/perl5/PVE/Storage.pm line 1151.
Oct 19 09:21:46 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $free in addition (+) at /usr/share/perl5/PVE/Storage/RBDPlugin.pm line 557.
Oct 19 09:21:46 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $used in addition (+) at /usr/share/perl5/PVE/Storage/RBDPlugin.pm line 557.
Oct 19 09:21:46 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $avail in int at /usr/share/perl5/PVE/Storage.pm line 1150.
Oct 19 09:21:46 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $used in int at /usr/share/perl5/PVE/Storage.pm line 1151.
Oct 19 09:21:56 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $free in addition (+) at /usr/share/perl5/PVE/Storage/RBDPlugin.pm line 557.
Oct 19 09:21:56 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $used in addition (+) at /usr/share/perl5/PVE/Storage/RBDPlugin.pm line 557.
Oct 19 09:21:56 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $avail in int at /usr/share/perl5/PVE/Storage.pm line 1150.
Oct 19 09:21:56 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $used in int at /usr/share/perl5/PVE/Storage.pm line 1151.

The 'destroy' removes the pool from Ceph, but not the /etc/pve/storage.cfg. Manually removing the pool and restarting services clears the errors.

Code:

proxmox-ve: 6.2-2 (running kernel: 5.4.65-1-pve)
pve-manager: 6.2-12 (running version: 6.2-12/b287dd27)
pve-kernel-5.4: 6.2-7
pve-kernel-helper: 6.2-7
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.4.60-1-pve: 5.4.60-2
ceph: 14.2.11-pve1
ceph-fuse: 14.2.11-pve1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-8
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
openvswitch-switch: 2.12.0-1
proxmox-backup-client: 0.9.0-2
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.3-1
pve-cluster: 6.2-1
pve-container: 3.2-2
pve-docs: 6.2-6
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-1
pve-qemu-kvm: 5.1.0-3
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-15
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve2

@Alwin can you investigate? We should be able to create a 2/1 replicated pool in Nautilus.

AKA · Oct 19, 2020

@RokaKen ~~, i can also confirm the behaviour with the pool not being removed from storage.cfg when it is destroyed.~~
Update to this below.

Aaaaaand, i figured out at least, why i can't create a pool with min_size = 1. Or any pool size besides min_size > 2 as seen here:

Code:

root@proxmox-01:/usr/share/perl5/PVE/API2# pveceph  pool create vm_pool_new --size 3 --min_size 3
mon_command failed - pool min_size must be between 1 and size, which is set to 2
root@proxmox-01:/usr/share/perl5/PVE/API2# pveceph pool ls
Name                       size   min_size     pg_num     %-used                 used
vm_pool                       2          2        128       0.26        1954478221071
vm_pool_new                   2          2        128       0.00                    0
root@proxmox-01:/usr/share/perl5/PVE/API2# pveceph pool destroy vm_pool_new
root@proxmox-01:/usr/share/perl5/PVE/API2# pveceph  pool create vm_pool_new --size 4 --min_size 3
mon_command failed - pool min_size must be between 1 and size, which is set to 2
root@proxmox-01:/usr/share/perl5/PVE/API2# pveceph pool ls
Name                       size   min_size     pg_num     %-used                 used
vm_pool                       2          2        128       0.26        1954479007503
vm_pool_new                   2          2        128       0.00                    0

The Problem lays in: /usr/share/perl5/PVE/Ceph/Tools.pm starting at line 205

Code:

sub create_pool {
    my ($pool, $param, $rados) = @_;

    if (!defined($rados)) {
        $rados = PVE::RADOS->new();
    }

    my $pg_num = $param->{pg_num} || 128;
    my $size = $param->{size} || 3;
    my $min_size = $param->{min_size} || 2;
    my $application = $param->{application} // 'rbd';

    $rados->mon_command({
        prefix => "osd pool create",
        pool => $pool,
        pg_num => int($pg_num),
        format => 'plain',
    });

    $rados->mon_command({
        prefix => "osd pool set",
        pool => $pool,
        var => 'min_size',
        val => "$min_size",
        format => 'plain',
    });

    $rados->mon_command({
        prefix => "osd pool set",
        pool => $pool,
        var => 'size',
        val => "$size",
        format => 'plain',
    });

The code is trying to set min_size before the actual size. Switching those two code blocks fixes the error in both scenarios.
As a confirmation i did that in my setup to test it and got the following results:

Code:

root@proxmox-01:/usr/share/perl5/PVE# pveceph  pool create vm_pool_new --size 2 --min_size 1
root@proxmox-01:/usr/share/perl5/PVE# pveceph pool ls
Name                       size   min_size     pg_num     %-used                 used
vm_pool                       2          2        128       0.26        1954484021007
vm_pool_new                   2          1        128       0.00                    0
root@proxmox-01:/usr/share/perl5/PVE# pveceph pool destroy vm_pool_new
root@proxmox-01:/usr/share/perl5/PVE# pveceph  pool create vm_pool_new --size 4 --min_size 3
root@proxmox-01:/usr/share/perl5/PVE# pveceph pool ls
Name                       size   min_size     pg_num     %-used                 used
vm_pool                       2          2        128       0.26        1954484021007
vm_pool_new                   4          3        128       0.00                    0
root@proxmox-01:/usr/share/perl5/PVE# pveceph pool destroy vm_pool_new
root@proxmox-01:/usr/share/perl5/PVE# pveceph  pool create vm_pool_new --size 3 --min_size 3
root@proxmox-01:/usr/share/perl5/PVE# pveceph pool ls
Name                       size   min_size     pg_num     %-used                 used
vm_pool                       2          2        128       0.26        1954484021007
vm_pool_new                   3          3        128       0.00                    0

So it seems like a logic Bug that just needs to be fixed.

~~The "not removing from storage" issue still happens though. Maybe i can figure that one out too.~~

@Alwin , do i need to create a bug report at bugzilla, or is this thread enough?

EDIT:

I figured it out. @RokaKen
Is was an user error. When destroying a ceph pool with pveceph you need the --remove_storages 1 parameter.
Then it works as intended.

Alwin · Oct 20, 2020

AKA said:
@Alwin , do i need to create a bug report at bugzilla, or is this thread enough?

I posted patches yesterday that redo that part. With the intention to allow the changing of pool values. In most cases this will set size before min_size.
https://lists.proxmox.com/pipermail/pve-devel/2020-October/045573.html

And see the commit, the min_size is now calculated. This means if you set size=2 it will have min_size=1.
https://github.com/ceph/ceph/pull/34585/commits/e8b79077eb6e93821a1f3619d09fc707c8beb0bd

In addition the osd_pool_default_min_size in the ceph.conf seems to be honored in the calculation.

AKA · Oct 20, 2020

Hi Alwin,

thanks for the Update

I guess this is resolved then. Everyone that needs a fix in the meantime can either swap those code blocks i mentioned above (I do not recommend)
or just change the CEPH Pools values after creation with the ceph commands found here: https://docs.ceph.com/en/latest/rados/operations/pools/

Alwin · Oct 20, 2020

AKA said:
I guess this is resolved then. Everyone that needs a fix in the meantime can either swap those code blocks i mentioned above (I do not recommend)
or just change the CEPH Pools values after creation with the ceph commands found here: https://docs.ceph.com/en/latest/rados/operations/pools/

As you said yourself, I want to emphasize. I STRONGLY ADVISE AGAINST IT! As disaster strikes, data may be lost. With only one copy of the data, they may be in-flight and a subsequent failure might just lose the data.

If storage usage is an issue, the lowest option is size=2 and min_size=2. While the pool is in read-only when the min_size is breached, the pool will only seek to have two copies.

AKA · Oct 20, 2020

I agree.
I posted it just for the users, that want to create a pool with > 2 min_size.
It's useful to know how to fix the state of having a 2/2 pool after wanting to create a 3/3 or 4/3 or 5/4 etc. Pool.

Just a last hint: changing the values with the ceph commands does not break proxmox. It recognizes it and displays it correctly too.
For more information why min_size=1 is bad i advise you to read here:
https://forum.proxmox.com/threads/ceph-pool-size-is-2-1-really-a-bad-idea.68939/
Just as a reference.

Search

Search

Cannot create CEPH Pool with min_size=1 or min_size > 2

AKA

Active Member

Klaus Steinberger

Renowned Member

RokaKen

Active Member

AKA

Active Member

Alwin

Proxmox Retired Staff

AKA

Active Member

Alwin

Proxmox Retired Staff

AKA

Active Member

We value your privacy