Cannot create CEPH Pool with min_size=1 or min_size > 2

AKA · Oct 19, 2020

Hello everyone,

i'm running a 3 Node Proxmox Cluster on Version 6.2-4.
I'm having CEPH configured as my primary storage system.

One of the Nodes is a VM acting as a tiebreaker / CEPH Monitor restricted from running VMs.
I used a private 10GbE Network as Cluster_network for CEPH for syncing the OSDs.

Here's my config:

Code:

[global]
     auth_client_required = cephx
     auth_cluster_required = cephx
     auth_service_required = cephx
     cluster_network = 192.168.100.52/24
     fsid = 0b4b4108-0d40-44d7-916b-0546e915ddb5
     mon_allow_pool_delete = true
     mon_host = 10.23.0.52 10.23.0.53 10.23.0.56
     osd_pool_default_min_size = 2
     osd_pool_default_size = 2
     public_network = 10.23.0.52/17

[client]
     keyring = /etc/pve/priv/$cluster.$name.keyring

[mds]
     keyring = /var/lib/ceph/mds/ceph-$id/keyring

[mon.proxmox-01]
     public_addr = 10.23.0.52

[mon.proxmox-02]
     public_addr = 10.23.0.53

[mon.tiebreaker-proxmox-01]
     public_addr = 10.23.0.56

My Issue is, that i can not create a ceph_pool with min_size = 1.

I tried it over the web-UI and the command line with the same result.
The command was:
pveceph pool create vm_pool_new --add_storages 1 --min_size 1 --size 2
It does not throw any errors

But it creates a pool with min_size = 2 as it shows in the Web-UI.

Code:

root@proxmox-02:~# pveceph  pool create vm_pool_new --add_storages 1 --min_size 1 --size 2
root@proxmox-02:~# pveceph pool ls
Name                       size   min_size     pg_num     %-used                 used
vm_pool                       2          2        128       0.24        1799372792324
vm_pool_new                   2          2        128       0.00                    0

Any idea why this occurs or how i can fix this?

Thanks,
AKA

EDIT:
I just realised, that there are default values in the global section of the ceph configuration.
However, i thought those could be overwritten and if not, i'm at least expecting an error message.

Klaus Steinberger · Oct 19, 2020

every Documentation of CEPH tells you cleanly why min_size 1 is a very bad idea, keep your fingers from it.

min_size 2 and size 3 is fine!

RokaKen · Oct 19, 2020

@AKA I agree this should work -- it did in 5.4, but I get the same result as you in 6.2, plus doing so results in the following errors when the pool is 'destroyed':

Code:

Oct 19 09:20:38 ash01-rs-prmxt04 pveceph[1057791]: <root@pam> starting task UPID:ash01-rs-prmxt04:00102400:05B0665D:5F8D92A6:cephcreatepool:vm_pool_new:root@pam:
Oct 19 09:20:43 ash01-rs-prmxt04 pveceph[1057791]: <root@pam> end task UPID:ash01-rs-prmxt04:00102400:05B0665D:5F8D92A6:cephcreatepool:vm_pool_new:root@pam: OK
Oct 19 09:20:49 ash01-rs-prmxt04 pveproxy[1047606]: Clearing outdated entries from certificate cache
Oct 19 09:21:00 ash01-rs-prmxt04 systemd[1]: Starting Proxmox VE replication runner...
Oct 19 09:21:01 ash01-rs-prmxt04 systemd[1]: pvesr.service: Succeeded.
Oct 19 09:21:01 ash01-rs-prmxt04 systemd[1]: Started Proxmox VE replication runner.
Oct 19 09:21:23 ash01-rs-prmxt04 pveceph[1058111]: <root@pam> starting task UPID:ash01-rs-prmxt04:00102540:05B077E3:5F8D92D3:cephdestroypool:vm_pool_new:root@pam:
Oct 19 09:21:24 ash01-rs-prmxt04 pveceph[1058111]: <root@pam> end task UPID:ash01-rs-prmxt04:00102540:05B077E3:5F8D92D3:cephdestroypool:vm_pool_new:root@pam: OK
Oct 19 09:21:26 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $free in addition (+) at /usr/share/perl5/PVE/Storage/RBDPlugin.pm line 557.
Oct 19 09:21:26 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $used in addition (+) at /usr/share/perl5/PVE/Storage/RBDPlugin.pm line 557.
Oct 19 09:21:26 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $avail in int at /usr/share/perl5/PVE/Storage.pm line 1150.
Oct 19 09:21:26 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $used in int at /usr/share/perl5/PVE/Storage.pm line 1151.
Oct 19 09:21:36 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $free in addition (+) at /usr/share/perl5/PVE/Storage/RBDPlugin.pm line 557.
Oct 19 09:21:36 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $used in addition (+) at /usr/share/perl5/PVE/Storage/RBDPlugin.pm line 557.
Oct 19 09:21:36 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $avail in int at /usr/share/perl5/PVE/Storage.pm line 1150.
Oct 19 09:21:36 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $used in int at /usr/share/perl5/PVE/Storage.pm line 1151.
Oct 19 09:21:46 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $free in addition (+) at /usr/share/perl5/PVE/Storage/RBDPlugin.pm line 557.
Oct 19 09:21:46 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $used in addition (+) at /usr/share/perl5/PVE/Storage/RBDPlugin.pm line 557.
Oct 19 09:21:46 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $avail in int at /usr/share/perl5/PVE/Storage.pm line 1150.
Oct 19 09:21:46 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $used in int at /usr/share/perl5/PVE/Storage.pm line 1151.
Oct 19 09:21:56 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $free in addition (+) at /usr/share/perl5/PVE/Storage/RBDPlugin.pm line 557.
Oct 19 09:21:56 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $used in addition (+) at /usr/share/perl5/PVE/Storage/RBDPlugin.pm line 557.
Oct 19 09:21:56 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $avail in int at /usr/share/perl5/PVE/Storage.pm line 1150.
Oct 19 09:21:56 ash01-rs-prmxt04 pvestatd[2072]: Use of uninitialized value $used in int at /usr/share/perl5/PVE/Storage.pm line 1151.

The 'destroy' removes the pool from Ceph, but not the /etc/pve/storage.cfg. Manually removing the pool and restarting services clears the errors.

Code:

proxmox-ve: 6.2-2 (running kernel: 5.4.65-1-pve)
pve-manager: 6.2-12 (running version: 6.2-12/b287dd27)
pve-kernel-5.4: 6.2-7
pve-kernel-helper: 6.2-7
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.4.60-1-pve: 5.4.60-2
ceph: 14.2.11-pve1
ceph-fuse: 14.2.11-pve1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-8
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
openvswitch-switch: 2.12.0-1
proxmox-backup-client: 0.9.0-2
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.3-1
pve-cluster: 6.2-1
pve-container: 3.2-2
pve-docs: 6.2-6
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-1
pve-qemu-kvm: 5.1.0-3
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-15
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve2

@Alwin can you investigate? We should be able to create a 2/1 replicated pool in Nautilus.

AKA · Oct 19, 2020

@RokaKen ~~, i can also confirm the behaviour with the pool not being removed from storage.cfg when it is destroyed.~~
Update to this below.

Aaaaaand, i figured out at least, why i can't create a pool with min_size = 1. Or any pool size besides min_size > 2 as seen here:

Code:

root@proxmox-01:/usr/share/perl5/PVE/API2# pveceph  pool create vm_pool_new --size 3 --min_size 3
mon_command failed - pool min_size must be between 1 and size, which is set to 2
root@proxmox-01:/usr/share/perl5/PVE/API2# pveceph pool ls
Name                       size   min_size     pg_num     %-used                 used
vm_pool                       2          2        128       0.26        1954478221071
vm_pool_new                   2          2        128       0.00                    0
root@proxmox-01:/usr/share/perl5/PVE/API2# pveceph pool destroy vm_pool_new
root@proxmox-01:/usr/share/perl5/PVE/API2# pveceph  pool create vm_pool_new --size 4 --min_size 3
mon_command failed - pool min_size must be between 1 and size, which is set to 2
root@proxmox-01:/usr/share/perl5/PVE/API2# pveceph pool ls
Name                       size   min_size     pg_num     %-used                 used
vm_pool                       2          2        128       0.26        1954479007503
vm_pool_new                   2          2        128       0.00                    0

The Problem lays in: /usr/share/perl5/PVE/Ceph/Tools.pm starting at line 205

Code:

sub create_pool {
    my ($pool, $param, $rados) = @_;

    if (!defined($rados)) {
        $rados = PVE::RADOS->new();
    }

    my $pg_num = $param->{pg_num} || 128;
    my $size = $param->{size} || 3;
    my $min_size = $param->{min_size} || 2;
    my $application = $param->{application} // 'rbd';

    $rados->mon_command({
        prefix => "osd pool create",
        pool => $pool,
        pg_num => int($pg_num),
        format => 'plain',
    });

    $rados->mon_command({
        prefix => "osd pool set",
        pool => $pool,
        var => 'min_size',
        val => "$min_size",
        format => 'plain',
    });

    $rados->mon_command({
        prefix => "osd pool set",
        pool => $pool,
        var => 'size',
        val => "$size",
        format => 'plain',
    });

The code is trying to set min_size before the actual size. Switching those two code blocks fixes the error in both scenarios.
As a confirmation i did that in my setup to test it and got the following results:

Code:

root@proxmox-01:/usr/share/perl5/PVE# pveceph  pool create vm_pool_new --size 2 --min_size 1
root@proxmox-01:/usr/share/perl5/PVE# pveceph pool ls
Name                       size   min_size     pg_num     %-used                 used
vm_pool                       2          2        128       0.26        1954484021007
vm_pool_new                   2          1        128       0.00                    0
root@proxmox-01:/usr/share/perl5/PVE# pveceph pool destroy vm_pool_new
root@proxmox-01:/usr/share/perl5/PVE# pveceph  pool create vm_pool_new --size 4 --min_size 3
root@proxmox-01:/usr/share/perl5/PVE# pveceph pool ls
Name                       size   min_size     pg_num     %-used                 used
vm_pool                       2          2        128       0.26        1954484021007
vm_pool_new                   4          3        128       0.00                    0
root@proxmox-01:/usr/share/perl5/PVE# pveceph pool destroy vm_pool_new
root@proxmox-01:/usr/share/perl5/PVE# pveceph  pool create vm_pool_new --size 3 --min_size 3
root@proxmox-01:/usr/share/perl5/PVE# pveceph pool ls
Name                       size   min_size     pg_num     %-used                 used
vm_pool                       2          2        128       0.26        1954484021007
vm_pool_new                   3          3        128       0.00                    0

So it seems like a logic Bug that just needs to be fixed.

~~The "not removing from storage" issue still happens though. Maybe i can figure that one out too.~~

@Alwin , do i need to create a bug report at bugzilla, or is this thread enough?

EDIT:

I figured it out. @RokaKen
Is was an user error. When destroying a ceph pool with pveceph you need the --remove_storages 1 parameter.
Then it works as intended.

Alwin · Oct 20, 2020

AKA said:
@Alwin , do i need to create a bug report at bugzilla, or is this thread enough?

I posted patches yesterday that redo that part. With the intention to allow the changing of pool values. In most cases this will set size before min_size.
https://lists.proxmox.com/pipermail/pve-devel/2020-October/045573.html

And see the commit, the min_size is now calculated. This means if you set size=2 it will have min_size=1.
https://github.com/ceph/ceph/pull/34585/commits/e8b79077eb6e93821a1f3619d09fc707c8beb0bd

In addition the osd_pool_default_min_size in the ceph.conf seems to be honored in the calculation.

AKA · Oct 20, 2020

Hi Alwin,

thanks for the Update

I guess this is resolved then. Everyone that needs a fix in the meantime can either swap those code blocks i mentioned above (I do not recommend)
or just change the CEPH Pools values after creation with the ceph commands found here: https://docs.ceph.com/en/latest/rados/operations/pools/

Alwin · Oct 20, 2020

AKA said:
I guess this is resolved then. Everyone that needs a fix in the meantime can either swap those code blocks i mentioned above (I do not recommend)
or just change the CEPH Pools values after creation with the ceph commands found here: https://docs.ceph.com/en/latest/rados/operations/pools/

As you said yourself, I want to emphasize. I STRONGLY ADVISE AGAINST IT! As disaster strikes, data may be lost. With only one copy of the data, they may be in-flight and a subsequent failure might just lose the data.

If storage usage is an issue, the lowest option is size=2 and min_size=2. While the pool is in read-only when the min_size is breached, the pool will only seek to have two copies.

AKA · Oct 20, 2020

I agree.
I posted it just for the users, that want to create a pool with > 2 min_size.
It's useful to know how to fix the state of having a 2/2 pool after wanting to create a 3/3 or 4/3 or 5/4 etc. Pool.

Just a last hint: changing the values with the ceph commands does not break proxmox. It recognizes it and displays it correctly too.
For more information why min_size=1 is bad i advise you to read here:
https://forum.proxmox.com/threads/ceph-pool-size-is-2-1-really-a-bad-idea.68939/
Just as a reference.

Search

Search

Cannot create CEPH Pool with min_size=1 or min_size > 2

AKA

Member

Klaus Steinberger

Renowned Member

RokaKen

Active Member

AKA

Member

Alwin

Proxmox Retired Staff

AKA

Member

Alwin

Proxmox Retired Staff

AKA

Member