Ceph Upgrade To Quincy .. "Ceph is not installed on this node"

L

leex12

Member
Mar 11, 2022
36
1
13
53
Hi,

I have just upgraded my first node from 16.2.13 to Quincy. After the update/reboot node has returned but the ceph services have not come back. i.e. i can see the ceoh mounts but server and OSD are showing as out. When you go to the server it looks like the first time you go to install ceph ..
"Ceph is not installed on this node. Would you like to install it now?"

This is obviously not what is supposed to happen (at least its not mentioned in the guide)? Should I confirm the install or is there a way to back out?
 
Can you please post the output of the following commands inside [CODE][/CODE] tags?

  • pveversion -v
  • ceph -s
  • apt search ceph | grep installed
  • ceph osd df tree
 
proxmox-ve: 7.4-1 (running kernel: 5.15.108-1-pve) pve-manager: 7.4-15 (running version: 7.4-15/a5d2a31e) pve-kernel-5.15: 7.4-4 pve-kernel-5.13: 7.1-9 pve-kernel-5.15.108-1-pve: 5.15.108-1 pve-kernel-5.15.107-2-pve: 5.15.107-2 pve-kernel-5.13.19-6-pve: 5.13.19-15 pve-kernel-5.13.19-2-pve: 5.13.19-4 ceph-fuse: 16.2.13-pve1 corosync: 3.1.7-pve1 criu: 3.15-1+pve-1 glusterfs-client: 9.2-1 ifupdown2: 3.1.0-1+pmx4 ksm-control-daemon: 1.4-1 libjs-extjs: 7.0.0-1 libknet1: 1.24-pve2 libproxmox-acme-perl: 1.4.4 libproxmox-backup-qemu0: 1.3.1-1 libproxmox-rs-perl: 0.2.1 libpve-access-control: 7.4.1 libpve-apiclient-perl: 3.2-1 libpve-common-perl: 7.4-2 libpve-guest-common-perl: 4.2-4 libpve-http-server-perl: 4.2-3 libpve-rs-perl: 0.7.7 libpve-storage-perl: 7.4-3 libspice-server1: 0.14.3-2.1 lvm2: 2.03.11-2.1 lxc-pve: 5.0.2-2 lxcfs: 5.0.3-pve1 novnc-pve: 1.4.0-1 proxmox-backup-client: 2.4.2-1 proxmox-backup-file-restore: 2.4.2-1 proxmox-kernel-helper: 7.4-1 proxmox-mail-forward: 0.1.1-1 proxmox-mini-journalreader: 1.3-1 proxmox-offline-mirror-helper: 0.5.2 proxmox-widget-toolkit: 3.7.3 pve-cluster: 7.3-3 pve-container: 4.4-6 pve-docs: 7.4-2 pve-edk2-firmware: 3.20230228-4~bpo11+1 pve-firewall: 4.3-4 pve-firmware: 3.6-5 pve-ha-manager: 3.6.1 pve-i18n: 2.12-1 pve-qemu-kvm: 7.2.0-8 pve-xtermjs: 4.16.0-2 qemu-server: 7.4-4 smartmontools: 7.2-pve3 spiceterm: 3.2-2 swtpm: 0.8.0~bpo11+3 vncterm: 1.7-1 zfsutils-linux: 2.1.11-pve1

Code: 
cluster:
    id:     1e8245d2-c907-490c-98e0-cddf1c2dea80
    health: HEALTH_WARN
            mons are allowing insecure global_id reclaim
            1 clients failing to respond to cache pressure
            1 MDSs report slow metadata IOs
            1/3 mons down, quorum pve02,pve04
            noout flag(s) set
            5 osds down
            1 host (5 osds) down
            1 nearfull osd(s)
            Reduced data availability: 44 pgs inactive
            Degraded data redundancy: 10496759/42306983 objects degraded (24.811%), 622 pgs degraded, 637 pgs undersized
            5 pool(s) nearfull
            1 subtrees have overcommitted pool target_size_bytes
 
  services:
    mon: 3 daemons, quorum pve02,pve04 (age 57m), out of quorum: pve03
    mgr: pve01(active, since 2h), standbys: pve04
    mds: 2/2 daemons up, 1 standby
    osd: 20 osds: 15 up (since 54m), 20 in (since 9w)
         flags noout
 
  data:
    volumes: 2/2 healthy
    pools:   9 pools, 817 pgs
    objects: 14.88M objects, 55 TiB
    usage:   119 TiB used, 37 TiB / 156 TiB avail
    pgs:     5.386% pgs not active
             10496759/42306983 objects degraded (24.811%)
             593 active+undersized+degraded
             178 active+clean
             29  undersized+degraded+peered
             15  undersized+peered
             2   active+clean+scrubbing+deep
 
  io:
    client:   29 KiB/s rd, 346 KiB/s wr, 5 op/s rd, 24 op/s wr

Code: 
libjaeger/now 16.2.13-pve1 amd64 [installed,local]


Code: 
ID    CLASS  WEIGHT     REWEIGHT  SIZE     RAW USE  DATA     OMAP     META     AVAIL    %USE   VAR   PGS  STATUS  TYPE NAME       
  -1         212.61264         -   56 TiB   42 TiB   41 TiB  422 MiB   69 GiB   15 TiB      0     0    -          root default    
  -3          46.34888         -   46 TiB   36 TiB   36 TiB  381 MiB   60 GiB   11 TiB  77.14  1.01    -              host pve01  
1201    hdd   16.37109   1.00000   16 TiB   12 TiB   12 TiB   86 KiB   20 GiB  3.9 TiB  76.15  1.00  144      up          osd.1201
1202    hdd   18.19040   1.00000   18 TiB   14 TiB   14 TiB   95 KiB   25 GiB  4.0 TiB  77.94  1.02  165      up          osd.1202
1203    hdd   10.91409   1.00000   11 TiB  8.7 TiB  8.6 TiB   47 KiB   14 GiB  2.3 TiB  79.34  1.04  100      up          osd.1203
1101    ssd    0.87329   1.00000  894 GiB  461 GiB  460 GiB  381 MiB  1.5 GiB  433 GiB  51.60  0.68  105      up          osd.1101
  -5          53.62625         -   54 TiB   42 TiB   42 TiB  371 MiB   71 GiB   12 TiB  77.96  1.02    -              host pve02  
2201    hdd   16.37109   1.00000   16 TiB   14 TiB   14 TiB  126 KiB   24 GiB  2.4 TiB  85.51  1.12  159      up          osd.2201
2202    hdd   18.19040   1.00000   18 TiB   14 TiB   14 TiB  109 KiB   23 GiB  4.5 TiB  75.44  0.99  161      up          osd.2202
2203    hdd   10.91409   1.00000   11 TiB  8.4 TiB  8.4 TiB   63 KiB   14 GiB  2.5 TiB  76.92  1.01   95      up          osd.2203
2302    hdd    3.63869   1.00000  3.6 TiB  2.3 TiB  2.3 TiB   34 KiB  3.7 GiB  1.3 TiB  64.46  0.85   26      up          osd.2302
2303    hdd    3.63869   1.00000  3.6 TiB  2.9 TiB  2.9 TiB   21 KiB  4.5 GiB  761 GiB  79.58  1.04   33      up          osd.2303
2101    ssd    0.87329   1.00000  894 GiB  462 GiB  460 GiB  370 MiB  1.3 GiB  433 GiB  51.61  0.68  105      up          osd.2101
  -7          56.31876         -      0 B      0 B      0 B      0 B      0 B      0 B      0     0    -              host pve03  
3201    hdd   10.91409   1.00000      0 B      0 B      0 B      0 B      0 B      0 B      0     0    0    down          osd.3201
3202    hdd   14.55269   1.00000      0 B      0 B      0 B      0 B      0 B      0 B      0     0    0    down          osd.3202
3203    hdd   14.55269   1.00000      0 B      0 B      0 B      0 B      0 B      0 B      0     0    0    down          osd.3203
3204    hdd   14.55269   1.00000      0 B      0 B      0 B      0 B      0 B      0 B      0     0    0    down          osd.3204
3101    ssd    1.74660   1.00000      0 B      0 B      0 B      0 B      0 B      0 B      0     0    0    down          osd.3101
  -9          56.31876         -   56 TiB   42 TiB   41 TiB  422 MiB   69 GiB   15 TiB  73.74  0.97    -              host pve04  
4201    hdd   10.91409   1.00000   11 TiB  8.2 TiB  8.2 TiB   50 KiB   14 GiB  2.7 TiB  75.17  0.99   95      up          osd.4201
4202    hdd   14.55269   1.00000   15 TiB   11 TiB   11 TiB   66 KiB   18 GiB  3.5 TiB  75.65  0.99  128      up          osd.4202
4203    hdd   14.55269   1.00000   15 TiB   11 TiB   11 TiB   85 KiB   18 GiB  3.6 TiB  75.06  0.99  127      up          osd.4203
4204    hdd   14.55269   1.00000   15 TiB   11 TiB   11 TiB   63 KiB   17 GiB  4.0 TiB  72.66  0.95  123      up          osd.4204
4101    ssd    1.74660   1.00000  1.7 TiB  840 GiB  837 GiB  421 MiB  2.6 GiB  948 GiB  46.97  0.62  169      up          osd.4101
 -21                 0         -      0 B      0 B      0 B      0 B      0 B      0 B      0     0    -              host pve08  
                           TOTAL  156 TiB  119 TiB  119 TiB  1.1 GiB  200 GiB   37 TiB  76.20                                     


[/CODE}
 
leex12 said:
libjaeger/now 16.2.13-pve1 amd64 [installed,local]
Click to expand...
interesting that there are not other packages shown. Did you run exactly this command? apt search ceph | grep installed

Are the repositories configured correctly? Can you post the output of apt update please?
 
This is on another node which I have tried to update ceph on ..
Code: 
root@pve01:~# apt update
Hit:1 http://ftp.uk.debian.org/debian bullseye InRelease
Get:2 http://ftp.uk.debian.org/debian bullseye-updates InRelease [44.1 kB]      
Hit:3 http://security.debian.org bullseye-security InRelease                         
Hit:4 http://download.proxmox.com/debian/ceph-pacific bullseye InRelease
Hit:5 http://download.proxmox.com/debian/pve bullseye InRelease
Fetched 44.1 kB in 0s (112 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
root@pve01:~# apt search ceph | grep installed

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

ceph/stable,now 16.2.13-pve1 amd64 [installed]
ceph-base/stable,now 16.2.13-pve1 amd64 [installed,automatic]
ceph-common/stable,now 16.2.13-pve1 amd64 [installed]
ceph-fuse/stable,now 16.2.13-pve1 amd64 [installed]
ceph-mds/stable,now 16.2.13-pve1 amd64 [installed]
ceph-mgr/stable,now 16.2.13-pve1 amd64 [installed,automatic]
ceph-mgr-modules-core/stable,now 16.2.13-pve1 all [installed,automatic]
ceph-mon/stable,now 16.2.13-pve1 amd64 [installed,automatic]
ceph-osd/stable,now 16.2.13-pve1 amd64 [installed,automatic]
libcephfs2/stable,now 16.2.13-pve1 amd64 [installed]
libjaeger/stable,now 16.2.13-pve1 amd64 [installed,automatic]
librados2/stable,now 16.2.13-pve1 amd64 [installed]
librbd1/stable,now 16.2.13-pve1 amd64 [installed]
librgw2/stable,now 16.2.13-pve1 amd64 [installed]
libsqlite3-mod-ceph/stable,now 16.2.13-pve1 amd64 [installed,automatic]
python3-ceph-argparse/stable,now 16.2.13-pve1 all [installed]
python3-ceph-common/stable,now 16.2.13-pve1 all [installed]
python3-cephfs/stable,now 16.2.13-pve1 amd64 [installed]
python3-rados/stable,now 16.2.13-pve1 amd64 [installed]
python3-rbd/stable,now 16.2.13-pve1 amd64 [installed]
python3-rgw/stable,now 16.2.13-pve1 amd64 [installed]

This is on the failed upgraded node
Code: 
apt update
Hit:1 http://security.debian.org bullseye-security InRelease
Hit:2 http://ftp.uk.debian.org/debian bullseye InRelease                                  
Hit:3 http://ftp.uk.debian.org/debian bullseye-updates InRelease                          
Hit:4 http://download.proxmox.com/debian/ceph-quincy bookworm InRelease
Hit:5 http://download.proxmox.com/debian/pve bullseye InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
15 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@pve03:~# apt upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
  python3-ceph-argparse python3-ceph-common
0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
root@pve03:~# apt search ceph | grep installed

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libjaeger/now 16.2.13-pve1 amd64 [installed,local]
 
yes noticed that as i posted .. going 'snow blind'

updated but still the same 'install prompt' for ceph

Code: 
root@pve03:~# apt upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  cryptsetup-bin libjaeger
Use 'apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root@pve03:~# apt search ceph | grep installed

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

ceph-base/stable,now 17.2.6-pve1 amd64 [installed,automatic]
ceph-common/stable,now 17.2.6-pve1 amd64 [installed]
ceph-fuse/stable,now 17.2.6-pve1 amd64 [installed]
ceph-mds/stable,now 17.2.6-pve1 amd64 [installed]
libcephfs2/stable,now 17.2.6-pve1 amd64 [installed]
libjaeger/now 16.2.13-pve1 amd64 [installed,local]
librados2/stable,now 17.2.6-pve1 amd64 [installed]
librbd1/stable,now 17.2.6-pve1 amd64 [installed]
librgw2/stable,now 17.2.6-pve1 amd64 [installed]
python3-ceph-argparse/stable,now 17.2.6-pve1 all [installed]
python3-ceph-common/stable,now 17.2.6-pve1 all [installed]
python3-cephfs/stable,now 17.2.6-pve1 amd64 [installed]
python3-rados/stable,now 17.2.6-pve1 amd64 [installed]
python3-rbd/stable,now 17.2.6-pve1 amd64 [installed]
python3-rgw/stable,now 17.2.6-pve1 amd64 [installed]
 
The other thing is, the ceph repo was set to quincy (17) and not pacific (16).

Try to remove all the Ceph packages that are version 17. Change the repo to bullseye and pacific and then run pveceph install --version pacific. That will hopefully get the correct versions installed again :)
 
Somehow I missed that you were in the process of upgrading to quincy ;)
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom