pve6to7 update repository confusion caused ceph to go down on updated node

I

inxamc

Active Member
Proxmox Subscriber
Dec 2, 2020
44
6
28
Hi Forum,

we run a hyperconverged 3 node pve-cluster still in eval mode (latest 6.4 pve and ceph 15.2.13)
I started an pve6to7-update on the first node of a healthy ceph-cluster with a faulty repository configuration, which I can't exactly recall anymore.
In addition I set the noout flag for the upgrade - after the update and a reboot and unsetting the noout flag ceph was down ( mgr and mon on this node did not start)
Because of the following message, I saw that there must have been an error in the repository config

1627467026889.png
I corrected the repoconfig and rerun the update but ceph still failed, while the other nodes are healthy, the ceph cluster is degraded.

The current repo-confing is as follows:

# in /etc/apt/
cat sources.list
deb http://ftp.de.debian.org/debian bullseye main contrib
deb http://ftp.de.debian.org/debian bullseye-updates main contrib
# security updates
deb http://security.debian.org bullseye-security main contrib
# proxmox
deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription
# in /etc/apt/sources.list.d/
cat ceph.list
deb http://download.proxmox.com/debian/ceph-octopus bullseye main

And this is ceph state
1627467640157.png



1627467793272.png
ceph -s
cluster:
id: ae713943-83f3-48b4-a0c2-124c092c250b
health: HEALTH_WARN
1/3 mons down, quorum amcvh12,amcvh13
3 osds down
1 host (15 osds) down
Degraded data redundancy: 448166/1344447 objects degraded (33.335%), 257 pgs degraded, 257 pgs undersized

services:
mon: 3 daemons, quorum amcvh12,amcvh13 (age 2d), out of quorum: amcvh11
mgr: amcvh13(active, since 5d), standbys: amcvh12
osd: 45 osds: 30 up (since 2d), 33 in (since 16h)

data:
pools: 3 pools, 257 pgs
objects: 448.17k objects, 1.6 TiB
usage: 1013 GiB used, 8.0 TiB / 9.0 TiB avail
pgs: 448166/1344447 objects degraded (33.335%)
257 active+undersized+degraded

io:
client: 0 B/s rd, 41 KiB/s wr, 0 op/s rd, 7 op/s wr


Any idea how to repair this?
Can somebody pls. comment on the use of setting the noout flag for upgrading a node - is this necessary or not?
If more inforamtion is neede, pls ask. Any help is highly appreciated!
 
Last edited:
Can't you manually start mon and mgr?

Setting noout during the upgrade prevents Ceph from doing any backfilling due to unexpected long reboot/down times. It's not directly necessary, but often pretty handy.
 
Apart from MON and MGR not starting, also your OSDs are not up. What is the output of systemctl status ceph-osd@X.service having X one of the OSD IDs on the failing host?
 
Hello ph0x - thanks for your support!
Output is:
# systemctl status ceph-osd@14.service
● ceph-osd@14.service - Ceph object storage daemon osd.14
Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled-runtim>
Drop-In: /lib/systemd/system/ceph-osd@.service.d
└─ceph-after-pve-cluster.conf
Active: inactive (dead) since Wed 2021-07-28 00:00:01 CEST; 13h ago
Process: 75043 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --clust>
Process: 75080 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} --i>
Main PID: 75080 (code=killed, signal=HUP)
CPU: 249ms

Jul 27 23:56:14 amcvh11 systemd[1]: Starting Ceph object storage daemon os>
Jul 27 23:56:15 amcvh11 systemd[1]: Started Ceph object storage daemon osd>
Jul 28 00:00:01 amcvh11 systemd[1]: ceph-osd@14.service: Succeeded.
 
Can you start this service (systemctl start ceph-osd@14.service) and what is the output of the status after that?
 
Last edited:
Yes :
root@amcvh11:~# systemctl start ceph-osd@14.service
root@amcvh11:~# systemctl status ceph-osd@14.service
● ceph-osd@14.service - Ceph object storage daemon osd.14
Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled-runtim>
Drop-In: /lib/systemd/system/ceph-osd@.service.d
└─ceph-after-pve-cluster.conf
Active: active (running) since Wed 2021-07-28 13:35:49 CEST; 14s ago
Process: 298824 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --clus>
Main PID: 298830 (ceph-osd)
Tasks: 9
Memory: 9.4M
CPU: 88ms
CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@14.service
└─298830 /usr/bin/ceph-osd -f --cluster ceph --id 14 --setuse>

Jul 28 13:35:49 amcvh11 systemd[1]: Starting Ceph object storage daemon os>
Jul 28 13:35:49 amcvh11 systemd[1]: Started Ceph object storage daemon osd>
 
Done - but it did not change anything - ceph on this updated node is somehow stuck -
the ceph -s command on this node does not finish - cursor is blinking
the output from second node does not show any diffrence after executing the restart
root@amcvh12:~# ceph -s
cluster:
id: ae713943-83f3-48b4-a0c2-124c092c250b
health: HEALTH_WARN
1/3 mons down, quorum amcvh12,amcvh13
3 osds down
1 host (15 osds) down
Degraded data redundancy: 448166/1344447 objects degraded (33.335%), 257 pgs degraded, 257 pgs undersized

services:
mon: 3 daemons, quorum amcvh12,amcvh13 (age 2d), out of quorum: amcvh11
mgr: amcvh13(active, since 5d), standbys: amcvh12
osd: 45 osds: 30 up (since 2d), 33 in (since 18h)

data:
pools: 3 pools, 257 pgs
objects: 448.17k objects, 1.6 TiB
usage: 1013 GiB used, 8.0 TiB / 9.0 TiB avail
pgs: 448166/1344447 objects degraded (33.335%)
257 active+undersized+degraded

io:
client: 0 B/s rd, 30 KiB/s wr, 0 op/s rd, 3 op/s wr



although systemctl status does show the 3 OSD to be active running
 
Last edited:
The cluster communication is offline, therefore the online OSDs are not recognized.
Try starting MON and MGR once more and provide the status output.

Please put the output in [ code ] tags, that way it's more readable. :)
 
Nice, wasn't aware of the code tag - thx!

Code: 
root@amcvh12:~# systemctl status ceph-mgr.target
● ceph-mgr.target - ceph target allowing to start/stop all ceph-mgr@.ser
   Loaded: loaded (/lib/systemd/system/ceph-mgr.target; enabled; vendor
   Active: active since Thu 2021-07-22 16:02:21 CEST; 5 days ago
Jul 22 16:02:21 amcvh12 systemd[1]: Reached target ceph target allowing

root@amcvh12:~# systemctl restart ceph-mgr.target

root@amcvh12:~# systemctl status ceph-mgr.target
● ceph-mgr.target - ceph target allowing to start/stop all ceph-mgr@.ser
   Loaded: loaded (/lib/systemd/system/ceph-mgr.target; enabled; vendor
   Active: active since Wed 2021-07-28 13:59:23 CEST; 9s ago
Jul 28 13:59:23 amcvh12 systemd[1]: Reached target ceph target allowing

and

Code: 
Jul 28 13:59:23 amcvh12 systemd[1]: Reached target ceph target allowing
root@amcvh12:~# systemctl status ceph-mon.target
● ceph-mon.target - ceph target allowing to start/stop all ceph-mon@.ser
   Loaded: loaded (/lib/systemd/system/ceph-mon.target; enabled; vendor
   Active: active since Thu 2021-07-22 16:02:21 CEST; 5 days ago
Jul 22 16:02:21 amcvh12 systemd[1]: Reached target ceph target allowing

root@amcvh12:~# systemctl restart ceph-mon.target

root@amcvh12:~# systemctl status ceph-mon.target
● ceph-mon.target - ceph target allowing to start/stop all ceph-mon@.ser
   Loaded: loaded (/lib/systemd/system/ceph-mon.target; enabled; vendor
   Active: active since Wed 2021-07-28 14:01:27 CEST; 3s ago
Jul 28 14:01:27 amcvh12 systemd[1]: Reached target ceph target allowing

I can't see any diffrence -

Code: 
root@amcvh12:~# ceph -s
  cluster:
    id:     ae713943-83f3-48b4-a0c2-124c092c250b
    health: HEALTH_WARN
            1/3 mons down, quorum amcvh12,amcvh13
            3 osds down
            1 host (15 osds) down
            Degraded data redundancy: 448166/1344447 objects degraded (33.335%), 257 pgs degraded, 257 pgs undersized
 
  services:
    mon: 3 daemons, quorum amcvh12,amcvh13 (age 2m), out of quorum: amcvh11
    mgr: amcvh13(active, since 5d), standbys: amcvh12
    osd: 45 osds: 30 up (since 2d), 33 in (since 18h)
 
  data:
    pools:   3 pools, 257 pgs
    objects: 448.17k objects, 1.6 TiB
    usage:   1013 GiB used, 8.0 TiB / 9.0 TiB avail
    pgs:     448166/1344447 objects degraded (33.335%)
             257 active+undersized+degraded
 
  io:
    client:   0 B/s rd, 71 KiB/s wr, 0 op/s rd, 7 op/s wr
 
Last edited:
Please check the network connection in your Ceph networks, then. Maybe the network problems that some people talk about here after the upgrade, also caught you.
 
Ok, it makes sense, because ceph manager on the updated node does not show an ip - since it is a client in the meshed-ceph-network it should have 192.168.228.11 like the other node (.12 and. .13)
and the network tab reports the following pending changes:

Code: 
--- /etc/network/interfaces    2021-07-12 14:32:27.146235890 +0200
+++ /etc/network/interfaces.new    2021-07-28 14:13:41.655934015 +0200
@@ -47,7 +47,7 @@
     bond-slaves ens2 ens3
     bond-miimon 100
     bond-mode broadcast
-#n1-0-pbulic: bond_ceph-meshed_10GbE
+#n1-0-public: bond_ceph-meshed_10GbE
 
 auto vmbr0
 iface vmbr0 inet static

Seems to be a part of the interfaces config file Shall I reboot to execute these?
 
Ok - after reboot which did not change anything - I checked the network-connections and now it is obvious: the updated node has lost connection to the other nodes in the ceph-meshed-network (public-network), because the bond is down (while the nics are up):

Code: 
9: bond0: <NO-CARRIER,BROADCAST,MULTICAST,MASTER,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 4a:e7:d0:5e:4d:8c brd ff:ff:ff:ff:ff:ff
    inet 192.168.228.11/24 brd 192.168.228.255 scope global bond0
       valid_lft forever preferred_lft forever

After serveral attemps bond0 came up again and ceph is working as expected. It changed immediately from degraded to active and clean, is just doing some scrubbing in 5 pgs.
In essence - the whole problem arose just from a bond which wasn't activated after the update form 6 to 7.
As ph0x mentioned before, this seems to be a known issue - Can someone pls. elaborate on this.
Is there a know solution how to avoid this behavior updating the other nodes?
 
Last edited:
I didn't have any problems, so I can only guess. Maybe you can check /etc/network/interfaces of the upgraded host to differences compared to one of the others.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom