[SOLVED] Ceph offline, interface says 500 timeout

DrillSgtErnst

DrillSgtErnst

Active Member
Jun 29, 2020
91
6
28
Hi,
yesterday out of the sudden on his first birthday my cluster just killed ceph. With birthday I mean, that the cluster was set up on 1st of December 2019.

ceph, ceph -s, ceph status do only hang up the session and I have to reconnect.

His last words were.
2020-12-01 13:07:02.542277 mon.pve1 (mon.0) 1652487 : cluster [INF] osd.4 marked down after no beacon for 901.965439 seconds
2020-12-01 13:07:02.543567 mon.pve1 (mon.0) 1652488 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2020-12-01 13:07:02.560437 mon.pve1 (mon.0) 1652489 : cluster [DBG] osdmap e201: 12 total, 11 up, 12 in
2020-12-01 13:07:03.567454 mon.pve1 (mon.0) 1652490 : cluster [DBG] osdmap e202: 12 total, 11 up, 12 in
2020-12-01 13:07:05.587868 mon.pve1 (mon.0) 1652491 : cluster [WRN] Health check failed: Degraded data redundancy: 6537/674457 objects degraded (0.969%), 30 pgs degraded (PG_DEGRADED)
2020-12-01 13:07:07.333601 mon.pve1 (mon.0) 1652492 : cluster [WRN] Health check failed: Reduced data availability: 97 pgs peering (PG_AVAILABILITY)
2020-12-01 13:07:07.548041 mon.pve1 (mon.0) 1652493 : cluster [WRN] Health check update: 437 slow ops, oldest one blocked for 560 sec, daemons [mon.pve2,mon.pve3] have slow ops. (SLOW_OPS)
2020-12-01 13:07:03.278567 mgr.pv


All SSDs are working fine

I rebooted
nothing
heard about his being an common error on nautilus
therefore I updated all servers with apt update && apt dist-upgrade
rebooted again
none of the ceph osds are online getting 500 timeout once again
the Log says something similar to auth failure auth_id

I can't manually start the ceph services.
the ceph target service is up and running.


I restored the VMs on an NFS share via backup and everything works for now.

Now I need my Ceph back up and running.
I would like to save my data from yesterday, but if not possible I am able to kill the ceph and do it all over.

What can I do?
Please tell me which additional Information you need and where to find, or give me some debugging tips.
I tried to manually push a mon list, but that did nothing.
I'm out of ideas
 
Please check if pve-cluster service and corosync are running.
 
You mean like this, or do you need further details, if so please provide the correct command for retrieving this info

Bash: 
root@pve3:~# systemctl status corosync
● corosync.service - Corosync Cluster Engine
   Loaded: loaded (/lib/systemd/system/corosync.service; enabled; vendor preset: enable
   Active: active (running) since Tue 2020-12-01 17:51:35 CET; 18h ago
     Docs: man:corosync
           man:corosync.conf
           man:corosync_overview
 Main PID: 2634 (corosync)
    Tasks: 9 (limit: 39321)
   Memory: 150.6M
   CGroup: /system.slice/corosync.service
           └─2634 /usr/sbin/corosync -f

Dec 02 10:00:15 pve3 corosync[2634]:   [TOTEM ] Retransmit List: 364ac 364ad
Dec 02 10:00:15 pve3 corosync[2634]:   [TOTEM ] Retransmit List: 364ac 364ad
Dec 02 10:00:15 pve3 corosync[2634]:   [TOTEM ] Retransmit List: 364ac 364ad
Dec 02 10:00:15 pve3 corosync[2634]:   [TOTEM ] Retransmit List: 364ac 364ad
Dec 02 10:00:15 pve3 corosync[2634]:   [TOTEM ] Retransmit List: 364ac 364ad
Dec 02 10:00:15 pve3 corosync[2634]:   [TOTEM ] Retransmit List: 364ac 364ad
Dec 02 10:00:15 pve3 corosync[2634]:   [TOTEM ] Retransmit List: 364ac 364ad
Dec 02 10:00:15 pve3 corosync[2634]:   [TOTEM ] Retransmit List: 364ac 364ad
Dec 02 10:00:15 pve3 corosync[2634]:   [TOTEM ] Retransmit List: 364ac 364ad

Bash: 
root@pve3:~# systemctl status pve-cluster
● pve-cluster.service - The Proxmox VE cluster filesystem
   Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; vendor preset: ena
   Active: active (running) since Tue 2020-12-01 17:51:35 CET; 18h ago
  Process: 2451 ExecStart=/usr/bin/pmxcfs (code=exited, status=0/SUCCESS)
 Main PID: 2464 (pmxcfs)
    Tasks: 8 (limit: 39321)
   Memory: 62.1M
   CGroup: /system.slice/pve-cluster.service
           └─2464 /usr/bin/pmxcfs

Dec 02 10:56:02 pve3 pmxcfs[2464]: [status] notice: received log
Dec 02 11:11:02 pve3 pmxcfs[2464]: [status] notice: received log
Dec 02 11:26:02 pve3 pmxcfs[2464]: [status] notice: received log
Dec 02 11:41:02 pve3 pmxcfs[2464]: [status] notice: received log
Dec 02 11:52:53 pve3 pmxcfs[2464]: [dcdb] notice: data verification successful
Dec 02 11:56:02 pve3 pmxcfs[2464]: [status] notice: received log
Dec 02 12:11:02 pve3 pmxcfs[2464]: [status] notice: received log
Dec 02 12:26:02 pve3 pmxcfs[2464]: [status] notice: received log
Dec 02 12:29:08 pve3 pmxcfs[2464]: [status] notice: received log
Dec 02 12:29:08 pve3 pmxcfs[2464]: [status] notice: received log


Just in case
Bash: 
root@pve3:~# pveversion -v
proxmox-ve: 6.3-1 (running kernel: 5.4.73-1-pve)
pve-manager: 6.3-2 (running version: 6.3-2/22f57405)
pve-kernel-5.4: 6.3-1
pve-kernel-helper: 6.3-1
pve-kernel-5.3: 6.1-6
pve-kernel-5.0: 6.0-11
pve-kernel-5.4.73-1-pve: 5.4.73-1
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph: 14.2.15-pve2
ceph-fuse: 14.2.15-pve2
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libproxmox-backup-qemu0: 1.0.2-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-6
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.3-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.5-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-3
pve-cluster: 6.2-1
pve-container: 3.3-1
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-7
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-1
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.5-pve1
 
DrillSgtErnst said:
Dec 02 10:00:15 pve3 corosync[2634]: [TOTEM ] Retransmit List: 364ac 364ad
Click to expand...
Seems there may be network issues with the corosync link(s). This can impact the start of Ceph, since the /etc/ceph/ceph.conf is a symlink to /etce/pve/ceph.conf. And if the pve-cluster isn't available during boot, it might not start the Ceph services.

Can you start the OSD manually? And what is the state of ceph, ceph -s?
 
I tried pinging all addresses
Host 1
10.1.1.1/24 public network2x1GB bond balance alb
10.1.2.1/24 corosync dedicated 1x1GB dedicated switch
10.1.3.1/24 ceph pub 2x10GB bond balance alb dedicated switch
10.1.4.1/24 ceph cluster2x10GB bond balance alb dedicated switch

Host 2
10.1.1.2/24 public network2x1GB bond balance alb
10.1.2.2/24 corosync dedicated1x1GB dedicated switch
10.1.3.2/24 ceph pub2x10GB bond balance alb dedicated switch
10.1.4.2/24 ceph cluster2x10GB bond balance alb dedicated switch

Host 3
10.1.1.3/24 public network 2x1GB bond balance alb
10.1.2.3/24 corosync dedicated1x1GB dedicated switch
10.1.3.3/24 ceph pub 2x10GB bond balance alb dedicated switch
10.1.4.3/24 ceph cluster 2x10GB bond balance alb dedicated switch


pinging each other on all subnets works in under 1msec (avg 0,33msec)


Bash: 
root@pve3:~# systemctl start ceph-osd@0
Job for ceph-osd@0.service failed because the control process exited with error code.
See "systemctl status ceph-osd@0.service" and "journalctl -xe" for details.

Bash: 
root@pve3:~# systemctl status ceph-osd@0
● ceph-osd@0.service - Ceph object storage daemon osd.0
   Loaded: loaded (/lib/systemd/system/ceph-osd@.service; disabled; vendor preset: enab
  Drop-In: /usr/lib/systemd/system/ceph-osd@.service.d
           └─ceph-after-pve-cluster.conf
   Active: failed (Result: exit-code) since Wed 2020-12-02 14:38:14 CET; 1min 1s ago
  Process: 1304644 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER}

Dec 02 14:38:14 pve3 systemd[1]: ceph-osd@0.service: Service RestartSec=100ms expired,
Dec 02 14:38:14 pve3 systemd[1]: ceph-osd@0.service: Scheduled restart job, restart cou
Dec 02 14:38:14 pve3 systemd[1]: Stopped Ceph object storage daemon osd.0.
Dec 02 14:38:14 pve3 systemd[1]: ceph-osd@0.service: Start request repeated too quickly
Dec 02 14:38:14 pve3 systemd[1]: ceph-osd@0.service: Failed with result 'exit-code'.
Dec 02 14:38:14 pve3 systemd[1]: Failed to start Ceph object storage daemon osd.0.

Bash: 
journalctl -xe

Dec 02 14:38:13 pve3 systemd[1]: Starting Ceph object storage daemon osd.0...
-- Subject: A start job for unit ceph-osd@0.service has begun execution
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit ceph-osd@0.service has begun execution.
--
-- The job identifier is 72962.
Dec 02 14:38:13 pve3 ceph-osd-prestart.sh[1304549]: OSD data directory /var/lib/ceph/os
Dec 02 14:38:13 pve3 systemd[1]: ceph-osd@0.service: Control process exited, code=exite
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- An ExecStartPre= process belonging to unit ceph-osd@0.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.


As mentioned earlier ceph -s leaves me haning. I'm going to let it work now, but I guess in about five minutes it will present me a timeout.
 
DrillSgtErnst said:
Dec 02 14:38:14 pve3 systemd[1]: ceph-osd@0.service: Start request repeated too quickly
Click to expand...
Use systemctl reset-failed ceph-osd@0.service first, since systemd tried too often to restart the OSD. Then a start might work and if it fails shows more in the journal.
 
I did,
service still can't come up

journal says
Bash: 
Dec 02 17:58:02 pve3 pvestatd[2706]: status update time (5.137 seconds)
Dec 02 18:02:00 pve3 systemd[1]: Starting Proxmox VE replication runner...
-- Subject: A start job for unit pvesr.service has begun execution
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit pvesr.service has begun execution.
-- 
-- The job identifier is 84803.
Dec 02 18:02:00 pve3 systemd[1]: pvesr.service: Succeeded.
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- The unit pvesr.service has successfully entered the 'dead' state.
Dec 02 18:02:00 pve3 systemd[1]: Started Proxmox VE replication runner.
-- Subject: A start job for unit pvesr.service has finished successfully
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit pvesr.service has finished successfully.
-- 
-- The job identifier is 84803.
Dec 02 18:02:02 pve3 pvestatd[2706]: got timeout
Dec 02 18:02:02 pve3 pvestatd[2706]: status update time (5.128 seconds)
Dec 02 18:02:03 pve3 pvedaemon[1494765]: VM 106 qmp command failed - VM 106 qmp command 'guest-ping' failed 
Dec 02 18:02:12 pve3 pvestatd[2706]: got timeout
Dec 02 18:02:12 pve3 pvestatd[2706]: status update time (5.130 seconds)
Dec 02 18:02:22 pve3 pvestatd[2706]: got timeout
Dec 02 18:02:22 pve3 systemd[1]: Starting Ceph object storage daemon osd.0...
-- Subject: A start job for unit ceph-osd@0.service has begun execution
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit ceph-osd@0.service has begun execution.
-- 
-- The job identifier is 84858.
Dec 02 18:02:22 pve3 ceph-osd-prestart.sh[1519360]: OSD data directory /var/lib/ceph/osd/ceph-0 does not exi
Dec 02 18:02:22 pve3 systemd[1]: ceph-osd@0.service: Control process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- An ExecStartPre= process belonging to unit ceph-osd@0.service has exited.
-- 
-- The process' exit code is 'exited' and its exit status is 1.
Dec 02 18:02:22 pve3 systemd[1]: ceph-osd@0.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- The unit ceph-osd@0.service has entered the 'failed' state with result 'exit-code'.
Dec 02 18:02:22 pve3 systemd[1]: Failed to start Ceph object storage daemon osd.0.
-- Subject: A start job for unit ceph-osd@0.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit ceph-osd@0.service has finished with a failure.
-- 
-- The job identifier is 84858 and the job result is failed.
Dec 02 18:02:22 pve3 pvestatd[2706]: status update time (5.136 seconds)
Dec 02 18:02:22 pve3 pvedaemon[1517739]: VM 106 qmp command failed - VM 106 qmp command 'guest-ping' failed 
Dec 02 18:02:23 pve3 systemd[1]: ceph-osd@0.service: Service RestartSec=100ms expired, scheduling restart.
Dec 02 18:02:23 pve3 systemd[1]: ceph-osd@0.service: Scheduled restart job, restart counter is at 1.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Automatic restarting of the unit ceph-osd@0.service has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Dec 02 18:02:23 pve3 systemd[1]: Stopped Ceph object storage daemon osd.0.
-- Subject: A stop job for unit ceph-osd@0.service has finished
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A stop job for unit ceph-osd@0.service has finished.
-- 
-- The job identifier is 84927 and the job result is done.
Dec 02 18:02:23 pve3 systemd[1]: Starting Ceph object storage daemon osd.0...
-- Subject: A start job for unit ceph-osd@0.service has begun execution
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit ceph-osd@0.service has begun execution.
-- 
-- The job identifier is 84927.
Dec 02 18:02:23 pve3 ceph-osd-prestart.sh[1519373]: OSD data directory /var/lib/ceph/osd/ceph-0 does not exi
Dec 02 18:02:23 pve3 systemd[1]: ceph-osd@0.service: Control process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- An ExecStartPre= process belonging to unit ceph-osd@0.service has exited.
-- 
-- The process' exit code is 'exited' and its exit status is 1.
Dec 02 18:02:23 pve3 systemd[1]: ceph-osd@0.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- The unit ceph-osd@0.service has entered the 'failed' state with result 'exit-code'.
Dec 02 18:02:23 pve3 systemd[1]: Failed to start Ceph object storage daemon osd.0.
-- Subject: A start job for unit ceph-osd@0.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit ceph-osd@0.service has finished with a failure.
-- 
-- The job identifier is 84927 and the job result is failed.
Dec 02 18:02:23 pve3 systemd[1]: ceph-osd@0.service: Service RestartSec=100ms expired, scheduling restart.
Dec 02 18:02:23 pve3 systemd[1]: ceph-osd@0.service: Scheduled restart job, restart counter is at 2.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Automatic restarting of the unit ceph-osd@0.service has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Dec 02 18:02:23 pve3 systemd[1]: Stopped Ceph object storage daemon osd.0.
-- Subject: A stop job for unit ceph-osd@0.service has finished
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A stop job for unit ceph-osd@0.service has finished.
-- 
-- The job identifier is 84996 and the job result is done.
Dec 02 18:02:23 pve3 systemd[1]: Starting Ceph object storage daemon osd.0...
-- Subject: A start job for unit ceph-osd@0.service has begun execution
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit ceph-osd@0.service has begun execution.
-- 
-- The job identifier is 84996.
Dec 02 18:02:23 pve3 ceph-osd-prestart.sh[1519380]: OSD data directory /var/lib/ceph/osd/ceph-0 does not exi
Dec 02 18:02:23 pve3 systemd[1]: ceph-osd@0.service: Control process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- An ExecStartPre= process belonging to unit ceph-osd@0.service has exited.
-- 
-- The process' exit code is 'exited' and its exit status is 1.
Dec 02 18:02:23 pve3 systemd[1]: ceph-osd@0.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- The unit ceph-osd@0.service has entered the 'failed' state with result 'exit-code'.
Dec 02 18:02:23 pve3 systemd[1]: Failed to start Ceph object storage daemon osd.0.
-- Subject: A start job for unit ceph-osd@0.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit ceph-osd@0.service has finished with a failure.
-- 
-- The job identifier is 84996 and the job result is failed.
Dec 02 18:02:23 pve3 systemd[1]: ceph-osd@0.service: Service RestartSec=100ms expired, scheduling restart.
Dec 02 18:02:23 pve3 systemd[1]: ceph-osd@0.service: Scheduled restart job, restart counter is at 3.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Automatic restarting of the unit ceph-osd@0.service has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Dec 02 18:02:23 pve3 systemd[1]: Stopped Ceph object storage daemon osd.0.
-- Subject: A stop job for unit ceph-osd@0.service has finished
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A stop job for unit ceph-osd@0.service has finished.
-- 
-- The job identifier is 85065 and the job result is done.
Dec 02 18:02:23 pve3 systemd[1]: ceph-osd@0.service: Start request repeated too quickly.
Dec 02 18:02:23 pve3 systemd[1]: ceph-osd@0.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- The unit ceph-osd@0.service has entered the 'failed' state with result 'exit-code'.
Dec 02 18:02:23 pve3 systemd[1]: Failed to start Ceph object storage daemon osd.0.
-- Subject: A start job for unit ceph-osd@0.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit ceph-osd@0.service has finished with a failure.
-- 
-- The job identifier is 85065 and the job result is failed.
 
DrillSgtErnst said:
I did,
service still can't come up

journal says
<snip>
Click to expand...
DrillSgtErnst said:
Dec 02 18:02:22 pve3 ceph-osd-prestart.sh[1519360]: OSD data directory /var/lib/ceph/osd/ceph-0 does not exi
<snip>
Dec 02 18:02:23 pve3 ceph-osd-prestart.sh[1519373]: OSD data directory /var/lib/ceph/osd/ceph-0 does not exi
<snip>
Dec 02 18:02:23 pve3 ceph-osd-prestart.sh[1519380]: OSD data directory /var/lib/ceph/osd/ceph-0 does not exi
Click to expand...

That's bad. Is /var a separate partition? Is it mounted? Does ls -lh /var/lib/ceph show the parent directories?
 
/var is not on a seperate partition

it is accessible
Bash: 
root@pve3:~# cd /var/
root@pve3:/var# ls
backups  cache  lib  local  lock  log  mail  opt  run  spool  tmp

seems okay to me
Bash: 
root@pve3:/var# ls -lh /var/lib/ceph
total 6.0K
drwxr-xr-x 2 ceph ceph 2 Sep 19  2019 bootstrap-mds
drwxr-xr-x 2 ceph ceph 2 Sep 19  2019 bootstrap-mgr
drwxr-xr-x 2 ceph ceph 3 Nov 30  2019 bootstrap-osd
drwxr-xr-x 2 ceph ceph 2 Sep 19  2019 bootstrap-rbd
drwxr-xr-x 2 ceph ceph 2 Sep 19  2019 bootstrap-rbd-mirror
drwxr-xr-x 2 ceph ceph 2 Sep 19  2019 bootstrap-rgw
drwxr-xr-x 3 ceph ceph 3 Nov 30  2019 crash
drwxr-xr-x 2 ceph ceph 2 Sep 19  2019 mds
drwxr-xr-x 2 ceph ceph 2 Sep 19  2019 mgr
drwxr-xr-x 3 ceph ceph 3 Nov 30  2019 mon
drwxr-xr-x 6 ceph ceph 6 Nov 30  2019 osd
drwxr-xr-x 2 ceph ceph 2 Sep 19  2019 tmp
 
Last edited:
Well I guess it was 6.2-13 but not quite that sure.

I attached all logs available.
systemd seems to be clear.
ceph-volume seems to be missing the ceph.conf.

I checked under /etc/ceph there is a ceph.conf. owner root 777 (But it is an Link)


Under ceph log you can observe a "timewarp" from 14:07 to 13:07 in the log timers.


@Klaus Steinberger I did now, I use 2,2 of 220GB. I guess that's fairly enough, but thanks for the suggestion.
 

Attachments

  • ceph-volume.log
    49 KB · Views: 2
  • ceph-volume-systemd.log
    29.7 KB · Views: 1
  • ceph-osd.8.log
    373.4 KB · Views: 2
  • ceph-mon.pve3.log
    86.5 KB · Views: 1
  • ceph.audit.log
    272.9 KB · Views: 0
  • ceph.log
    678.9 KB · Views: 1
Last edited:
ceph.log said:
2020-12-01 14:07:23.659051 mon.pve1 (mon.0) 1654734 : cluster [WRN] Health check update: 9754 slow ops, oldest one blocked for 4142 sec, daemons [osd.2,osd.3,osd.5,mon.pve2,mon.pve3] have slow ops. (SLOW_OPS)
2020-12-01 14:07:28.659670 mon.pve1 (mon.0) 1654735 : cluster [WRN] Health check update: 9766 slow ops, oldest one blocked for 4147 sec, daemons [osd.2,osd.3,osd.5,mon.pve2,mon.pve3] have slow ops. (SLOW_OPS)
Click to expand...
Slow ops are not a good sign. How is the hardware setup?

ceph-osd.8.log said:
2020-12-01 14:08:38.181 7fa3ffe8b700 0 bluestore(/var/lib/ceph/osd/ceph-8) log_latency_fn slow operation observed for _do_read, latency = 31.5051s, num_ios = 0
2020-12-01 14:08:38.181 7fa3ffe8b700 0 bluestore(/var/lib/ceph/osd/ceph-8) log_latency slow operation observed for read, latency = 31.5053s
2020-12-01 14:08:38.181 7fa3ffe8b700 1 heartbeat_map reset_timeout 'OSD::eek:sd_op_tp thread 0x7fa3ffe8b700' had timed out after 15
Click to expand...
The OSDs seem not to come up, because it seems the time out on reading from the disk.
https://ceph.io/planet/dealing-with-some-osd-timeouts/

ceph-mon.pve3.log said:
2020-12-01 14:07:38.874 7f996097c700 -1 mon.pve3@2(peon).paxos(paxos updating c 26638129..26638738) lease_expire from mon.0 v2:10.1.3.1:3300/0 is 0.046515 seconds in the past; mons are probably laggy (or possibly clocks are too skewed)
2020-12-01 14:07:39.310 7f996097c700 -1 mon.pve3@2(peon).paxos(paxos updating c 26638129..26638739) lease_expire from mon.0 v2:10.1.3.1:3300/0 is 0.008459 seconds in the past; mons are probably laggy (or possibly clocks are too skewed)
Click to expand...
I don't know if that was just around the upgrade, but do you have a local NTP where all the nodes sync their time from?
 
I have an Supermicro Server with AMD EPYC 7302P
Intel XL710 Quad 10GBit Adapter (2x10 Ceph public, 2x10 Ceph Cluster)
DDR4 3200MHz 8x32GB
Intel DC P4510 (x4)

Well the Link you provided is nice, but I can not start the ceph services.

My Firewall does provide NTP. All Nodes do use the Firewall, I made sure they do.



But I do not plan on fixing ceph here anymore. The machines are running on backup repo and it's "fine".
Now I am searching for how to purge and redo ceph.
If I redo Ceph from scratch I can just move the machines back to the ceph and I am fine.

How do I achieve this?
 
DrillSgtErnst said:
Well the Link you provided is nice, but I can not start the ceph services.
Click to expand...
Just to clarify, these settings can be made in the ceph.conf (see the first part of the article). Hopefully then they can start.

DrillSgtErnst said:
My Firewall does provide NTP. All Nodes do use the Firewall, I made sure they do.
Click to expand...
The probably the RTC is not up to the current time (hwclock).

DrillSgtErnst said:
If I redo Ceph from scratch I can just move the machines back to the ceph and I am fine.
Click to expand...
I suppose, since only the OSDs are affected, you could just destroy and re-create them. You can start with one and see if it comes up again.

Otherwise pveceph purge is your friend.
 
Sooooo

Purge was indeed my friend

stop all remaining ceph-services
systemctl stop ceph-mon.target
systemctl stop ceph-mgr.target
systemctl stop ceph-mds.target
systemctl stop ceph-osd.target

avoid that they're being restarted by systemd the next boot (the low level way)
rm -rf /etc/systemd/system/ceph*

be really sure they're stopped:
killall -9 ceph-mon ceph-mgr ceph-mds

then do
rm -rf /var/lib/ceph/mon/ /var/lib/ceph/mgr/ /var/lib/ceph/mds/

then retry purge
pveceph purge


ceph-volume lvm zap /dev/nvme2n1 --destroy && ceph-volume lvm zap /dev/nvme3n1 --destroy && ceph-volume lvm zap /dev/nvme4n1 --destroy && ceph-volume lvm zap /dev/nvme5n1 --destroy

Installed freshly by GUI
Works like a charm.
Once again. Backup is Backup.

Thnkas for the help.

On a side note. Clockskew was about 30 Sec.
The timesyncd conf on one host was empty. Like completely empty. populated it, restartet the service and the clockskew is gone.
 
  • Like
Reactions: Alwin
Hi,

I had very similar problem, maybe will help someone. Ceph -s hung, Long story short, 2 of 3 nodes / was disk full, monitors down and no quorum. Fortunately quick fix.
 
otoce said:
Hi,

I had very similar problem, maybe will help someone. Ceph -s hung, Long story short, 2 of 3 nodes / was disk full, monitors down and no quorum. Fortunately quick fix.
Click to expand...
I have the same problem, I already released logs that the node disks were occupying, but the ceph is still down.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back