[SOLVED] URGENT Proxmox gui not accessible ceph system died

lodperera · Oct 7, 2017

Hi Guys,

I'm on panic mode, any help is appreciated.

have 4 nodes in production

Code:

root@s2n1:~# pveversion -v
proxmox-ve: 4.4-76 (running kernel: 4.4.35-1-pve)
pve-manager: 4.4-2 (running version: 4.4-2/80259e05)
pve-kernel-4.4.6-1-pve: 4.4.6-48
pve-kernel-4.4.13-1-pve: 4.4.13-56
pve-kernel-4.4.35-1-pve: 4.4.35-76
pve-kernel-4.2.6-1-pve: 4.2.6-36
pve-kernel-4.4.13-2-pve: 4.4.13-58
pve-kernel-4.4.15-1-pve: 4.4.15-60
pve-kernel-4.2.8-1-pve: 4.2.8-41
pve-kernel-4.4.16-1-pve: 4.4.16-64
pve-kernel-4.4.19-1-pve: 4.4.19-66
pve-kernel-4.4.10-1-pve: 4.4.10-54
lvm2: 2.02.116-pve3
corosync-pve: 2.4.0-1
libqb0: 1.0-1
pve-cluster: 4.0-48
qemu-server: 4.0-102
pve-firmware: 1.1-10
libpve-common-perl: 4.0-84
libpve-access-control: 4.0-19
libpve-storage-perl: 4.0-70
pve-libspice-server1: 0.12.8-1
vncterm: 1.2-1
pve-docs: 4.4-1
pve-qemu-kvm: 2.7.0-9
pve-container: 1.0-89
pve-firewall: 2.0-33
pve-ha-manager: 1.0-38
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u2
lxc-pve: 2.0.6-2
lxcfs: 2.0.5-pve1
criu: 1.6.0-1
novnc-pve: 0.5-8
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.8-pve13~bpo80
ceph: 0.94.9-1~bpo80+1
root@s2n1:~#

I do not have access to proxmox gui

Ceph is giving me errors when I try to do something on ceph.

What can I do in this instance?

lodperera · Oct 7, 2017

I got access to gui. used another browser could be cache related.
Still having issues with ceph
I'm guessing I might have to remove and re-add them

lodperera · Oct 7, 2017

OK so I'm just writing stuff what happened.

I stopped all nodes and rebooted one by one

I did noout via GUI

then I had to manually start OSDs
However 6 out of 12 didn't like to start

I removed OSD and put them back using GUI

2 disks didn't want to put in as OSDs
Those two disks were giving data corrupted errors, didn't have the time to figure out why so I left those two

then I unset nout and let it recover
at this time I have shut down all VMs, well they were having issues because close to 50% data is in recovery

it was slow so I had to increase the recovery using

https://forum.proxmox.com/threads/increase-ceph-recovery-speed.36728/

this happened when proxmox was doing backups in the night. I have set to backup all VMs which takes like 6 hours.
I'm guessing IO was high then proxmox nmi watchdog started restarting the nodes because I can see each node was restarted several times until everyone was fenced or who knows

If some one can point me to figure out where to look to understand what actually happened. I would appreciate a lot.

lodperera · Oct 10, 2017

Update:

I found out that our backup server started updating itself at the same time the VM backup was happening and backup server restarted itself.
Since proxmox couldn't send the backup it went all suicide it seems.

This happened at around 1.40 am OCT 7.
I have attached logs for someone who can let me know actually what happened.

Both of other node logs are too big to upload. but if someone needs them I can upload by other means.

udo · Oct 10, 2017

lodperera said:
Hi Guys,

I'm on panic mode, any help is appreciated.

have 4 nodes in production

Code:

root@s2n1:~# pveversion -v proxmox-ve: 4.4-76 (running kernel: 4.4.35-1-pve)

Hi,
this shows, that you don't make "right" updates.

With proxmox-ve you must allways use "apt dist-upgrade" (or the old "apt-get dist-upgrade").
"apt-get upgrade" isn't enough!

To see what happens witzh ceph are following commands helpfull:

Code:

ceph -s

ceph -w

Udo

lodperera · Oct 11, 2017

udo said:
Hi,
this shows, that you don't make "right" updates.

With proxmox-ve you must allways use "apt dist-upgrade" (or the old "apt-get dist-upgrade").
"apt-get upgrade" isn't enough!

To see what happens witzh ceph are following commands helpfull:

Code:

ceph -s ceph -w

Udo

Thank you for getting back to me.
Now it's all good.
For record I didn't update proxmox nodes.
My backup server synology started doing it's automated update and restarted itself while the VM backup was happening.

I'm going to list some codes I ran to bring a corrupted OSD up so the nubes like me would know what to do.

Code:

oot@s1n2:~#  ceph-disk list
mount: mount /dev/sdf1 on /var/lib/ceph/tmp/mnt.2Pvo85 failed: Structure needs cleaning
/dev/sda :
 /dev/sda1 other, 21686148-6449-6e6f-744e-656564454649
 /dev/sda2 other, vfat
 /dev/sda3 other, LVM2_member
/dev/sdb :
 /dev/sdb4 ceph journal
 /dev/sdb6 ceph journal, for /dev/sde1
 /dev/sdb7 ceph journal, for /dev/sdc1
 /dev/sdb8 ceph journal, for /dev/sdd1
/dev/sdc :
 /dev/sdc1 ceph data, active, cluster ceph, osd.4, journal /dev/sdb7
/dev/sdd :
 /dev/sdd1 ceph data, active, cluster ceph, osd.5, journal /dev/sdb8
/dev/sde :
 /dev/sde1 ceph data, active, cluster ceph, osd.6, journal /dev/sdb6
/dev/sdf :
mount: mount /dev/sdf1 on /var/lib/ceph/tmp/mnt.JqXaut failed: Structure needs cleaning
 /dev/sdf1 ceph data, unprepared
root@s1n2:~#

root@s1n2:~# ls -l /var/lib/ceph/osd/ceph-7
total 0
root@s1n2:~#

h-m, looks like it’s crashed completely
do the same for /var/lib/ceph/osd/ceph-6
 just in case - want to check structure


 root@s1n2:~# ls -l /var/lib/ceph/osd/ceph-6
total 76
-rw-r--r--   1 root root   610 Oct  7 11:29 activate.monmap
-rw-r--r--   1 root root     3 Oct  7 11:29 active
-rw-r--r--   1 root root    37 Oct  7 11:29 ceph_fsid
drwxr-xr-x 450 root root 16384 Oct  7 13:49 current
-rw-r--r--   1 root root    37 Oct  7 11:29 fsid
lrwxrwxrwx   1 root root    58 Oct  7 11:29 journal -> /dev/disk/by-partuuid/37a3ac56-767e-4303-9786-cb92744380f5
-rw-r--r--   1 root root    37 Oct  7 11:29 journal_uuid
-rw-------   1 root root    56 Oct  7 11:29 keyring
-rw-r--r--   1 root root    21 Oct  7 11:29 magic
-rw-r--r--   1 root root     6 Oct  7 11:29 ready
-rw-r--r--   1 root root     4 Oct  7 11:29 store_version
-rw-r--r--   1 root root    53 Oct  7 11:29 superblock
-rw-r--r--   1 root root     0 Oct  7 11:29 sysvinit
-rw-r--r--   1 root root     2 Oct  7 11:29 whoami
root@s1n2:~#

check if you still have osd running for this disk
root@s1n2:~# ps ax|grep osd.7
13124 pts/0    S+     0:00 grep osd.7
root@s1n2:~#


root@s1n2:~# xfs_repair /dev/sdf1
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed.  Mount the filesystem to replay the log, and unmount it before
re-running xfs_repair.  If you are unable to mount the filesystem, then use
the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.
root@s1n2:~#

root@s1n2:~# ceph-disk zap /dev/sdf
Caution: invalid backup GPT header, but valid main header; regenerating
backup header from main header.

Warning! Main and backup partition tables differ! Use the 'c' and 'e' options
on the recovery & transformation menu to examine the two tables.

Warning! One or more CRCs don't match. You should repair the disk!

****************************************************************************
Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.
****************************************************************************
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
Creating new GPT entries.
The operation has completed successfully.
root@s1n2:~#


check if you have osd.7 mounted:

root@s1n2:~# mount|grep ceph-7
root@s1n2:~#

root@s1n2:~# pveceph createosd /dev/sdf -journal_dev /dev/sdb4
create OSD on /dev/sdf (xfs)
using device '/dev/sdb4' for journal
Caution: invalid backup GPT header, but valid main header; regenerating
backup header from main header.

****************************************************************************
Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.
****************************************************************************
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
Creating new GPT entries.
The operation has completed successfully.
WARNING:ceph-disk:OSD will not be hot-swappable if journal is not the same device as the osd data
Setting name!
partNum is 0
REALLY setting name!
The operation has completed successfully.
meta-data=/dev/sdf1              isize=2048   agcount=4, agsize=61013951 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0        finobt=0
data     =                       bsize=4096   blocks=244055803, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal log           bsize=4096   blocks=119167, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot.
The operation has completed successfully.
root@s1n2:~#

jeffwadsworth · Oct 11, 2017

I hope you replaced that /dev/sdf disk. May be failing mechanically. Do you have smart set up on it?

lodperera · Oct 16, 2017

jeffwadsworth said:
I hope you replaced that /dev/sdf disk. May be failing mechanically. Do you have smart set up on it?

Hi J,

I do not have smarts due to the RAID card I use I believe.

Code:

root@s1n2:/var/log# smartctl -a /dev/sda
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.4.35-1-pve] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               AVAGO
Product:              SMC3108
Revision:             4.62
User Capacity:        79,456,894,976 bytes [79.4 GB]
Logical block size:   512 bytes
Logical Unit id:      0x600304801a3b65001e90bd5ed3518ef2
Serial number:        00f28e51d35ebd901e00653b1a800403
Device type:          disk
Local Time is:        Mon Oct 16 12:47:48 2017 AEST
SMART support is:     Unavailable - device lacks SMART capability.

=== START OF READ SMART DATA SECTION ===
Current Drive Temperature:     0 C
Drive Trip Temperature:        0 C

Error Counter logging not supported

Device does not support Self Test logging

Still trying to figure out the exact issue. Had time to go through kernal log. and I'm seeing a lot of

Code:

 EXT4-fs (rbd3): mounted filesystem without journal. Opts: noload

like messages all across the cluster everyday at the time of backups are happening.

udo · Oct 16, 2017

lodperera said:
Hi J,

I do not have smarts due to the RAID card I use I believe.

Hi,
to see the smartvalues you must do something like this (for many lsi-controller):

Code:

smartctl -a -d megaraid,N  /dev/sda

# where N is the device number, like 0-5

Udo

lodperera · Oct 17, 2017

udo said:
Hi,
to see the smartvalues you must do something like this (for many lsi-controller):

Code:

smartctl -a -d megaraid,N /dev/sda # where N is the device number, like 0-5

Udo

Hi Udo,

Thank you for that.

Attached.

Hard disk smart on node 1 and 2

RAID Adapter info on node 1 and 2

PS : I can see some errors on node 2 hard disk smart not sure what to do.

lodperera · Oct 17, 2017

The errors seems to be increasing but I still do not understand why it rebooted the whole node and send the cluster to a zombie state.

Code:

root@s1n1:/var/log# smartctl --scan-open
/dev/sda -d scsi # /dev/sda, SCSI device
/dev/sdb -d scsi # /dev/sdb, SCSI device
/dev/sdc -d scsi # /dev/sdc, SCSI device
/dev/sdd -d scsi # /dev/sdd, SCSI device
/dev/sde -d scsi # /dev/sde, SCSI device
/dev/sdf -d scsi # /dev/sdf, SCSI device
/dev/bus/0 -d megaraid,16 # /dev/bus/0 [megaraid_disk_16], SCSI device
/dev/bus/0 -d megaraid,17 # /dev/bus/0 [megaraid_disk_17], SCSI device
/dev/bus/0 -d megaraid,18 # /dev/bus/0 [megaraid_disk_18], SCSI device
/dev/bus/0 -d megaraid,19 # /dev/bus/0 [megaraid_disk_19], SCSI device
/dev/bus/0 -d sat+megaraid,24 # /dev/bus/0 [megaraid_disk_24] [SAT], ATA device
/dev/bus/0 -d sat+megaraid,25 # /dev/bus/0 [megaraid_disk_25] [SAT], ATA device
root@s1n1:/var/log# smartctl -a -d megaraid,16 /dev/bus/0 | grep Non-medium
Non-medium error count:        1
root@s1n1:/var/log# smartctl -a -d megaraid,17 /dev/bus/0 | grep Non-medium
Non-medium error count:        0
root@s1n1:/var/log# smartctl -a -d megaraid,18 /dev/bus/0 | grep Non-medium
Non-medium error count:        0
root@s1n1:/var/log# smartctl -a -d megaraid,19 /dev/bus/0 | grep Non-medium
Non-medium error count:        0
root@s1n1:/var/log#



root@s1n2:/var/log# smartctl --scan-open
/dev/sda -d scsi # /dev/sda, SCSI device
/dev/sdb -d scsi # /dev/sdb, SCSI device
/dev/sdc -d scsi # /dev/sdc, SCSI device
/dev/sdd -d scsi # /dev/sdd, SCSI device
/dev/sde -d scsi # /dev/sde, SCSI device
/dev/sdf -d scsi # /dev/sdf, SCSI device
/dev/bus/0 -d megaraid,16 # /dev/bus/0 [megaraid_disk_16], SCSI device
/dev/bus/0 -d megaraid,17 # /dev/bus/0 [megaraid_disk_17], SCSI device
/dev/bus/0 -d megaraid,18 # /dev/bus/0 [megaraid_disk_18], SCSI device
/dev/bus/0 -d megaraid,19 # /dev/bus/0 [megaraid_disk_19], SCSI device
/dev/bus/0 -d sat+megaraid,24 # /dev/bus/0 [megaraid_disk_24] [SAT], ATA device
/dev/bus/0 -d sat+megaraid,25 # /dev/bus/0 [megaraid_disk_25] [SAT], ATA device
root@s1n2:/var/log#
root@s1n2:/var/log#
root@s1n2:/var/log# smartctl -a -d megaraid,16 /dev/bus/0 | grep Non-medium
Non-medium error count:        7
root@s1n2:/var/log# smartctl -a -d megaraid,17 /dev/bus/0 | grep Non-medium
Non-medium error count:        1
root@s1n2:/var/log# smartctl -a -d megaraid,18 /dev/bus/0 | grep Non-medium
Non-medium error count:        3
root@s1n2:/var/log# smartctl -a -d megaraid,19 /dev/bus/0 | grep Non-medium
Non-medium error count:       26
root@s1n2:/var/log#




root@s2n1:/var/log# smartctl --scan-open
/dev/sda -d scsi # /dev/sda, SCSI device
/dev/sdb -d scsi # /dev/sdb, SCSI device
/dev/sdc -d scsi # /dev/sdc, SCSI device
/dev/sdd -d scsi # /dev/sdd, SCSI device
/dev/sde -d scsi # /dev/sde, SCSI device
/dev/sdf -d scsi # /dev/sdf, SCSI device
/dev/bus/0 -d megaraid,18 # /dev/bus/0 [megaraid_disk_18], SCSI device
/dev/bus/0 -d megaraid,27 # /dev/bus/0 [megaraid_disk_27], SCSI device
/dev/bus/0 -d megaraid,29 # /dev/bus/0 [megaraid_disk_29], SCSI device
/dev/bus/0 -d megaraid,30 # /dev/bus/0 [megaraid_disk_30], SCSI device
/dev/bus/0 -d sat+megaraid,34 # /dev/bus/0 [megaraid_disk_34] [SAT], ATA device
/dev/bus/0 -d sat+megaraid,35 # /dev/bus/0 [megaraid_disk_35] [SAT], ATA device
root@s2n1:/var/log# smartctl -a -d megaraid,18 /dev/bus/0 | grep Non-medium
Non-medium error count:        6
root@s2n1:/var/log# smartctl -a -d megaraid,27 /dev/bus/0 | grep Non-medium
Non-medium error count:        0
root@s2n1:/var/log# smartctl -a -d megaraid,29 /dev/bus/0 | grep Non-medium
Non-medium error count:        0
root@s2n1:/var/log# smartctl -a -d megaraid,30 /dev/bus/0 | grep Non-medium
Non-medium error count:        0
root@s2n1:/var/log# smartctl -a -d sat+megaraid,34 /dev/bus/0 | grep Non-medium
root@s2n1:/var/log#



root@s2n2:/var/log# smartctl --scan-open
/dev/sda -d scsi # /dev/sda, SCSI device
/dev/sdb -d scsi # /dev/sdb, SCSI device
/dev/sdc -d scsi # /dev/sdc, SCSI device
/dev/sdd -d scsi # /dev/sdd, SCSI device
/dev/sde -d scsi # /dev/sde, SCSI device
/dev/sdf -d scsi # /dev/sdf, SCSI device
/dev/bus/0 -d megaraid,23 # /dev/bus/0 [megaraid_disk_23], SCSI device
/dev/bus/0 -d megaraid,25 # /dev/bus/0 [megaraid_disk_25], SCSI device
/dev/bus/0 -d megaraid,26 # /dev/bus/0 [megaraid_disk_26], SCSI device
/dev/bus/0 -d megaraid,27 # /dev/bus/0 [megaraid_disk_27], SCSI device
/dev/bus/0 -d sat+megaraid,36 # /dev/bus/0 [megaraid_disk_36] [SAT], ATA device
/dev/bus/0 -d sat+megaraid,37 # /dev/bus/0 [megaraid_disk_37] [SAT], ATA device
root@s2n2:/var/log# smartctl -a -d megaraid,23 /dev/bus/0 | grep Non-medium
Non-medium error count:        0
root@s2n2:/var/log# smartctl -a -d megaraid,25 /dev/bus/0 | grep Non-medium
Non-medium error count:        0
root@s2n2:/var/log# smartctl -a -d megaraid,26 /dev/bus/0 | grep Non-medium
Non-medium error count:        0
root@s2n2:/var/log# smartctl -a -d megaraid,27 /dev/bus/0 | grep Non-medium
Non-medium error count:        0
root@s2n2:/var/log#

lodperera · Feb 19, 2018

Well just closing this thread.

It seems that the node ran out of memory so it crashed.
Nothing to do with physical drives even though they showed some errors.

Thank you @udo @jeffwadsworth for replying and taking your time.

Search

Search

[SOLVED] URGENT Proxmox gui not accessible ceph system died

lodperera

Member

lodperera

Member

lodperera

Member

lodperera

Member

Attachments

udo

Distinguished Member

lodperera

Member

jeffwadsworth

Member

lodperera

Member

udo

Distinguished Member

lodperera

Member

Attachments

lodperera

Member

lodperera

Member