ceph issues [mostly operator errors ]

[RobFantini]

I tried a few different things, still issue, I can not get this node updated. and I have other osd only nodes.

it aways gets stuck here:

# dpkg --configure ceph-common:amd64
Setting up ceph-common (10.2.6-1~bpo80+1) ...
Setting system user ceph properties..usermod: no changes
..done
Fixing /var/run/ceph ownership....done

that hangs. here is the process list

root     19494 15213  0 14:39 pts/0    00:00:00 dpkg --configure ceph-common:amd64
root     19495 19494  0 14:39 pts/0    00:00:00 /bin/sh /var/lib/dpkg/info/ceph-common.postinst configure 10.2.5-1~bpo80+1
root     19524 19495  0 14:39 pts/0    00:00:00 /bin/systemctl start ceph.target
root     19525 19524  0 14:39 pts/0    00:00:00 /bin/systemd-tty-ask-password-agent --watch

 

[RobFantini]

I tried this: on the next osd only node to update - added is as a mon.

that did not fix the issue:

Setting up librgw2 (10.2.6-1~bpo80+1) ...
Setting up python-rados (10.2.6-1~bpo80+1) ...
Setting up libcephfs1 (10.2.6-1~bpo80+1) ...
Setting up python-cephfs (10.2.6-1~bpo80+1) ...
Setting up python-rbd (10.2.6-1~bpo80+1) ...
Setting up ceph-common (10.2.6-1~bpo80+1) ...
Setting system user ceph properties..usermod: no changes
..done
Fixing /var/run/ceph ownership....done


ps -ef
root     13067 13066  1 03:54 pts/0    00:00:00 apt-get dist-upgrade
root     15546 13067  0 03:55 pts/1    00:00:00 /usr/bin/dpkg --force-confdef --force-confold --status-fd 25 --configure librados2:amd64 librbd1:amd64 libradosstriper1:amd64 librgw2:amd64 python-rados:amd64 libcephfs1:amd64 python-cephfs:amd64 python-rbd:amd64 ceph-common:amd64 ceph-base:amd64 ceph-osd:amd64 ceph-mon:amd64 ceph:amd64 python-ceph:amd64 libappconfig-perl:all libtemplate-perl:amd64 lxc-pve:amd64 novnc-pve:amd64 pve-kernel-4.4.44-1-pve:amd64 libpve-common-perl:all pve-container:all pve-manager:amd64 proxmox-ve:all libcorosync4-pve:amd64 corosync-pve:amd64
root     15574 15546  0 03:55 pts/1    00:00:00 /bin/sh /var/lib/dpkg/info/ceph-common.postinst configure 10.2.5-1~bpo80+1

it is odd that ceph-common amd64 is not trying to configure 10.2.6-1~bpo80+1 . and instead configuring 10.2.5-1~bpo80+1

 

[RobFantini]

I submitted a bug report at https://bugzilla.proxmox.com/show_bug.cgi?id=1312

 

[RobFantini]

to get around the issue this works and does not seem to cause any issues.

find and kill these 2 processes

21148 pts/2    S+     0:00  |                       \_ /bin/systemctl start ceph.target
21149 pts/2    S+     0:00  |                           \_ /bin/systemd-tty-ask-password-agent --watch

 

[RobFantini]

well there are some other issues. I posted a new thread ' no such cluster node 'sys3' (500)'

AND
at pve > vm > hardware - ceph disks do not show up.

at pve > ceph-kvm > content - the disks do show up.


Update - 10 minutes later the disks do show up at vm > hardware ..

 

[fabian]

this has already been discussed here on the forum: https://forum.proxmox.com/threads/upgrade-hanging.33477/

 

[RobFantini]

this has already been discussed here on the forum: https://forum.proxmox.com/threads/upgrade-hanging.33477/

thank you .

 

[RobFantini]

and there is some data loss, i use dovecot . we use this cronjob every 10 minutes to backup imap mail:


after every system hang , the system needs to be rebooted and due to my settings running dovecot or vm settings , we end up with errors like this

Date: Tue, 14 Mar 2017 12:10:43
..
Subject: Cron <root@imap> doveadm backup -f -A remote:10.192.54.18 | grep -v Warning

dsync-remote(user1): Error: Mailbox INBOX sync: mailbox_delete failed: INBOX can't be deleted.
dsync-remote(user2): Error: Mailbox INBOX sync: mailbox_delete failed: INBOX can't be deleted.
dsync-remote(user4): Error: Mailbox INBOX sync: mailbox_delete failed: INBOX can't be deleted.

Note for future reference, the -f in the doveadm command was wrong . -f forces a full sync of our 20GB mail storage .
'Full synchronization (-f parameter) scans through all the messages in all the mailboxes. This guarantees that everything will be synchronized, but it's unnecessarily slow for incremental synchronization.'

doveadm-backup - is Dovecot's one-way mailbox synchronization utility.