PVEceph configuration not enabled (500)

[PiotrD]

Hi,
We are running proxmox cluster with 7 nodes, 3 of them are compute nodes and 4 are osds. In Proxmox panel on all osd nodes and on 1 compute I am able to see Ceph tab in panel, however for compute02 and compute03 I get PVEceph configuration not enabled (500).
Those nodes are part of the cluster and are also running vms using ceph storage. What can be a reason of that ? I have also noticed some weird things happening to vms on those nodes - instant freezes etc.

Kind regards,
Piotr D

 

[brad_mssw]

Are you sure your cluster communication is in good health? You can have really weird issues if not, such as if you are having multicast issues. You should check things like "fence_tool ls", "clustat", "pvecm status", "group_tool ls" and make sure everything appears in good health. If not, you're probably having multicast issues and you should see this FAQ: https://pve.proxmox.com/wiki/Multicast_notes

Mulitcast kicked my butt for the longest time until I figured it out.

 

[dietmar]

You need to run:

# pveceph init

on each node you want to enable the ceph management GUI.

 

[PiotrD]

Are you sure your cluster communication is in good health? You can have really weird issues if not, such as if you are having multicast issues. You should check things like "fence_tool ls", "clustat", "pvecm status", "group_tool ls" and make sure everything appears in good health. If not, you're probably having multicast issues and you should see this FAQ: https://pve.proxmox.com/wiki/Multicast_notes

Mulitcast kicked my butt for the longest time until I figured it out.

Hmm multicast seems ok, but I am suspecting that those freezes are connected to some ceph bug. Probably I am going to upgrade to Ceph giant.

You need to run:

# pveceph init

on each node you want to enable the ceph management GUI.

Thank you, that helped. Regarding Ceph, have you tested latest Ceph Giant release with proxmox ? and can you recommend to upgrade to this version ?

 

[brad_mssw]

Hmm multicast seems ok, but I am suspecting that those freezes are connected to some ceph bug. Probably I am going to upgrade to Ceph giant.

Thank you, that helped. Regarding Ceph, have you tested latest Ceph Giant release with proxmox ? and can you recommend to upgrade to this version ?

Can you describe your freezes a little more? Is it the VM guest that is freezing? If so, I've had that issue too when using the 3.10 kernel, and it goes away if you use the 2.6.32 kernel. I've got a recent forum thread on it and ceph bugs filed but ceph blames the kernel.

 

[PiotrD]

Can you describe your freezes a little more? Is it the VM guest that is freezing? If so, I've had that issue too when using the 3.10 kernel, and it goes away if you use the 2.6.32 kernel. I've got a recent forum thread on it and ceph bugs filed but ceph blames the kernel.

I am using proxmox-ve-2.6.32.
One of the freezes happend when I tried to start new docker container inside the vm running on ceph. The vm freezed, even from console it was not responding, only the reset helped. In vm logs i found those nice bats:
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@


Another issue happend during backup job of ceph vm:

138: Nov 19 01:41:09 INFO: Starting Backup of VM 138 (qemu)
138: Nov 19 01:41:09 INFO: status = running
138: Nov 19 01:41:10 INFO: update VM 138: -lock backup
138: Nov 19 01:41:10 INFO: backup mode: snapshot
138: Nov 19 01:41:10 INFO: ionice priority: 7
138: Nov 19 01:41:10 INFO: creating archive '/mnt/pve/backup/dump/vzdump-qemu-138-2014_11_19-01_41_09.vma.lzo'
138: Nov 19 01:41:10 INFO: started backup task 'e736783e-5c27-4690-827e-6b16c6b4cc89'
138: Nov 19 01:41:13 INFO: status: 0% (118423552/201863462912), sparse 0% (23613440), duration 3, 39/31 MB/s
138: Nov 19 01:41:44 INFO: status: 1% (2038497280/201863462912), sparse 0% (1427709952), duration 34, 61/16 MB/s
138: Nov 19 01:42:12 INFO: status: 2% (4056547328/201863462912), sparse 1% (3232280576), duration 62, 72/7 MB/s
138: Nov 19 01:42:39 INFO: status: 3% (6078070784/201863462912), sparse 2% (5102952448), duration 89, 74/5 MB/s
138: Nov 19 01:43:07 INFO: status: 4% (8155299840/201863462912), sparse 3% (7044325376), duration 117, 74/4 MB/s
138: Nov 19 01:43:31 INFO: status: 5% (10128195584/201863462912), sparse 4% (9016020992), duration 141, 82/0 MB/s
138: Nov 19 01:43:56 INFO: status: 6% (12179668992/201863462912), sparse 5% (11064979456), duration 166, 82/0 MB/s
138: Nov 19 01:44:28 INFO: status: 7% (14189920256/201863462912), sparse 6% (12619145216), duration 198, 62/14 MB/s
138: Nov 19 01:44:54 INFO: status: 8% (16160653312/201863462912), sparse 7% (14456373248), duration 224, 75/5 MB/s
138: Nov 19 01:45:31 INFO: status: 9% (18175098880/201863462912), sparse 7% (15701368832), duration 261, 54/20 MB/s
138: Nov 19 01:45:46 ERROR: VM 138 not running
138: Nov 19 01:45:46 INFO: aborting backup job
138: Nov 19 01:45:46 ERROR: VM 138 not running
138: Nov 19 01:45:47 ERROR: Backup of VM 138 failed - VM 138 not running

It was run in the middle of the night and no one stopped the vm. It just went down. Nothing in the logs - it just died - like power failure.

I suspect that in both these cases there were some issues in ceph due to bigger io load. Maybe upgrading to the latest Giant release would help.
On the other hand, those could be caused by some network failures - I am going to upgrade the switches in the near future so maybe if not upgrading ceph this could help with this issues.

 

[PiotrD]

Hmm it looks like one of the problem was caused by kernel issue inside vm, not by ceph. And another one probably by some network problems. So I guess the latest version of Proxmox and Ceph firefly is fine.
Solved or not a bug

Kind regards,
Piotr D