[SOLVED] Add node to cluster -> can't access ceph-vm

F

fips

Renowned Member
Proxmox Subscriber
May 5, 2014
178
10
83
Hi,

I have a 3 node cluster with ceph, which is running fine.
Now I wanted to add nodes which just run VMs and store their data on the ceph storage.

The 4. node is now part of the cluster and can access the ceph-lxc pool (and see its content) but not the ceph-vm (can't see the content).
I already restarted the 4. node but nothing happend.

4. node was just added to the pvecm cluster not ceph cluster.

What to do?

EDIT:

maybe "pveceph install --version jewel" on the new 4th node is needed?
Even if the 4th node doesn't be of the ceph cluster and it either not holding any OSDs??
 
Last edited:
Please explain your setup further. Which version of PVE are you using? How das your storage.cfg look like? What error message do you receive?
 
My setup:
3 nodes with proxmox 4.4-92 and Ceph Jewel, it was setup like in the video described.
So I have 2 pools: ceph-vm and a ceph-lxc.

Cluster is fine, ceph is running over sperate 10Gb network.

new node is a fresh installed and patched 4.4-92 proxmox, in the same cluster and storage network like the others.
I just executed pvecm add 192.168.3.21 on the new node.
Afterward it was part of the cluster:

Code: 
Quorum information
------------------
Date:             Wed Aug  9 14:19:56 2017
Quorum provider:  corosync_votequorum
Nodes:            4
Node ID:          0x00000004
Ring ID:          4/244
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   4
Highest expected: 4
Total votes:      4
Quorum:           3 
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000004          1 192.168.3.11 (local)
0x00000001          1 192.168.3.21
0x00000002          1 192.168.3.22
0x00000003          1 192.168.3.23


If I open the ceph-lxc pool on new 4th node, I can see the lxc disk images. They are all listed.
But if I click on ceph-vm pool nothing is listed, no vm disk images.

Correct me if I am wrong: but nodes access the ceph-lxc pool with krbd drivers which are loaded.
So If the node want to access the ceph-vm node, the proxmox rbd drivers has to be installed?
This I would do with a "pveceph install --version jewel"?

Can you approve my suggestion?

I just don't want to crash my ceph cluster ;-)

Ceph nodes:
Code: 
proxmox-ve: 4.4-92 (running kernel: 4.4.67-1-pve)
pve-manager: 4.4-15 (running version: 4.4-15/7599e35a)
pve-kernel-4.4.35-1-pve: 4.4.35-77
pve-kernel-4.4.67-1-pve: 4.4.67-92
lvm2: 2.02.116-pve3
corosync-pve: 2.4.2-2~pve4+1
libqb0: 1.0.1-1
pve-cluster: 4.0-52
qemu-server: 4.0-110
pve-firmware: 1.1-11
libpve-common-perl: 4.0-95
libpve-access-control: 4.0-23
libpve-storage-perl: 4.0-76
pve-libspice-server1: 0.12.8-2
vncterm: 1.3-2
pve-docs: 4.4-4
pve-qemu-kvm: 2.7.1-4
pve-container: 1.0-101
pve-firewall: 2.0-33
pve-ha-manager: 1.0-41
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u3
lxc-pve: 2.0.7-4
lxcfs: 2.0.6-pve1
criu: 1.6.0-1
novnc-pve: 0.5-9
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.9-pve15~bpo80
ceph: 10.2.9-1~bpo80+1

Storage.cfg
Code: 
dir: local

    path /var/lib/vz

    content vztmpl,backup,iso


zfspool: local-zfs

    pool rpool/data

    content images,rootdir

    sparse 1


rbd: ceph-vm

    monhost 172.30.3.21 172.30.3.22 172.30.3.23

    content images

    krbd 0

    pool ceph-vm

    username admin


rbd: ceph-lxc

    monhost 172.30.3.21 172.30.3.22 172.30.3.23

    content rootdir

    krbd 1

    pool ceph-lxc

    username admin


nfs: Backup

    export /mnt/Backup

    path /mnt/pve/Backup

    server 172.30.3.50

    content rootdir,images,iso,vztmpl,backup

    maxfiles 10

    options vers=3,async
 
PVE 4.4 is by default ceph hammer release.
 
You must log in or register to reply here.
Top Bottom