Expose Ceph Public Network to VM

[manofoz]

Hello, I am trying to execute the following steps to expose Ceph running on my Proxmox host to a VM running K8:

1 - give access to the public ceph network to the VM. Try jumbo frames if you use them.
2 - create ceph.conf and admin key file on vm.
3 - try to see ceph -s
4 - mount RBD or CephFS inside the vm

I am having trouble exposing the ceph network to the VM. My Ceph network is using a tunderbolt ring network with openfabric routing. eno5 and eno6 are the TB interfaces and 10.0.0.81 is the IP from this node:



When I create a linux bridge like this and assign it to an adapter on the VM the network will not connect:



Not sure how to proceed, been googling for a while but am thoroughly stuck.

 

[manofoz]

I was going about this wrong, though I don't know if I did it right. I ended up changing the public network of ceph to my already bridged netwrok the VMs use. Then I copy pasted ceph.conf and a keyring file and boom!

thaynes@kubevip:/etc/ceph$ ceph -s
  cluster:
    id:     9b0628e1-1fe9-49d2-b65b-746d05215e3d
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum pve01,pve02,pve03 (age 39m)
    mgr: pve03(active, since 39m), standbys: pve01, pve02
    mds: 2/2 daemons up, 1 standby
    osd: 6 osds: 6 up (since 63m), 6 in (since 6h)
 
  data:
    volumes: 2/2 healthy
    pools:   6 pools, 161 pgs
    objects: 9.11k objects, 35 GiB
    usage:   102 GiB used, 10 TiB / 10 TiB avail
    pgs:     161 active+clean
 
  io:
    client:   51 KiB/s wr, 0 op/s rd, 10 op/s wr