Best way to use Cephfs in a VM

[jamesthetechie]

I'm sure this has been covered before, but i wasn't able to find quite what i was looking for in other posts.

i have created my proxmox 8 cluster and i have created a cephfs. cephfs needs to go to the VM's hosted within proxmox and im trying to avoid having internet traffic go outside the host and back in to the VM's.

would the best way to accomplish this with the SDN? or am i over thinking it?

apologies for the question, i'm very new to ceph and its a lot to learn.

 

[alexskysilk]

There are two ways to do this.

the "proper way is to attach your ceph public interface to a bridge, and then give your vm a nic attached to it. that way, it will become a full fledged client for the filesystem, with its own cephx security settings.

If you're ok with the hypervisor having access to the file system, you can mount the file system to the host, and then use virtiofs to pass it to the guest.

ephfs needs to go to the VM's hosted within proxmox and im trying to avoid having internet traffic go outside the host and back in to the VM's.

there is no avoiding that. ceph requires resources outside any one host no matter what you do- no reason to "avoid" it.

 

[jamesthetechie]

mounting it to the hypervisor seems like an interesting way of doing it, but would probably save me some network traffic that way.

all im using it as is a basic file share so it should be ok if the host has access to it. VM's just need to be able to change perms and chown files.

i have ceph going over IB using IPoIB if that makes a difference.

 

[jamesthetechie]

There are two ways to do this.

the "proper way is to attach your ceph public interface to a bridge, and then give your vm a nic attached to it. that way, it will become a full fledged client for the filesystem, with its own cephx security settings.

If you're ok with the hypervisor having access to the file system, you can mount the file system to the host, and then use virtiofs to pass it to the guest.

there is no avoiding that. ceph requires resources outside any one host no matter what you do- no reason to "avoid" it.

i was looking at the virtiofs and what a cool piece of software, unfortunately it doesn't support migration yet, so i wont be able to use it in a cluster

ill be testing out using a bridge next and see if i can get it working that way. if the traffic goes over my IB network then its fine, the Ethernet network is all 1gb.

 

[alexskysilk]

i was looking at the virtiofs and what a cool piece of software, unfortunately it doesn't support migration yet, so i wont be able to use it in a cluster

it never will. anything that ties a vm to a specific host will preclude migration.

 

[alexskysilk]

if the traffic goes over my IB network then its fine

if you are using ipoib for your ceph transport, no can do. ipoib does not have ethernet l2 capabilities which means you cant use it for a linux bridge. Your only option is to have a participating server bridge cephfs to nfs.

 

[jamesthetechie]

if you are using ipoib for your ceph transport, no can do. ipoib does not have ethernet l2 capabilities which means you cant use it for a linux bridge. Your only option is to have a participating server bridge cephfs to nfs.

thats fair, how about using a VF from the IB card to the VM using SRIOV and getting ceph data that way?

 

[alexskysilk]

thats fair, how about using a VF from the IB card to the VM using SRIOV and getting ceph data that way?

theoretically possible, but I've never tried/heard of someone using this. if you do go down that road, please be sure to document the process and post of your experience.