shared storage for VMs on cephfs

vasilis

New Member
Jan 31, 2022
2
0
1
38
Hi
First time posting here. I have a proxmox cluster with 3 nodes. All identical HP proliant (older models) server computers with 10G networking for CEPH.
The vm's are a mix of win10 and centos 8. What i am trying to achieve is to have a part of cephfs treated as a directory which can be shared with read/write access among all the vms for fast data transfers instead of moving data over the 1G network which is currently used for vm access.
Any suggestions will be greatly appreciated. THanks
 

Moayad

Proxmox Staff Member
Staff member
Jan 2, 2020
2,159
175
68
29
Vienna
shop.maurer-it.com
Hi,

You can mount a directory from CephFS to a Linux VM by creating a Directory in let's say VMs, first create VMs on /mnt/pve/cepfhs and create a key for a client user who manage the mount with full r/w permission. e.g.:

Bash:
mkdir -p /mnt/pve/cephfs/VMs
ceph fs authorize cephfs client.foo /VMs rw
mount -t ceph 192.168.1.2:6789,192.168.1.3:6789,192.168.1.4:6789:/VMs /mnt/bar/ -o name=foo,secret=AQDtOk1iEj50IhAABN2Sy9WEKk9aMYYrodnCkQ==

Maybe you have to install ceph-common on the Linux VM. Regarding Windows VM, I didn't try to mount an CephFS on Win system before, however the Ceph Docs mentioned that [0]. See also here [1] about caps.

[0] https://docs.ceph.com/en/latest/cephfs/ceph-dokan/
[1] https://docs.ceph.com/en/latest/rados/operations/user-management/
 

vasilis

New Member
Jan 31, 2022
2
0
1
38
Hi,

You can mount a directory from CephFS to a Linux VM by creating a Directory in let's say VMs, first create VMs on /mnt/pve/cepfhs and create a key for a client user who manage the mount with full r/w permission. e.g.:

Bash:
mkdir -p /mnt/pve/cephfs/VMs
ceph fs authorize cephfs client.foo /VMs rw
mount -t ceph 192.168.1.2:6789,192.168.1.3:6789,192.168.1.4:6789:/VMs /mnt/bar/ -o name=foo,secret=AQDtOk1iEj50IhAABN2Sy9WEKk9aMYYrodnCkQ==

Maybe you have to install ceph-common on the Linux VM. Regarding Windows VM, I didn't try to mount an CephFS on Win system before, however the Ceph Docs mentioned that [0]. See also here [1] about caps.

[0] https://docs.ceph.com/en/latest/cephfs/ceph-dokan/
[1] https://docs.ceph.com/en/latest/rados/operations/user-management/
Hi
Thanks so much for taking the time to reply to this. Your wording confused me a little bit so to make sure i understand the concept of what you are suggesting:
All my VM's reside on CEPH FS.
Create a vm
then create a directory on /mnt/pve/cephfs ( all vms can see this directory)
The IP address you used on mount -t is the ceph network (which in my is separate as per the proxmox mesh network instructions) or the vm network?
Will the vms need an IP on the ceph network in addition to the vm access network?

Many thanks again
 

Moayad

Proxmox Staff Member
Staff member
Jan 2, 2020
2,159
175
68
29
Vienna
shop.maurer-it.com
Hi,

The IP address you used on mount -t is the ceph network (which in my is separate as per the proxmox mesh network instructions) or the vm network?
The IPs should be for the Ceph monitors.
Will the vms need an IP on the ceph network in addition to the vm access network?
If the VMs can ping/access the Ceph mon IPs, then not need an additional IP.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!