Proxmox VE Ceph/RBD Usage

E

emregozturk

New Member
Proxmox Subscriber
Dec 1, 2022
17
0
1
Hello,

Currently we are using NFS storage for our VMs. However, there are some performance issues. We want to try Ceph/RBD storage.
We have three proxmox nodes (2 physical proxmox VE + 1 tiebreaker). We have LVM(Block) storage with SAN configuration. There is a NVMe storage and two SAN switches. There are some VMs that are working in LVM storage. We are wondering if we can configure Ceph/RBD with SAN configuration. I know Ceph/RBD is used for especially hyperconverged systems/local disks but we want to try this solution. Is it possible to use SAN LUNs for Ceph/RBD ?

Regards
 
Its possible, however it wasnt meant to be used this way. Depending on your SAN you have some sort of RAID implementation there. The LUNs are likely slices of that RAID. By placing Ceph on top of it you will be replicating data across the same RAID group again, increase write amplification of each block. I would not expect a great performance results out of it.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
bbgeek17 said:
Its possible, however it wasnt meant to be used this way. Depending on your SAN you have some sort of RAID implementation there. The LUNs are likely slices of that RAID. By placing Ceph on top of it you will be replicating data across the same RAID group again, increase write amplification of each block. I would not expect a great performance results out of it.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...
Hello bbgeek17,

Thank you for your message. You are right, the new and existing data will be balanced across the nodes and it will consume some network latency and storage space.
The thing is, we want to use our SAN configuration with proxmox VE features such as snapshot, backup. The performance on using NFS in NVMe storage is not that good.I did some research and found that Ceph/RBD is best choice for this. I would want to hear if you recommend different things.

Regards
 
If you must stay within your existing infrastructure and you want to have snapshots, then your most straightforward solution would be to implement a Clustered File System. You can then utilize it in conjunction with QCOW, which will give you snapshots/clones, etc.
There is no built-in/supported CFS that comes with PVE, you will need to research/install/configure one yourself.

I would also look into what is "not good" in regards to NFS performance. Is the issue with your vendor NFS implementation? Network? Clients? The Ceph replication will presumably go over the same network. If the storage is iSCSI - more than doubling amount of traffic.

Good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
It may also will be possible, to use zfs with iSCSI backed vdevs. But only for the features. Not for performance i think.
 
When you have a NVMe Storage, you can use NVMe over TCP or over FC and LVM on top.
This works also great.
 
Falk R. said:
When you have a NVMe Storage, you can use NVMe over TCP or over FC and LVM on top.
Click to expand...
The NVMe over Fabrics protocols provide a way to expose the base NVMe functionality over a network (i.e., controls, commands, and data transport). NVMe and the associated transport protocols do not provide virtualized storage management capabilities such as snapshots.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
bbgeek17 said:
The NVMe over Fabrics protocols provide a way to expose the base NVMe functionality over a network (i.e., controls, commands, and data transport). NVMe and the associated transport protocols do not provide virtualized storage management capabilities such as snapshots.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...
On iSCSI with LVM, you can’t made Snapshot's. With NVMe you have better Performance and Latency than iSCSI or FC, with same restrictions.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back