Hello,
I'm new to both Proxmox and CEPH... I'm trying to set up a cluster for long-term temporary use (Like 1-2 years) for a small organization that has most of their servers in AWS, but has a couple legacy VMs that are still hosted in a 3rd party data center running VMware ESXi. We also plan to host a few other things on these servers that may go beyond that timeline. The datacenter that is currently providing the hosting is being phased out at the end of the month, and I am trying to migrate those few VMs to Proxmox until those systems can be phased out. We purchased some relatively high end (though previous gen) servers for reasonably cheap, servers that are actually a fair bit better than the ones they're currently hosted on. However, because of budget and issues I was seeing online with people claiming Proxmox and SAS connected SANs didn't really work well together, and the desire to have the 3 server minimum for a cluster/HA etc, I decided to go with CEPH for storage. The drives are 1.6TB Dell NVME U.2 drives, I have a Mesh network using 25GB links between the 3 servers for CEPH, and there's a 10GB connection to the switch for networking. Currently 1 network port is unused, however I had planned to use it as a secondary connection to the switch for redundancy. Currently, I've only added 1 of these drives from each server to the CEPH setup, however I have more I want to add to once it's performing correctly. I was ideally trying to get the most redundancy/HA as possible with what hardware we were able to get a hold of and the short timeline. However things took longer just to get the hardware etc than I'd hoped, and although I did some testing, I didn't have hardware close enough to test some of this stuff with.
As far as I can tell, I followed instructions I could find for setting up CEPH with a Mesh network using the routed setup with fallback. However, it's running really slow. If I run something like CrystalDiskMark on a VM, I'm seeing around 76MB/sec for sequential reads and 38MB/sec for Seq writes. The random read/writes are around 1.5-3.5MB/sec.
At the same time, on the rigged test environment I set up prior to having the servers on hand, (which is just 3 old Dell workstations from 2016 with old SSDs in them and a 1GB shared network connection) I'm seeing 80-110MB/sec for SEQ reads, and 40-60 on writes, and on some of the random reads I'm seeing 77MB/sec compared to 3.5 on the new server.
I've done IPERF3 tests on the 25GB connections that go between the 3 servers and they're all running just about 25GB speeds.
Here is my /etc/network/interfaces file. It's possible I've overcomplicated some of this. My intention was to have separate interfaces for mgmt, VM traffic, cluster traffic, and ceph cluster and ceph osd/replication traffic. Some of these are set up as virtual interfaces as each server has 2 network cards, both with 2 ports, so not enough to give everything its own physical interface, and hoping virtual ones on separate vlans are more than adequate for the traffic that doesn't need high performance.
Ceph Config File:
My /etc/frr/frr.conf file:
If I do the same disk benchmarking with another of the same NVME U.2 drives just as an LVM storage, I get 600-900MB/sec on SEQ reads and writes.
Any help is greatly appreciated, like I said setting up CEPH and some of this networking stuff is a bit out of my comfort zone, and I need to be off the old set up by July 1. I can just load the VMs onto local storage/LVM for now, but I'd rather do it correctly the first time.
Also, if anyone even has a link to a video or directions you think might help, I'd also be open to them. A lot of the videos and things I find are just "Install Ceph" and that's it, without much on the actual configuration of it.
Thanks
I'm new to both Proxmox and CEPH... I'm trying to set up a cluster for long-term temporary use (Like 1-2 years) for a small organization that has most of their servers in AWS, but has a couple legacy VMs that are still hosted in a 3rd party data center running VMware ESXi. We also plan to host a few other things on these servers that may go beyond that timeline. The datacenter that is currently providing the hosting is being phased out at the end of the month, and I am trying to migrate those few VMs to Proxmox until those systems can be phased out. We purchased some relatively high end (though previous gen) servers for reasonably cheap, servers that are actually a fair bit better than the ones they're currently hosted on. However, because of budget and issues I was seeing online with people claiming Proxmox and SAS connected SANs didn't really work well together, and the desire to have the 3 server minimum for a cluster/HA etc, I decided to go with CEPH for storage. The drives are 1.6TB Dell NVME U.2 drives, I have a Mesh network using 25GB links between the 3 servers for CEPH, and there's a 10GB connection to the switch for networking. Currently 1 network port is unused, however I had planned to use it as a secondary connection to the switch for redundancy. Currently, I've only added 1 of these drives from each server to the CEPH setup, however I have more I want to add to once it's performing correctly. I was ideally trying to get the most redundancy/HA as possible with what hardware we were able to get a hold of and the short timeline. However things took longer just to get the hardware etc than I'd hoped, and although I did some testing, I didn't have hardware close enough to test some of this stuff with.
As far as I can tell, I followed instructions I could find for setting up CEPH with a Mesh network using the routed setup with fallback. However, it's running really slow. If I run something like CrystalDiskMark on a VM, I'm seeing around 76MB/sec for sequential reads and 38MB/sec for Seq writes. The random read/writes are around 1.5-3.5MB/sec.
At the same time, on the rigged test environment I set up prior to having the servers on hand, (which is just 3 old Dell workstations from 2016 with old SSDs in them and a 1GB shared network connection) I'm seeing 80-110MB/sec for SEQ reads, and 40-60 on writes, and on some of the random reads I'm seeing 77MB/sec compared to 3.5 on the new server.
I've done IPERF3 tests on the 25GB connections that go between the 3 servers and they're all running just about 25GB speeds.
Here is my /etc/network/interfaces file. It's possible I've overcomplicated some of this. My intention was to have separate interfaces for mgmt, VM traffic, cluster traffic, and ceph cluster and ceph osd/replication traffic. Some of these are set up as virtual interfaces as each server has 2 network cards, both with 2 ports, so not enough to give everything its own physical interface, and hoping virtual ones on separate vlans are more than adequate for the traffic that doesn't need high performance.
Bash:
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!
auto lo
iface lo inet loopback
auto eno1np0
iface eno1np0 inet manual
#Daughter Card - NIC1 10G to network switch
auto eno2np1
iface eno2np1 inet manual
#Daughter Card - NIC2 10G to network switch
iface ens6f0np0 inet manual
mtu 9000
#PCIx - NIC1 25G Storage direct attached
iface ens6f1np1 inet manual
mtu 9000
#PCIx - NIC2 25G Storage direct attached
auto bond0
iface bond0 inet manual
bond-slaves eno1np0 eno2np1
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer3+4
mtu 1500
#Network bond of both 10GB interfaces (Currently 1 is not plugged in)
auto vmbr0
iface vmbr0 inet manual
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
post-up /usr/bin/systemctl restart frr.service
#Bridge to network switch
auto vmbr0.6
iface vmbr0.6 inet static
address 10.6.247.1/24
#VM network
auto vmbr0.1247
iface vmbr0.1247 inet static
address 172.30.247.1/24
#Regular Non-CEPH Cluster Communication
auto vmbr0.254
iface vmbr0.254 inet static
address 10.254.247.1/24
gateway 10.254.254.1
#Mgmt-Interface
source /etc/network/interfaces.d/*Ceph Config File:
Bash:
[global]
auth_client_required = cephx
auth_cluster_required = cephx
auth_service_required = cephx
cluster_network = 192.168.0.1/24
fsid = 68593e29-22c7-418b-8748-852711ef7361
mon_allow_pool_delete = true
mon_host = 10.6.247.1 10.6.247.2 10.6.247.3
ms_bind_ipv4 = true
ms_bind_ipv6 = false
osd_pool_default_min_size = 2
osd_pool_default_size = 3
public_network = 10.6.247.1/24
[client]
keyring = /etc/pve/priv/$cluster.$name.keyring
[client.crash]
keyring = /etc/pve/ceph/$cluster.$name.keyring
[mon.PM01]
public_addr = 10.6.247.1
[mon.PM02]
public_addr = 10.6.247.2
[mon.PM03]
public_addr = 10.6.247.3My /etc/frr/frr.conf file:
Bash:
# default to using syslog. /etc/rsyslog.d/45-frr.conf places the log in
# /var/log/frr/frr.log
#
# Note:
# FRR's configuration shell, vtysh, dynamically edits the live, in-memory
# configuration while FRR is running. When instructed, vtysh will persist the
# live configuration to this file, overwriting its contents. If you want to
# avoid this, you can edit this file manually before starting FRR, or instruct
# vtysh to write configuration to a different file.
frr defaults traditional
hostname PM01
log syslog warning
ip forwarding
no ipv6 forwarding
service integrated-vtysh-config
!
interface lo
ip address 192.168.0.1/32
ip router openfabric 1
openfabric passive
!
interface ens6f0np0
ip router openfabric 1
openfabric csnp-interval 2
openfabric hello-interval 1
openfabric hello-multiplier 2
!
interface ens6f1np1
ip router openfabric 1
openfabric csnp-interval 2
openfabric hello-interval 1
openfabric hello-multiplier 2
!
line vty
!
router openfabric 1
net 49.0001.1111.1111.1111.00
lsp-gen-interval 1
max-lsp-lifetime 600
lsp-refresh-interval 180If I do the same disk benchmarking with another of the same NVME U.2 drives just as an LVM storage, I get 600-900MB/sec on SEQ reads and writes.
Any help is greatly appreciated, like I said setting up CEPH and some of this networking stuff is a bit out of my comfort zone, and I need to be off the old set up by July 1. I can just load the VMs onto local storage/LVM for now, but I'd rather do it correctly the first time.
Also, if anyone even has a link to a video or directions you think might help, I'd also be open to them. A lot of the videos and things I find are just "Install Ceph" and that's it, without much on the actual configuration of it.
Thanks
Last edited:
