ceph

I'm seeing crazy low performance in Ceph. I've traced it to OSD's sometimes being very slow...see below. In 3 tests, the first is fairly slow, then fast (as fast as I expect this OSD to be), then the 3rd is crazy slow. root@pm3:~# ceph tell osd.3 bench { "bytes_written": 1073741824...

In our cluster there are currently three hosts with four 3.84TB RI OSDs each. I want to replace the four OSDs with 3.2TB MU SSDs, eventually adding a fifth OSD (later on). Currently there is only around 2.2TB used per OSD so this should work. CEPH version is 19.2.3-pve2...

Hello, this is my first post. I have 3 Dell OptiPlex 7060 Micro which I have upgraded and each of the computers has the following specifications: - 4TB M.2 NVMe - 4TB SATA SSD - 512GB M.2 NVMe Now what I want is to make a CEPH pool with the 3 machines and using only the 512GB disks so I...

Dear, I am facing to a choice to share the existing ceph storage from proxmox cluster to K3S cluster. Is it best practice to do that or need to SoC and each cluster has his own storage ? If proxmox manage the ceph storage is it a good governance ? Some clarifications are welcome. Best Regards,

Hey all, I'm the administrator for a bunch of servers here at the university (both students and researchers). We have quite a lot of data science / machine learning projects, and we store the data in locally hosted S3 buckets. For the last few years, I have just been spinning up a separate...

We're in the process of validating a PVE cluster setup that will be deployed to prod some time in 2026, and for that purpose, we've spun up the MVC (Minimum Viable Cluster) that mimicks, except in node count, what we're planning to have by then. As a result, we have three modern Dell boxen with...

Hello everyone, I'm new to Proxmox and would like to set up Ceph, but my server doesn't have direct internet access. Has anyone done this before or have any tips on how to install Ceph offline? Any advice or experiences would be greatly appreciated! Thanks in advance and best regards,

Hello everyone, I have following issue where I would like to define Ceph-Pool quota-thresholds for all thresholds defined within my Ceph-Cluster. These quota-definitions work, but my issue is that warnings and critical notices within the Proxmox-Cpeh dashboard tab are not visible. When following...

A while back I had an event that caused my Ceph cluster to crash. By design I had backups of everything that mattered. However, I wanted to see if I could fix the cluster and maybe try to bring back some VMs that won't start due to the crash. I had a lot of `pgs inactive`, but managed to get...

Hello :) my ceph dashboard shows duplicate entries for the ceph mon/mgr on node01 for some reason, it doesnt seem to affect anything but Id like to get rid of it :D pveversion 9.0.10 ceph 19.2.3 I found this thread with almost the same problem as me, changing /etc/hostname from fqdn to...

Hi, I have 2 clusters ceph one is hyper converged proxmox and the other one is cephadm managed. I want to mirrored them to test if it's possible. But by following the proxmox documentation about rbd-mirroring, rbd mirror wont start am I missing something ? Thanks for your help :) !

Hello Everyone I seem to struggle when it comes to the Ceph full-mesh cluster created. I made one using a bond on each node using the fiber NIC's two interfaces of 25Gbps fiber card. Later I made an additional corosync link, not sure if this picks up if the other is down. I have created a pool...

Hello all, I am adding Ceph to an 3 node cluster. On each machine, I have one 10GbE and one 1GbE link available. What would be the better way of configuring my network? - 10 GbE Public network, 1 GbE Cluster network - 10 GbE Cluster network, 1 GbE Public network - 10 GbE both networks Thank...

Hello everyone, I’m currently facing a rather strange issue on my Proxmox cluster, which uses Ceph for storage. My infrastructure consists of 8 nodes, each equipped with 7 NVMe drives of 7.68 TB. Each node therefore hosts 7 OSDs (one per drive), for a total of 56 OSDs across the cluster...

Hello everyone, I’m currently facing a rather strange issue on my Proxmox cluster, which uses Ceph for storage. My infrastructure consists of 8 nodes, each equipped with 7 NVMe drives of 7.68 TB. Each node therefore hosts 7 OSDs (one per drive), for a total of 56 OSDs across the cluster...

1. Can you take 3 servers with 4 port networking and connect them in this fashion: A - B A - C B - C with 3 cables and have redundant networking? (literal redundant would require 6 cables and eat all ports I think) 2. Are there three nets in the end? (and then possibly connect them all together...

Hi all, i am facing a strange issue, after using having a proxmox pc for my self hosted app I decided to play around and create a cluter to dive deeper into the HA topics, i dowloaded the latest ISO and build up a cluster from scratch. My Cluster works, i can see every node, my ceph storage says...

Hi there, We're planning on migrating one of our Ceph platforms to new Ceph versions, this currently runs 'regular' Ceph from the official download()ceph()com repository on Ubuntu 22.04. Now we'd like to change this out for Proxmox VE Ceph, primarily because of the extra bug-fixes & cherry...

Because someone (hint: me) designed the ceph network poorly, I would like to make changes to it. For that, I need to move it from running a seperate public/cluster network on 2x25gbe bond, to my primary interface, thats is a 2x10 gbe bond. While it will be a slower network, its a temporary...

Hi all, is it correct that by default Proxmox 9 in a 3 node cluster configuration with CEPH for VM migration use VMBR0 network? Current node network config does use a separate network for CEPH Cluster (10.10.10.x/24), CEPH Public (10.10.20.x/24), COROSYNC (172.16.1.x/24) and VMBR0...