ceph

We are using Ceph on three nodes (10g). There are one HDD pool (3 OSDs per node) and one NVMe pool. The NVMes are used for the HDD WALs and DBs, too. For each OSD osd_max_scrubs is set to 1. During the deep scrubbing phases (I have limited this to some hours during the night) the cluster is...

Hi all, Recently the latest version of PVE-Ceph Quincy released (17.2.8)* within the Enterprise repository. However on another (non-PVE ceph) storage cluster we experienced quite the nasty bug related to BluestoreDB (https://tracker.ceph.com/issues/69764). Causing OSD's to crash and recover...

Hello! I have another issue I'd appreciate a community feedback from. One of our nodes crashed during last Friday inexplicably during migration of a vm disk from ceph storage to a local one. I'm attaching ceph log on pastebin as it's too long: https://pastebin.com/wkxHP7rt proxmox-ve: 8.3.0...

Hello all! Recently we have experienced a power outage and loss network connectivity (Junpier switch that was used by Ceph cluster). Some proxmox/ceph nodes have been restarted as well. Network traffic and nodes have been restored but our cluster is in critical condition. On the monitors we...

Hi, I’m running Plex in an LXC container with the root disk on my local-zfs storage. However, when I try to move the storage to my ceph pool, my local OSD drives disconnect during the process. I tried doing something similar with a larger VM disk (300GB) with no issues. Likewise, when I move...

Hi all disclaimer: I work for Fujitsu, this is a semi-official posting but I hope it is not considered as advertising. ;) I found a couple of discussions in this forum around recommended hardware and components for Primergy servers, so maybe this will be helpful for some of you. Of course we...

Hello, I have a cluster of 9 nodes, with a default crush map and a replication (size) of 3, and min_size of 2. Are there any points against increasing the size to 5 and min_size 3 during operation to handle a higher failure of individual nodes? Enough disk space is available, I am aware of...

We are using Open vSwitch (OVS) for networking in our Proxmox VE cluster. We are experiencing a network issue after restarting a node – Ceph is not immediately ready, and HEALTH_OK status takes several minutes to appear instead of just a few seconds. Observed behavior: • Ping with MTU...

There’s not a whole lot of support forms on the Internet for ceph, so if this is misplaced, I apologize. This is my little one page platform I'm putting together https://github.com/rlewkowicz/micro-platform I use ceph nano to launch a standalone ceph cluster...

Hi ALL, I have a requirement where i need to install & Configure a 3 node Proxmox Cluster with HCI. As part of this setup, I need to configure Ceph storage and enable High Availability (HA). 3*server Configuration: 2CPU - Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz (18cores) with 10 RAM (32GB)...

Moin zusammen, aktuell setzen wir einen kleinen Proxmox-Cluster jeweils mit lokalem Storage via ZFS ein. Da die Hardware zur Reinvestition ansteht, plane ich einen neuen PVE-Cluster mit CEPH und habe dazu noch ein paar Fragen, die ich trotz Recherche hier und via Google nicht oder teilweise...

Hello, I just want to ask, is it possible for me to install Ceph with Cephadm on a Proxmox node? then I install the RBD from Ceph as a storage disk image / container? I think the Ceph that is included in Proxmox is less customizable, and cannot implement Orchestrator on Ceph Proxmox.

I am relatively new to Proxmox. I had Ceph installed and fully configured and for learning purposes i tried removing it. I removed the Ceph install and rebuilt the whole Ceph setup. I have all the nodes online, but i can't create an OSD. The drives are wiped. NAME MAJ:MIN RM...

Hello all, Here is the situation : We have a Ceph cluster on top of proxmox on Dell Hardware. On of the DELL virtual disk failed, hence the corresponding OSD failed. This is a HDD disk not NVME and "thankfully" the bluestore was not split out on local NVME disks. Anyway, we followed the...

Hello, just after a bit of implementation advice for planning out a cluster. Scenario - VMs running on ceph as a high availability cluster, but attached to each VM is an additional drive which is just for storing ephemeral data as a high-speed cache. This data does not need to survive a...

Mit dem Titel fängt es ja schon mal sehr gut an. Wir haben ein kleines Problem: Wir müssen eine VM aus dem Backup wiederherstellen und leider sind unsere Backups failed. Warum fragen wir trotzdem? Im Backup liegen 5 SCSi Abbilder, welche ursprünglich aus einem Ceph Cluster stammen. Nur die...

Hello, I had OSD 23 on one of my hosts but the drive failed so we stopped the services and removed it. We did not need to replace it immediately (lots of OSDs in the cluster so not a lot of capacity lost) so I didn't pay much attention to it. Sometime in the last couple of months another disk...

I hope this message finds you well. I would like to gain a clearer understanding of the Monitor permissions (mon caps) in Ceph. Specifically, could you explain what actions these permissions allow, and what exactly can be executed or written with them? Thank you in advance :)

Hi, I'm trying to figure out how to upgrade Ceph from 18.2.2 to 18.2.4. I've found this document describing going from Reef to Squid but I want to get to the latest Reef 18.2.4 first. I know I can use that procedure to get to 18.2.4 if I don't update my ceph repo file to squid but then I'll...

What are the best storage configurations in Proxmox to access a SAN with multiple nodes while ensuring high performance, redundancy, and simultaneous access? How do options like ZFS over iSCSI, ZFS over NVMe/TCP, Ceph compare in terms of latency, scalability, and complexity? The goal is to...