ceph

Yesterday I was greeted with numerous unreachable services stemming from a Ceph health error on our VM cluster due to "1 full osd(s)" and "1 backfillfull osd(s)" resulting in "4 pool(s) full" and solved it. This was the ceph status panel: As a result, our Ceph ended up in read-only state. The...

Hello, I have an 3 node full mesh ceph cluster with 4x 100 GE NIC per Node. With the SDN stack, I use openfabric for the NICs and on top ceph. For corosync, there are 2 dedicated NICS via switch (no lacp, bonding/teaming), each corosync link is in his own small network. VM traffic and management...

Hi everyone, we are using Proxmox VE with Ceph and were wondering how much storage we should use from our available storage before ceph starts acting up. Our Ceph Cluster consists of three nodes that each contribute a capacity of 27,94TB, so we reach a total of 83,82TB We were wondering now...

Hi all, I’m putting together a Proxmox cluster with Ceph for HA and wanted to get some feedback before I go ahead and deploy everything. What I’m aiming for is fairly simple: I want proper HA with no data loss and automatic failover, but at the same time I’d still like one node (an R640) to...

Hallo zusammen, derzeit betreibe ich ein Proxmox Cluster mit Ceph als Shared Storage. Hardware pro Node: i5-12500 64GB DDR5 RAM 3x 1,92TB PM883 SSD LACP 2x 10Gbit für Cluster Network LACP 2x 10Gbit für VM Network Bei Benchmarks des Storages bekomme ich maximale Schreibwerte von rund...

Dear all, I am running PVE in a what i consider being a rather typical "IT nerd" setup, a 3-node-Cluster on consumer hardware. Private use only. 2 nodes are built identically: AMD 3200G (2C/4T) APU, 32GB DDR4 RAM, M2 NVMe for OS, 2x4TB HDD for storage (connected to SATA onboard), GbE NIC 1...

Hello, I have several Proxmox clusters of 3 dedicated servers each. Previously, I used GlusterFS to replicate the VM disks between the three servers in the cluster so that if one server went down, the VMs would restart on the two remaining servers without data loss. This method worked very...

I've got a working 6 nodes stretched Proxmox VE9.1 + Ceph cluster, with hosts split over two datacenters (3 in each). In a third datacenter I've got a Proxmox VE 9.1 virtual machine (on vsphere) which acts as proxmox + ceph tie-breaker. Each host has 4x25 Gbits/s interfaces in LACP bond defined...

Hello everyone, I have spent a lot of time to figure out what is provoking that IO wait. on this cluster VM who do a high amount of IO have a lot of IO wait ( like 30k I/O read, 50% IO wait) Summary of my setup: PVE proxmox-ve: 9.0.0 (running kernel: 6.17.2-2-pve) pve-manager: 9.0.18...

Since last minor upgrade end of january, we have an crashing osd every few days. OSDs recovering from itself. Journalctl looks very similar every time. It started a few days after the last minor update. My assumption is, it`s maybe related. Do you have any ideas or tips which information/logs...

I'm playing with a cluster composed by 3 machines with 3 hdd osd per machine (9 osd in total). It is a test environment to learn doing stuff, but I don't want to destroy it with this "expansion". I have a VM with an application that is not tolerating HDD slow performance, and since I have 3 ssd...

I've noticed VMs that show much higher usage via rbd du than in the VM, for example: NAME PROVISIONED USED vm-119-disk-0 500 GiB 413 GiB vm-122-disk-0 140 GiB 131 GiB Inside the VM, df shows 95G and 63G used space, respectively. Both of these are Debian 12 which has...

Hi everyone, I've run into a serious issue while managing a Proxmox VE Ceph environment. A user created a lot of VMs and ended up filling the entire Ceph cluster. The problem is, when I look at the RBD storage in the WebUI, I can only see the "Provisioned Size" of each disk. I can't tell which...

Hi there, the Title may be a bit deceptive as I know there are good solutions that work for many but for me/my workplace we face a bit of a dilemma. I know I'm opening this can of worms again and this is also partly me venting a bit of my frustration and I'm sorry about that. We wanna use...

I am on the latest proxmox 9.1.5 with ceph 19.2.3 squid. These warnings bothering me a lot and I am scared to touch anything related to ceph. systemd‑sysv‑generator: SysV service '/etc/init.d/ceph' lacks a native systemd unit file… Please update package to include a native systemd unit file...

Environment Proxmox VE cluster with 2 nodes (node94: 10.129.56.94, node107: 10.129.56.107) Ceph cluster running on the Proxmox nodes (public_network: 10.129.56.0/24) Proxmox SDN EVPN zone (madp) for VM networking VMs are on the EVPN overlay network: 172.16.0.0/16 Goal: Configure Rook External...

Hi! I’m planning to migrate from TrueNAS to Ceph because I want node-based redundancy. I was planning to use HDDs with an SSD cache tier on top to boost performance, since I work with heavy file sequences and want to saturate my 10 Gbit network. However, I found that the cache tier is...

Hello all i have a 3 node proxmox cluster with ceph. each node has 2x 4tb samsung 870 QVO ssds. I have noticed my vms being really slow and i was wondering how much of that is because of the ssds. I have checked my network and everything else. Im here to just confirm if what AI assistant is...

Hello everyone, I need your support. We recently updated our cluster to versions PVE 8.4.16 and CEPH 17.2.8, and only after the update did we read an article that said we need to urgently upgrade from this version because there's a critical error with bluestore. Can you tell me if the error is...

Schönen guten Morgen, ich habe hier eine HA Cluster mit CEPH-Storage im Betrieb. Auf 3 Nodes verteilt laufen ca. 30 VEs (Linux, Windows). Hardware:CPUs, RAM und Storage ausreichend vorhanden. Seit einiger Zeit kann ich, auch bei ausgeschalteter VE die Größe des RAMs nicht ändern. Weder...