ceph

I experience the same issue mentioned here: https://forum.proxmox.com/threads/ceph-client-still-using-luminous-while-on-squid.165370/ and here: https://forum.proxmox.com/threads/unable-to-mount-cephfs-when-set-require-min-compat-client-squid.159502/ (both of which seem to have remained...

Hi everyone, During testing we are seeing behaviour we haven't seen mentioned elsewhere, but hope somebody here might have ideas on how to solve it. Proxmox is connected to an external 5-node ceph cluster with each node having 3 HDD OSDs using DB/WAL on SSD. Proxmox is running version 8, while...

so...been using proxmox for many years (almost 20). have several existing server clusters. Just building a new server stack. 6x Dell R760 768G ram, dual 32 core cpu. each server with 11 850G 24GPS SA SCSI SSD, 4x 1G nic plus 2X dual 100G nic The drives are 3,400 MB/s capable Build cluster...

I want to connect my PVE datacenter to an external Ceph storage. On a pool on the external Ceph cluster, I created a namespace. Let's say my pool is called `testpool`, my namespace is called `testspace` and my user is called `testuser`. So what i did on my Ceph cluster is rbd namespace create...

Yesterday I was greeted with numerous unreachable services stemming from a Ceph health error on our VM cluster due to "1 full osd(s)" and "1 backfillfull osd(s)" resulting in "4 pool(s) full" and solved it. This was the ceph status panel: As a result, our Ceph ended up in read-only state. The...

Hello, I have an 3 node full mesh ceph cluster with 4x 100 GE NIC per Node. With the SDN stack, I use openfabric for the NICs and on top ceph. For corosync, there are 2 dedicated NICS via switch (no lacp, bonding/teaming), each corosync link is in his own small network. VM traffic and management...

Hi everyone, we are using Proxmox VE with Ceph and were wondering how much storage we should use from our available storage before ceph starts acting up. Our Ceph Cluster consists of three nodes that each contribute a capacity of 27,94TB, so we reach a total of 83,82TB We were wondering now...

Hi all, I’m putting together a Proxmox cluster with Ceph for HA and wanted to get some feedback before I go ahead and deploy everything. What I’m aiming for is fairly simple: I want proper HA with no data loss and automatic failover, but at the same time I’d still like one node (an R640) to...

Hallo zusammen, derzeit betreibe ich ein Proxmox Cluster mit Ceph als Shared Storage. Hardware pro Node: i5-12500 64GB DDR5 RAM 3x 1,92TB PM883 SSD LACP 2x 10Gbit für Cluster Network LACP 2x 10Gbit für VM Network Bei Benchmarks des Storages bekomme ich maximale Schreibwerte von rund...

Dear all, I am running PVE in a what i consider being a rather typical "IT nerd" setup, a 3-node-Cluster on consumer hardware. Private use only. 2 nodes are built identically: AMD 3200G (2C/4T) APU, 32GB DDR4 RAM, M2 NVMe for OS, 2x4TB HDD for storage (connected to SATA onboard), GbE NIC 1...

Hello, I have several Proxmox clusters of 3 dedicated servers each. Previously, I used GlusterFS to replicate the VM disks between the three servers in the cluster so that if one server went down, the VMs would restart on the two remaining servers without data loss. This method worked very...

I've got a working 6 nodes stretched Proxmox VE9.1 + Ceph cluster, with hosts split over two datacenters (3 in each). In a third datacenter I've got a Proxmox VE 9.1 virtual machine (on vsphere) which acts as proxmox + ceph tie-breaker. Each host has 4x25 Gbits/s interfaces in LACP bond defined...

Hello everyone, I have spent a lot of time to figure out what is provoking that IO wait. on this cluster VM who do a high amount of IO have a lot of IO wait ( like 30k I/O read, 50% IO wait) Summary of my setup: PVE proxmox-ve: 9.0.0 (running kernel: 6.17.2-2-pve) pve-manager: 9.0.18...

Since last minor upgrade end of january, we have an crashing osd every few days. OSDs recovering from itself. Journalctl looks very similar every time. It started a few days after the last minor update. My assumption is, it`s maybe related. Do you have any ideas or tips which information/logs...

I'm playing with a cluster composed by 3 machines with 3 hdd osd per machine (9 osd in total). It is a test environment to learn doing stuff, but I don't want to destroy it with this "expansion". I have a VM with an application that is not tolerating HDD slow performance, and since I have 3 ssd...

I've noticed VMs that show much higher usage via rbd du than in the VM, for example: NAME PROVISIONED USED vm-119-disk-0 500 GiB 413 GiB vm-122-disk-0 140 GiB 131 GiB Inside the VM, df shows 95G and 63G used space, respectively. Both of these are Debian 12 which has...

Hi everyone, I've run into a serious issue while managing a Proxmox VE Ceph environment. A user created a lot of VMs and ended up filling the entire Ceph cluster. The problem is, when I look at the RBD storage in the WebUI, I can only see the "Provisioned Size" of each disk. I can't tell which...

Hi there, the Title may be a bit deceptive as I know there are good solutions that work for many but for me/my workplace we face a bit of a dilemma. I know I'm opening this can of worms again and this is also partly me venting a bit of my frustration and I'm sorry about that. We wanna use...

I am on the latest proxmox 9.1.5 with ceph 19.2.3 squid. These warnings bothering me a lot and I am scared to touch anything related to ceph. systemd‑sysv‑generator: SysV service '/etc/init.d/ceph' lacks a native systemd unit file… Please update package to include a native systemd unit file...

Environment Proxmox VE cluster with 2 nodes (node94: 10.129.56.94, node107: 10.129.56.107) Ceph cluster running on the Proxmox nodes (public_network: 10.129.56.0/24) Proxmox SDN EVPN zone (madp) for VM networking VMs are on the EVPN overlay network: 172.16.0.0/16 Goal: Configure Rook External...