ceph

  1. powersupport

    PVE upgrade 6 to 7 issue

    HI, Recently we performed an upgrade on our proxmox cluster and we faced an outage. we have 4 node clusters, after completed the upgrade in 3 rd node, we tried to reboot node 3, but what happened is the whole cluster, which means all nodes went rebooted. We suffered a lot due to the downtime...
  2. [SOLVED] proxmox ceph ( Nautilus to Octopus) upgrade issue

    As part of upgrading to proxmox 7 i done the folling steps: upgraded all nodes to latest 6.4-13 rebooting everything upgrading cepth to Octopus , based on (https://pve.proxmox.com/wiki/Ceph_Nautilus_to_Octopus) after running ( ceph osd pool set POOLNAME pg_autoscale_mode on) on a pool it...
  3. OSD replacement and adding minimizing rebalances

    Hello! I have a 10 nodes hyperconverged cluster. Each node has 4 OSDs (total 40 OSDs). I have 2 different questions: QUESTION 1 - OSD REPLACEMENT (with identical SSD) Since I need to replace an SSD (one OSD has crashed 3 times in latest months so I prefer to replace it with a brand new one)...
  4. Ceph 16.2.6 - CEPHFS failed after upgrade from 16.2.5

    TL;DR - Upgrade from 16.2.5 to 16.2.6 - CEPHFS fails to start after upgrade, all MDS in "standby" - requires ceph fs compat <fs name> add_incompat 7 "mds uses inline data" to work again. Longer version : pve-manager/7.0-11/63d82f4e (running kernel: 5.11.22-5-pve) apt dist-upgraded, CEPH...
  5. [SOLVED] Ghost monitors, managers and meta data servers

    Hi. I am running Proxmox 7.0-11 with CEPH version 16.2.6. I have some trouble with the GUI Datacenter -> Ceph -> Services. I see a few services marked as unknown for some reason that is related to CEPH.. See screenshot below. I've tried systemctl restart pveproxy on all nodes and clearing my...
  6. CephFS user/permission conflict between different nodes (best practices?)

    Hi, we are using CephFS on a 3-node proxmox cluster. We have mounted the CephFS to /home on several different Debian clients. All Debian clients (server) see the files of the other Debian clients in the cephfs mount (/home). It happens that client XY has services on Debian client 1 and Debian...
  7. Ceph Update Nautilus -> Octopus on pve6.1 in preparation to pve7 upgrade

    Hi, we would update our pve-cluster to pve7 and like to reduce the reboots to one for each node. Ceph is allready on 14.2.22. The howto say: "We assume that all nodes are on the latest Proxmox VE 6.3 (or higher)", which isn't on most nodes in our cluster. Is it still posible to upgrade ceph to...
  8. Strange Ceph behavior

    Hi everyone! I'm having this strange issue with Ceph on my Proxmox cluster. The *ceph* nodes have OSD, mon, mds and mgr on each of them. The *pve* nodes have only OSD on them. Recently every Saturday exactly at 06:00 (±couple minutes) ceph goes to Warning state. Logs show that one of the tree...
  9. Proxmox ceph pacific cluster becomes unstable after rebooting a node

    Hi everyone, I have a simple 3 node cluster that has always worked for many years and successfully passed the updates starting from proxmox 4. After updating to version 7 of proxmox and pacific ceph, the system is affected by this issue: every time I reboot a node for any reason (ie updating to...
  10. Node reboot when a disk is removed

    Hi everyone, I have setup a 3 nodes Proxmox 7.0-11 cluster, each node has 3 HDD drives, the first drive is used for Proxmox and the others are OSDs. I wanted to test HA by removing the last disk of the first node (which doesn't contain Proxmox), but when I remove it the node reboot. Is this...
  11. No disks unused when installing the ceph

    I am trying to install the ceph in 2 nodes of the cluster to make a mirror node, and when the ceph finishes the OSD disk storage is not created, when creating it manually it gives me the following error, what can be happening? I need help please and thank you very much for your attention
  12. Proxmox only with CEPH

    Hello, I decided to use the functionality of proxmox to support only CEPH without any VE for RBD block. I installed the "Proxmox Virtual Environment" option for that experiment. I notice that the system run a lot of things responsible for e.g. VE replication (pvest). What do you recommend to...
  13. How to change ceph cluster and network ip?

    I was setting up ceph on my dev cluster but I accidently set ceph to use nmbr0 instead of vmbr1. Is their a way I can change that? Is their a way I can confirm that I am using the wrong bridge instead of looking at the ips?
  14. Dedicated Ceph Setup for Multiple PVE Clusters

    New to Ceph on PVE, have a couple of questions. Is it possible to setup one large Ceph cluster that is dedicated to just running Ceph, but using PVE to set it up and manage it? If so, can I then present this ceph cluster to multiple clusters of PVE? End goal here is one large ceph cluster...
  15. How to recover from loosing all but 1 ceph monitor

    Hi all, Today out of the blue my ceph cluster had all clients disconnected. Ceph dashboard still shows healthy (lies), but proxmox shows both my vm storage (based on rdb) and cephfs as in an "unknown state". When I started to dive in further, I found that on every node, ceph health hangs. osd's...
  16. DynFi User

    [SOLVED] Relaunching Ceph installation wizard in GUI

    I have noticed that the CEPH public / private network weren't properly created during the execution of the Ceph installation wizard. Unfortunately reloading the page killed the process and I can't seem to find a way to relaunch the Ceph installation wizard ? Any idea how I could do that, I'd...
  17. Setup Ceph object storage with S3

    Hello, I'm looking for help on below points 1) Installing and configuring the RADOS Gateway 2) How to create a RADOS Gateway user for the S3 access 3) Configuring DNS & S3 client to access ceph object storage. Update: I followed steps given in below post but not able to access gateway on port...
  18. Ceph Error

    Good morning everyone, I have a cluster of 3 Proxmox servers under version 6.4-13. Last Friday I updated Ceph from nautilus to octupus, since it is one of the requirements to upgrade Proxmox to version 7. At first everything worked wonders. But today when I check I find that it is giving me the...
  19. Not maxing out ceph performance?

    Not maxing out ceph performance? I got a three node AMD epic 7702 installed in a Gigabyte R272-Z31 hyper converged proxmox cluster running. With a ceph storage cluster. The setup is as described in https://pve.proxmox.com/wiki/Deploy_Hyper-Converged_Ceph_Cluster. There is a separate ceph...
  20. VM Disk cache config for Ceph

    Hi We are in the process of migrating services from servers running a local ZFS-mirror on 4TB nVME SSD drives. We are migrating to new servers with the same specifications and hardware as what is currently running, but instead of spreading services over two nodes (a 3rd node runs some internal...

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!