ceph cluster

  1. CEPH Performance issues after upgrade from 15.2.8 to 15.2.10

    Hello, A few information about the System: Its a Hyperconverged Cluster of 5 Supermicro AS -1114S-WN10RT: 4 of the Servers have: CPU: 128 x AMD EPYC 7702P 64-Core Processor (1 Socket) RAM: 512 GB 1 of the Servers has: 64 x AMD EPYC 7502P 32-Core Processor (1 Socket) RAM: 256 GB Network: All...
  2. Advice needed for new small cluster install

    We recently purchased three small servers and need advice on the network setup. What would be recommended with regard to distributing the network services amongst the adapter/ports? Both adapters are npar capable. 3x Dell 6515 with: 1x AMD 7542 128 GB memory 2x 1Gb nic 2x 25GbE sfp28 (Broadcom...
  3. Ceph Cluster performance

    Hi All, I've a ceph Cluster with 3 nodes HPE each with 10xSAS 1TB and 2xnvme 1TB below the config. The replica and ceph network is 10Gb but the performance are very low... in VM I got (in sequential mode) Read: 230MBps Write: 65MBps What I can do/check to tune my storage environment? # begin...
  4. [SOLVED] PGs not being deep scrubbed in time; After replacing disks

    This week we have been balancing storage across our 5 node cluster; Everything is going relatively smoothly but am getting a warning in CEPH: "pgs not being deep-scrubbed in time" This only began happening AFTER we made changes to the disks on one of our nodes; CEPH is still healing properly...
  5. Proxmox cluster - disk layout for ceph

    Hi, I plan to build my first ceph cluster and have some newbie questions. In the beginning I will start with 5 nodes, and plan to reach 50 nodes. Those nodes quite old (CPU E3,16GB RAM, 2x1Gbps network), so I think to gain the performance in adding more nodes but not upgrading RAM or CPU. I...
  6. Proxmox on Ceph: After PG_NUM increase high read io on NVMEs - VMs unusable

    Hi, the autoscaler increased the number of PGs on our Ceph storage (Hardware like this but 5 nodes). As soon as the backfill starts the VMs become unusable and we startet killing OSD processes that cause high read io load. So as in this picture we would kill the ceph-osd process working on...
  7. Ceph becomes unresponsive when one node-mon goes down

    I have 2 ceph nodes. Each has a mon and mgr installed. Whenever I shutdown any one mon-instance on any of my nodes ceph becomes completely unresponsive until I start that mon again. Is this normal or can I fix this?
  8. Ceph Stuck with active+clean+remapped pgs

    I don't know how to fix this. I'm just starting out with ceph. It just keeps on showing active+clean+remapped. It doesn't fix it over time. How do I fix this? I just use the default replication rule for my pools.
  9. [SOLVED] Ceph & iSCSI HA - How to configure the network?

    I am looking to some guidance to finalize the setup of a 3-nodes Proxmox cluster with Ceph and shared ISCSI storage. While it's working, I am not really happy with the ceph cluster resilienc and I am looking for some guidance. Each nodes have 2x10GbE ports and 2x480GB SSD dedicated for ceph...
  10. bug in ui? ceph public network ip

    When I read the doc I can see the public network ip set to 10.10.10.0/24: I would expect to have 10.10..10.0/24. Looks like an error
  11. [SOLVED] cannot use ceph after joining cluster if already installed

    Hi Team! I reconfigured a server from scratch. Then installed the ceph package but did cancel the configuration after the install, so it could use the setup of the already configured setup. Then made it join the cluster. now I cannot configure it with the GUI, and have the got timeout (500)'...
  12. [SOLVED] Trim/discard With CEPH Rbd

    Hello I realized that I'm deleting data from the vms but this space is not being released in Ceph. I found in the documentation that I should do a Fstrim in the RDB but I can't find its assembly text such as: fstrim /mnt/ myrbd. Any idea? Thank you
  13. Proxmox Ceph Converged (HCI) or external ceph

    Hello, On this moment we have: 6 x Proxmox Nodes 2 x 10 cores (2 nodes have 2 x 14 cores) 512 GB RAM 4 x 10 GB (2 x 10 GB LACP for network en corosync and 2 x 10 GB LACP for Storage) 3 x Ceph Monitor Dual Core 4 GB RAM 2 x 10 GB LACP 4 x Ceph OSD 2 x 6 Core 2,6 Ghz 96 GB RAM 4 x 10 GB (2 x...
  14. [SOLVED] Ceph : delta in reported total space

    Hello, We now have a well working Ceph cluster with 5 nodes (PVE5.4, Ceph luminous). OSDs are distributed like this : * host1 : osd.12 : 1750 GB osd.13 : 1750 GB * host2 : osd.0 : 894 GB osd.1 : 1750 GB * host3 : osd.5 : 1750 GB * host4 : osd.4 : 1750 GB osd.5...
  15. reinstall ceph after pveceph purge

    Hi, After I installed proxmox I decided to tinker around with ceph. Some things didn't work out and I removed ceph from the proxmox node. After stopping all the ceph services I removed it with 'pveceph purge'. That worked! Now when I tried to reconfigure ceph I keep getting this error "Could...
  16. CEPH inaccurate OSD count displayed

    I have had this issue for a while now, and after upgrading to Proxmox 6 and the new Ceph it is still there. The problem is that the Ceph Display page shows that I have 17 OSD's when I only have 16. It shows the extra one as being down and out. (Side note, I do in fact have one OSD that is down...
  17. Proxmox Ceph OSD Partition Created With Only 10GB

    How do you define the Ceph OSD Disk Partition Size? It always creates with only 10 GB usable space. Disk size = 3.9 TB Partition size = 3.7 TB Using *ceph-disk prepare* and *ceph-disk activate* (See below) OSD created but only with 10 GB, not 3.7 TB Commands Used root@proxmox:~#...
  18. CEPH cluster. Wanted a comment from other PVE ceph users on ram usage per node.

    Currently as all nodes are under load and memory consumption is around 90-95% on each of them. CEPH cluster details: * 5 nodes in total, all 5 used for OSD's 3 of them also used as monitors * All 5 nodes currently have 64G ram * OSD's 12 disks in total per node - 6x6TB hdd and 6x500G ssd. *...
  19. ceph osd apply latency is high

    Hello Sirs. Has there anyone encountered the same issue as mine? I found one of OSDs in our production proxmox CEPH cluster environment which had high apply latency(around 500ms.) It caused our CEPH cluster performance to degrade. After I restarted the OSD, the cluster performance is back to...
  20. PVE WEB GUI commnucation failure(0) when list CEPH Storage

    Hi there... I have 2 PVE nodes and 5 servers as CEPH Storage, also building under PVE Servers. So I have two cluster: 1 cluster with 2 PVE nodes, named PROXMOX01 and PROXMOX02. * PROXMOX01 runs proxmox-ve: 5.3-1 (running kernel: 4.15.18-11-pve) pve-manager: 5.3-11 (running version...

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!