ceph

  1. A

    Hardware Feedback - Proxmox Ceph Cluster (3-4 nodes)

    After going down many rabbit holes, I have finally come to the conclusion that the best solution (for my office) is a Proxmox cluster with 4 nodes. Depending on my final build, I might be able to only have 3. For now, I will use both proxmox backup and veeam to backup my VMs to a TrueNAS box...
  2. M

    PVE 8.2.2 and Ceph: all OSDs report slow ops

    Hello! I'm testing Ceph setup on single node, 12 HDDs are connected via SAS9211-4i contoller: root@pve-1:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 32.74786 root default -3 32.74786 host pve-1 0 hdd 2.72899 osd.0...
  3. D

    [SOLVED] Ceph Public/Private And What Goes Over the network

    Good Day All. Im new to ceph. i have set up a 6 node cluster. Its a mixture of SSD AND sas drives. With the sas drives i use a ssd partition for the db. Now what im experiencing is that my VMs are slow. Boot is slow Opening programs are slow etc etc. the 10.0.45.0/24 network is 10Gig the...
  4. M

    CEPH on three separate nodes

    Hi, A client want to have 3 nodes in 3 datacenter with iSCSI local storage connected, it's possible to configure on the top one shared storage for all hosts?? CEPH??
  5. L

    Had to remove 1 Node from Proxmox/Ceph

    Once I had 4 nodes running with Ceph and one could fail. Now that one has failed and is removed. Is it possible to establish redundancy with the remaining 3 nodes? From ceph -s data: volumes: 2/2 healthy pools: 7 pools, 193 pgs objects: 139.48k objects, 538 GiB usage: 1.6...
  6. F

    [SOLVED] CEPH Reef osd still shutdown

    Hi everyone, I'm working with a 3 node cluster running with ceph 17 and I'm about to upgrade. I also add a new node to the cluster and install ceph 18.2 . The first OSD i'm creating seems OK yet after a few moments it's shut down. In the logs here is what I can find : May 18 15:34:44 node4...
  7. M

    Number of disks on Ceph storage?

    Currently, I have 1 disk in each server(total 3) used for Ceph shared storage. I noticed that if 2 out of 3 disks are not working then complete ceph will also stop working. I am not sure if the same (n/2 +1) formula works here as well. Why It is concerning is because we are using CephFS as...
  8. F

    CEPH vs RAID

    Hello, I would like to ask you a few questions about CEPH. Currently, I'm working on critical infrastructure for my client, and I'm planning to use servers with hardware RAID. My first question is whether I should opt for CEPH or classic RAID on these servers? The second question is whether it's...
  9. D

    Understanding Ceph Fundamentals

    Good Day All. I have just set up a 4 node cluster with Ceph. So a few Questions to help me understand what is happening 1. I know data is stored over multiple servers. However, when I run a VM is the "Hard Drive" on the local machine copy for IO. Or does the VM have to read and write over the...
  10. D

    Reducing Size of Ceph fs

    Greetings, I currently have a 1.12 TB CEPH fs which has 60GB of data on it that I don’t want to lose. However 1.11TB is much too large. I need to reduce it to 512GB or even 256GB. I have set the target size to 512GB however the available size of the fs refuses to move from 1.11TB. Why...
  11. G

    [SOLVED] Three-nodes Ceph mesh network with bond and VLANs

    Hi everyone, I'm designing a new 3-node cluster. It's not the first cluster I make but it's the first with a full-mesh Ceph network (no dedicated switch) . Each node has a double 10G NIC and they're already physically connected each other. So I've been reading the docs and I think the best...
  12. M

    Having trouble clearing some ceph warnings.. Reduced data availability & Slow ops

    Hey all, I'm having trouble clearing some warnings from my ceph cluster. 1.) HEALTH_WARN: Reduced data availability: 1 pg inactive pg 1.0 is stuck inactive for 5m, current state unknown, last acting [] 2.) HEALTH_WARN: 2 slow ops, oldest one blocked for 299 sec, daemons [osd.0,osd.1] have...
  13. B

    Ceph failure - missing /etc/pve/ceph directory

    I have an 8.1.4 ProxMox Ceph cluster. I'm trying to add a new host that is running the latest 8.2.2 to the cluster but when I try and add the OSD I get the failure "pveceph configuration not initialized - missing '/etc/pve/ceph'". There's a commit that went in on Feb 16, 2024 that looks like...
  14. J

    Promox + Ceph: VMs won't start + dreadful performance

    I have been searching far and wide for this. But I can't seem to find a solution. In one of our testing clusters we're experimenting with Ceph. But so far the journey has been bumpy at best. The actual trigger for this post is the fact that one of our VMs locked up and I can't seem to get it to...
  15. T

    8.2.2 upgrade breaks 1st node. Manual network start needed

    Hey all, I have a 3 node Proxmox/ceph cluster and I decided to update. After updating the 1st node, networking no longer works on it. Others have said their 8.2.x upgrade changed the device name but that's not what I'm seeing. Upon booting up and and logging in, I can run manually run...
  16. H

    pool ceph very slow write

    tenho um sistema com 3 nós e 1 OSD em cada nó. Velocidade muito lenta, principalmente escrita CrystalDiskMask ~600MBs leitura / ~30MBs escrita. Coloquei DB Disk com 2% e WAL Disk com 10% No Pool_HDD em questão, tem apenas uma VM de teste. O problema também acontece no pool_ssd. Porém tem VMs...
  17. F

    [SOLVED] VMWare -> Proxmox | Storage Ceph - Fragen?

    Hallo in die Runde! Auch wir überlegen, einen Übergang von VMWare zu Proxmox zu realisieren. Stand bisher: VMWare Cluster 5 Nodes, 2 Storages mit SAS Anbindung Nun zu unseren Überlegungen: Unter Beibehaltung der bestehenden 5 Nodes wollen wir die bestehenden Storages gegen ein Ceph-Cluster...
  18. L

    Ceph install Configuration not set. How to fix or remove & re-install Ceph

    I have an older Proxmox node that is on 7.4.17. It is running on an older 2nd Gen i7-2960XM & is having stability issues so I don't want to upgrade it until I have everything duplicated elsewhere. I got 3 new(er) nodes running 7th gen i5 & i7s. I can't add & setup the 3rd new node until I remove...
  19. J

    Best way to use Cephfs in a VM

    I'm sure this has been covered before, but i wasn't able to find quite what i was looking for in other posts. i have created my proxmox 8 cluster and i have created a cephfs. cephfs needs to go to the VM's hosted within proxmox and im trying to avoid having internet traffic go outside the host...
  20. I

    [SOLVED] Could you please suggest optimal Proxmox HA cluster with Ceph NIC configuration?

    Hi there. I have a PX cluster built on 4 nodes, they have pretty much the same hardware spec which is 2xCPU sockets, 256GB of RAM and 3 network cards: 1x 2-ports 40Gb 1x 2-ports 10Gb 1x 4-ports 1Gb 1x 2,4TB SSD hardware RAID5 1x 2TB NVMe m.2 Because some configs are difficult to change and I...