ceph

What are the best storage configurations in Proxmox to access a SAN with multiple nodes while ensuring high performance, redundancy, and simultaneous access? How do options like ZFS over iSCSI, ZFS over NVMe/TCP, Ceph compare in terms of latency, scalability, and complexity? The goal is to...

I’ve read many threads but haven’t quite figured it out… I know one should not use RAID with Ceph. Do drives connected as JBOD/single drives on a RAID controller (3108) still fall into that warning category? We’re hoping to reuse some older hardware.

Hi all! I have a pve cluster and a cef storage of 3 nodes. On one node, both OSDs are in the down and out status. I need to replace this node with another node. I added a new node to the proxmox cluster, installed cef. Tell me what are my next steps? Do I need to delete the OSD on the node I...

I tried to move my ceph network to another subnet and now all osd's are not picking up the new network and staying down. I fear i may have hosed ceph, but as a learning experience, would like to see if i can recover them. this is what they look like in the cluster: this is the contents of...

We've got a quite serious problem over here. We have added a new node to our cluster. However, when installing ceph on the new node, there was a problem due to the fact that the VLANs for ceph and osd could not communicate correctly (network problem). As a result, we tried to uninstall ceph...

Good Day, I'm seeing a somewhat weird issue specifically with 8.3.3. I am trying to install ceph to convert a cluster running off of local storage to a hyper-converged cluster running on ceph. The issue I'm seeing is timeouts when performing the ceph install wizard. On the first page I'm...

Hi, I'm going to be upgrading my ProxMox hosts from 8.2.4 up to 8.3.1 and my Ceph from Reef 18.2.2 to Squid. I'm following the directions here (https://pve.proxmox.com/wiki/Ceph_Reef_to_Squid) for ceph and I'm wondering if I can go from Reef 18.2.2 up to Squid directory? In the Introduction...

Hello everyone, I am kinda new into all this so be patient with me. I am having a particular container with the name CT213 in my organization and I have the task to bring it back up. From the HA cluster pve trick we have , its the only container that doesnt want to start up. When i try to start...

This 3-node cluster also had a 4th node (r730), which didn't have any OSDs assigned. This is what I have to recover:- /etc/ceph/ceph.conf and /etc/ceph/ceph.client.admin.keyring available from a previous node that was part of cluster. /var/lib/pve-cluster/config.db file from r730 node Now...

I'm stupid :/, and I really need your help. I was following the thread to clear a dead monitor here https://forum.proxmox.com/threads/ceph-cant-remove-monitor-with-unknown-status.63613/post-452396 And as instructed, I deleted the folder named "ceph-nuc10" where nuc10 is my node name under...

Hello, I recently configured Ceph supervision with Zabbix. to make the superversion working, i applied the following permissions on each mgr: ceph auth caps $mgr mds 'allow *' mon 'allow * 'osd 'allow *' However, I'm concerned about the excessive permissions. Should I be worried about the...

Hi, I plan to build a new proxmox ceph cluster (3 nodes). LACP 2x25 Gbps for ceph LACP 2x10 Gbps for VMs My question is related to the switches, I want mlag switches indeed. Mikrotik was my first choice due to functionalities and low budget as well. I have 2 CRS310 and have settled a very...

Hello all, I am trying to set up a 3-node cluster with failover, nodes 1 and 2 have my OSDs and nodes 1, 2, and 3 are monitors, nodes 1 and 2 are my production servers and node 3 is just for negotiation. If I gracefully power off node 1 the VM will move to node 2 (using HA fail over) and start...

Good day all, Short description of the issue: inability to start containers consistently. Context in which the problem arose: 7-node cluster recently upgraded from version 7 to 8 (I believe the issue did not occur before the upgrade); no PVE subscription. Scope: All the containers (three in...

Hi , i have a 3-node cluster , with ceph all current nodes have 4x 3,4TB Nve in the ceph OSD i want to add a node the host is the same hardware but with 3x 1.9TB Nvme my questions are; What is the best way of expanding the OSD and can it be done with different storage, or is it better to...

Hello, I want to configure ceph to use msgr2 and not to use msgr1, to encrypt ceph traffic. So I firstly set ms_bind_msgr1 = false and ms_bind_msgr2 = true into /etc/ceph/ceph.conf under the section [global], and changed IP addresses into v2-only addresses. The full configuration is: [global]...

I am looking to let Proxmox manage Ceph so that I can mount CephFS in a bunch of VM's. However, I don't know what the best practice is for authorization. I would like to follow least privilege so that a VM only has r/w access to CephFS. How would I generate a key that only has access to one...

Hello, After adding the new node to the cluster, I installed ceph on it. Ceph installed itself with version 18.2.4, on the others it is version 18.2.2. I would like to do a ceph update on the other nodes. What is the safest way to do this? If there is data on osd's can it be done live or should...

I am building a cluster with 2 PVE hosts and a quorum host. Each PVE host is running local storage with hardware raid. I would like have it set up in such a way that in the event of a node failure, all VMs running on one host can be restarted on another. Ideally with clustering, but replication...

Hi all, I am looking for a fast storage for HA cluster. Currently I use Ceph, but it is terribly slow (I have 3 nodes with enterprise class nvme drives. I read many articles about it and its performace. By my opinion the bottleneck is in network synchronization. So I am looking for another...