Issues with New Install, loosing access to VMs

N

neil2048

New Member
May 3, 2026
5
0
1
Hi All

Brand new to Proxmox and mid first install, have used VMware for many years with out issues, so having a few teething issues with Proxmox, assume born from stupidity and trying to implement beyond my current level of understanding... So here goes

We have a 6 node Proxmox cluster, 32 Core/256GB ram each. Each host has 5 x 3.2tb NVMe's. Now here was my first attempt to do something funky in regards to resilience within Ceph. So using crush maps (Device Classes) with the "Host" bucket option, i have split these drives in to 3 pools. Pool 1 has drives 1,2,3 from the first 3 hosts, Pool 2 has drive 1,2,3 from the last 3 hosts, and Pool 3 has the remaining 2 drives from all 6 hosts. All the pools use a replication factor of 3 and a minimum of 2. From what i can tell this should be valid. The reasoning for this was around VM's that had pairs, AD, SQL Always on Nodes etc would live on Pool 1 and 2 respectively for the partners, and HA is configured to run the VM's on the same hosts as the pools live and then Pool 3 was for VM's that are only single copies.

When we shut down any single host, the VMs migrate to hosts specified in the HA config, we then see slow ops in Ceph, as we have OSD's missing from the powered off host and we loose access to some VM's (still testing to see how far spread this issue is) that live on the pools controlled by that host (console and network access), even though we still have the minimum 2 replication requirement for Ceph to function. It make no difference if we set "No Out" or not. Access to the VM's returns instantly once the host comes back online. The VM's do not reboot during this time as we don't get the unclean shutdown messages, they appear to cease functioning and "pause", for the duration.

Please can any one shed any light of why this is the case? or suggest any troubleshooting steps we can take?

As a note we are only using <3% of each pool currently, so space is not an issue, and we are running 4 x 25GB dedicated nics in LACP for the backend Ceph networking, so storage IO should also be good here.

1790082640770.png

We have allocated 3 Mons (Nodes 2/4/6) and 3 Mgrs (Nodes 1/3/5) to the cluster also.

Thanking you in advance for the assistance
 
My first thought is you’re overcomplicating Ceph. Why not one pool with lots of drives for resilience and more distributed I/O?

What does the top of the Ceph status page show when one host is off? Should not be worse than yellow.
 
  • Like
Reactions: UdoB
Hi

You are correct Ceph Health goes Yellow, its probably worth stating that i have also finagled the OSD ID's to correspond to the Host and Drive bay number, but when all hosts are online, Ceph is Green.

So if we shut down node 2 (which has drives from Pool 1 and Pool 3) we lose access to all VM's from Pool1 and Pool 3.
1790087496400.png

Regards
 
So you are absolutely correct that is the issue, stopping the OSD's first keeps the VM's online, however once all back and working, taking this one stage further and simulating a host failure (hard power pull) exhibits the same behavior where access is lost, until the OSD's are "stopped" from another host, or maybe reaches the time out value (Will test this one in a bit)

I saw today that Ceph 19 is EOL, however we downgraded to Ceph 19, after walking straight into PVE 9.2 + Ceph Tentacle 20 RBD Auth Issue. That also only has a workaround not a fix as yet...
 
I had not seen/encountered that yet. The last comment says "AFAICT this should be fixed in c1290c4d1334350a3ba84c8745eb230f077bf9a3 / libpve-storage-perl >= 9.1.10" ...? Which is what we have installed.

In the other thread about the shutdown bug I mentioned there has been at least one case I can think of where I have seen VMs pause when I didn't expect, so might be that same issue in hindsight. Normally when installing updates and restarting nodes we haven't seen that hang. But we set various Ceph options like noout, norecover, and it's only a couple minutes to restart. A hang does rather defeat the purpose of a high availability cluster/storage though. What should happen is nothing, other than VMs actively running on the powered-off node boot up elsewhere.
 
So on a test tonight, before leaving the office we shut down a host unclean from a Ceph perspective to see how long it would take to fail the OSD's and bring the VM's back online, almost 4 hours later we still didnt have access back. To your note above, this does kinda defeat the purpose of HA, as a node failure will not be so kind to stop the OSD's, prior to failing...

From what i saw on the Ceph 20 issue, was a fix was being provided in the new release coming soon.. not that it was released already.

Does Ceph 20 still suffer from this IO stall? or has that been fixed?

As it stands currently i cant put this platform into production, knowing that a node failure will knock out over 1/2 the platform VM's. Technically the only saving grace is that the third pool actually remains online, rather than what i would assume would be a complete system failure if all drives were part of one pool....

Thanks for your assistance..
 
You must log in or register to reply here.
Top Bottom
Back