[SOLVED] Must migrate vm's from specific host to delete them

[celadon-one]

I hope the title give some clue to what is going on...
I could swear I saw a thread on this issue before but could not find it.

I have a 3 node cluster with Ceph. All 3 see a quorum. If I try to delete a VM from a specific virtual machine host, I get the error attached.


If I migrate the VM off of that node (either of the other 2) then I can delete the VM without issue.

Side note: I've noticed that when logging into the web UI, 2 of the three show the green checkmark instantly, but the problem node takes a few seconds. Including this in case it provides a clue about what is going on.

 

[ph0x]

Are the three hosts running the same Ceph version?

 

[celadon-one]

They are.

 

[ph0x]

And also the same pve version?

 

[celadon-one]

No...
I'll resolve that now and see if the issue persists.

 

[ph0x]

Good luck.
If I remember correctly, this feature came with 6.4 and led to a second check box in the "remove" panel. So, you should be fine with updating the offending host.

 

[celadon-one]

I can delete VM's on that node now if I need to, but all OSD's on the problem host (node2) were down/in upon reboot. Right now they are going up and down, triggering constant rebalancing.

The other two nodes (node1, node3) are flooding the log with slow ops and active+undersized+degraded messages. The backend should be 10Gb, but I'm not convinced it's configured correctly. The IOPS I've seen on this network make sense for a 10Gb network, but the throughput is terribly slow.

Even without the SSD cache, 3 rotating disks I should be able to break past 100MB/s and that never happens. Unfortunately both Ceph's and Proxmox's documentation spends too much time in the land of theory and doesn't break much down into application, and the Proxmox installer doesn't give you an opportunity to define more than one network at install; reconfiguring can horribly break a cluster if not done carefully with understanding.

 

[ph0x]

It's just the vmbr0 at install time, that's true. It takes a bit of experience to roll out several different networks. But at least with Ceph you get to choose a public and a cluster network.

 

[celadon-one]

Is the host key connected to any of this? I had the hosts regenerate a ed25519 host key and then went through the tedium of setting up the authorized and known hosts with each new key for each host.

 

[ph0x]

A lot of the GUI stuff is done via ssh. But if there are no problems on the command line, i.e. authorized_keys and known_hosts are in order, I would rather rule that out for Ceph.

 

[celadon-one]

I'm going to call this solved since the original issue has been resolved.

 

[celadon-one]

Real quick I wanted to update the Ceph trouble and show this has a happy ending.

ceph -s shows it was trying to rebalance, but no progress was being made.

  cluster:
    id:     3f468525-b2dc-4706-9f31-50a4166bde5a
    health: HEALTH_WARN
            3 osds down
            1 host (3 osds) down
            Degraded data redundancy: 121351/364053 objects degraded (33.333%), 188 pgs degraded, 257 pgs undersized
            18 slow ops, oldest one blocked for 372 sec, mon.hv1 has slow ops
 
  services:
    mon: 3 daemons, quorum hv1,hv3,hv2 (age 93m)
    mgr: hv3(active, since 93m), standbys: hv1, hv2
    mds: cephfs-ISO:1 {0=hv3=up:active} 2 up:standby
    osd: 9 osds: 6 up (since 2m), 9 in (since 83m)
 
  data:
    pools:   6 pools, 257 pgs
    objects: 121.35k objects, 463 GiB
    usage:   130 GiB used, 33 TiB / 33 TiB avail
    pgs:     121351/364053 objects degraded (33.333%)
             188 active+undersized+degraded
             69  active+undersized
 
  progress:
    Rebalancing after osd.4 marked in (83m)
      [............................]

"ceph health detail" showed a bunch of pg's stuck peering. In my experience, this is usually because a monitor is down and communication between the remaining monitors is disrupted, or OSD's are in an inconsistent state. The former wasn't the case here so time to check the OSD's.

One of the OSD's that was down was osd.4 so I checked it's logs (tail -f /var/log/ceph/ceph-osd.4.log) and saw this...
... 2021-06-03T16:12:42.200-0500 7f801d5e4700 1 osd.4 1467 is_healthy false -- only 0/6 up peers (less than 33%) 2021-06-03T16:12:42.200-0500 7f801d5e4700 1 osd.4 1467 not healthy; waiting to boot 2021-06-03T16:12:43.152-0500 7f801d5e4700 1 osd.4 1467 is_healthy false -- only 0/6 up peers (less than 33%) 2021-06-03T16:12:43.152-0500 7f801d5e4700 1 osd.4 1467 not healthy; waiting to boot 2021-06-03T16:12:44.164-0500 7f801d5e4700 1 osd.4 1467 is_healthy false -- only 0/6 up peers (less than 33%) 2021-06-03T16:12:44.164-0500 7f801d5e4700 1 osd.4 1467 not healthy; waiting to boot ...

"waiting to boot". I see this kind of thing a lot when Ceph upgrades...an extra reboot is required after system has come back up from the upgrade.
So I did a "ceph osd set noout" to keep OSD's in while it rebooted in case the reboot took a while or got hung up for some reason, then rebooted. When the system came back up, "ceph osd unset noout".


And now ceph -s shows we are progressing nicely.

  cluster:
    id:     3f468525-b2dc-4706-9f31-50a4166bde5a
    health: HEALTH_WARN
            Degraded data redundancy: 100115/364053 objects degraded (27.500%), 104 pgs degraded, 104 pgs undersized
            18 slow ops, oldest one blocked for 1377 sec, mon.hv1 has slow ops
 
  services:
    mon: 3 daemons, quorum hv1,hv3,hv2 (age 9m)
    mgr: hv3(active, since 2h), standbys: hv1, hv2
    mds: cephfs-ISO:1 {0=hv3=up:active} 2 up:standby
    osd: 9 osds: 9 up (since 9m), 9 in (since 114m); 104 remapped pgs
 
  data:
    pools:   6 pools, 257 pgs
    objects: 121.35k objects, 463 GiB
    usage:   128 GiB used, 33 TiB / 33 TiB avail
    pgs:     100115/364053 objects degraded (27.500%)
             153 active+clean
             103 active+recovery_wait+undersized+degraded+remapped
             1   active+recovering+undersized+degraded+remapped
 
  io:
    recovery: 0 B/s, 27 objects/s
 
  progress:
    Rebalancing after osd.4 marked in (114m)
      [=======================.....] (remaining: 22m)