Proxmox Ceph Image Cannot Be Found

I

infinityM

Well-Known Member
Dec 7, 2019
179
1
58
31
Hey Guys,

Can someone please assist? I have googled this thing to death now...
When I try and migrate a VM I am getting a message as follows:

Task viewer: VM 107 - Migrate

OutputStatus

Stop
task started by HA resource agent
2020-01-27 13:03:36 starting migration of VM 107 to node 'c2' (129.232.xxx.xxx)
2020-01-27 13:03:37 ERROR: Failed to sync data - rbd error: rbd: listing images failed: (2) No such file or directory
2020-01-27 13:03:37 aborting phase 1 - cleanup resources
2020-01-27 13:03:37 ERROR: migration aborted (duration 00:00:01): Failed to sync data - rbd error: rbd: listing images failed: (2) No such file or directory
TASK ERROR: migration aborted

When I run: rbd ls -l Default
I get the message: rbd: error opening vm-106-disk-1: (2) No such file or directory


The problem is, this image is gone, if I try and run "rbd rm vm-106-disk-1 -p Default", it runs up to 99% and just stays there... How can I get this removed? I'm getting desperate since the live migrations and all arent working anymore now...

Can you guys please advise?
 
Any other error messages in the logs? How does the ceph -s look like?
 
I'm not sure what I should be looking for in the logs?
What I did notice though is there were a couple of drives with IO errors, So we replaced them one by one, and there is one drive left (osd16) which has 3 objects unfound which I can't seem to resolve?
That may be the issue?

ceph -s
cluster:
id: 248fab2c-bd08-43fb-a562-08144c019785
health: HEALTH_ERR
3/715723 objects unfound (0.000%)
Possible data damage: 2 pgs recovery_unfound, 1 pg backfill_unfound
Degraded data redundancy: 4084/1431446 objects degraded (0.285%), 3 pgs degraded, 3 pgs undersized

services:
mon: 3 daemons, quorum pve,c1,c2 (age 3h)
mgr: pve(active, since 14h)
osd: 21 osds: 19 up (since 24m), 18 in (since 6h); 24 remapped pgs

data:
pools: 1 pools, 512 pgs
objects: 715.72k objects, 2.7 TiB
usage: 4.9 TiB used, 9.6 TiB / 15 TiB avail
pgs: 4084/1431446 objects degraded (0.285%)
32995/1431446 objects misplaced (2.305%)
3/715723 objects unfound (0.000%)
488 active+clean
20 active+remapped+backfill_wait
2 active+recovery_unfound+undersized+degraded+remapped
1 active+backfill_unfound+undersized+degraded+remapped
1 active+remapped+backfilling

io:
client: 170 KiB/s rd, 22 MiB/s wr, 70 op/s rd, 67 op/s wr
recovery: 18 MiB/s, 5 objects/s

progress:
Rebalancing after osd.16 marked out
[============================..]

root@pve:~#
 
Alwin said:
These three object do not actively exist anymore in the cluster. They may have been on the replaced OSDs. If those are the last objects for recovery and still aren't found, then you can try to mark them as lost.
https://docs.ceph.com/docs/nautilus/rados/troubleshooting/troubleshooting-pg/#unfound-objects
Click to expand...

I have marked the osd as lost which they were on since nothing was working. Then I am sitting with 3pg's inactive/incomplete...
How can I just rebuild them to fix them? I honestly don't care if the data on them is lost,just want to be able to use the space, Right now backups and all are failing and I am assuming it's because it's hanging looking for those pg's?
 
Alwin said:
As it says in the link, you need to 'mark_unfound_lost', so the cluster doesn't consider those objects anymore.
https://docs.ceph.com/docs/nautilus/rados/troubleshooting/troubleshooting-pg/#unfound-objects
Click to expand...
have done just that and it still reports 3 pg's inactive / incomplete. Please see below output?

root@pve:~# ceph health detail
HEALTH_WARN Reduced data availability: 3 pgs inactive, 3 pgs incomplete; 71 daemons have recently crashed; 29 slow ops, oldest one blocked for 4902 sec, daemons [osd.13,osd.14,osd.3] have slow ops.
PG_AVAILABILITY Reduced data availability: 3 pgs inactive, 3 pgs incomplete
pg 1.ad is incomplete, acting [13,11]
pg 1.c3 is incomplete, acting [14,11]
pg 1.1b6 is incomplete, acting [3,11]
RECENT_CRASH 71 daemons have recently crashed
osd.8 crashed on host c1 at 2020-01-27 08:40:43.484936Z
osd.8 crashed on host c1 at 2020-01-16 00:19:24.821060Z
osd.8 crashed on host c1 at 2020-01-26 16:53:44.474854Z
osd.8 crashed on host c1 at 2020-01-24 15:23:35.030625Z
osd.8 crashed on host c1 at 2020-01-26 08:45:52.903788Z
osd.8 crashed on host c1 at 2020-01-25 11:37:38.712350Z
osd.8 crashed on host c1 at 2020-01-17 00:34:23.826575Z
osd.8 crashed on host c1 at 2020-01-25 06:28:44.768879Z
osd.8 crashed on host c1 at 2020-01-16 22:50:01.288727Z
osd.16 crashed on host c4 at 2020-01-26 00:15:28.921768Z
osd.16 crashed on host c4 at 2020-01-25 03:25:17.819939Z
osd.8 crashed on host c1 at 2020-01-27 11:13:34.417378Z
osd.8 crashed on host c1 at 2020-01-18 23:38:43.665995Z
osd.8 crashed on host c1 at 2020-01-18 00:36:54.073513Z
osd.8 crashed on host c1 at 2020-01-23 03:51:45.634132Z
osd.16 crashed on host c4 at 2020-01-24 14:02:31.669665Z
osd.8 crashed on host c1 at 2020-01-18 23:35:31.914101Z
osd.8 crashed on host c1 at 2020-01-16 22:45:47.556311Z
osd.8 crashed on host c1 at 2020-01-19 01:00:11.066376Z
osd.8 crashed on host c1 at 2020-01-26 10:33:17.449749Z
osd.8 crashed on host c1 at 2020-01-19 01:03:54.141456Z
osd.8 crashed on host c1 at 2020-01-26 08:45:59.204403Z
osd.8 crashed on host c1 at 2020-01-25 06:28:38.792457Z
osd.8 crashed on host c1 at 2020-01-27 11:13:46.041183Z
osd.8 crashed on host c1 at 2020-01-19 03:53:12.883454Z
osd.13 crashed on host c1 at 2020-01-25 12:06:38.165035Z
osd.8 crashed on host c1 at 2020-01-26 16:53:49.990468Z
osd.8 crashed on host c1 at 2020-01-24 14:15:31.771421Z
osd.8 crashed on host c1 at 2020-01-18 09:21:47.383252Z
osd.8 crashed on host c1 at 2020-01-27 08:40:32.366058Z
and 41 more
SLOW_OPS 29 slow ops, oldest one blocked for 4902 sec, daemons [osd.13,osd.14,osd.3] have slow ops.
root@pve:~# ceph pg 1.1b6 mark_unfound_lost delete
pg has no unfound objects
root@pve:~# ceph pg 1.c3 mark_unfound_lost delete
pg has no unfound objects
root@pve:~# ceph pg 1.ad mark_unfound_lost delete
pg has no unfound objects
root@pve:~#
 
infinityM said:
pg 1.ad is incomplete, acting [13,11]
pg 1.c3 is incomplete, acting [14,11]
pg 1.1b6 is incomplete, acting [3,11]
Click to expand...
These are incomplete, they don't have anything lost. A third OSD is needed to recover the a good copy.

infinityM said:
71 daemons have recently crashed;
Click to expand...
This sound like the complete cluster failed at some point. What led up to the daemons crashing?

infinityM said:
29 slow ops, oldest one blocked for 4902 sec, daemons [osd.13,osd.14,osd.3] have slow ops.
Click to expand...
Is there also a network issue? These OSDs couldn't work all requests in time. You should see more in their log.


See for incomplete PG:
incomplete
Ceph detects that a placement group is missing information about writes that may have occurred, or does not have any healthy copies. If you see this state, try to start any failed OSDs that may contain the needed information. In the case of an erasure coded pool temporarily reducing min_size may allow recovery.
Click to expand...
https://docs.ceph.com/docs/nautilus/rados/operations/pg-states/
 
Alwin said:
These are incomplete, they don't have anything lost. A third OSD is needed to recover the a good copy.


This sound like the complete cluster failed at some point. What led up to the daemons crashing?


Is there also a network issue? These OSDs couldn't work all requests in time. You should see more in their log.


See for incomplete PG:

https://docs.ceph.com/docs/nautilus/rados/operations/pg-states/
Click to expand...


Thanks for the quick reply... What do you mean a third osd is needed? I have 17 OSD's in the server... There's ample osd's available

As for the crashes. Yes we had a couple of drives that kept failing and I eventually got them out. the trouble was getting their data secured first obviously come pg's were shared between them and needed to be replicated again, Currently OSD's don't down anymore except when I down them.

AS for the response times. I don't understand why though the network was running 700 megs before now it's dead slow... But nothing's changed... I doubt it's a network issue, I have disabled osd 13,14 and 11 to see if it happens on other drives...

But ok so the main issue is, How can we get the ceph actually usable? How do i add that 3rd osd?
 
infinityM said:
What do you mean a third osd is needed?
Click to expand...
By default the pool size is 3/2. Which gives 3x OSDs connected to a PG. As incomplete means, that the PG doesn't have a healthy copy, the third OSD that was connected with that PG might have it. But that OSD is replaced if I am not mistaken?

infinityM said:
As for the crashes. Yes we had a couple of drives that kept failing and I eventually got them out.
Click to expand...
The crash reports might reveal some more information, to why the services crashed. Change in MTU, link speed or availability can trigger such a cascading effect.

infinityM said:
AS for the response times. I don't understand why though the network was running 700 megs before now it's dead slow... But nothing's changed... I doubt it's a network issue, I have disabled osd 13,14 and 11 to see if it happens on other drives...
Click to expand...
Whatever the reason, check the syslog and ceph logs. They might reveal some more about what happend too.

infinityM said:
But ok so the main issue is, How can we get the ceph actually usable? How do i add that 3rd osd?
Click to expand...
If you don't have the old OSD available, then no healthy copy might exist. But you could try to run a deep scrub on the PGs in question to see if the state might change.
 
Alwin said:
These are incomplete, they don't have anything lost. A third OSD is needed to recover the a good copy.


This sound like the complete cluster failed at some point. What led up to the daemons crashing?


Is there also a network issue? These OSDs couldn't work all requests in time. You should see more in their log.


See for incomplete PG:

https://docs.ceph.com/docs/nautilus/rados/operations/pg-states/
Click to expand...

Thank you the issue appears to maybe be the network, I've set the server's to sync and monitoring it now disablng the drives causing delays...
One question though...

In my initial question I mentioned I can't move servers between nodes because of missing images... How am I to resolve that part?

2020-01-27 13:03:36 starting migration of VM 107 to node 'c2' (129.232.xxx.xxx)
2020-01-27 13:03:37 ERROR: Failed to sync data - rbd error: rbd: listing images failed: (2) No such file or directory
2020-01-27 13:03:37 aborting phase 1 - cleanup resources
2020-01-27 13:03:37 ERROR: migration aborted (duration 00:00:01): Failed to sync data - rbd error: rbd: listing images failed: (2) No such file or directory
TASK ERROR: migration aborted

When I run: rbd ls -l Default
I get the message: rbd: error opening vm-106-disk-1: (2) No such file or directory
 
infinityM said:
In my initial question I mentioned I can't move servers between nodes because of missing images... How am I to resolve that part?
Click to expand...
This image may have data (or metadata) located on the incomplete PGs. If that's the case then the only way may be to restore from backup.

Run, rados -p <pool> list-inconsistent-obj <pgid>, to see which objects are inconsistent. You can verify which image it is by using rbd info <disk-image>.
 
Alwin said:
This image may have data (or metadata) located on the incomplete PGs. If that's the case then the only way may be to restore from backup.

Run, rados -p <pool> list-inconsistent-obj <pgid>, to see which objects are inconsistent. You can verify which image it is by using rbd info <disk-image>.
Click to expand...
I am busy querying it now. But for some reason it's taking FOREVER long. Is that normal?
Also, is there anyway to get the images mounted from the disks? With 3 incomplete pg's I can't imagine all the disks are completely corrupted? But yet I can't get anyne backed up or mounted or started at all?
 
infinityM said:
I am busy querying it now. But for some reason it's taking FOREVER long. Is that normal?
Click to expand...
This greatly depends on how many objects are in that PG.

infinityM said:
Also, is there anyway to get the images mounted from the disks?
Click to expand...
If there are enough objects of an image in that PG, then that image might just be unrecoverable. You can try to map the image, rbd map <pool>/<image>. Or you could try to export it, by rbd export <pool>/<image>. But in both cases I doubt it, as a rbd ls already fails on that image.
 
Alwin said:
This greatly depends on how many objects are in that PG.


If there are enough objects of an image in that PG, then that image might just be unrecoverable. You can try to map the image, rbd map <pool>/<image>. Or you could try to export it, by rbd export <pool>/<image>. But in both cases I doubt it, as a rbd ls already fails on that image.
Click to expand...

Hey Alwin,

So I found a script: https://gitlab.lbader.de/kryptur/ceph-recovery/tree/master
But it seems ceph changed... The problem is the rbd export does not work, it just freezes, so I'm hoping this will help, But along the way ceph changed something and now it does not look to be working...

Any advise? Maybe there's an expert who can help me tweak the script hidden somewhere in the forum?
 
If you’re not able to migrate vm b/w nodes it can be issue with local ceph.conf file located in /etc/ceph as symbolic link:

In order to fix the issue we need to rename the ceph.conf file symbolic link and it will work like charm.

Don’t Touch /etc/pve/ceph.conf as its required




After rename vm migrated


2021-12-10 22:24:48 migration status: active (transferred 420475910, remaining 5021327360), total 8607571968)
2021-12-10 22:24:48 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2021-12-10 22:24:49 migration status: active (transferred 970023784, remaining 4470259712), total 8607571968)
2021-12-10 22:24:49 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2021-12-10 22:24:50 migration status: active (transferred 1572524552, remaining 3866808320), total 8607571968)
2021-12-10 22:24:50 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2021-12-10 22:24:51 migration status: active (transferred 2164497346, remaining 3239002112), total 8607571968)
2021-12-10 22:24:51 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2021-12-10 22:24:52 migration status: active (transferred 2755754099, remaining 2599751680), total 8607571968)
2021-12-10 22:24:52 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2021-12-10 22:24:53 migration status: active (transferred 3342202891, remaining 2012876800), total 8607571968)
2021-12-10 22:24:53 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2021-12-10 22:24:54 migration status: active (transferred 3921808748, remaining 1422364672), total 8607571968)
2021-12-10 22:24:54 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2021-12-10 22:24:55 migration status: active (transferred 4512280675, remaining 816013312), total 8607571968)
2021-12-10 22:24:55 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2021-12-10 22:24:56 migration status: active (transferred 5103732827, remaining 201519104), total 8607571968)
2021-12-10 22:24:56 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2021-12-10 22:24:57 migration status: active (transferred 5162689485, remaining 139595776), total 8607571968)
2021-12-10 22:24:57 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2021-12-10 22:24:57 migration status: active (transferred 5222714605, remaining 79687680), total 8607571968)
2021-12-10 22:24:57 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2021-12-10 22:24:57 migration speed: 819.20 MB/s - downtime 67 ms
2021-12-10 22:24:57 migration status: completed
2021-12-10 22:25:00 migration finished successfully (duration 00:00:15)
TASK OK

Please note down for any future migration issue.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom