rbd error: rbd: listing images failed: (2) No such file or directory (500)

Kaboom

Member
Mar 5, 2019
96
9
8
48
Hi All,

I got this error when I want to see Storage 'ceph_ssd' (Ceph pool name) on node 'node002' (on all nodes).
rbd error: rbd: listing images failed: (2) No such file or directory (500)

When I run rbd ls -l ceph_ssd, I get this at the end:
rbd: listing images failed: (2) No such file or directory

Anyone familiar with this error? Seen some other topics about this, but they all came with a 'vm disk name', but not in my situation.

Thanks!
 

Kaboom

Member
Mar 5, 2019
96
9
8
48
I don't get any errors about a VM:


.......
vm-223-disk-0 50 GiB 2 excl
vm-246-disk-0 50 GiB 2 excl
vm-247-disk-0 50 GiB 2 excl
vm-248-disk-0 50 GiB 2 excl
rbd: listing images failed: (2) No such file or directory
 

Alwin

Proxmox Staff Member
Staff member
Aug 1, 2017
3,867
361
88
Possibly there is a defekt image in that pool. :/

Do you see any entries in the ceph logs? Somewhere it should say if a object is defekt.
 

Kaboom

Member
Mar 5, 2019
96
9
8
48
The Ceph logs are looking healthy and no errors or defects. I can see 'summary' but can't see 'content'.
 

Alwin

Proxmox Staff Member
Staff member
Aug 1, 2017
3,867
361
88
Take the image list, sort it and run through them. It should hang at the image that might have an issue.
 

Kaboom

Member
Mar 5, 2019
96
9
8
48
I also notice just now when I want to migrate a VM to another node I get this error:

task started by HA resource agent
2020-03-25 21:59:43 starting migration of VM 108 to node 'node004' (10.0.0.1)
2020-03-25 21:59:43 ERROR: Failed to sync data - rbd error: rbd: listing images failed: (2) No such file or directory
2020-03-25 21:59:43 aborting phase 1 - cleanup resources
2020-03-25 21:59:43 ERROR: migration aborted (duration 00:00:00): Failed to sync data - rbd error: rbd: listing images failed: (2) No such file or directory
TASK ERROR: migration aborted

When I want to migrate a container to another node I have no (!) problems.
 

Alwin

Proxmox Staff Member
Staff member
Aug 1, 2017
3,867
361
88
When I want to migrate a container to another node I have no (!) problems.
Is the container storage a different pool?

What do you mean with sort it? Stop the VM and run list again?
No. Its one liner. Since rbd ls works, you get its output. Sort it and run the rbd info <image> in a loop through the sorted list. The command will hang when it encounters the faulty image.
 
  • Like
Reactions: Kaboom

Kaboom

Member
Mar 5, 2019
96
9
8
48
Thanks, that did the trick! I found the image and deleted it.

rbd rm vm-xxx-disk-1 -p cephpool
 

Lephisto

Member
Jun 22, 2019
51
3
8
43
Out of curiosity: how can an rbd image become "faulty" if there was no power or hardware related crash?
 

Alwin

Proxmox Staff Member
Staff member
Aug 1, 2017
3,867
361
88
Out of curiosity: how can an rbd image become "faulty" if there was no power or hardware related crash?
Hard to say afterwards. But as it's a network distributed storage, a network fault may do that. Or the VM was killed during a process that wrote the header of the RBD image.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!