[SOLVED] rbd error: rbd: listing images failed: (2) No such file or directory (500)

[kofik]

Hi

I'm seeing this error randomly when trying to attempt (usually live migration works for the same VM after 1-2 attempts) and consistently when I'm trying to show the content of second Ceph pool. I have 2 Ceph pools, within Proxmox, one has a replication rule where it replicated to SSDs only (called vm_ssd, based on device class) where this doesn't pop up whereas a HDD-based pool vm_hdd consistently shows this message in the UI: "rbd error: rbd: listing images failed: (2) No such file or directory (500)"

Disks can be created and moved between both pools and even an NFS storage for what I can tell. The cluster was updated from PVE 6.2 and Ceph Nautilus to 6.3 and Octopus lately. No issues like these after the update to Octopus either. Listing the rbd volumes on the shell seems to work: 'rbd ls -p vm_hdd' doesn't show errors.

What could be the next logical step to investigate?

 

[aaron]

Does the rbd ls command show any images that shouldn't be there? Have you tried it on the other pool?

 

[kofik]

Hi Aaron

Thank you for your reply! Simply entering rbd ls yields:

Ensure that the default pool has been created or specify an alternate pool name.
rbd: listing images failed: (2) No such file or directory

There is definitely one interesting part, i.e. there is one VM with the ID 140 which I have added disks onto and remove a couple of times so here is the result on the node where the VM is currently run. It should be noted that the error shown with vm_hdd is only visible when the option --long is used:

# qm status 140
status: running

# grep disk /etc/pve/qemu-server/140.conf
hotplug: disk,network,usb,memory,cpu
scsi0: vm_ssd:vm-140-disk-1,cache=writeback,size=126G,ssd=1
scsi1: vm_ssd:vm-140-disk-0,cache=writeback,size=356G,ssd=1

# grep disk /etc/pve/qemu-server/140.conf
hotplug: disk,network,usb,memory,cpu
scsi0: vm_ssd:vm-140-disk-1,cache=writeback,size=126G,ssd=1
scsi1: vm_ssd:vm-140-disk-0,cache=writeback,size=356G,ssd=1

# rbd ls --long -p vm_ssd
NAME                       SIZE     PARENT  FMT  PROT  LOCK
[base-1000-disk-0            50 GiB            2
[...]
vm-140-disk-0              356 GiB            2        excl
vm-140-disk-1              126 GiB            2        excl
[...]

# VM 140 definitely had virtual disk which I previously removed, but is still there
# rbd ls --long -p vm_hdd
2021-02-22T18:17:15.691+0100 7fbcd1ffb700 -1 librbd::io::AioCompletion: 0x556cd24c6020 fail: (2) No such file or directory
rbd: error opening vm-140-disk-0: (2) No such file or directory
NAME           SIZE     PARENT  FMT  PROT  LOCK
[...]
vm-140-disk-1  500 GiB            2        excl
[...]
rbd: listing images failed: (2) No such file or directory

Would I have to (cautiously) use rbd rm on all those volumes or should I expect other hidden dragons?

 

[Alwin Antreich]

Would I have to (cautiously) use rbd rm on all those volumes or should I expect other hidden dragons?

As long as that disk isn't used by a VM/CT anymore, you should be good to delete it.

 

[kofik]

OK, I've doubled checked that the listed image vm-140-disk-1 wasn't used nowhere, deleted that one without issues, the error remained though.

Then I removed the erroneous mentioned vm-140-disk-0 and voilà: A couple of minutes later this not properly-listed device was gone as well. After that rbd ls --long -p vm_hdd doesn't yield any errors and listing the content of said pool works again also within the UI.

Obviously that was a leftover from a deleted disk for which I don't know the exact reason other than that I tried copying a very large disk and deletion hadn't properly terminated back then. I've done a couple of random life migration with no hickup afterwards, things seem OK now.

Thank you for your reassuring pointers and help!