PVE-Cluster Ceph: "rbd: delete error" "Structure needs cleaning"

Noah0302

Member
Jul 21, 2022
62
8
13
Hello guys,

once again I encountered the issue
Code:
rbd error: rbd: listing images failed: (2) No such file or directory (500)
Only that last time I was able to fix it via:
Code:
rbd rm -p CEPH-POOL-NAME vm-ID-disk-ID
This however results in:
Code:
Removing image: 0% complete...failed.
rbd: delete error: 2023-07-07T15:08:57.361+0200 7fd6e6771340 -1 librbd::api::Trash: remove: error: image is pending moving to the trash.(117) Structure needs cleaning



Does anyone know how I might be able to fix this?
The VM that had the disk was deleted and is not even showing up in the listing of the Ceph Pool anymore...



Code:
rbd ls --long -p SSD-Pool01
rbd: error opening vm-611-disk-2: (2) No such file or directory
NAME                                  SIZE      PARENT  FMT  PROT  LOCK
base-7999-disk-0                         1 MiB            2          
base-7999-disk-0@__base__                1 MiB            2  yes      
base-7999-disk-1                         4 MiB            2          
base-7999-disk-1@__base__                4 MiB            2  yes      
base-7999-disk-2                       256 GiB            2          
base-7999-disk-2@__base__              256 GiB            2  yes      
vm-100-disk-0                           32 GiB            2          
vm-100-disk-1                           32 GiB            2        excl
vm-1001-disk-0                         500 GiB            2        excl
vm-1001-disk-0@update2006              500 GiB            2          
vm-1001-state-update2006                16 GiB            2          
vm-101-disk-0                          250 GiB            2        excl
vm-102-disk-0                           50 GiB            2        excl
vm-102-disk-1                          150 GiB            2        excl
vm-102-disk-2                           64 GiB            2        excl
vm-103-disk-0                           32 GiB            2        excl
vm-1301-disk-0                          32 GiB            2        excl
vm-201-disk-0                           72 GiB            2        excl
vm-202-disk-0                          128 GiB            2        excl
vm-203-disk-0                           64 GiB            2        excl
vm-204-disk-0                           60 GiB            2        excl
vm-222-disk-0                            2 TiB            2        excl
vm-223-disk-0                          500 GiB            2        excl
vm-224-disk-0                          200 GiB            2        excl
vm-225-disk-0                           50 GiB            2        excl
vm-226-disk-0                          100 GiB            2        excl
vm-227-disk-0                           50 GiB            2        excl
vm-228-disk-0                         1000 GiB            2        excl
vm-228-disk-0@GCCapAlma               1000 GiB            2          
vm-229-disk-0                         1000 GiB            2        excl
vm-229-disk-0@GCCapAlmaRate           1000 GiB            2          
vm-281-disk-0                           64 GiB            2        excl
vm-282-disk-0                           32 GiB            2        excl
vm-283-disk-0                           32 GiB            2        excl
vm-284-disk-0                           32 GiB            2        excl
vm-285-disk-0                           32 GiB            2        excl
vm-286-disk-0                           64 GiB            2        excl
vm-287-disk-0                           32 GiB            2        excl
vm-288-disk-0                           32 GiB            2        excl
vm-288-disk-0@ohnevoiptools             32 GiB            2          
vm-290-disk-0                           16 GiB            2        excl
vm-293-disk-0                           32 GiB            2        excl
vm-300-disk-0                           32 GiB            2        excl
vm-303-disk-0                           80 GiB            2        excl
vm-304-disk-0                           64 GiB            2        excl
vm-305-disk-0                          200 GiB            2        excl
vm-306-disk-0                           32 GiB            2        excl
vm-307-disk-0                          150 GiB            2        excl
vm-308-disk-0                          150 GiB            2        excl
vm-309-disk-0                           20 GiB            2        excl
vm-309-disk-1                          150 GiB            2        excl
vm-310-disk-0                            8 GiB            2        excl
vm-311-disk-0                           40 GiB            2        excl
vm-311-disk-1                           70 GiB            2        excl
vm-312-disk-0                          300 GiB            2        excl
vm-313-disk-0                           35 GiB            2        excl
vm-315-disk-0                           50 GiB            2        excl
vm-401-disk-0                           32 GiB            2        excl
vm-402-disk-0                          250 GiB            2        excl
vm-410-disk-0                           64 GiB            2        excl
vm-410-disk-0@update                    64 GiB            2          
vm-410-state-update                    4.0 GiB            2          
vm-411-disk-0                          200 GiB            2        excl
vm-412-disk-0                          200 GiB            2        excl
vm-413-disk-0                          200 GiB            2        excl
vm-420-disk-0                           64 GiB            2        excl
vm-421-disk-0                          150 GiB            2        excl
vm-422-disk-0                            8 GiB            2        excl
vm-423-disk-0                           64 GiB            2        excl
vm-427-disk-0                            1 MiB            2        excl
vm-427-disk-1                            4 MiB            2        excl
vm-427-disk-2                          256 GiB            2        excl
vm-427-disk-3                          140 GiB            2        excl
vm-436-disk-0                          1.2 TiB            2        excl
vm-438-disk-0                          250 GiB            2        excl
vm-439-disk-0                           64 GiB            2        excl
vm-441-disk-0                           50 GiB            2        excl
vm-443-disk-0                            1 MiB            2        excl
vm-443-disk-1                            4 MiB            2        excl
vm-444-disk-1                            1 MiB            2        excl
vm-444-disk-2                            4 MiB            2        excl
vm-445-disk-0                            1 MiB            2        excl
vm-445-disk-1                            4 MiB            2        excl
vm-445-disk-2                          256 GiB            2        excl
vm-445-disk-3                           50 GiB            2        excl
vm-446-disk-0                          100 GiB            2        excl
vm-446-disk-1                            1 MiB            2        excl
vm-446-disk-2                            4 MiB            2        excl
vm-446-disk-3                          100 GiB            2        excl
vm-451-disk-0                           64 GiB            2        excl
vm-452-disk-0                           32 GiB            2        excl
vm-461-disk-0                          150 GiB            2        excl
vm-462-disk-0                           64 GiB            2        excl
vm-465-disk-0                           64 GiB            2        excl
vm-473-disk-0                           32 GiB            2        excl
vm-474-disk-0                           32 GiB            2        excl
vm-600-disk-0                           16 GiB            2        excl
vm-601-disk-0                           64 GiB            2        excl
vm-6010-disk-0                          16 GiB            2        excl
vm-6010-disk-0@Init                     16 GiB            2          
vm-6010-disk-0@PreUpdater               16 GiB            2          
vm-6012-disk-0                          32 GiB            2        excl
vm-6012-disk-0@Update                    8 GiB            2          
vm-605-disk-0                           64 GiB            2        excl
vm-606-disk-0                           32 GiB            2        excl
vm-607-disk-0                          200 GiB            2        excl
vm-608-disk-0                           16 GiB            2        excl
vm-610-disk-0                            1 MiB            2        excl
vm-610-disk-1                            4 MiB            2        excl
vm-610-disk-3                          200 GiB            2        excl
vm-6145-disk-0                          16 GiB            2        excl
vm-6200-disk-0                          32 GiB            2        excl
vm-6201-disk-0                        1000 GiB            2          
vm-6201-disk-0@PreExpand              1000 GiB            2          
vm-6205-disk-0                          16 GiB            2        excl
vm-6205-disk-0@WorkingNoCustomPW        16 GiB            2          
vm-6206-disk-0                           8 GiB            2        excl
vm-6225-disk-0                          32 GiB            2        excl
vm-6225-disk-0@PreReleaseUpgrade        32 GiB            2          
vm-6230-disk-0                         128 GiB            2        excl
vm-6245-disk-0                          32 GiB            2        excl
vm-6245-disk-0@Init                     32 GiB            2          
vm-6245-disk-0@Paperless                32 GiB            2          
vm-6245-disk-0@ldap                     32 GiB            2          
vm-6245-state-ldap                     1.8 GiB            2          
vm-6246-disk-0                          32 GiB            2        excl
vm-6246-disk-0@BookStack                32 GiB            2          
vm-6246-disk-0@UserPermissions          32 GiB            2          
vm-6246-disk-0@Gliederung               32 GiB            2          
vm-6246-disk-0@VerbesserteGliederung    32 GiB            2          
vm-6246-disk-0@VorADconfig              32 GiB            2          
vm-7102-disk-2                         256 GiB            2          
rbd: listing images failed: (2) No such file or directory
Code:
rbd rm -p SSD-Pool01 vm-611-disk-2

Removing image: 0% complete...failed.

rbd: delete error: 2023-07-07T15:08:57.361+0200 7fd6e6771340 -1 librbd::api::Trash: remove: error: image is pending moving to the trash.(117) Structure needs cleaning

Also "rbd --pool SSD-Pool01 trash ls" has no output:
Code:
root@PVE50:~# rbd --pool SSD-Pool01 trash ls
root@PVE50:~#

Thank you for reading!
 
Last edited:
Structure needs cleaning is typical for file system corruption, which in this case is likely referring to an OSD file system. hows the health of your cluster? any OSD issues? any outstanding scrubs/deep scrubs?
 
Structure needs cleaning is typical for file system corruption, which in this case is likely referring to an OSD file system. hows the health of your cluster? any OSD issues? any outstanding scrubs/deep scrubs?
Everything is fine according to the Health Status. I even deep scrubbed most OSDs.
Ass Ceph is still running on v16, I might try to upgrade it to 17 and see if it fixes anything
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!