3.4.0 Upgrade – Detected X Index Files Outside Expected Directory Structure

F

fmoreira86

Renowned Member
Aug 7, 2013
47
1
73
Hi!

Since upgrading to 3.4.0, I’ve been getting the following message when the GC runs:

2025-04-12T11:53:29+01:00: marked 100% (1004 of 1004 index files)
2025-04-12T11:53:29+01:00: found 3 index files outside of expected directory scheme

Any idea what might be causing this?

I tried to run:

find /mnt/datastore/BackupStore/vm -type f -name "*.fidx" | wc -l
and got:
1004

But I don't see any fidx outside the expected directory scheme...
 
Last edited:
Hi,
fmoreira86 said:
Hi!

Since upgrading to 3.4.0, I’ve been getting the following message when the GC runs:

2025-04-12T11:53:29+01:00: marked 100% (1004 of 1004 index files)
2025-04-12T11:53:29+01:00: found 3 index files outside of expected directory scheme

Any idea what might be causing this?
Click to expand...
yes, there was recent changes on how garbage collection handles iteration over the index files [0].

fmoreira86 said:
I tried to run:

find /mnt/datastore/BackupStore/vm -type f -name "*.fidx" | wc -l
and got:
1004

But I don't see any fidx outside the expected directory scheme...
Click to expand...
What about didx files? Also, you should run the find command from the datastore's top level path, not just the vm folder. Garbage collection looks into all folders (hidden ones are excluded however).

Also, please do share the complete task log, the strange path locations should be marked by garbage collection as well and logged. If there are no further warnings in the log, maybe identify if there was a concurrent backup or prune job while garbage collection was running.

[0] https://git.proxmox.com/?p=proxmox-backup.git;a=commit;h=74361da8557d38d4533428453eab52f0db8b1b32
 
Last edited:
fmoreira86 said:
@Chris hi, how can I debug this to solve it?
Click to expand...

Please do check as suggested above:

Chris said:
What about didx files? Also, you should run the find command from the datastore's top level path, not just the vm folder. Garbage collection looks into all folders (hidden ones are excluded however).

Also, please do share the complete task log, the strange path locations should be marked by garbage collection as well and logged. If there are no further warnings in the log, maybe identify if there was a concurrent backup or prune job while garbage collection was running.
Click to expand...

I suspect this to be caused by a concurrent prune job deleting snapshots, which are than incorrectly interpreted/listed as strange path. I am working on a patch for that.
 
Chris said:
Please do check as suggested above:



I suspect this to be caused by a concurrent prune job deleting snapshots, which are than incorrectly interpreted/listed as strange path. I am working on a patch for that.
Click to expand...
find /mnt/datastore/BackupStore -type f -name "*.fidx" | wc -l
1004

find /mnt/datastore/BackupStore -type f -name "*.didx" | wc -l
0

Right now I've:

2025-04-13T23:17:00+01:00: marked 100% (1004 of 1004 index files)
2025-04-13T23:17:00+01:00: found 9 index files outside of expected directory scheme

But yesterday I only had 1:

2025-04-13T19:23:18+01:00: marked 100% (1004 of 1004 index files)
2025-04-13T19:23:18+01:00: found 1 index files outside of expected directory scheme

and also:

2025-04-12T22:03:11+01:00: marked 100% (1004 of 1004 index files)
2025-04-12T22:03:11+01:00: found 4 index files outside of expected directory scheme

I don't see any other job with errors...

Only GC with this warning...

The only concurrent jobs I see are backup jobs...
 
fmoreira86 said:
Do you want me to try something else? :)
Click to expand...
See the patch above, this will cover the edge cases leading to the incorrect log message you are seeing. Until that is applied, you can safely ignore this warning, there is currently nothing wrong with your datastore layout.
 
Chris said:
See the patch above, this will cover the edge cases leading to the incorrect log message you are seeing. Until that is applied, you can safely ignore this warning, there is currently nothing wrong with your datastore layout.
Click to expand...
Can you please guide me on how to apply that patch?
 
  • Like
Reactions: fmoreira86
Chris said:
Please do check as suggested above:



I suspect this to be caused by a concurrent prune job deleting snapshots, which are than incorrectly interpreted/listed as strange path. I am working on a patch for that.
Click to expand...
I am getting the same phase 1 GC errors now last 3 days since 3.4 upgrade. The number is not growing, first day it was 94 chunks, then next day 56, then last night 52.

I definitely have concurrent backup jobs happening. And one of the two daily prune jobs occurs during the GC task.
 
sienar said:
I am getting the same phase 1 GC errors
Click to expand...
This log messages are warnings, not errors ;). If you only have the Detected X Index Files Outside Expected Directory Structure, with no other lines indicating that these index files have been marked, than you are safe to ignore the warning, as stated above this log is due to incorrect accounting for newly created and/or pruned snapshots while GC is ongoing.

sienar said:
The number is not growing, first day it was 94 chunks, then next day 56, then last night 52.
Click to expand...
I would assume you mean index files, not chunks. Can you please post one of the garbage collection task logs, so I might check we are on the same page here?
 
OsvaldoP said:
Same problem here.

Possible to estimate when the patch will released?

kr
Click to expand...
Patch has already been applied and packaged with proxmox-backup-server version 3.4.1-1, which is currently available on the pbstest repo.
 
  • Like
Reactions: sienar and OsvaldoP
Chris said:
Patch has already been applied and packaged with proxmox-backup-server version 3.4.1-1, which is currently available on the pbstest repo.
Click to expand...
Ben running this update a few days now. Eliminated the errors for my install.

Thanks!
 
Also, just want to say, loving the speed improvements to the GC process. I've gone from a weekly GC that ran for most of a day to a nightly GC that finishes in about 3 hours.
 
You must log in or register to reply here.
Top Bottom