Huge differences in reported usage on two PBS instances (and very large .chunks directory)

[Razva]

Hello,

Three months ago I've moved everything to a new PBS server and set an S3 provider as datastore. I see no errors being generated, everything seems to be fine, so I'm thinking to decommission the old server. On the new server, the .chunks directory size is 1TB. Both PBS and the S3 provider report 1TB of usage.

What worries me is that the old server has no less than 6TB in the .chunks directory. No backups were executed in the last 3 months. I've executed garbage collection and pruning tasks, but the size of the .chunks directory remains the same.

Retention policies are identical for both PBS instances: 7x Daily, 4x Weekly. There were no huge changes made inside the VMs, there are practically the same VMs running for 2 years.

Could you please let me know what could be the root cause of this inconsistency?

Thank you!

 

[Onslow]

I'm thinking to decommission the old server. The .chunks directory size is 1TB. Both PBS and the S3 provider report 1TB of usage.

What worries me is that the old server has no less than 6TB in the .chunks directory.

Hi, @Razva .
I don't understand. 1 TB or 6 TB?
And what, exactly, (what commands) do show these sizes?

 

[Razva]

Hey. I've updated the thread so it's more clear.

The new server reports 1TB while the old server reports 6TB.

I'm using `df` to check physical usage, and `Datastore -> Summary" in the PBS UI.

 

[Onslow]

df reports the filesystem usage. Depending on the situation, comparing one server's with the second server's fs usage may be appropriate or no.

Better view of the situation could be got with du -hcs . in the directories of the datastores (there are more files/directories than .chunks). And note that the results may be also not identical due to other circumstances, like block sizes of the particular filesystems.

 

[Razva]

Sorry, my bad, I was referring to `du -csh`, not 'df'.

New PBS:

# du -csh .chunks/
1017G   .chunks/
1017G   total

Old PBS:

# du -csh .chunks/
6.5T    .chunks/
6.5T    total

This is not a ±10 Gb difference. We're talking about a full-blown 5.5TB difference, which is practically 5x more.

 

[Impact]

Can you share the last GC log from the old PBS? This might also be useful. The datastore Summary of both might also be interesting.

 

[Razva]

Here's something prelimilary:

Old:

Per-guest summary (sorted by size):
  /vm/120  :   1.5TiB
  /vm/102  : 376.8GiB
  /vm/101  : 273.9GiB
  /vm/104  : 232.9GiB
  /vm/119  : 149.7GiB
  /vm/114  : 134.1GiB
  /vm/117  : 121.0GiB
  /vm/115  :  86.7GiB
  /vm/107  :  36.5GiB
  /vm/201  :  27.6GiB
  /vm/108  :  14.8GiB
  /vm/113  :  13.1GiB
  /vm/803  :  12.6GiB
  /vm/810  :  11.8GiB
  /vm/109  :  10.4GiB
  /vm/116  :   9.7GiB
  /vm/112  :   9.0GiB
  /vm/110  :   8.2GiB
  /vm/812  :   8.1GiB
  /vm/106  :   7.8GiB
  /vm/100  :   6.9GiB
  /vm/103  :   4.0GiB
  /vm/105  :   3.6GiB
  /vm/124  :   2.8GiB
  /vm/118  :   2.6GiB
  /vm/111  :   2.6GiB
  /vm/121  :   2.6GiB
  /vm/125  :   2.5GiB

This means roughly 3.5 TB, which is 50% of the 6.5 TB reported by `du` or PBS UI.

The new server is OK.