[SOLVED] Weird disk usage stats and data

DynFi User

DynFi User

Renowned Member
Apr 18, 2016
152
17
83
49
dynfi.com
I am pushing my tests with PBS and am now encountering a weird situation.

I'll try to describe briefly the situation.

I have two pools in my system one for the system and one for the backups / data.

Code: 
root@dc1-pbs01:/mnt/datastore/backup/.chunks# zfs list
NAME               USED  AVAIL     REFER  MOUNTPOINT
backup            5.57T   159T     2.68T  /mnt/datastore/backup
backup/duplicacy  2.89T   159T     2.89T  /mnt/datastore/backup/duplicacy
rpool              148G  81.0G       96K  /rpool
rpool/ROOT         148G  81.0G       96K  /rpool/ROOT
rpool/ROOT/pbs-1   148G  81.0G      148G  /

My problem is that I shouldn't have 148G of data on the rpool but more likely 2G.

When I try to du -sh / the system is telling me that there is 2.7G (which seems like the right value) and the two remaining MP on /rpool have close to 0 data.

Code: 
root@dc1-pbs01:~# du -sh /
du: cannot access '/proc/17038/task/17038/fd/3': No such file or directory
du: cannot access '/proc/17038/task/17038/fdinfo/3': No such file or directory
du: cannot access '/proc/17038/fd/4': No such file or directory
du: cannot access '/proc/17038/fdinfo/4': No such file or directory
du: cannot access '/proc/20602': No such file or directory
du: cannot access '/proc/20603': No such file or directory
du: cannot access '/proc/20605': No such file or directory
du: cannot access '/proc/20607': No such file or directory
du: cannot access '/proc/20608': No such file or directory
2.7T    /
root@dc1-pbs01:/mnt/datastore/backup/.chunks# du -sh /rpool/
1.5K    /rpool/
root@dc1-pbs01:/mnt/datastore/backup/.chunks# du -sh /rpool/ROOT/
1.0K    /rpool/ROOT/


So where are the 148G of "ghost" referenced data ??

So do you have any clue where this data might be ?? or how I can identify this somehow.
 
Launched a scrub to figure out what might be the issue : 0 error.

Code: 
  pool: rpool
 state: ONLINE
  scan: scrub repaired 0B in 00:01:54 with 0 errors on Wed Jun 23 10:32:42 2021
config:

    NAME                                                 STATE     READ WRITE CKSUM
    rpool                                                ONLINE       0     0     0
      mirror-0                                           ONLINE       0     0     0
        nvme-eui.00000000000000018ce38e010009f4e7-part3  ONLINE       0     0     0
        nvme-eui.00000000000000018ce38e010009f4e6-part3  ONLINE       0     0     0

errors: No known data errors
 
can you do
Code: 
du -shx /

?
the -x skips other filesystems
 
Yes :

Code: 
root@dc1-pbs01:~# du -shx /
1.2G    /
 
rpool history is also very clean :

Code: 
History for 'rpool':
2020-01-28.16:04:55 zpool create -f -o cachefile=none -o ashift=12 rpool mirror /dev/disk/by-id/nvme-eui.00000000000000018ce38e010009f4e7-part3 /dev/disk/by-id/nvme-eui.00000000000000018ce38e010009f4e6-part3
2020-01-28.16:04:55 zfs create rpool/ROOT
2020-01-28.16:04:55 zfs create rpool/ROOT/pbs-1
2020-01-28.16:04:55 zfs set atime=off rpool
2020-01-28.16:04:55 zfs set compression=on rpool
2020-01-28.16:05:01 zfs set sync=disabled rpool
2020-01-28.16:05:49 zfs set sync=standard rpool
2020-01-28.16:05:49 zfs set mountpoint=/ rpool/ROOT/pbs-1
2020-01-28.16:05:49 zpool set bootfs=rpool/ROOT/pbs-1 rpool
2020-01-28.16:05:49 zpool export rpool
2020-01-28.16:11:08 zpool import -N rpool
2020-01-29.09:57:50 zpool import -N rpool
2020-02-04.15:31:02 zpool import -N rpool
2020-02-05.09:51:39 zpool import -N rpool
2020-02-16.12:40:20 zpool import -N rpool
2021-06-09.10:41:04 zpool import -N rpool
2021-06-14.13:34:31 zpool import -N rpool
2021-06-17.11:58:28 zpool import -N rpool
2021-06-17.12:09:07 zpool import -N rpool
2021-06-17.12:19:03 zpool import -N rpool
2021-06-17.12:48:52 zpool import -N rpool
2021-06-23.09:32:39 zpool import -N rpool
2021-06-23.10:30:54 zpool scrub rpool
 
ok thanks, and 'zfs get all rpool/ROOT/pbs-1' ?
 
Here it is :

Code: 
root@dc1-pbs01:~# zfs get all rpool/ROOT/pbs-1
NAME              PROPERTY              VALUE                  SOURCE
rpool/ROOT/pbs-1  type                  filesystem             -
rpool/ROOT/pbs-1  creation              Tue Jan 28 16:04 2020  -
rpool/ROOT/pbs-1  used                  148G                   -
rpool/ROOT/pbs-1  available             81.0G                  -
rpool/ROOT/pbs-1  referenced            148G                   -
rpool/ROOT/pbs-1  compressratio         1.02x                  -
rpool/ROOT/pbs-1  mounted               yes                    -
rpool/ROOT/pbs-1  quota                 none                   default
rpool/ROOT/pbs-1  reservation           none                   default
rpool/ROOT/pbs-1  recordsize            128K                   default
rpool/ROOT/pbs-1  mountpoint            /                      local
rpool/ROOT/pbs-1  sharenfs              off                    default
rpool/ROOT/pbs-1  checksum              on                     default
rpool/ROOT/pbs-1  compression           on                     inherited from rpool
rpool/ROOT/pbs-1  atime                 off                    inherited from rpool
rpool/ROOT/pbs-1  devices               on                     default
rpool/ROOT/pbs-1  exec                  on                     default
rpool/ROOT/pbs-1  setuid                on                     default
rpool/ROOT/pbs-1  readonly              off                    default
rpool/ROOT/pbs-1  zoned                 off                    default
rpool/ROOT/pbs-1  snapdir               hidden                 default
rpool/ROOT/pbs-1  aclmode               discard                default
rpool/ROOT/pbs-1  aclinherit            restricted             default
rpool/ROOT/pbs-1  createtxg             9                      -
rpool/ROOT/pbs-1  canmount              on                     default
rpool/ROOT/pbs-1  xattr                 on                     default
rpool/ROOT/pbs-1  copies                1                      default
rpool/ROOT/pbs-1  version               5                      -
rpool/ROOT/pbs-1  utf8only              off                    -
rpool/ROOT/pbs-1  normalization         none                   -
rpool/ROOT/pbs-1  casesensitivity       sensitive              -
rpool/ROOT/pbs-1  vscan                 off                    default
rpool/ROOT/pbs-1  nbmand                off                    default
rpool/ROOT/pbs-1  sharesmb              off                    default
rpool/ROOT/pbs-1  refquota              none                   default
rpool/ROOT/pbs-1  refreservation        none                   default
rpool/ROOT/pbs-1  guid                  8668317700026287756    -
rpool/ROOT/pbs-1  primarycache          all                    default
rpool/ROOT/pbs-1  secondarycache        all                    default
rpool/ROOT/pbs-1  usedbysnapshots       0B                     -
rpool/ROOT/pbs-1  usedbydataset         148G                   -
rpool/ROOT/pbs-1  usedbychildren        0B                     -
rpool/ROOT/pbs-1  usedbyrefreservation  0B                     -
rpool/ROOT/pbs-1  logbias               latency                default
rpool/ROOT/pbs-1  objsetid              91                     -
rpool/ROOT/pbs-1  dedup                 off                    default
rpool/ROOT/pbs-1  mlslabel              none                   default
rpool/ROOT/pbs-1  sync                  standard               inherited from rpool
rpool/ROOT/pbs-1  dnodesize             legacy                 default
rpool/ROOT/pbs-1  refcompressratio      1.02x                  -
rpool/ROOT/pbs-1  written               148G                   -
rpool/ROOT/pbs-1  logicalused           151G                   -
rpool/ROOT/pbs-1  logicalreferenced     151G                   -
rpool/ROOT/pbs-1  volmode               default                default
rpool/ROOT/pbs-1  filesystem_limit      none                   default
rpool/ROOT/pbs-1  snapshot_limit        none                   default
rpool/ROOT/pbs-1  filesystem_count      none                   default
rpool/ROOT/pbs-1  snapshot_count        none                   default
rpool/ROOT/pbs-1  snapdev               hidden                 default
rpool/ROOT/pbs-1  acltype               off                    default
rpool/ROOT/pbs-1  context               none                   default
rpool/ROOT/pbs-1  fscontext             none                   default
rpool/ROOT/pbs-1  defcontext            none                   default
rpool/ROOT/pbs-1  rootcontext           none                   default
rpool/ROOT/pbs-1  relatime              on                     temporary
rpool/ROOT/pbs-1  redundant_metadata    all                    default
rpool/ROOT/pbs-1  overlay               on                     default
rpool/ROOT/pbs-1  encryption            off                    default
rpool/ROOT/pbs-1  keylocation           none                   default
rpool/ROOT/pbs-1  keyformat             none                   default
rpool/ROOT/pbs-1  pbkdf2iters           0                      default
rpool/ROOT/pbs-1  special_small_blocks  0                      default


When the system was first installed there might have been couple of backups triggered on the rpool filesystem instead of the backup one.
But datastore were destroyed and removed and data cleared so unless there is a "inode leak" somewhere, I don't see any reason why this data is still referenced… and I can't figure out where it is located.
 
ok, i am very sure that something is overmounted

to check you can bind mount '/' somewhere else
e.g.

Code: 
mkdir /mnt/tmproot
mount --bind / /mnt/tmproot

now you can do 'du -shx /mnt/tmproot'
check the output of 'mount' to see where the other mountpoints are to check what directory might contain data that was overmounted

after cleanup you can unmount it again
Code: 
umount /mnt/tmproot
 
I have done what you have suggested and I have found the data !
You were right.

In this thread I have described my problem with ZFS not mounting my second pool (backup).
So the initial datastore that I have created have been setup on the rpool and not on backup.

Thanks to your command the files which were obfuscated by my new mount were revealed and i could properly remove the files left by old datastore.

Suggestion : maybe passing an argument to destroy the data when you delete a datastore on CLI could be nice to have.


Thanks a lot !
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom