vzdumps in backup dir eating space

[mitrich]

Hello everyone
We are using PVE 5.3-5 and recently I found that there are numerous dumps in vzdump backup directory located in directories

.snapshot/hourly.YYYY-DD-MM
.snapshot/daily.YYYY-DD-MM
.snapshot/weekly.YYYY-DD-MM

Dumps are eating enormous amount of disk space, if the things keep this way very soon I will face serious difficulties with free space. Here is a directory listing:

Spoiler: ls -lR

.:
total 40
drwxr-xr-x 3 root root 4096 Apr 8 12:15 daily.2019-04-09_0010
drwxr-xr-x 3 root root 4096 Apr 8 12:15 daily.2019-04-10_0010
drwxr-xr-x 3 root root 4096 Apr 8 12:15 hourly.2019-04-10_0405
drwxr-xr-x 3 root root 4096 Apr 8 12:15 hourly.2019-04-10_0505
drwxr-xr-x 3 root root 4096 Apr 8 12:15 hourly.2019-04-10_0605
drwxr-xr-x 3 root root 4096 Apr 8 12:15 hourly.2019-04-10_0705
drwxr-xr-x 3 root root 4096 Apr 8 12:15 hourly.2019-04-10_0805
drwxr-xr-x 3 root root 4096 Apr 8 12:15 hourly.2019-04-10_0905
drwxr-xr-x 3 root root 4096 Aug 7 2018 weekly.2019-03-31_0015
drwxr-xr-x 3 root root 4096 Aug 7 2018 weekly.2019-04-07_0015

./daily.2019-04-09_0010:
total 60
drwxr-xr-x 2 root root 57344 Apr 8 21:50 dump

./daily.2019-04-09_0010/dump:
total 1053890060
-rw-r--r-- 1 root root 1494 Feb 7 21:30 vzdump-qemu-100-2019_02_07-21_30_02.log
-rw-r--r-- 1 root root 65805621 Feb 7 21:30 vzdump-qemu-100-2019_02_07-21_30_02.vma.lzo
-rw-r--r-- 1 root root 1374 Feb 14 21:30 vzdump-qemu-100-2019_02_14-21_30_02.log
-rw-r--r-- 1 root root 65796765 Feb 14 21:30 vzdump-qemu-100-2019_02_14-21_30_02.vma.lzo
...

./daily.2019-04-10_0010:
total 60
drwxr-xr-x 2 root root 57344 Apr 9 16:25 dump

./daily.2019-04-10_0010/dump:
total 200272384
-rw-r--r-- 1 root root 11716264335 Apr 7 21:51 vzdump-qemu-102-2019_04_07-21_30_02.vma.lzo
-rw-r--r-- 1 root root 11907640288 Apr 8 21:50 vzdump-qemu-102-2019_04_08-21_30_02.vma.lzo
...

./hourly.2019-04-10_0405:
total 60
drwxr-xr-x 2 root root 57344 Apr 9 16:25 dump

./hourly.2019-04-10_0405/dump:
total 200272384
-rw-r--r-- 1 root root 11716264335 Apr 7 21:51 vzdump-qemu-102-2019_04_07-21_30_02.vma.lzo
-rw-r--r-- 1 root root 11907640288 Apr 8 21:50 vzdump-qemu-102-2019_04_08-21_30_02.vma.lzo
....

./hourly.2019-04-10_0505:
total 60
drwxr-xr-x 2 root root 57344 Apr 9 16:25 dump

./hourly.2019-04-10_0505/dump:
total 200272384
-rw-r--r-- 1 root root 11716264335 Apr 7 21:51 vzdump-qemu-102-2019_04_07-21_30_02.vma.lzo
-rw-r--r-- 1 root root 11907640288 Apr 8 21:50 vzdump-qemu-102-2019_04_08-21_30_02.vma.lzo
...

./hourly.2019-04-10_0605:
total 60
drwxr-xr-x 2 root root 57344 Apr 9 16:25 dump

./hourly.2019-04-10_0605/dump:
total 200272384
-rw-r--r-- 1 root root 11716264335 Apr 7 21:51 vzdump-qemu-102-2019_04_07-21_30_02.vma.lzo
-rw-r--r-- 1 root root 11907640288 Apr 8 21:50 vzdump-qemu-102-2019_04_08-21_30_02.vma.lzo
...

./hourly.2019-04-10_0705:
total 60
drwxr-xr-x 2 root root 57344 Apr 9 16:25 dump

./hourly.2019-04-10_0705/dump:
total 200272384
-rw-r--r-- 1 root root 11716264335 Apr 7 21:51 vzdump-qemu-102-2019_04_07-21_30_02.vma.lzo
-rw-r--r-- 1 root root 11907640288 Apr 8 21:50 vzdump-qemu-102-2019_04_08-21_30_02.vma.lzo
...

./hourly.2019-04-10_0805:
total 60
drwxr-xr-x 2 root root 57344 Apr 9 16:25 dump

./hourly.2019-04-10_0805/dump:
total 200272384
-rw-r--r-- 1 root root 11716264335 Apr 7 21:51 vzdump-qemu-102-2019_04_07-21_30_02.vma.lzo
-rw-r--r-- 1 root root 11907640288 Apr 8 21:50 vzdump-qemu-102-2019_04_08-21_30_02.vma.lzo
...

./hourly.2019-04-10_0905:
total 60
drwxr-xr-x 2 root root 57344 Apr 9 16:25 dump

./hourly.2019-04-10_0905/dump:
total 200272384
-rw-r--r-- 1 root root 11716264335 Apr 7 21:51 vzdump-qemu-102-2019_04_07-21_30_02.vma.lzo
-rw-r--r-- 1 root root 11907640288 Apr 8 21:50 vzdump-qemu-102-2019_04_08-21_30_02.vma.lzo
...

./weekly.2019-03-31_0015:
total 60
drwxr-xr-x 2 root root 57344 Mar 30 21:48 dump

./weekly.2019-03-31_0015/dump:
total 2114099060
-rw-r--r-- 1 root root 1494 Feb 7 21:30 vzdump-qemu-100-2019_02_07-21_30_02.log
-rw-r--r-- 1 root root 65805621 Feb 7 21:30 vzdump-qemu-100-2019_02_07-21_30_02.vma.lzo
-rw-r--r-- 1 root root 1374 Feb 14 21:30 vzdump-qemu-100-2019_02_14-21_30_02.log
-rw-r--r-- 1 root root 65796765 Feb 14 21:30 vzdump-qemu-100-2019_02_14-21_30_02.vma.lzo
...

./weekly.2019-04-07_0015:
total 60
drwxr-xr-x 2 root root 57344 Apr 6 21:51 dump

./weekly.2019-04-07_0015/dump:
total 880860688
-rw-r--r-- 1 root root 1494 Feb 7 21:30 vzdump-qemu-100-2019_02_07-21_30_02.log
-rw-r--r-- 1 root root 65805621 Feb 7 21:30 vzdump-qemu-100-2019_02_07-21_30_02.vma.lzo
-rw-r--r-- 1 root root 1374 Feb 14 21:30 vzdump-qemu-100-2019_02_14-21_30_02.log
-rw-r--r-- 1 root root 65796765 Feb 14 21:30 vzdump-qemu-100-2019_02_14-21_30_02.vma.lzo
...

I cannot delete these dumps directly, I get error message saying that this is read only file system. Searched everywhere in the web interface, found nothing similar to my problem.

Backups are scheduled in the PVE interface as shown in the picture.


How can I get rid out of these dumps?

 

[dcsapak]

it seems that your storage has snapshots, since the folder '.snapshot' is not something we create -> check your storage for who makes those snapshots

 

[mitrich]

it seems that your storage has snapshots, since the folder '.snapshot' is not something we create -> check your storage for who makes those snapshots

Please, make some tips how to monitor processes that create dirs, files, etc on nfs.

 

[dcsapak]

Please, tell some tips how to monitor processes that create dirs, files, etc on nfs.

this is not a normal 'create dir or file' operation, someone makes snapshot on your storage which gets referenced in those '.snapshot' directories

 

[mitrich]

this is not a normal 'create dir or file' operation, someone makes snapshot on your storage which gets referenced in those '.snapshot' directories

There are very old snapshots, some are 1-2 months old. Seems that PVE only remounts dirs containing snapshots with different names (e.g. hourly.2019-04-10_0605, hourly.2019-04-10_0705, etc).

 

[tom]

There are very old snapshots, some are 1-2 months old. Seems that PVE only remounts dirs containing snapshots with different names (e.g. hourly.2019-04-10_0605, hourly.2019-04-10_0705, etc).

Someone installed / configured something non standard. So we do not know what is going on here.

 

[mitrich]

Someone installed / configured something non standard. So we do not know what is going on here.

We found it. The problem was in the settings of NetApp that made those snapshots.
Thank you for attention and excuse me for waisting your time!