local pve full short period

[sda17]

Hello,

My local pve disk filled up very quickly last night.


cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
content iso,backup,vztmpl

lvmthin: local-lvm
thinpool data
vgname pve
content images,rootdir

lvmthin: SSD1To
thinpool data
vgname data
content rootdir,images

df -h /var/lib/vz
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pve-root 68G 68G 0 100% /

du -sh /var/lib/vz
24K /var/lib/vz

At 2am, this command was run via the crontab
00 2 * * * cp -ur /etc/pve/ /media/DD2To/pve/ > /tmp/cp_pve.txt

My pve:
pve-manager/7.4-18/b1f94095 (running kernel: 5.15.158-2-pve)


I rebooted the server, umount and mount the usb disks without success. If you have any ideas.
Thank you

 

[gurubert]

You should run

du -schx /* | sort -h

to see which directories contain all that data.
Maybe it's a logfile going crazy

 

[sda17]

Hi,
du -schx /* | sort -h
du: cannot access '/proc/556310/task/556310/fd/4': No such file or directory
du: cannot access '/proc/556310/task/556310/fdinfo/4': No such file or directory
du: cannot access '/proc/556310/fd/3': No such file or directory
du: cannot access '/proc/556310/fdinfo/3': No such file or directory
0 /bin
0 /dev
0 /lib
0 /lib32
0 /lib64
0 /libx32
0 /proc
0 /sbin
0 /sys
4.0K /media
4.0K /opt
4.0K /srv
8.0K /home
16K /lost+found
20K /mnt
52K /tmp
192K /root
5.4M /etc
169M /run
258M /boot
3.1G /usr
3.4G /var
6.8G total

here's the result, thanks for your help

 

[gurubert]

That's only a tenth of the usage reported.
Do you have many open deleted files?

lsof -n | grep -i del

 

[sda17]

I'm not sure.

kvm 2581 706116 iou-wrk-2 root DEL REG 0,1 2076 /dev/zero
kvm 2581 706117 iou-wrk-2 root DEL REG 0,1 2076 /dev/zero
kvm 2581 706118 iou-wrk-2 root DEL REG 0,1 2076 /dev/zero
kvm 2581 706119 iou-wrk-2 root DEL REG 0,1 2076 /dev/zero
apache2 251584 100033 DEL REG 0,1 18 /dev/zero
apache2 251584 251614 apache2 100033 DEL REG 0,1 18 /dev/zero
apache2 251584 251615 apache2 100033 DEL REG 0,1 18 /dev/zero
apache2 251584 251616 apache2 100033 DEL REG 0,1 18 /dev/zero
apache2 251584 251617 apache2 100033 DEL REG 0,1 18 /dev/zero
apache2 251584 251618 apache2 100033 DEL REG 0,1 18 /dev/zero
apache2 251584 251619 apache2 100033 DEL REG 0,1 18 /dev/zero
apache2 251584 251620 apache2 100033 DEL REG 0,1 18 /dev/zero
apache2 251584 251621 apache2 100033 DEL REG 0,1 18 /dev/zero
apache2 251584 251622 apache2 100033 DEL REG 0,1 18 /dev/zero
apache2 251584 251623 apache2 100033 DEL REG 0,1 18 /dev/zero
apache2 251584 251624 apache2 100033 DEL REG 0,1 18 /dev/zero
apache2 251584 251625 apache2 100033 DEL REG 0,1 18 /dev/zero
apache2 251584 251626 apache2 100033 DEL REG 0,1 18 /dev/zero
apache2 251584 251627 apache2 100033 DEL REG 0,1 18 /dev/zero
apache2 251584 251628 apache2 100033 DEL REG 0,1 18 /dev/zero
.................................

I have a few of them

 

[gurubert]

Ignore /dev/zero, are there any other large files open and deleted?

 

[sda17]

it's not easy to see, there are a lot of lines and I can't put the result in a file. I found this
iscsid 926 root DEL REG 0,1 2 /SYSV00000000
iscsid 926 root DEL REG 0,1 1 /SYSV00000000
iscsid 926 root DEL REG 0,1 0 /SYSV00000000
iscsid 927 root DEL REG 0,1 2 /SYSV00000000
iscsid 927 root DEL REG 0,1 1 /SYSV00000000
iscsid 927 root DEL REG 0,1 0 /SYSV00000000

 

[Neobin]

At 2am, this command was run via the crontab
00 2 * * * cp -ur /etc/pve/ /media/DD2To/pve/ > /tmp/cp_pve.txt

umount and mount the usb disks without success.

Did/Can you confirm, that there is no data in that/those mount point(s) after unmounting the disk(s)?

 

[sda17]

I can confirm that there is no data after unmounting the disk.
is there a link? I have a 10% wearout