Warning: unable to close filehandle GEN4 properly: No space left on device at /usr/share/perl5/PVE/APIServer/Any

[GAS]

Can anyone guide me how to reduce the local hd space it's almost full and my server restarted this morning I think this is the issue...

root@pve4:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 63G 0 63G 0% /dev
tmpfs 13G 26M 13G 1% /run
/dev/mapper/pve-root 27G 26G 0 100% /
tmpfs 63G 57M 63G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 63G 0 63G 0% /sys/fs/cgroup
/dev/sdf2 511M 324K 511M 1% /boot/efi
tmpfs 63G 24K 63G 1% /var/lib/ceph/osd/ceph-12
tmpfs 63G 24K 63G 1% /var/lib/ceph/osd/ceph-10
tmpfs 63G 24K 63G 1% /var/lib/ceph/osd/ceph-13
tmpfs 63G 24K 63G 1% /var/lib/ceph/osd/ceph-9
tmpfs 63G 24K 63G 1% /var/lib/ceph/osd/ceph-11
/dev/fuse 30M 132K 30M 1% /etc/pve
tmpfs 13G 0 13G 0% /run/user/0

 

[Ernst T.]

You can try to reduce the logfile size: journalctl --vacuum-size 64M

 

[GAS]

You can try to reduce the logfile size: journalctl --vacuum-size 64M

Still same

 

[gfngfn256]

Show output for:

du -h -x -d1 /

 

[GAS]

Have a look at this thread: https://forum.proxmox.com/threads/local-pve-unknown-storage-ve-8-0.136553/post-605716 and use du -h -x -d1 starting from the root and going down into (unexpectedly) large directories to find out what it taking up space. A fresh install only takes around 3GB.

/dev/mapper/pve-root 27G 26G 0 100% this directory is taking too much space, but I don't know how to clean this

 

[GAS]

Show output for:

du -h -x -d1 /

 

[gfngfn256]

So your var folder @24G has most of your space. Now check with:

du -h -x -d1 /var

 

[Ernst T.]

Still same

Ah OK ... you are using PVE6.x (syslog).

There was no convenient way to reduce the size of the log file, except to delete the logfiles manually

 

[GAS]

So your var folder @24G has most of your space. Now check with:

du -h -x -d1 /var

 

[gfngfn256]

You should use the forum code-editor to enter ouput (the </> button on the forum response bar) as I have done.

Now see which logs are filling your space with:

du -h -x -d1 /var/log/

 

[GAS]

You should use the forum code-editor to enter ouput (the </> button on the forum response bar) as I have done.

Now see which logs are filling your space with:

du -h -x -d1 /var/log/

 

[gfngfn256]

So ceph logging is your culprit.
I think looking here - will help you.

 

[GAS]

So ceph logging is your culprit.
I think looking here - will help you.

Should I delete the log file that is in tar.gz format to resolve the issue?

 

[gfngfn256]

Should I delete the log file that is in tar.gz format to resolve the issue?

Probably - but I don't use ceph - so I'm not going to directly advise you - I would follow the advice I linked to above.

Interestingly - I don't see where your default node logging is going.
Have you already deleted it - or do you have some exotic configuration for logging?

 

[GAS]

Probably - but I don't use ceph - so I'm not going to directly advise you - I would follow the advice I linked to above.

Interestingly - I don't see where your default node logging is going.
Have you already deleted it - or do you have some exotic configuration for logging?

I have cleared the large log files from /var/log/ceph, but the issue persists.

 

[gfngfn256]

Show output again for:

du -h -x -d1 /var/log/

#  also show again:

du -h -x -d1 /var

#  also show again:

du -h -x -d1 /

You should use the forum code-editor to enter ouput (use the </> button on the forum response bar) as I have done.

 

[GAS]

Show output again for:

du -h -x -d1 /var/log/

#  also show again:

du -h -x -d1 /var

#  also show again:

du -h -x -d1 /

You should use the forum code-editor to enter ouput (use the </> button on the forum response bar) as I have done.

 

[gfngfn256]

Looks good to me. What does this show now:

df -h

You should use the forum code-editor to enter ouput (use the </> button on the forum response bar) as I have done.

 

[GAS]

Looks good to me. What does this show now:

df -h

You should use the forum code-editor to enter ouput (use the </> button on the forum response bar) as I have done.

 

[bbgeek17]

https://serverfault.com/questions/2...showing-correct-free-space-after-file-removal

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox