No Space Left on Device

Nixellion · May 30, 2024

Hi!

I'm getting "No Space Left On Device" whenever I try to run apt update, open SSH in the web ui, or perform any operation through the GUI, and others.

However

Code:

df -h

shows there's plenty of space everywhere:

Code:

Filesystem         Size  Used Avail Use% Mounted on
udev                16G     0   16G   0% /dev
tmpfs              3.2G  9.1M  3.2G   1% /run
/dev/nvme0n1p3     221G  173G   49G  79% /
tmpfs               16G   43M   16G   1% /dev/shm
tmpfs              5.0M     0  5.0M   0% /run/lock
efivarfs           320K   51K  265K  17% /sys/firmware/efi/efivars
/dev/nvme0n1p2     511M  584K  511M   1% /boot/efi
/dev/sdb1          113G   91G   17G  85% /mnt/backup_usb
/dev/fuse          128M   20K  128M   1% /etc/pve
//192.168.x.xx/BB   22T   18T  4.5T  80% /mnt/pve/remote-bb
tmpfs              3.2G     0  3.2G   0% /run/user/0

Code:

root@sigrun:~# df -i
Filesystem         Inodes IUsed   IFree IUse% Mounted on
udev              4083415   516 4082899    1% /dev
tmpfs             4091822   876 4090946    1% /run
/dev/nvme0n1p3          0     0       0     - /
tmpfs             4091822   106 4091716    1% /dev/shm
tmpfs             4091822    17 4091805    1% /run/lock
efivarfs                0     0       0     - /sys/firmware/efi/efivars
/dev/nvme0n1p2          0     0       0     - /boot/efi
/dev/sdb1         7512064   339 7511725    1% /mnt/backup_usb
/dev/fuse          262144    38  262106    1% /etc/pve
//192.168.1.18/Bk       0     0       0     - /mnt/pve/brunhilda-bk
tmpfs              818364    20  818344    1% /run/user/0

Code:

# apt update
<...>
E: Some index files failed to download. They have been ignored, or old ones used instead.
E: Problem renaming the file /var/cache/apt/srcpkgcache.bin.n3EwrQ to /var/cache/apt/srcpkgcache.bin - rename (28: No space left on device)
E: Problem renaming the file /var/cache/apt/pkgcache.bin.h4dzcf to /var/cache/apt/pkgcache.bin - rename (28: No space left on device)
W: You may want to run apt-get update to correct these problems
E: The package cache file is corrupted

This happened after upgrading from 7 to 8. Initially space on /dev/nvme0n1p3 was used up, but I freed up some space, and yet it still shows errors.

Is there any way to fix it that does not involve reinstalling Proxmox?

Thanks

alexskysilk · May 30, 2024

something is eating up all your inodes. you can do something like

find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n

to find the culprit(s) and delete them.

donhwyo · May 30, 2024

Also try

Code:

apt clean all

.

Nixellion · May 30, 2024

alexskysilk said:
something is eating up all your inodes. you can do something like

find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n

to find the culprit(s) and delete them.

Here's the output, not sure what it means and what to do about it?

Code:

root@sigrun:~# find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n
      1 .bash_history
      1 .bashrc
      1 .forward
      1 .profile
      1 .rnd
      1 .wget-hsts
      3 .ssh

donhwyo, I tried apt clean before, didn't help

notKakarot · May 30, 2024

Do you have backups running periodically? I had that problem with HomeAssistant making a backup before every upgrade and I had to go an clean them up.

Nixellion · May 30, 2024

Yes, though they've been disabled for about a month, and backups are going to an external USB drive.
I've also posted the df -h output which shows that there's free space available on all locations.

alexskysilk · May 30, 2024

sorry, that command was relative.

find / -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n

make sure to dismount any extraneous file systems (eg /mnt/backup_usb) or the command will take a very long time and produce output on other filesystems.

Nixellion · May 30, 2024

alexskysilk said:
sorry, that command was relative.

find / -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n

make sure to dismount any extraneous file systems (eg /mnt/backup_usb) or the command will take a very long time and produce output on other filesystems.

Code:

root@sigrun:~# find / -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n
      9 root
    601 etc
    632 boot
   4839 var
  58504 usr

There you go.

alexskysilk · May 30, 2024

so we know the problem is in the /usr directory. keep digging- now run the command in /usr (eg:

find /usr -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n

Nixellion · May 31, 2024

Well, that does not seem right.

Code:

root@sigrun:~# find /usr -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n
  58504 usr

alexskysilk · May 31, 2024

the one liner I constructed was built to run inside the directory to examine, which made the cut command not effective for subdirectories. run the original command inside the directory to examine.

More importantly, examine the command and understand whats its doing so you dont have to wait for me (or someone else) to do the work for you.

Nixellion · Jun 1, 2024

Sorry, brain melting. I'll try to learn how it works exactly, but as of right now it does now work when run from the directory to examine, neither one of them

Code:

root@sigrun:~# cd /usr
root@sigrun:/usr# find /usr -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n
  58504 usr
root@sigrun:/usr# find / -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n
      1 tmp
     10 root
    601 etc
    632 boot
   4845 var
  58504 usr

I did find another command

Code:

{ find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n; } 2>/dev/null

Which outputs, if it tells you anything... But I am not sure if this looks like inodes are overused somewhere:

Code:

<...>
  
    139 /usr/lib/modules/5.15.30-2-pve/kernel/net/netfilter
    139 /usr/lib/modules/5.15.64-1-pve/kernel/net/netfilter
    140 /usr/share/ca-certificates/mozilla
    140 /usr/share/nmap/nselib
    143 /usr/share/terminfo/p
    145 /usr/share/proxmox-acme/dnsapi
    146 /usr/share/terminfo/h
    146 /usr/share/zoneinfo/America
    146 /usr/share/zoneinfo/right/America
    155 /usr/share/terminfo/v
    163 /usr/share/terminfo/x
    171 /usr/lib/python3.11/__pycache__
    175 /usr/lib/python3.9/__pycache__
    176 /usr/lib/modules/5.15.152-1-pve/kernel/drivers/hwmon
    176 /usr/lib/modules/5.15.30-2-pve/kernel/drivers/hwmon
    176 /usr/lib/modules/5.15.64-1-pve/kernel/drivers/hwmon
    187 /etc
    191 /usr/share/locale
    193 /usr/share/lintian/overrides
    195 /usr/lib/modules/6.8.4-3-pve/kernel/drivers/hwmon
    199 /usr/share/terminfo/n
    204 /usr/lib/python3.9
    205 /usr/lib/python3.11
    223 /usr/lib/modules/5.15.30-2-pve/kernel/sound/soc/codecs
    227 /usr/lib/modules/5.15.152-1-pve/kernel/sound/soc/codecs
    227 /usr/lib/modules/5.15.64-1-pve/kernel/sound/soc/codecs
    233 /usr/share/i18n/charmaps
    236 /usr/share/terminfo/w
    246 /usr/lib/firmware/radeon
    256 /usr/lib/x86_64-linux-gnu/gconv
    260 /usr/share/man/man5
    266 /usr/share/terminfo/t
    267 /usr/lib/firmware
    278 /boot/grub/x86_64-efi
    279 /usr/lib/grub/x86_64-efi
    283 /etc/ssl/certs
    286 /usr/lib/modules/6.8.4-3-pve/kernel/sound/soc/codecs
    289 /boot/grub/i386-pc
    293 /usr/share/terminfo/d
    303 /usr/lib/grub/i386-pc
    333 /usr/share/terminfo/a
    334 /usr/lib/systemd/system
    339 /usr/share/javascript/sencha-touch/resources/themes/images/pictos
    361 /usr/share/i18n/locales
    394 /usr/share/man/man7
    456 /usr/share/consolefonts
    470 /usr/sbin
    511 /usr/share/man/man3
    556 /usr/lib/firmware/amdgpu
    605 /usr/share/nmap/scripts
    757 /usr/share/doc
    936 /usr/bin
    937 /usr/lib/x86_64-linux-gnu
    939 /usr/share/bash-completion/completions
    961 /usr/share/man/man1
    967 /usr/share/man/man8
   3129 /var/lib/dpkg/info

Nixellion · Jun 1, 2024

Another curious thing is that

Code:

E: Problem renaming the file /var/cache/apt/srcpkgcache.bin.GVbB1J to /var/cache/apt/srcpkgcache.bin - rename (28: No space left on device)
E: Problem renaming the file /var/cache/apt/pkgcache.bin.V6zHBf to /var/cache/apt/pkgcache.bin - rename (28: No space left on device)

This, I tried to manually move those files and also got no space left on device. However if I use `cp` instead of `mv` then it copies it just fine. Which would be using more space and inodes, right? So for some reason it thinks that there's no space when there is?

alexskysilk · Jun 1, 2024

regardless of what's eating your inodes, one thing is for certain. your filesystem was created with an overly limited number of inodes (58000 inodes isnt a lot.) Assuming this to be an ext(4) file system, there is no mechanism to remedy in place.

The remedy would be to recreate the filesystem. in your case, simplest thing would be to wipe and reinstall the whole thing.

You CAN live with this through judicial application of separate filesystems (eg for var) but unless you're comfortable managing partitions and file systems manually... just reinstall.

Nixellion · Jun 1, 2024

It's BTRFS

I'm fine with managing partitions and file systems, but that just sounds like tweaking something on the host that should not be tweaked, as in - better to keep the host clean from any manual modifications.

I think I can recreate all containers and VMs from this node from older backups + manually copied latest .raw files. Will also get a larger SSD for it.

Not sure why it's created with so little inodes though? It's the first time I tried BTRFS, but all I did was select it during install, and x2 128GB SSDs.

alexskysilk · Jun 1, 2024

Nixellion said:
It's BTRFS

read this: https://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html

Nixellion · Jun 3, 2024

Makes sense now. Well, I've reinstalled it, and upgraded from 2x 128GB SSDs in raid to 1TB SSD (main) + 1TB HDD (backups) for this server. So not going to use BTRFS with this setup, but I'll save this for when I need to use BTRFS again.

Thanks a lot for your time and help.

Search

Search

No Space Left on Device

Nixellion

Active Member

alexskysilk

Distinguished Member

donhwyo

Member

Nixellion

Active Member

donhwyo, I tried apt clean before, didn't help

notKakarot

New Member

Nixellion

Active Member

alexskysilk

Distinguished Member

Nixellion

Active Member

alexskysilk

Distinguished Member

Nixellion

Active Member

alexskysilk

Distinguished Member

Nixellion

Active Member

Nixellion

Active Member

alexskysilk

Distinguished Member

Nixellion

Active Member

alexskysilk

Distinguished Member

Nixellion

Active Member

No Space Left on Device

Active Member

Distinguished Member

Member

Active Member

donhwyo, I tried apt clean before, didn't help​

New Member

Active Member

Distinguished Member

Active Member

Distinguished Member

Active Member

Distinguished Member

Active Member

Active Member

Distinguished Member

Active Member

Distinguished Member

Active Member

donhwyo, I tried apt clean before, didn't help