No space left on device

S

seelenvirtuose

New Member
May 30, 2026
3
1
3
Hello.

I need some help with the above mentioned issue. I have a simple Proxmox installation on a single Mini PC. Yesterday, I spotted some unusual problems, like not getting into the web UI. After several reboots, the system seems to work again and is accessible. But there are still some issues. I guess, the cause is "no space left on device", as the title says. The problem might already exist for several days. I noticed, that the last automatic backup is five days ago.

Ok. Jumping into the problem: When trying to do a simple apt update, I get the following output:

Code: 
root@pve:/# apt update
Hit:1 http://deb.debian.org/debian trixie InRelease
Get:2 http://security.debian.org/debian-security trixie-security InRelease [43.4 kB]
Ign:2 http://security.debian.org/debian-security trixie-security InRelease
Get:3 http://download.proxmox.com/debian/pve trixie InRelease [3,534 B]
Ign:3 http://download.proxmox.com/debian/pve trixie InRelease
Get:4 http://deb.debian.org/debian trixie-updates InRelease [47.3 kB]
Ign:4 http://deb.debian.org/debian trixie-updates InRelease
Get:2 http://security.debian.org/debian-security trixie-security InRelease [43.4 kB]
Ign:2 http://security.debian.org/debian-security trixie-security InRelease
Get:4 http://deb.debian.org/debian trixie-updates InRelease [47.3 kB]
Ign:4 http://deb.debian.org/debian trixie-updates InRelease
Get:3 http://download.proxmox.com/debian/pve trixie InRelease [3,534 B]
Ign:3 http://download.proxmox.com/debian/pve trixie InRelease
Get:2 http://security.debian.org/debian-security trixie-security InRelease [43.4 kB]
Ign:2 http://security.debian.org/debian-security trixie-security InRelease
Get:3 http://download.proxmox.com/debian/pve trixie InRelease [3,534 B]
Ign:3 http://download.proxmox.com/debian/pve trixie InRelease
Get:4 http://deb.debian.org/debian trixie-updates InRelease [47.3 kB]
Ign:4 http://deb.debian.org/debian trixie-updates InRelease
Get:2 http://security.debian.org/debian-security trixie-security InRelease [43.4 kB]
Err:2 http://security.debian.org/debian-security trixie-security InRelease
  Error writing to file - write (28: No space left on device) [IP: 146.75.122.132 80]
Get:3 http://download.proxmox.com/debian/pve trixie InRelease [3,534 B]
Err:3 http://download.proxmox.com/debian/pve trixie InRelease
  Error writing to file - write (28: No space left on device) [IP: 185.219.221.167 80]
Get:4 http://deb.debian.org/debian trixie-updates InRelease [47.3 kB]
Err:4 http://deb.debian.org/debian trixie-updates InRelease
  Error writing to file - write (28: No space left on device) [IP: 146.75.118.132 80]
118 packages can be upgraded. Run 'apt list --upgradable' to see them.
Warning: Failed to fetch http://deb.debian.org/debian/dists/trixie-updates/InRelease  Error writing to file - write (28: No space left on device) [IP: 146.75.118.132 80]
Warning: Failed to fetch http://security.debian.org/debian-security/dists/trixie-security/InRelease  Error writing to file - write (28: No space left on device) [IP: 146.75.122.132 80]
Warning: Failed to fetch http://download.proxmox.com/debian/pve/dists/trixie/InRelease  Error writing to file - write (28: No space left on device) [IP: 185.219.221.167 80]
Warning: Some index files failed to download. They have been ignored, or old ones used instead.

As I am a typical Windows user, I somewhat struggle with Linux systems, and do not know how to handle this. Googling the phrase "no space left on device" reveals a ton of articles and forum posts. But I fear, that they do not help me. Most of them explain, how to find out, which directories and/or files take that space, and to delete them. Here my problem: I do not find them.

So I will post here as much information as possible, so that you could possibly help me.

Maybe I first explain, what disks I have attached to that PC and how I use them. I have put a 1TB SATA SSD and a 4TB M.2 (NVMe) SSD into the PC. Additionally, I have four 4TB SATA HDDs inside a USB enclosure, which itself is attached to the PC. All 4TB disks (SSD and HDDs) are used inside ZFS pools for various purposes. Proxmox itself was installed with the standard installation procedure onto the 1 TB SATA SSD. This uses LVM. Sic:

Code: 
root@pve:~# lsblk
NAME                           MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda                              8:0    0 931.5G  0 disk
├─sda1                           8:1    0  1007K  0 part
├─sda2                           8:2    0     1G  0 part /boot/efi
└─sda3                           8:3    0   930G  0 part
  ├─pve-swap                   252:0    0     8G  0 lvm  [SWAP]
  ├─pve-root                   252:1    0    96G  0 lvm  /
  ├─pve-data_tmeta             252:2    0   8.1G  0 lvm
  │ └─pve-data-tpool           252:4    0 793.8G  0 lvm
  │   ├─pve-data               252:5    0 793.8G  1 lvm
  │   ├─pve-vm--100--disk--0   252:6    0     4G  0 lvm
  │   ├─pve-vm--120--disk--0   252:7    0     4G  0 lvm
  │   ├─pve-vm--300--cloudinit 252:8    0     4M  0 lvm
  │   ├─pve-vm--300--disk--0   252:9    0     8G  0 lvm
  │   ├─pve-vm--301--cloudinit 252:10   0     4M  0 lvm
  │   ├─pve-vm--301--disk--0   252:11   0     8G  0 lvm
  │   ├─pve-vm--302--cloudinit 252:12   0     4M  0 lvm
  │   ├─pve-vm--302--disk--0   252:13   0     8G  0 lvm
  │   ├─pve-vm--510--disk--0   252:14   0   128G  0 lvm
  │   ├─pve-vm--199--disk--0   252:15   0     4G  0 lvm
  │   ├─pve-vm--510--disk--1   252:16   0     4M  0 lvm
  │   └─pve-vm--590--disk--0   252:17   0     8G  0 lvm
  └─pve-data_tdata             252:3    0 793.8G  0 lvm
    └─pve-data-tpool           252:4    0 793.8G  0 lvm
      ├─pve-data               252:5    0 793.8G  1 lvm
      ├─pve-vm--100--disk--0   252:6    0     4G  0 lvm
      ├─pve-vm--120--disk--0   252:7    0     4G  0 lvm
      ├─pve-vm--300--cloudinit 252:8    0     4M  0 lvm
      ├─pve-vm--300--disk--0   252:9    0     8G  0 lvm
      ├─pve-vm--301--cloudinit 252:10   0     4M  0 lvm
      ├─pve-vm--301--disk--0   252:11   0     8G  0 lvm
      ├─pve-vm--302--cloudinit 252:12   0     4M  0 lvm
      ├─pve-vm--302--disk--0   252:13   0     8G  0 lvm
      ├─pve-vm--510--disk--0   252:14   0   128G  0 lvm
      ├─pve-vm--199--disk--0   252:15   0     4G  0 lvm
      ├─pve-vm--510--disk--1   252:16   0     4M  0 lvm
      └─pve-vm--590--disk--0   252:17   0     8G  0 lvm
sdb                              8:16   0   3.6T  0 disk
├─sdb1                           8:17   0   3.6T  0 part
└─sdb9                           8:25   0     8M  0 part
sdc                              8:32   0   3.6T  0 disk
├─sdc1                           8:33   0   3.6T  0 part
└─sdc9                           8:41   0     8M  0 part
sdd                              8:48   0   3.6T  0 disk
├─sdd1                           8:49   0   3.6T  0 part
└─sdd9                           8:57   0     8M  0 part
sde                              8:64   0   3.6T  0 disk
├─sde1                           8:65   0   3.6T  0 part
└─sde9                           8:73   0     8M  0 part
nvme0n1                        259:0    0   3.6T  0 disk
├─nvme0n1p1                    259:1    0   3.6T  0 part
└─nvme0n1p9                    259:2    0     8M  0 part

Now to the file system space:

Code: 
root@pve:/# df -h
Filesystem                         Size  Used Avail Use% Mounted on
udev                                32G     0   32G   0% /dev
tmpfs                              6.3G   13M  6.3G   1% /run
/dev/mapper/pve-root                94G   94G     0 100% /
tmpfs                               32G   19M   32G   1% /dev/shm
efivarfs                           128K   25K   99K  20% /sys/firmware/efi/efivars
tmpfs                              5.0M     0  5.0M   0% /run/lock
tmpfs                              1.0M     0  1.0M   0% /run/credentials/systemd-journald.service
tmpfs                               32G     0   32G   0% /tmp
/dev/sda2                         1022M  8.8M 1014M   1% /boot/efi
cpool/media                        6.0T  256K  6.0T   1% /zfs/cpool/media
cpool/backup                       6.2T  240G  6.0T   4% /zfs/cpool/backup
cpool/eon                          6.0T  4.2G  6.0T   1% /zfs/cpool/eon
cpool/documents                    6.0T   96M  6.0T   1% /zfs/cpool/documents
cpool/misc                         6.0T  256K  6.0T   1% /zfs/cpool/misc
cpool/entertainment/video          6.0T  256K  6.0T   1% /zfs/cpool/entertainment/video
cpool/entertainment/lego           6.0T  256K  6.0T   1% /zfs/cpool/entertainment/lego
cpool/software                     6.0T   21G  6.0T   1% /zfs/cpool/software
hpool/apps/kubernetes              3.6T  3.8G  3.6T   1% /zfs/hpool/apps/kubernetes
cpool/cloud/sarah_jonas            6.0T  256K  6.0T   1% /zfs/cpool/cloud/sarah_jonas
cpool/temp                          11T  4.3T  6.0T  42% /zfs/cpool/temp
cpool/cloud/jenny_mirko            6.0T  256K  6.0T   1% /zfs/cpool/cloud/jenny_mirko
cpool/entertainment/audio          6.0T  256K  6.0T   1% /zfs/cpool/entertainment/audio
cpool/cloud/ronja                  6.0T  256K  6.0T   1% /zfs/cpool/cloud/ronja
hpool/apps/postgres                3.6T   15M  3.6T   1% /zfs/hpool/apps/postgres
hpool/apps/nexus                   3.6T  2.1G  3.6T   1% /zfs/hpool/apps/nexus
cpool/studies                      6.0T 1010M  6.0T   1% /zfs/cpool/studies
/dev/fuse                          128M   32K  128M   1% /etc/pve
bkp.fritz.box:/mnt/HD/HD_a2/Store   15T  4.7T  9.8T  33% /mnt/bkp.fritz.box/store
tmpfs                              1.0M     0  1.0M   0% /run/credentials/getty@tty1.service
tmpfs                              6.3G  4.0K  6.3G   1% /run/user/0

This shows that the root filesystem (correct wording?) seems to be 100% full. But where? Disk usage reports:

Code: 
root@pve:/# du -sh /*
0       /bin
309M    /boot
19M     /dev
5.7M    /etc
4.0K    /home
0       /lib
0       /lib64
16K     /lost+found
4.0K    /media
4.7T    /mnt
4.0K    /opt
du: cannot access '/proc/58210/task/58210/fd/4': No such file or directory
du: cannot access '/proc/58210/task/58210/fdinfo/4': No such file or directory
du: cannot access '/proc/58210/fd/3': No such file or directory
du: cannot access '/proc/58210/fdinfo/3': No such file or directory
0       /proc
72K     /root
13M     /run
0       /sbin
4.0K    /srv
0       /sys
0       /tmp
5.4G    /usr
1.2G    /var
4.6T    /zfs

root@pve:/# du -sh /mnt/*
4.7T    /mnt/bkp.fritz.box
4.0K    /mnt/vzsnap0

Some explanation to the /zfs directory: All ZFS pools and datasets are mounted here. Remember: These are the other disks. A similar fact for the /mnt directory: I have mounted an NFS share from a NAS device into /mnt/bkp.fritz.box. This is, why you see such high disk usage here.

As you can see, space consumption of all other directories does not sum up to the nearly 100GB of the root filesystem. So ... How can I narrow this down? How can I find out, what actually takes up the space? How can I solve my problem(s)?

Let me know, what else I should provide, or what steps I should do for analyzing and troubleshooting.

Many thanks.
Mirko
 
Last edited:
Let me add my 5cents:

First ncdu:

Bash: 
apt install ncdu

Then traverse filesystem with:

Bash: 
ncdu --exclude-kernfs --one-file-system /

Also post output of
Bash: 
df -hT
 
diffway said:
Let me add my 5cents:

First ncdu:

Bash: 
apt install ncdu

Then traverse filesystem with:

Bash: 
ncdu --exclude-kernfs --one-file-system /

Also post output of
Bash: 
df -hT
Click to expand...

Hi. I guess that ncdu is another (yet better) analyze tool. Unfortunately, I can't install it either:

Code: 
root@pve:~# apt install ncdu
Installing:
  ncdu

Summary:
  Upgrading: 0, Installing: 1, Removing: 0, Not Upgrading: 118
  Download size: 51.5 kB
  Space needed: 122 kB / 0 B available

Warning: More space needed than available: 122 kB > 0 B, installation may fail
Error: You don't have enough free space in /var/cache/apt/archives/.

I will have a look at the other suggestion first. But thanks anyway.
 
Impact said:
This was just recently a topic: https://forum.proxmox.com/threads/high-boot-disk-usage-on-proxmox-9-2-node-ext4.183879/#post-854741
My guess is you have the same issue. Make sure to disable/unmount /mnt/bkp.fritz.box/store then check again with du -shc /* | sort -h.
Click to expand...

My apologies. I searched in this forum, but didn't find that.

You really nailed the point. My backups weren't stored in the NAS, but inside the directory. After unmounting, I could see them and delete them. Now df -h shows this for the root filesystem:

Code: 
/dev/mapper/pve-root        94G  6.9G   83G   8% /

Many thanks.
Mirko
 
  • Like
Reactions: Johannes S
You must log in or register to reply here.
Top Bottom
Back