Proxmox claims /var/tmp has no space despite it does

[fransesco]

Hello everyone,

Running Proxmox 8.4.5 otherwise happily on a server that is supposed to have plenty of disk space across filesystems.

However, when trying to upload a new .iso file, the upload tool claims "Uploads are stored temporarily in '/var/tmp/', make sure there is enough free space." and seems not to start uploading the file. Earlier uploads have been just fine, but it's been a while since an iso was needed to be uploaded. So I'm kind of thinking that something has changed but can't figure out what.

I've been trying to figure out for hours what could becausing this and circulating across the host's filesystem, like:

root@proliant:/var# df -Th /var/tmp
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/pve-root ext4 94G 4.6G 85G 6% /

According to my understanding, filesystem that holds /var/tmp is only 6% used and have 85 Gb free. Still Proxmox claims it can't upload a typical iso because of lack of disk space. The final destination of the iso would be somewhere else in the storages allocated to Proxmox than the OS itself.

Does anyone have any ideas where to look next?

 

[SteveITS]

I would guess the error message is a default message when the write is failing. Is /var/tmp writeable?

 

[fransesco]

I would guess the error message is a default message when the write is failing. Is /var/tmp writeable?

Thanks for suggesting this. However, seems to be writeable (I tried to write file also with non root privileges and it works):

root@proliant:/var/tmp# fallocate -l 5G testfile.txt
root@proliant:/var/tmp# ls -la
total 5242904
drwxrwxrwt 4 root root 4096 Jul 25 16:56 .
drwxr-xr-x 12 root root 4096 Jul 3 20:52 ..
-rw-r--r-- 1 root root 16 Jul 25 13:35 pve-reserved-ports
drwx------ 3 root root 4096 Jul 25 13:45 systemd-private-53fe7d6e64d74c948d4ef833474490d1-chrony.service-bs24T1
drwx------ 3 root root 4096 Jul 25 13:45 systemd-private-53fe7d6e64d74c948d4ef833474490d1-systemd-logind.service-qDlvzK
-rw-r--r-- 1 root root 5368709120 Jul 25 16:56 testfile.txt

root@proliant:/var/tmp# dd if=/dev/zero of=./testfile2.txt bs=4k iflag=fullblock,count_bytes count=5G
1310720+0 records in
1310720+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 5.93413 s, 905 MB/s

root@proliant:/var/tmp# ls -la
total 10485788
drwxrwxrwt 4 root root 4096 Jul 25 16:57 .
drwxr-xr-x 12 root root 4096 Jul 3 20:52 ..
-rw-r--r-- 1 root root 16 Jul 25 13:35 pve-reserved-ports
drwx------ 3 root root 4096 Jul 25 13:45 systemd-private-53fe7d6e64d74c948d4ef833474490d1-chrony.service-bs24T1
drwx------ 3 root root 4096 Jul 25 13:45 systemd-private-53fe7d6e64d74c948d4ef833474490d1-systemd-logind.service-qDlvzK
-rw-r--r-- 1 root root 5368709120 Jul 25 16:57 testfile2.txt
-rw-r--r-- 1 root root 5368709120 Jul 25 16:56 testfile.txt

root@proliant:/var/tmp# df -Th /var/tmp
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/pve-root ext4 94G 15G 75G 17% /

 

[Impact]

This is very hard to read. Please use code blocks and try in another browser.

 

[janus57]

Hi,

what is the return of :

df -Hi /var/tmp && df -Hi

Best regards,

 

[fransesco]

Hi,

what is the return of :

df -Hi /var/tmp && df -Hi

Best regards,

Hi,

Here it comes:

root@proliant:/var/tmp# df -Hi /var/tmp && df -Hi
Filesystem           Inodes IUsed IFree IUse% Mounted on
/dev/mapper/pve-root   6.3M   76k  6.3M    2% /
Filesystem                                 Inodes IUsed IFree IUse% Mounted on
udev                                          17M   851   17M    1% /dev
tmpfs                                         17M  1.6k   17M    1% /run
/dev/mapper/pve-root                         6.3M   76k  6.3M    2% /
tmpfs                                         17M   106   17M    1% /dev/shm
tmpfs                                         17M    26   17M    1% /run/lock
efivarfs                                        0     0     0     - /sys/firmware/efi/efivars
/dev/md0                                      63M    21   63M    1% /mnt/ssdmdraid
/dev/sdb1                                     25M    22   25M    1% /mnt/pve/raid1ssd
/dev/sda2                                       0     0     0     - /boot/efi

 

[fba]

Why do you thinkt the stalling upload is related to /var/tmp? This warning is totally common. Do you have any other messages indicating an issue with this folder?
Try to copy the iso via CLI instead. Local storage uses /var/lib/vz/template/iso to store iso files.

 

[fransesco]

Why do you thinkt the stalling upload is related to /var/tmp? This warning is totally common. Do you have any other messages indicating an issue with this folder?

Because the error message says so. See the screenshot a couple of posts up.

There's no other indication anywhere that would say the filesystem is full. And yes, of course one can transfer the iso using other methods, but I wonder what's wrong with the gui assuming the filesystem in question running out of disk space (i.e from where does it take this info and does it calculate it wrong after lates updates, or something, as an example?). Like said before, iso uploads using Proxmox gui have been working perfectly for ages and now all of a sudden it goes like this.

 

[janus57]

Hi,

What is the symptom ?
What the dev tool says when uploading ?

Note : the message is always displayed, it's not necessarily mean that your file system is full, just a reminder if you’re not supervising your storage.

Best regards,

 

[fransesco]

Hi,

What is the symptom ?

The symptom is that one cannot upload iso image using Proxmox gui because the dialogue box says /var/tmp is full.

What the dev tool says when uploading ?

Dev tool? You mean browser's dev tool or something else? I'm not a developer, so I can't say for sure, but browser's dev tool shows nothing particularly interesting that would point to an error.

 

[fransesco]

Hello everyone. Thanks for suggesting all the good ideas how to troubleshoot this. Surprisingly, the problem seems to have disappear on its own! It's still nagging to ensure /var/tmp has enough space (I guess the "nag" has been there always as someone mentioned but I haven't just noted it), but uploads starts flawlessly. Tried on multiple browsers on multiple clients and anything that failed yesterday, works again. Did absolutely no changes in the meanwhile to the system but tried again on day after.

 

[Impact]

Just out of curiosity, can you check ls -l /sys/class/net/*/device/driver? I want to see if this might be a case of the e1000e NIC causing trouble.
If it happens again I'd also try journalctl -f to follow the logs shortly before uploading.
As for dev tools (chrome as example) you can check the networking tab for network failures and the console for errors/warnings.

 

[fransesco]

Just out of curiosity, can you check ls -l /sys/class/net/*/device/driver? I want to see if this might be a case of the e1000e NIC causing trouble.

Here:

root@proliant:/var/log# ls -l /sys/class/net/*/device/driver
lrwxrwxrwx 1 root root 0 Jul 26 09:37 /sys/class/net/eno1/device/driver -> ../../../../bus/pci/drivers/tg3
lrwxrwxrwx 1 root root 0 Jul 25 17:46 /sys/class/net/eno2/device/driver -> ../../../../bus/pci/drivers/tg3
lrwxrwxrwx 1 root root 0 Jul 25 17:46 /sys/class/net/eno3/device/driver -> ../../../../bus/pci/drivers/tg3
lrwxrwxrwx 1 root root 0 Jul 25 17:46 /sys/class/net/eno49/device/driver -> ../../../../bus/pci/drivers/ixgbe
lrwxrwxrwx 1 root root 0 Jul 25 17:46 /sys/class/net/eno4/device/driver -> ../../../../bus/pci/drivers/tg3
lrwxrwxrwx 1 root root 0 Jul 25 17:46 /sys/class/net/eno50/device/driver -> ../../../../bus/pci/drivers/ixgbe

If it happens again I'd also try journalctl -f to follow the logs shortly before uploading.
As for dev tools (chrome as example) you can check the networking tab for network failures and the console for errors/warnings.

I'll keep this in mind if the problem pops up again. Thanks!