[SOLVED] No space left on device - Disk not full

[johninjax]

I am trying to upload an ISO using the API and it always fails with: Empty reply from server
I've done this from multiple server including from the shell in PVE. Always the same result.

From this:

# Create a small test file
dd if=/dev/zero of=test.iso bs=1M count=10

# Try uploading via API
curl -X POST \
  -H "Authorization: PVEAPIToken=image-builder@pve\!capi=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" \
  -v \
  -F "content=iso" \
  -F "filename=@test.iso" \
  "https://10.0.0.10:8006/api2/json/nodes/pve.lab/storage/local/upload"

When I do that. I see this in the system log. The only difference is the IP of the computer that called PVE. example:
Sep 28 17:47:30 pve pveproxy[1666]: problem with client ::ffff:10.0.0.124; No space left on device

I've also tried looking for what could be full. I am lost here. Does anyone have a next step for troubleshooting this?

root@pve:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
udev                   46G     0   46G   0% /dev
tmpfs                 9.2G  3.1M  9.2G   1% /run
/dev/mapper/pve-root   94G  8.1G   82G  10% /
tmpfs                  46G   43M   46G   1% /dev/shm
efivarfs              128K   49K   75K  40% /sys/firmware/efi/efivars
tmpfs                 5.0M     0  5.0M   0% /run/lock
tmpfs                  46G     0   46G   0% /tmp
/dev/nvme0n1p2       1022M  8.8M 1014M   1% /boot/efi
/dev/fuse             128M   20K  128M   1% /etc/pve
tmpfs                 1.0M     0  1.0M   0% /run/credentials/getty@tty1.service
tmpfs                 1.0M     0  1.0M   0% /run/credentials/systemd-journald.service
/dev/loop0             51M   51M     0 100% /snap/snapd/25202
/dev/loop1             74M   74M     0 100% /snap/core22/2133
/dev/loop2            7.8M  7.8M     0 100% /snap/yq/2726
tmpfs                 9.2G  8.0K  9.2G   1% /run/user/0

root@pve:~# du -h -d1 -x /
3.8G    /usr
191M    /boot
4.0K    /home
4.0K    /srv
3.6G    /var
4.0K    /mnt
8.4M    /etc
16K     /lost+found
601M    /root
4.0K    /media
24K     /snap
4.0K    /opt
8.1G    /

root@pve:~# lsblk
NAME                            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
loop0                             7:0    0 50.8M  1 loop /snap/snapd/25202
loop1                             7:1    0 73.9M  1 loop /snap/core22/2133
loop2                             7:2    0  7.7M  1 loop /snap/yq/2726
nvme0n1                         259:0    0  1.8T  0 disk
├─nvme0n1p1                     259:1    0 1007K  0 part
├─nvme0n1p2                     259:2    0    1G  0 part /boot/efi
└─nvme0n1p3                     259:3    0  1.8T  0 part
  ├─pve-swap                    252:0    0    8G  0 lvm  [SWAP]
  ├─pve-root                    252:1    0   96G  0 lvm  /
  ├─pve-data_tmeta              252:2    0 15.9G  0 lvm
  │ └─pve-data-tpool            252:4    0  1.7T  0 lvm
  │   ├─pve-data                252:5    0  1.7T  1 lvm
  │   ├─pve-base--9999--disk--0 252:6    0   10G  1 lvm
  │   ├─pve-vm--9999--cloudinit 252:7    0    4M  0 lvm
  │   ├─pve-vm--777--cloudinit  252:8    0    4M  0 lvm
  │   ├─pve-vm--777--disk--0    252:9    0   10G  0 lvm
  │   ├─pve-vm--100--cloudinit  252:24   0    4M  0 lvm
  │   └─pve-vm--100--disk--0    252:25   0  100G  0 lvm
  └─pve-data_tdata              252:3    0  1.7T  0 lvm
    └─pve-data-tpool            252:4    0  1.7T  0 lvm
      ├─pve-data                252:5    0  1.7T  1 lvm
      ├─pve-base--9999--disk--0 252:6    0   10G  1 lvm
      ├─pve-vm--9999--cloudinit 252:7    0    4M  0 lvm
      ├─pve-vm--777--cloudinit  252:8    0    4M  0 lvm
      ├─pve-vm--777--disk--0    252:9    0   10G  0 lvm
      ├─pve-vm--100--cloudinit  252:24   0    4M  0 lvm
      └─pve-vm--100--disk--0    252:25   0  100G  0 lvm

 

[fiona]

Hi,
please share the full output from the curl command and the full excerpt from the system journal around the time the issue happens. Anything noticeable in /var/log/pveproxy/access.log? Was your installation made on top of Debian or directly from the Proxmox VE ISO? Please also share the version pveversion -v. What does df -i say?

 

[johninjax]

Hi,
please share the full output from the curl command and the full excerpt from the system journal around the time the issue happens. Anything noticeable in /var/log/pveproxy/access.log? Was your installation made on top of Debian or directly from the Proxmox VE ISO? Please also share the version pveversion -v. What does df -i say?

Thank you for the response. I figured it out.

There were 2 problems. 1. The header wasn't working until I changed to single quotes and unescaped the !, i.e.
"Authorization: PVEAPIToken=image-builder@pve\!capi=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
'Authorization: PVEAPIToken=image-builder@pve!capi=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
2.
Wrong:
/api2/json/nodes/pve.lab/storage/local/upload
Right:
/api2/json/nodes/pve/storage/local/upload

 

[fiona]

Thank you for sharing the solution! You can edit the thread and select the [SOLVED] prefix to help other users find solutions more quickly.