[SOLVED] PveProxy : No space left on device - problem with storage api

cakeoats

Member
Oct 30, 2021
12
2
8
35
Hello!

I'm encountering an error with Proxmox API that does not allow me to upload files using API due to error thrown by PveProxy.
I did a fresh installation of Proxmox 7.2 using VM in Hyper-V with fixed disk size of 128gb and gen 2 VM. Proxmox is installed with LVM.
From my local syslog I can see this error:Aug 10 05:46:52 pvelab pveproxy[8637]: problem with client ::ffff:192.168.10.112; No space left on device

Looking for any help I can get troubleshooting this strange error!!

I have plenty of space. The iso is only 600M.
Code:
root@pvelab:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
udev                  3.9G     0  3.9G   0% /dev
tmpfs                 795M  648K  795M   1% /run
/dev/mapper/pve-root   31G  4.3G   26G  15% /
tmpfs                 3.9G   34M  3.9G   1% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
/dev/sda2             511M  328K  511M   1% /boot/efi
/dev/fuse             128M   16K  128M   1% /etc/pve
tmpfs                 795M     0  795M   0% /run/user/0

Memory status:
Code:
free -h
               total        used        free      shared  buff/cache   available
Mem:           7.8Gi       831Mi       6.5Gi        34Mi       418Mi       6.7Gi
Swap:          4.0Gi          0B       4.0Gi
root@pvelab:~#

Status of `local`-storage:

Code:
root@pvelab:~# pvesh get /storage/local/ --output-format=json
{"content":"snippets,iso,vztmpl,backup,rootdir","digest":"5820b6fbff46e80cc2d46da13d7c9354673bed22","path":"/var/lib/vz","shared":0,"storage":"local","type":"dir"}

root@pvelab:~# df -h /var/lib/vz
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/pve-root   31G  4.3G   26G  15% /

Client log for upload:
Code:
curl -k --verbose --cookie "$(<cookie)" --upload-file ./jammy-server-cloudimg-amd64.img https://192.168.10.113:8006/api2/json/nodes/pve/storage/local/upload
*   Trying 192.168.10.113:8006...
* Connected to 192.168.10.113 (192.168.10.113) port 8006 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: OU=PVE Cluster Node; O=Proxmox Virtual Environment; CN=pvelab
*  start date: Aug  8 14:55:38 2022 GMT
*  expire date: Aug  7 14:55:38 2024 GMT
*  issuer: CN=Proxmox Virtual Environment; OU=43f3a941-0e87-44c1-8d0e-2febbe8b46ca; O=PVE Cluster Manager CA
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> PUT /api2/json/nodes/pve/storage/local/upload HTTP/1.1
> Host: 192.168.10.113:8006
> User-Agent: curl/7.81.0
> Accept: */*
> Cookie: PVEAuthCookie=PVE:root@pam:62F326A5::0ZtYc7iOnjWJEuAaTiVdUFnRPxcd4Bp15W5AqEtD3NDcyeqlncsXfAJYjaJuxJ9H8eX2+TQZvYzkwbzRieEmGJ9AKf+yH4YiUC8wYyoC7wRN/Lti1j4DL2+dCnnjCThw1jsg7zZD6EAssjSDOdT8EZLxShV3IH0wPSCxxLtdSKn+3ug2WkbXxfRZ9XC9LelyFEoKIvlrlt1sZdok2rtj2cc0xMHoR/MfYxQfjOZbt2krivPdKEM/2Bb7rG4VzCKgg7ikY0stOXBYNfb0sZ70+2K8vnD7Y89HMZvcB5f2OFczKNNo8xF9T7wbNG7Fg/JbU34IR0IKDBPlITHuLevM6A==
> Content-Length: 630063104
> Expect: 100-continue
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Done waiting for 100-continue
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS alert, decode error (562):
* OpenSSL SSL_read: error:0A000126:SSL routines::unexpected eof while reading, errno 0
* Closing connection 0
curl: (56) OpenSSL SSL_read: error:0A000126:SSL routines::unexpected eof while reading, errno 0


Server logs from pveproxy (debug does not output the error??!)
Code:
pveproxy start --debug 1
worker[8637]: PVE::APIServer::AnyEvent +1731: (eval): ACCEPT FH10 CONN1
Use of uninitialized value $token in pattern match (m//) at /usr/share/perl5/PVE/AccessControl.pm line 336.
Use of uninitialized value $token in pattern match (m//) at /usr/share/perl5/PVE/Ticket.pm line 32.
worker[8637]: PVE::APIServer::AnyEvent +1781: client_do_disconnect: close connection AnyEvent::Handle=HASH(0x5589b5fce8c8)
worker[8637]: PVE::APIServer::AnyEvent +1781: client_do_disconnect: CLOSE FH10 CONN0

pveversion:
Code:
pveversion -v
proxmox-ve: 7.2-1 (running kernel: 5.15.39-3-pve)
pve-manager: 7.2-7 (running version: 7.2-7/d0dd0e85)
pve-kernel-5.15: 7.2-8
pve-kernel-helper: 7.2-8
pve-kernel-5.15.39-3-pve: 5.15.39-3
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph-fuse: 15.2.16-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-3
libpve-storage-perl: 7.2-7
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.5-1
proxmox-backup-file-restore: 2.2.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-2
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.5-1
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 6.2.0-11
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.5-pve1
 
Last edited:
Update: I noticed that I tried to upload to node pve when in fact it should've been pvelab
Fixing the URL I still get the same error.. Correct url with correct node:
https://192.168.10.113:8006/api2/json/nodes/pvelab/storage/local/upload

Error:
Aug 10 18:09:29 pvelab pveproxy[950]: problem with client ::ffff:192.168.10.112; No space left on device

EDIT:
I generated a test file of 16KB and still see the same error
Code:
dd if=/dev/urandom of=test.iso bs=16k count=1
1+0 records in
1+0 records out
16384 bytes (16 kB, 16 KiB) copied, 0.000545609 s, 30.0 MB/s

$ curl -k --verbose --cookie "$(<cookie)" -X POST -H "Content-type: multipart/form-data" --form "content=iso" --form "filename=@test.iso" https://192.168.10.113:8006/api2/json/nodes/pvelab71/storage/local/upload
bash: cookie: No such file or directory
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 192.168.10.113:8006...
* Connected to 192.168.10.113 (192.168.10.113) port 8006 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: OU=PVE Cluster Node; O=Proxmox Virtual Environment; CN=pvelab
*  start date: Aug  8 14:55:38 2022 GMT
*  expire date: Aug  7 14:55:38 2024 GMT
*  issuer: CN=Proxmox Virtual Environment; OU=43f3a941-0e87-44c1-8d0e-2febbe8b46ca; O=PVE Cluster Manager CA
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> POST /api2/json/nodes/pvelab71/storage/local/upload HTTP/1.1
> Host: 192.168.10.113:8006
> User-Agent: curl/7.81.0
> Accept: */*
> Content-Length: 16687
> Content-Type: multipart/form-data; boundary=------------------------e960e34dffd6d4a8
>
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* We are completely uploaded and fine
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 No ticket
< Cache-Control: max-age=0
< Connection: close
< Date: Wed, 10 Aug 2022 16:15:01 GMT
< Pragma: no-cache
< Server: pve-api-daemon/3.0
< Expires: Wed, 10 Aug 2022 16:15:01 GMT
<
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS alert, decode error (562):
* OpenSSL SSL_read: error:0A000126:SSL routines::unexpected eof while reading, errno 0
* Closing connection 0
curl: (56) OpenSSL SSL_read: error:0A000126:SSL routines::unexpected eof while reading, errno 0
 
Last edited:
the error says that you dont have space on your _client_.
The log shows you are connecting to proxmox at IP 113, while your client 192.168.10.112 is out of space. Perhaps it needs some temp space to encode the file , and you dont have it?


Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Same error on Windows as well, so that's not it!

Also, the error I'm seeing from syslog is on Proxmox :)

Code:
curl -k --verbose --cookie "$(gc cookie)" --upload-file test.iso https://192.168.10.113:8006/api2/json/nodes/pvelab/storage/local/upload
*   Trying 192.168.10.113:8006...
* Connected to 192.168.10.113 (192.168.10.113) port 8006 (#0)
* schannel: disabled automatic use of client certificate
* schannel: using IP address, SNI is not supported by OS.
* ALPN: offers http/1.1
* ALPN: server did not agree on a protocol. Uses default.
> PUT /api2/json/nodes/pvelab/storage/local/upload HTTP/1.1
> Host: 192.168.10.113:8006
> User-Agent: curl/7.83.1
> Accept: */*
> Cookie: PVEAuthCookie=PVE:root@pam:62F3E503::ZFxzH2c95eEuXsxB1MCb3Tm+o67yulBINtTxGaNyOPOHkknXcLYkGKoJEuAeVWrjdnt0jDShETK1iYREOVS4tIxg2paatNr757JqplW6ysKDZC6XVNjXO0dXnCGmo6cjRE+3qBgMi014DiDvo2Sx5u1Ho1CmSL06rYyzfHplpoBlUguATYTSaB21YZ6DEO0SSc4D9lXtkG0+S0ILpA2Afc4vUDDBUVfZAeEtqkrnTsh6njdep93R54os/Ge2H3RrGGDNqBDKcU4VxjQYe5FnNOxcbzYhQns+3PyyE+w3/WMnffkGdrsxo82Rt6VsGcco+7lo9HqG4domaBlPROhb7g==
> Content-Length: 1048576
> Expect: 100-continue
>
* Done waiting for 100-continue
* schannel: server closed abruptly (missing close_notify)
* Closing connection 0
* schannel: shutting down SSL/TLS connection with 192.168.10.113 port 8006
* Send failure: Connection was aborted
* schannel: failed to send close msg: Failed sending data to the peer (bytes written: -1)
curl: (56) Send failure: Connection was aborted

problem with client ::ffff:192.168.10.112; No space left on device
 
192.168.10.112 is client (uploader), 192.168.10.113 is server (proxmox)

EDIT:
I didn't pay much attention to how curl handles upload-file, it's PUT request.... so I switched to POST, and I still see the same error:
problem with client ::ffff:192.168.10.112; No space left on device

I think I'm doing something wrong with curl. I tried with HTTPie and it works.

CURL ERROR:
Code:
curl -k --verbose --cookie "PVEAuthCookie=MYVERYLONGHIDDENCOOKIE" -F "filename=@test.iso" -F "content=iso" https://192.168.10.113:8006/api2/json/nodes/pvelab/storage/local/upload
*   Trying 192.168.10.113:8006...
* Connected to 192.168.10.113 (192.168.10.113) port 8006 (#0)
* schannel: disabled automatic use of client certificate
* schannel: using IP address, SNI is not supported by OS.
* ALPN: offers http/1.1
* ALPN: server did not agree on a protocol. Uses default.
> POST /api2/json/nodes/pvelab/storage/local/upload HTTP/1.1
> Host: 192.168.10.113:8006
> User-Agent: curl/7.83.1
> Accept: */*
> Cookie: PVEAuthCookie=PVE:root@pam:62F3E503::ZFxzH2c95eEuXsxB1MCb3Tm+o67yulBINtTxGaNyOPOHkknXcLYkGKoJEuAeVWrjdnt0jDShETK1iYREOVS4tIxg2paatNr757JqplW6ysKDZC6XVNjXO0dXnCGmo6cjRE+3qBgMi014DiDvo2Sx5u1Ho1CmSL06rYyzfHplpoBlUguATYTSaB21YZ6DEO0SSc4D9lXtkG0+S0ILpA2Afc4vUDDBUVfZAeEtqkrnTsh6njdep93R54os/Ge2H3RrGGDNqBDKcU4VxjQYe5FnNOxcbzYhQns+3PyyE+w3/WMnffkGdrsxo82Rt6VsGcco+7lo9HqG4domaBlPROhb7g==
> Content-Length: 1048879
> Content-Type: multipart/form-data; boundary=------------------------6ab0052296932e93
> Expect: 100-continue
>
* Done waiting for 100-continue
* We are completely uploaded and fine
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 Parameter verification failed.
< Cache-Control: max-age=0
< Connection: close
< Date: Wed, 10 Aug 2022 18:16:09 GMT
< Pragma: no-cache
< Server: pve-api-daemon/3.0
< Content-Length: 79
< Content-Type: application/json;charset=UTF-8
< Expires: Wed, 10 Aug 2022 18:16:09 GMT
<
{"data":null,"errors":{"content":"property is missing and it is not optional"}}* Closing connection 0
* schannel: shutting down SSL/TLS connection with 192.168.10.113 port 8006


HTTPie OK:
Code:
http  --verbose  --verify=no -f POST https://192.168.10.113:8006/api2/json/nodes/pvelab/storage/local/upload "Cookie:PVEAuthCookie=MYVERYLONGHIDDENCOOKIE" content=iso filename@test.iso

POST /api2/json/nodes/pvelab/storage/local/upload HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 1048815
Content-Type: multipart/form-data; boundary=b11bd780056d480a93ce88d89d0fc4be
Cookie: PVEAuthCookie=PVE:root@pam:62F3E503::ZFxzH2c95eEuXsxB1MCb3Tm+o67yulBINtTxGaNyOPOHkknXcLYkGKoJEuAeVWrjdnt0jDShETK1iYREOVS4tIxg2paatNr757JqplW6ysKDZC6XVNjXO0dXnCGmo6cjRE+3qBgMi014DiDvo2Sx5u1Ho1CmSL06rYyzfHplpoBlUguATYTSaB21YZ6DEO0SSc4D9lXtkG0+S0ILpA2Afc4vUDDBUVfZAeEtqkrnTsh6njdep93R54os/Ge2H3RrGGDNqBDKcU4VxjQYe5FnNOxcbzYhQns+3PyyE+w3/WMnffkGdrsxo82Rt6VsGcco+7lo9HqG4domaBlPROhb7g==
Host: 192.168.10.113:8006
User-Agent: HTTPie/3.2.1

+-----------------------------------------+
| NOTE: binary data not shown in terminal |
+-----------------------------------------+

HTTP/1.1 200 OK
Cache-Control: max-age=0
Connection: close
Content-Length: 68
Content-Type: application/json;charset=UTF-8
Date: Wed, 10 Aug 2022 18:12:26 GMT
Expires: Wed, 10 Aug 2022 18:12:26 GMT
Pragma: no-cache
Server: pve-api-daemon/3.0

{
    "data": "UPID:pvelab:000046BF:000B6882:62F3F50A:imgcopy::root@pam:"
}

EDIT:
Added HTTPie test that works and made the output verbose.
 
Last edited:
The reason for starting this thread is because I'm using Proxmmox Packer plugin to download template from URL and this is where I first encountered the error, which made think there was a proxmox issue and I started testing with curl.
I noticed that if I use the packer plugin with along with a pre-defined token I encounter this error, see output from packer below.

I never encountered this error if I use authentication with username + password (I literally just tested this) and I wasn't aware of this working before now. So I think the issue with the packer plugin, as proper httpie request work then testing locally.

THREAD CAN BE CLOSED. NOT A PROXMOX ISSUE


Packer output with token as authentication failing
Code:
proxmox.ubuntu-server-jammy: output will be in this color.

==> proxmox.ubuntu-server-jammy: Retrieving ISO
==> proxmox.ubuntu-server-jammy: Trying https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img
==> proxmox.ubuntu-server-jammy: Trying https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img?checksum=sha256%3A5ba6b098d2e7b2c4b572812c1df5943187780b64aaa673a0ffde323adba8594b
==> proxmox.ubuntu-server-jammy: https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img?checksum=sha256%3A5ba6b098d2e7b2c4b572812c1df5943187780b64aaa673a0ffde323adba8594b => /home/mikae/.cache/packer/a0e0f623f60b007055b48b832ee127198c046372.iso
==> proxmox.ubuntu-server-jammy: Pausing after run of step 'StepDownload'. Press enter to continue. 
==> proxmox.ubuntu-server-jammy: Post "https://192.168.10.113:8006/api2/json/nodes/pvelab/storage/data/upload": write tcp 172.30.246.10:51110->192.168.10.113:8006: use of closed network connection
==> proxmox.ubuntu-server-jammy: Pausing before cleanup of step 'StepDownload'. Press enter to continue.
 
Hi @cakeoats,

I'm also using Packer and I encouter the same issue. After reading some threads, I think this really is a Proxmox issue.
Please have a look to the bug 4344 on the Proxmox bug tracker : https://bugzilla.proxmox.com/show_bug.cgi?id=4344
And other threads on this forum :
 
  • Like
Reactions: cakeoats