storage REST API error: No space left on device

[chacha]

Hi,

Using proxmoxer I am trying to upload lxc template to my hypervisor.
It used to work just fine but since a few weeks (or month) it fails.

On proxmoxer side it say:
Remote end closed connection without response

Then I checked on hypervisor side:
In /var/log/syslog :
Oct 30 22:01:55 hypervisor pveproxy[2126784]: problem with client ::ffff:XXX.XXX.XXX.XXX; No space left on device
In /var/log/pveproxy/ :
::ffff:XXX.XXX.XXX.XXX- dabfactory-bot@pam!XXXXXXXXXXXX[30/10/2022:22:04:34 +0100] "POST /api2/json/nodes/hypervisor/storage/WORK/upload HTTP/1.1" 500 -

There is plenty of space:
Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur
udev 16G 0 16G 0% /dev
tmpfs 3,2G 13M 3,2G 1% /run
/dev/mapper/pve-root 28G 13G 14G 50% /
tmpfs 16G 49M 16G 1% /dev/shm
tmpfs 5,0M 4,0K 5,0M 1% /run/lock
/dev/mapper/pve-data 65G 17G 49G 26% /var/lib/vz
/dev/sdc1 4,6T 4,2T 374G 92% /mnt/BRICK2
/dev/sdb1 4,6T 4,3T 345G 93% /mnt/BRICK1
/dev/fuse 128M 24K 128M 1% /etc/pve
tmpfs 3,2G 32K 3,2G 1% /run/user/0
localhost:TMP_FAST 100G 0 100G 0% /media/TMP_FAST
localhost:ARRAY 11T 11T 752G 94% /media/ARRAY
localhost:WORK 3,7T 3,4T 252G 94% /media/WORK

What I have try:
* Differents users: root@pam, or an authentification token, does not change anything.
* Tried with 2 differents storages, also not change anything
* Restarting the Node, also not change anything
* Other command like Str.status.get() work fine, for instance:
{'shared': 0, 'total': 3998318010368, 'active': 1, 'type': 'dir', 'content': 'vztmpl,rootdir', 'used': 3727743836160, 'enabled': 1, 'avail': 270574174208}
* It is working well using the proxmox WebGUI...

Other info:
proxmox VE version: 7.2-11
proxmoxer version: 1.3.1


EDIT 1:
test using pveproxy start --debug=1 :
worker[2323338]: PVE::APIServer::AnyEvent +1714: (eval): ACCEPT FH10 CONN0
worker[2323338]: PVE::APIServer::AnyEvent +1325: (eval): start upload /api2/json/nodes/hypervisor/storage/local/upload multipart/form-data e4e26ace364a4f9488cf9a9f06dbc860
worker[2323338]: PVE::APIServer::AnyEvent +1764: client_do_disconnect: close connection AnyEvent::Handle=HASH(0x559afd76bfd8)
worker[2323338]: PVE::APIServer::AnyEvent +1764: client_do_disconnect: CLOSE FH10 CONN0

Compared to through webGUI:
(...)
worker[2326306]: PVE::APIServer::AnyEvent +1325: (eval): start upload /api2/json/nodes/hypervisor/storage/local/upload multipart/form-data ---------------------------221127871937183401032155461004
(...)

EDIT2:
Seems to work using low-level API (requests + requests_toolbelt)....
So not a Proxmox issue, will make a ticket on proxmoxer-side.. !
Ticket

Working sample code for others interest:

import requests
from requests_toolbelt.multipart.encoder import MultipartEncoder

SERV_ADDR="https://XXX.XXX.XXX.XXX:8006"
NODE_NAME="XXXX"
STORAGE_NAME="XXXX"
FILENAME="XXXX"
FILEPATH=FILENAME #if same dir as script


mp_encoder = MultipartEncoder(
    fields={
        'filename'    : (FILENAME, open(FILEPATH, 'rb'), 'text/plain'),
    }
)

data = {
        'username'  : "root@pam",
        'password'  : "XXXXXXXXXX",
}

res = requests.post(    SERV_ADDR + '/api2/json/access/ticket',
                data = data,
                verify  =False)
ticket = res.json()["data"]["ticket"]
print(ticket)
CSRFPreventionToken = res.json()["data"]["CSRFPreventionToken"]
print(CSRFPreventionToken)

headers={
        'CSRFPreventionToken' : CSRFPreventionToken,
        'Content-Type': mp_encoder.content_type
 }
 
cookies={
    'PVEAuthCookie':ticket
    }

res = requests.post(    SERV_ADDR + '/api2/json/nodes/'+NODE_NAME+'/storage/'+STORAGE_NAME+'/upload',
                headers = headers,
                cookies = cookies,
                data    = mp_encoder,
                verify  =False)
print(res.json())

 

[jhollowe]

It seems like this is related to this forum thread: https://forum.proxmox.com/threads/pveproxy-no-space-left-on-device-problem-with-storage-api.113451/

After some testing, it appears this error appears somewhere between 7.2-7 (works correctly) and 7.2-11 (broken while using same code as -7)

 

[jhollowe]

I've done a packet capture against 7.1-7 (the latest ISO I had on hand) and 7.2-11 (same PVE node updated to latest). From what I can tell there is nothing wrong on the client side (the upload is the exact same request with different auth and multipart boundary). The request goes through fine and then hits the client_do_disconnect: close connection AnyEvent::Handle in the pveproxy which then sends an RST to the client.




Have there been any changes to how uploads work recently?

(7.1-7 zip parts just need to be cat-ed together to form the whole zip; working around file upload size limits, or just use captures.zip for both)

 

[chacha]

Sorry for late answer, I raised a ticket on proxmoxer GITHUB:
Issue 116

It appears that the problem is linked to Multipart transfert that now require 'text/plain' option.
Adding this field in the MultipartEncoder dict fixes the issue, PR not approved yet.

EDIT: did not figure out you were the same person

 

[jhollowe]

I've created bug 4344 and submitted a patch to the mailing list.