API Internal Server Error [500], but requests complete?

[s_sasha]

Hi everyone,

We're in need for some help here please.

In short, some of the API requests return with 500, but complete anyway.

We're on proxmox-ve: 6.2-1

There are no errors in /var/log/syslog in fact here is one of the successful attempts:
Jan 7 15:47:45 xxx.xxx.xxx pvedaemon[42603]: <root@pam> update VM 0000: resize --disk scsi1 --size +700G

API endpoint request details:
/nodes/xxx.xxx.xxx/qemu/0000/resize


The issue is intermittent, but causes lots of trouble in this particular case as users keep on resizing vm storage having impression that it failed.

It would be very helpful to know how else can we debug this? I did not find any log files specific to API and there isn't anything helpful in /var/log/syslog nor /var/log/pve/tasks

Thanks!

 

[Moayad]

Hi,

Could you post the request for the resized disk?

 

[s_sasha]

Hi,

Could you post the request for the resized disk?

Hi, thanks for the prompt reply.

Endpoint: /nodes/xxx.xxx.xxx/qemu/0000/resize

Request data: {:disk \"scsi0\" :size \"+10G\"}

Request type: PUT

 

[fabian]

the resize likely takes longer than the API request timeout, but as it has already been forwarded to the VM/storage layer, it will still proceed. what kind of storage is the disk on? how is the load situation when requests "fail"?

maybe we need to provide an async API endpoint as well (that spawns a real task worker that you can then poll for status/completion)...

 

[s_sasha]

That makes sense, to answer your questions

1) Load isn't any different between when requests are successful and when they fail and general node load on average is not higher than 49%
2) ZFS

Yeah, think async endpoints would solve this. Any work around you could suggest in the interim?

 

[fabian]

no workaround unfortunately (except doing those resizes on the CLI where there is no timeout, but I assume your customers don't get that level of access to the PVE system )

I just updated https://bugzilla.proxmox.com/show_bug.cgi?id=2315 , feel free to subscribe there to get progress updates! shouldn't be too hard to fix..

 

[s_sasha]

No problem, thanks - I'll monitor the thread

 

[SagnikS]

Heya, we just encountered this too. It has been a fair bit of time, any idea when we can expect to get a patch?