Api dont working in a node of the cluster

[petruzzo]

Hi
one node of ones of my clusters respond error 500 when I try to execute api by comandline.
The first time I known this is when I try to migrate a VM from another cluster on the node, the API respond me error and if, with the same command, only changing the target node all was fine.
Now had samo idea why I had this issue?
Before I reinstall the node anyone can give me an idea how fix?
The log in /var/log/pveproxy dont show errors
thamks

 

[bbgeek17]

Hi @petruzzo,

Can you provide the details:
- actual command line you run with full output, good and bad
- output of "pvecm status"
- output "tail -f /var/log/pveproxy/access.log" during API execution
- output of "journalctl -f " during command execution
- pveversion from across the nodes

please use text with CODE or SPOILER tags

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[petruzzo]

Hi sorry for the delay but I was sick
command

qm remote-migrate 7555 7555 apitoken='Authorization:' 'PVEAPIToken=root@pam!migrazione=',host=cl9kvm4.stellar.pro,port=8006' --target-bridge vmbr0 --target-storage VM-DISK --online

output
596 Connection timed out

Spoiler: pvecm status

Cluster information
-------------------
Name: CL9KVM
Config Version: 6
Transport: knet
Secure auth: on

Quorum information
------------------
Date: Wed Oct 1 10:37:43 2025
Quorum provider: corosync_votequorum
Nodes: 6
Node ID: 0x00000004
Ring ID: 1.c5
Quorate: Yes

Votequorum information
----------------------
Expected votes: 6
Highest expected: 6
Total votes: 6
Quorum: 4
Flags: Quorate

Membership information
----------------------
Nodeid Votes Name
0x00000001 1 10.10.10.225
0x00000002 1 10.10.10.226
0x00000003 1 10.10.10.227
0x00000004 1 10.10.10.228 (local)
0x00000005 1 10.10.10.229
0x00000006 1 10.10.10.230

Spoiler: tail -f /var/log/pveproxy/access.log

::ffff:10.103.252.251 - - [01/10/2025:10:47:05 +0200] "GET / HTTP/1.0" 200 2488
::ffff:10.103.252.251 - - [01/10/2025:10:47:07 +0200] "GET / HTTP/1.0" 200 2488
::ffff:10.103.252.251 - - [01/10/2025:10:47:09 +0200] "GET / HTTP/1.0" 200 2488
::ffff:10.103.252.251 - - [01/10/2025:10:46:11 +0200] "GET / HTTP/1.0" 200 2488
::ffff:10.103.252.251 - - [01/10/2025:10:46:13 +0200] "GET / HTTP/1.0" 200 2488
::ffff:10.103.252.251 - - [01/10/2025:10:46:15 +0200] "GET / HTTP/1.0" 200 2488
::ffff:10.103.252.28 - - [01/10/2025:10:46:16 +0200] "POST /api2/json/access/ticket HTTP/1.1" 200 1483
::ffff:10.103.252.251 - - [01/10/2025:10:46:17 +0200] "GET / HTTP/1.0" 200 2488
::ffff:10.1.253.103 - root@pam!migrazione [01/10/2025:10:46:18 +0200] "GET /api2/json/nodes/localhost/storage?enabled=1 HTTP/1.1" 596 -
::ffff:10.103.252.251 - - [01/10/2025:10:46:19 +0200] "GET / HTTP/1.0" 200 2488
::ffff:10.103.252.251 - - [01/10/2025:10:46:21 +0200] "GET / HTTP/1.0" 200 2488
::ffff:10.103.252.251 - - [01/10/2025:10:46:23 +0200] "GET / HTTP/1.0" 200 2488
::ffff:10.103.252.251 - - [01/10/2025:10:46:25 +0200] "GET / HTTP/1.0" 200 2488
::ffff:10.103.252.251 - - [01/10/2025:10:46:27 +0200] "GET / HTTP/1.0" 200 2488
::ffff:10.103.252.251 - - [01/10/2025:10:46:29 +0200] "GET / HTTP/1.0" 200 2488
::ffff:10.103.252.251 - - [01/10/2025:10:46:31 +0200] "GET / HTTP/1.0" 200 2488

Spoiler: journalctl -f

Oct 01 10:45:45 cl9kvm4 systemd[1]: Stopping user-runtime-dir@0.service - User Runtime Directory /run/user/0...
Oct 01 10:45:45 cl9kvm4 systemd[1]: run-user-0.mount: Deactivated successfully.
Oct 01 10:45:45 cl9kvm4 systemd[1]: user-runtime-dir@0.service: Deactivated successfully.
Oct 01 10:45:45 cl9kvm4 systemd[1]: Stopped user-runtime-dir@0.service - User Runtime Directory /run/user/0.
Oct 01 10:45:45 cl9kvm4 systemd[1]: Removed slice user-0.slice - User Slice of UID 0.
Oct 01 10:46:01 cl9kvm4 pmxcfs[1659]: [status] notice: received log
Oct 01 10:46:01 cl9kvm4 pmxcfs[1659]: [status] notice: received log
Oct 01 10:46:02 cl9kvm4 pmxcfs[1659]: [status] notice: received log
Oct 01 10:46:02 cl9kvm4 pmxcfs[1659]: [status] notice: received log
Oct 01 10:46:02 cl9kvm4 pmxcfs[1659]: [status] notice: received log
Oct 01 10:46:04 cl9kvm4 pmxcfs[1659]: [status] notice: received log
Oct 01 10:46:07 cl9kvm4 pvestatd[2315]: status update time (39.043 seconds)
Oct 01 10:46:16 cl9kvm4 pmxcfs[1659]: [status] notice: received log
Oct 01 10:46:16 cl9kvm4 pmxcfs[1659]: [status] notice: received log
Oct 01 10:46:16 cl9kvm4 pmxcfs[1659]: [status] notice: received log
Oct 01 10:46:16 cl9kvm4 pvedaemon[2359]: <root@pam> successful auth for user 'sygma-api@pve'
Oct 01 10:46:17 cl9kvm4 pmxcfs[1659]: [status] notice: received log
Oct 01 10:46:17 cl9kvm4 pmxcfs[1659]: [status] notice: received log
Oct 01 10:46:17 cl9kvm4 pmxcfs[1659]: [status] notice: received log

Spoiler: pveversion

cl9kvm1:/# pveversion
pve-manager/8.2.2/9355359cd7afbae4 (running kernel: 6.5.13-6-pve)
cl9kvm2:/# pveversion
pve-manager/8.2.2/9355359cd7afbae4 (running kernel: 6.5.13-6-pve)
cl9kvm3:/# pveversion
pve-manager/8.2.2/9355359cd7afbae4 (running kernel: 6.5.13-6-pve)
cl9kvm4:/# pveversion
pve-manager/8.2.2/9355359cd7afbae4 (running kernel: 6.5.13-6-pve)
cl9kvm5:~# pveversion
pve-manager/8.2.2/9355359cd7afbae4 (running kernel: 6.5.13-6-pve)