API problems 596 ERROR

[fokatoka]

For a month now proxmox has been working and the API access just CONSTANTLY hangs, that is, it is not there! To make everything work, restart helps, then he just decided to "fall off" and no longer work, attempts to restart pveproxy do nothing at all.

this is the log /var/log/pveproxy/access.log
::ffff:12.12.12.12 - root@pam [24/12/2021:20:44:31 +0300] "GET /api2/json/nodes/node/lxc/112/status/current HTTP/1.1" 596 -
::ffff:12.12.12.12 - root@pam [24/12/2021:20:44:31 +0300] "GET /api2/json/nodes/node/lxc/109/status/current HTTP/1.1" 596 -
::ffff:12.12.12.12 - root@pam [24/12/2021:20:45:02 +0300] "GET /api2/json/nodes/node/lxc/112/status/current HTTP/1.1" 596 -
::ffff:12.12.12.12 - root@pam [24/12/2021:20:45:02 +0300] "GET /api2/json/nodes/node/lxc/146/rrddata?timeframe=hour&cf=AVERAGE HTTP/1.1" 596 -
::ffff:13.13.13.13 - root@pam [24/12/2021:20:45:19 +0300] "GET /api2/json/nodes/node/lxc/112/status/current HTTP/1.1" 596 -
::ffff:12.12.12.12 - root@pam [24/12/2021:20:46:35 +0300] "GET /api2/json/nodes/node/lxc/112/status/current HTTP/1.1" 596 -
::ffff:13.13.13.13 - root@pam [24/12/2021:20:47:21 +0300] "GET /api2/json/nodes/node/lxc/112/status/current HTTP/1.1" 596 -

why the timeout? I do not understand anything at all, after restarting everything works, then it stops at any time.

while choosing another lxc container, I can get a response without errors (any other container, on the same server and node)

::ffff:12.12.12.12 - root@pam [24/12/2021:20:52:14 +0300] "GET /api2/json/nodes/node/lxc/143/status/current HTTP/1.1" 200 374
::ffff:12.12.12.12 - root@pam [24/12/2021:20:52:14 +0300] "GET /api2/json/nodes/node/lxc/143/rrddata?timeframe=hour&cf=AVERAGE HTTP/1.1" 200 1630


6.4-13

 

[Benjamin Hodgens]

Have you checked syslog for the associated/related failure is there? There should be some indication as to the underlying cause in that, or some other, log which could be used for correlation.

 

[fokatoka]

there is nothing.

the commands are also executed for a very long time.

time lxc-info -n 126 -p
PID: 30600

real 0m30,263s
user 0m0,003s
sys 0m0,000s

why is it taking so long?
I have no problems with the disk system, memory or processor, everything is practically no load.

help pls.

 

[oguz]

hi,

please post:

6.4-13

* full output from pveversion -v

there is nothing.

* and also please check journalctl -u pve* -b0

* do you happen to have a reverse proxy for the web GUI?

 

[fokatoka]

Spoiler: pveversion -v

proxmox-ve: 6.4-1 (running kernel: 5.4.140-1-pve)
pve-manager: 6.4-13 (running version: 6.4-13/9f411e79)
pve-kernel-5.4: 6.4-6
pve-kernel-helper: 6.4-6
pve-kernel-5.4.140-1-pve: 5.4.140-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.1.0-1
libpve-access-control: 6.4-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-3
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-3
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.13-2
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.6-1
pve-cluster: 6.4-1
pve-container: 3.3-6
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-4
pve-firmware: 3.3-1
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.5-pve1~bpo10+1

Spoiler: journalctl -u pve* -b0

https://transfer.sh/vMFGzx/jrn.txt

* do you happen to have a reverse proxy for the web GUI?

no

I want to note that the problem does not apply to all containers

Spoiler: time lxc-info -n 125

Name: 125
State: RUNNING
PID: 28613
IP: 212.76.129.204
CPU use: 179104.12 seconds
BlkIO use: 0 bytes
Memory use: 5.09 GiB
KMem use: 57.25 MiB
Link: veth125i0
TX bytes: 369.60 MiB
RX bytes: 382.91 MiB
Total bytes: 752.51 MiB

real 0m0,006s
user 0m0,006s
sys 0m0,000s

Spoiler: time lxc-info -n 126

Name: 126
State: RUNNING
PID: 4088
IP: 212.76.129.214
CPU use: 6.47 seconds
BlkIO use: 0 bytes
Memory use: 71.65 MiB
KMem use: 9.14 MiB
Link: veth126i0
TX bytes: 92.56 KiB
RX bytes: 2.12 MiB
Total bytes: 2.21 MiB

real 0m24,762s
user 0m0,006s
sys 0m0,000s

and on the problem container, pct enter also works for a very long time, even the ssh server does not start (although after rebooting the entire node everything is ok), well, the graphs are drawn in "fragments"

 

[oguz]

do you have an api client running somewhere?

in your log i see a lot of spam like:

янв 06 17:44:10 lc1 pvedaemon[31714]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:11 lc1 pvedaemon[2057]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:11 lc1 pvedaemon[20939]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:11 lc1 pvedaemon[2057]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:11 lc1 pvedaemon[2057]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:11 lc1 pvedaemon[31714]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:11 lc1 pvedaemon[31714]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:11 lc1 pvedaemon[2057]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:11 lc1 pvedaemon[31714]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:11 lc1 pvedaemon[31714]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:11 lc1 pvedaemon[2057]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:11 lc1 pvedaemon[20939]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:11 lc1 pvedaemon[20939]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:11 lc1 pvedaemon[2057]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:11 lc1 pvedaemon[20939]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:11 lc1 pvedaemon[20939]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:11 lc1 pvedaemon[2057]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:11 lc1 pvedaemon[2057]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:11 lc1 pvedaemon[20939]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:11 lc1 pvedaemon[2057]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:11 lc1 pvedaemon[31714]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:12 lc1 pvedaemon[2057]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:12 lc1 pvedaemon[31714]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:12 lc1 pvedaemon[31714]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:12 lc1 pvedaemon[31714]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:12 lc1 pvedaemon[20939]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:12 lc1 pvedaemon[2057]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:12 lc1 pvedaemon[31714]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:14 lc1 pvedaemon[31714]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:14 lc1 pvedaemon[20939]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:15 lc1 pvedaemon[2057]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:16 lc1 pvedaemon[20939]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:16 lc1 pvedaemon[31714]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:17 lc1 pvedaemon[20939]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:17 lc1 pvedaemon[2057]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:18 lc1 pvedaemon[31714]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:18 lc1 pvedaemon[31714]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:19 lc1 pvedaemon[31714]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:19 lc1 pvedaemon[20939]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:19 lc1 pvedaemon[31714]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:20 lc1 pvedaemon[31714]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:20 lc1 pvedaemon[2057]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:21 lc1 pvedaemon[31714]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:21 lc1 pvedaemon[31714]: <root@pam> successful auth for user 'root@pam'
янв 06 17:44:22 lc1 pvedaemon[20939]: <root@pam> successful auth for user 'root@pam'

which looks possibly like an api client is logging in over and over?

also, do you have 2fa setup on the root@pam user? that could be one reason for the spam above (bad api client)

 

[fokatoka]

yes, the work is done through the api
but even through the proxmox interface I see freezes and long execution of commands.

2fa is not included.
this is a module https://marketplace.whmcs.com/product/1198-proxmox-vps-for-whmcs