[SOLVED] Reload failed for PVE API Proxy Server

[joabe]

Hi, i have a cluster with two nodes. In one of them the web interface is not accessible, in pveproxy status I have:

~#systemctl status pveproxy
● pveproxy.service - PVE API Proxy Server
   Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2020-11-10 10:30:10 -03; 6 months 8 days ago
  Process: 1134 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)
  Process: 1136 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
 Main PID: 1139 (pveproxy)
    Tasks: 6 (limit: 4915)
   Memory: 338.4M
   CGroup: /system.slice/pveproxy.service
           ├─ 1139 pveproxy
           ├─ 5025 pveproxy worker
           ├─ 5892 pveproxy worker
           ├─16034 /usr/bin/perl -T /usr/bin/pveproxy restart
           ├─30043 pveproxy worker
           └─30340 pveproxy worker (shutdown)

May 20 22:57:23 pve01 pveproxy[5025]: Clearing outdated entries from certificate cache
May 20 23:01:04 pve01 pveproxy[1139]: worker 27284 finished
May 20 23:01:04 pve01 pveproxy[1139]: starting 1 worker(s)
May 20 23:01:04 pve01 pveproxy[1139]: worker 5892 started
May 20 23:01:04 pve01 pveproxy[5891]: got inotify poll request in wrong process - disabling inotify
May 20 23:01:05 pve01 pveproxy[5891]: worker exit
May 20 23:02:58 pve01 pveproxy[5892]: Clearing outdated entries from certificate cache
May 21 00:00:00 pve01 systemd[1]: Reloading PVE API Proxy Server.
May 21 00:01:30 pve01 systemd[1]: pveproxy.service: Reload operation timed out. Killing reload process.
May 21 00:01:30 pve01 systemd[1]: Reload failed for PVE API Proxy Server.

I tried to use pvecm updatecerts --force, but I got:

~# pvecm updatecerts --force
(re)generate node files
generate new node certificate
got timeout

 

[Moayad]

Hi,

have you checked the journalctl for any error? please post the output of the pveversion -v command

 

[joabe]

Hi, from errors I got that here in journalctl:

May 20 21:44:19 pve01 kernel: INFO: task pvesr:25689 blocked for more than 120 seconds.
May 20 21:44:19 pve01 kernel:       Tainted: P           O      5.4.65-1-pve #1
May 20 21:44:19 pve01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 20 21:46:20 pve01 kernel: INFO: task pvesr:25689 blocked for more than 241 seconds.
May 20 21:46:20 pve01 kernel:       Tainted: P           O      5.4.65-1-pve #1
May 20 21:46:20 pve01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 20 21:48:21 pve01 kernel: INFO: task pvesr:25689 blocked for more than 362 seconds.
May 20 21:48:21 pve01 kernel:       Tainted: P           O      5.4.65-1-pve #1
May 20 21:48:21 pve01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 20 21:50:22 pve01 kernel: INFO: task pvesr:25689 blocked for more than 483 seconds.
May 20 21:50:22 pve01 kernel:       Tainted: P           O      5.4.65-1-pve #1
May 20 21:50:22 pve01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 20 21:52:23 pve01 kernel: INFO: task pvesr:25689 blocked for more than 604 seconds.
May 20 21:52:23 pve01 kernel:       Tainted: P           O      5.4.65-1-pve #1
May 20 21:52:23 pve01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 20 21:54:23 pve01 kernel: INFO: task pvesr:25689 blocked for more than 724 seconds.
May 20 21:54:23 pve01 kernel:       Tainted: P           O      5.4.65-1-pve #1
May 20 21:54:23 pve01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 20 21:56:24 pve01 kernel: INFO: task pvesr:25689 blocked for more than 845 seconds.
May 20 21:56:24 pve01 kernel:       Tainted: P           O      5.4.65-1-pve #1
May 20 21:56:24 pve01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 20 21:58:25 pve01 kernel: INFO: task pvesr:25689 blocked for more than 966 seconds.
May 20 21:58:25 pve01 kernel:       Tainted: P           O      5.4.65-1-pve #1
May 20 21:58:25 pve01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 20 22:00:26 pve01 kernel: INFO: task pvesr:25689 blocked for more than 1087 seconds.
May 20 22:00:26 pve01 kernel:       Tainted: P           O      5.4.65-1-pve #1
May 20 22:00:26 pve01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 20 22:02:27 pve01 kernel: INFO: task pvesr:25689 blocked for more than 1208 seconds.
May 20 22:02:27 pve01 kernel:       Tainted: P           O      5.4.65-1-pve #1
May 20 22:02:27 pve01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 20 23:01:04 pve01 pveproxy[5891]: got inotify poll request in wrong process - disabling inotify
May 21 00:01:30 pve01 systemd[1]: Reload failed for PVE API Proxy Server.

pveversion:

~# pveversion -v
proxmox-ve: 6.2-2 (running kernel: 5.4.65-1-pve)
pve-manager: 6.2-15 (running version: 6.2-15/48bd51b6)
pve-kernel-5.4: 6.2-7
pve-kernel-helper: 6.2-7
pve-kernel-5.4.65-1-pve: 5.4.65-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-4
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-9
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 0.9.6-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.3-8
pve-cluster: 6.2-1
pve-container: 3.2-2
pve-docs: 6.2-6
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-6
pve-xtermjs: 4.7.0-2
pve-zsync: 2.0-3
qemu-server: 6.2-19
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve2

I tried restarting pveproxy and spiceproxy. Spiceproxy took a long time to start but it went, pveproxy timed out:

The unit pveproxy.service has entered the 'failed' state with result 'timeout'.

 

[joabe]

Hi, I was able to solve the problem with:

# systemctl restart pve-cluster.service
# pvecm updatecerts -f
# systemctl restart pveproxy.service

Finally it is working

 

[Moayad]

Glad you solved your issue yourself!

I'll go ahead and make this thread as solved to help other people who have the same issue.

Have a nice weekend