[SOLVED] issue with greyed out?

killmasta93 · Jun 22, 2020

Hi,
Recently having this server only showing grey out vms, It is not in a cluster only single node i checked these services
Rebooting fixes the issue but now its constant not sure why

Thank you

pvecm statusroot@prometheus:~# pvecm status
Corosync config '/etc/pve/corosync.conf' does not exist - is this node part of a cluster?
Cannot initialize CMAP service
root@prometheus:~# service pve-manager status
● pve-guests.service - PVE guests
Loaded: loaded (/lib/systemd/system/pve-guests.service; enabled; vendor preset: enabled)
Active: active (exited) since Mon 2020-05-18 23:13:31 -05; 1 months 3 days ago
Process: 2780 ExecStart=/usr/bin/pvesh --nooutput create /nodes/localhost/startall (code=exited, status=0/SUCCESS)
Main PID: 2780 (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 4915)
Memory: 0B
CPU: 0
CGroup: /system.slice/pve-guests.service

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
root@prometheus:~# service pvestatd status
● pvestatd.service - PVE Status Daemon
Loaded: loaded (/lib/systemd/system/pvestatd.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2020-05-18 23:13:11 -05; 1 months 3 days ago
Process: 2604 ExecStart=/usr/bin/pvestatd start (code=exited, status=0/SUCCESS)
Main PID: 2648 (pvestatd)
Tasks: 1 (limit: 4915)
Memory: 73.2M
CPU: 2h 21min 650ms
CGroup: /system.slice/pvestatd.service
└─2648 pvestatd

Jun 21 19:59:36 prometheus pvestatd[2648]: status update time (8.109 seconds)
Jun 21 20:59:40 prometheus pvestatd[2648]: status update time (51.866 seconds)
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

matrix · Jun 22, 2020

maybe you have upgraded your server without rebooting

killmasta93 · Jun 22, 2020

Thanks for the reply, but nope havent upgraded

matrix · Jun 23, 2020

if you got same issue check journalctl/dmesg

killmasta93 · Jun 26, 2020

i checked the system logs the only thing i found was this

Code:

Jun 26 07:16:01 prometheus pvestatd[16740]: start failed - can't acquire lock '/var/run/pvestatd.pid.lock' - Resource temporarily unavailable
Jun 26 07:16:01 prometheus pvestatd[16740]: start failed - can't acquire lock '/var/run/pvestatd.pid.lock' - Resource temporarily unavailable

Moayad · Jun 26, 2020

Hi,

Please post output of pveversion -v

killmasta93 · Jun 27, 2020

thanks for the reply this is what i get

root@prometheus:~# pveversion -v
proxmox-ve: 5.4-1 (running kernel: 4.15.18-12-pve)
pve-manager: 5.4-3 (running version: 5.4-3/0a6eaa62)
pve-kernel-4.15: 5.3-3
pve-kernel-4.15.18-12-pve: 4.15.18-35
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-8
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-50
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-13
libpve-storage-perl: 5.0-41
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-3
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-3
proxmox-widget-toolkit: 1.0-25
pve-cluster: 5.0-36
pve-container: 2.0-37
pve-docs: 5.4-2
pve-edk2-firmware: 1.20190312-1
pve-firewall: 3.0-19
pve-firmware: 2.0-6
pve-ha-manager: 2.0-9
pve-i18n: 1.1-4
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 2.12.1-3
pve-xtermjs: 3.12.0-1
qemu-server: 5.0-50
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.13-pve1~bpo2

Moayad · Jul 6, 2020

First of all i recommended you to upgrade to PVE 6 because PVE 5 will be EOL end of July [0].

What the output of:

Code:

~# pvestatd restart

~# pvestatd status

[0] https://pve.proxmox.com/wiki/FAQ

killmasta93 · Jul 6, 2020

Thanks for the reply, i guess ill have to update it then to see if that error stops as the restart option im getting this

Code:

Jul  6 07:52:04 prometheus pvestatd[14508]: start failed - can't acquire lock '/var/run/pvestatd.pid.lock' - Resource temporarily unavailable
Jul  6 07:52:04 prometheus pvestatd[14508]: start failed - can't acquire lock '/var/run/pvestatd.pid.lock' - Resource temporarily unavailable

matrix · Jul 6, 2020

Hi @killmasta93 can you try to make backup file pvestatd.pid.lock and try to restart the pvestatd?

`mv /var/run/pvestatd.pid.lock /var/run/pvestatd.pid.lock.bak`

then

➜ ~ pvestatd restart
➜ ~ pvestatd status

but first kill the process id of pvestatd

killmasta93 · Jul 6, 2020

Thanks that did the trick

Steve_B · Feb 9, 2021

Hi,

On my side, I have my proxmox greyed out everytime... (once I reboot the server it leasts less than 24h).

When it's greyed out, I have test everything that is in this post.... but nothing is working ....

I have to reboot the server .... trough command line... but often it does not work (failed to talt to init) and I have to reboot "hardly"...

Any idea what is the cause ?

Steve_B · Feb 9, 2021

WTF.... this time
pvestatd restart worked.... (I had to launch it twice)

Steve_B · Feb 9, 2021

and 30 minutes later... greyed out...

I found this in journalctl

Code:

Feb 09 16:42:00 proxmox systemd[1]: Starting Proxmox VE replication runner...
Feb 09 16:42:00 proxmox kernel: pvesr[2160]: segfault at 555377289e63 ip 0000555573a4
Feb 09 16:42:00 proxmox kernel: Code: 41 f6 40 22 04 74 47 48 8b 44 24 08 48 83 e8 01
Feb 09 16:42:00 proxmox systemd[1]: pvesr.service: Main process exited, code=killed,
Feb 09 16:42:00 proxmox systemd[1]: pvesr.service: Failed with result 'signal'.
Feb 09 16:42:00 proxmox systemd[1]: Failed to start Proxmox VE replication runner.
Feb 09 16:42:08 proxmox kernel: node[24444]: segfault at 311e804c0526 ip 0000308a6e40
Feb 09 16:42:08 proxmox kernel: Code: 4d 8b 48 07 4d 8b 49 0f 4d 39 4d d8 0f 84 f2 00
Feb 09 16:42:19 proxmox kernel: node[2213]: segfault at 2c7494480010 ip 0000000000cfc
Feb 09 16:42:19 proxmox kernel: Code: 89 f0 83 e0 03 48 83 f8 01 75 da 4d 89 f7 44 89
Feb 09 16:42:22 proxmox kernel: pvestatd[31162]: segfault at 0 ip 00007f31e36b0557 sp
Feb 09 16:42:22 proxmox kernel: Code: fc c3 90 4c 8b 5e ec 48 8b 4e f4 8b 56 fc 4c 89
Feb 09 16:42:22 proxmox systemd[1]: pvestatd.service: Main process exited, code=kille
Feb 09 16:42:22 proxmox systemd[1]: pvestatd.service: Failed with result 'signal'.
Feb 09 16:42:30 proxmox kernel: node[2292]: segfault at 371ed8ec15b5 ip 0000000000d03
Feb 09 16:42:30 proxmox kernel: Code: 01 00 00 4c 39 c6 0f 96 c2 4c 39 c1 48 8b b8 80
Feb 09 16:42:41 proxmox kernel: traps: node[2335] general protection fault ip:f10611
Feb 09 16:42:52 proxmox kernel: node[2374]: segfault at 380bb6f40824 ip 0000000000d03
Feb 09 16:42:52 proxmox kernel: Code: 01 00 00 4c 39 c6 0f 96 c2 4c 39 c1 48 8b b8 80
Feb 09 16:43:00 proxmox systemd[1]: Starting Proxmox VE replication runner...
Feb 09 16:43:01 proxmox systemd[1]: pvesr.service: Succeeded.

killmasta93 · Feb 10, 2021

Steve_B said:

and 30 minutes later... greyed out...

I found this in journalctl

Code:

Feb 09 16:42:00 proxmox systemd[1]: Starting Proxmox VE replication runner...
Feb 09 16:42:00 proxmox kernel: pvesr[2160]: segfault at 555377289e63 ip 0000555573a4
Feb 09 16:42:00 proxmox kernel: Code: 41 f6 40 22 04 74 47 48 8b 44 24 08 48 83 e8 01
Feb 09 16:42:00 proxmox systemd[1]: pvesr.service: Main process exited, code=killed,
Feb 09 16:42:00 proxmox systemd[1]: pvesr.service: Failed with result 'signal'.
Feb 09 16:42:00 proxmox systemd[1]: Failed to start Proxmox VE replication runner.
Feb 09 16:42:08 proxmox kernel: node[24444]: segfault at 311e804c0526 ip 0000308a6e40
Feb 09 16:42:08 proxmox kernel: Code: 4d 8b 48 07 4d 8b 49 0f 4d 39 4d d8 0f 84 f2 00
Feb 09 16:42:19 proxmox kernel: node[2213]: segfault at 2c7494480010 ip 0000000000cfc
Feb 09 16:42:19 proxmox kernel: Code: 89 f0 83 e0 03 48 83 f8 01 75 da 4d 89 f7 44 89
Feb 09 16:42:22 proxmox kernel: pvestatd[31162]: segfault at 0 ip 00007f31e36b0557 sp
Feb 09 16:42:22 proxmox kernel: Code: fc c3 90 4c 8b 5e ec 48 8b 4e f4 8b 56 fc 4c 89
Feb 09 16:42:22 proxmox systemd[1]: pvestatd.service: Main process exited, code=kille
Feb 09 16:42:22 proxmox systemd[1]: pvestatd.service: Failed with result 'signal'.
Feb 09 16:42:30 proxmox kernel: node[2292]: segfault at 371ed8ec15b5 ip 0000000000d03
Feb 09 16:42:30 proxmox kernel: Code: 01 00 00 4c 39 c6 0f 96 c2 4c 39 c1 48 8b b8 80
Feb 09 16:42:41 proxmox kernel: traps: node[2335] general protection fault ip:f10611
Feb 09 16:42:52 proxmox kernel: node[2374]: segfault at 380bb6f40824 ip 0000000000d03
Feb 09 16:42:52 proxmox kernel: Code: 01 00 00 4c 39 c6 0f 96 c2 4c 39 c1 48 8b b8 80
Feb 09 16:43:00 proxmox systemd[1]: Starting Proxmox VE replication runner...
Feb 09 16:43:01 proxmox systemd[1]: pvesr.service: Succeeded.

what proxmox version?

Steve_B · Feb 11, 2021

6.3-2

and today... it is greyed out....
and pvestatd restart does not work...
Failed to start pvestatd.service: Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)

and when I try to reboot:

root@proxmox:~# reboot
Failed to open initctl fifo: No such device or address
Failed to talk to init daemon.

I found a way to reboot thanks to https://www.svennd.be/systemd-doesnt-want-to-reboot/
systemctl --force --force reboot

killmasta93 · Feb 12, 2021

try the following

Code:

service pvestatd daemon status
   service pvestatd status
  service pve-cluster stop
 service corosync stop
   service pvestatd stop
   service pveproxy stop
   service pvedaemon stop
   service pve-cluster start
   service corosync start
    service pvestatd start
  service pveproxy start
    service pvedaemon start

Search

Search

[SOLVED] issue with greyed out?

killmasta93

Renowned Member

matrix

Active Member

killmasta93

Renowned Member

matrix

Active Member

killmasta93

Renowned Member

Moayad

Proxmox Staff Member

killmasta93

Renowned Member

Moayad

Proxmox Staff Member

killmasta93

Renowned Member

matrix

Active Member

killmasta93

Renowned Member

Steve_B

Member

Steve_B

Member

Steve_B

Member

killmasta93

Renowned Member

Steve_B

Member

killmasta93

Renowned Member

We value your privacy