Gray node status after disk cloning

B

berdenok

New Member
Nov 15, 2023
7
0
1
Hi. after cloning the system from disk to disk, it received the gray status of the node, no solutions found do not help.
 

Attachments

Hi,
such is typically caused by the pvestatd not running or having issues. Check the status via systemctl status pvestatd.service and check the systemd journal for errors, journalctl -b -r gives you a paginated view of the journal since boot in reverse order.
 
Thanks for your reply.

systemctl status pvestatd.service
● pvestatd.service - PVE Status Daemon
Loaded: loaded (/lib/systemd/system/pvestatd.service; enabled; preset: enabled)
Active: active (running) since Wed 2023-11-15 11:55:31 MSK; 4min 36s ago
Process: 976 ExecStart=/usr/bin/pvestatd start (code=exited, status=0/SUCCESS)
Main PID: 995 (pvestatd)
Tasks: 3 (limit: 18945)
Memory: 137.7M
CPU: 4min 27.411s
CGroup: /system.slice/pvestatd.service
├─ 995 pvestatd
└─1073 /usr/bin/kvm --version

Nov 15 11:55:30 pve1 systemd[1]: Starting pvestatd.service - PVE Status Daemon...
Nov 15 11:55:31 pve1 pvestatd[995]: starting server
Nov 15 11:55:31 pve1 systemd[1]: Started pvestatd.service - PVE Status Daemon.

journalctl -b -r
Of the errors, I see only this

Nov 15 11:55:29 pve1 pmxcfs[870]: [status] crit: can't initialize service
Nov 15 11:55:29 pve1 pmxcfs[870]: [status] crit: cpg_initialize failed: 2
Nov 15 11:55:29 pve1 pmxcfs[870]: [dcdb] crit: can't initialize service
Nov 15 11:55:29 pve1 pmxcfs[870]: [dcdb] crit: cpg_initialize failed: 2
Nov 15 11:55:29 pve1 pmxcfs[870]: [confdb] crit: can't initialize service
Nov 15 11:55:29 pve1 pmxcfs[870]: [confdb] crit: cmap_initialize failed: 2
Nov 15 11:55:29 pve1 pmxcfs[870]: [quorum] crit: can't initialize service
Nov 15 11:55:29 pve1 pmxcfs[870]: [quorum] crit: quorum_initialize failed: 2
 
berdenok said:
Thanks for your reply.

systemctl status pvestatd.service
● pvestatd.service - PVE Status Daemon
Loaded: loaded (/lib/systemd/system/pvestatd.service; enabled; preset: enabled)
Active: active (running) since Wed 2023-11-15 11:55:31 MSK; 4min 36s ago
Process: 976 ExecStart=/usr/bin/pvestatd start (code=exited, status=0/SUCCESS)
Main PID: 995 (pvestatd)
Tasks: 3 (limit: 18945)
Memory: 137.7M
CPU: 4min 27.411s
CGroup: /system.slice/pvestatd.service
├─ 995 pvestatd
└─1073 /usr/bin/kvm --version

Nov 15 11:55:30 pve1 systemd[1]: Starting pvestatd.service - PVE Status Daemon...
Nov 15 11:55:31 pve1 pvestatd[995]: starting server
Nov 15 11:55:31 pve1 systemd[1]: Started pvestatd.service - PVE Status Daemon.

journalctl -b -r
Of the errors, I see only this

Nov 15 11:55:29 pve1 pmxcfs[870]: [status] crit: can't initialize service
Nov 15 11:55:29 pve1 pmxcfs[870]: [status] crit: cpg_initialize failed: 2
Nov 15 11:55:29 pve1 pmxcfs[870]: [dcdb] crit: can't initialize service
Nov 15 11:55:29 pve1 pmxcfs[870]: [dcdb] crit: cpg_initialize failed: 2
Nov 15 11:55:29 pve1 pmxcfs[870]: [confdb] crit: can't initialize service
Nov 15 11:55:29 pve1 pmxcfs[870]: [confdb] crit: cmap_initialize failed: 2
Nov 15 11:55:29 pve1 pmxcfs[870]: [quorum] crit: can't initialize service
Nov 15 11:55:29 pve1 pmxcfs[870]: [quorum] crit: quorum_initialize failed: 2
Click to expand...
Did you create a cluster on this node? It seems like you have no quorum and the node is in read only. Check the status of pvecm status and post the output of cat /etc/pve/corosync.conf.
 
Chris said:
Did you create a cluster on this node? It seems like you have no quorum and the node is in read only. Check the status of pvecm status and post the output of cat /etc/pve/corosync.conf.
Click to expand...
Apparently yes. This is the only node.

pvecm status
Cluster information
-------------------
Name: 1
Config Version: 1
Transport: knet
Secure auth: on

Quorum information
------------------
Date: Wed Nov 15 12:18:37 2023
Quorum provider: corosync_votequorum
Nodes: 1
Node ID: 0x00000001
Ring ID: 1.19
Quorate: Yes

Votequorum information
----------------------
Expected votes: 1
Highest expected: 1
Total votes: 1
Quorum: 1
Flags: Quorate

Membership information
----------------------
Nodeid Votes Name
0x00000001 1 192.168.100.153 (local)

cat /etc/pve/corosync.conf
logging {
debug: off
to_syslog: yes
}

nodelist {
node {
name: pve1
nodeid: 1
quorum_votes: 1
ring0_addr: 192.168.100.153
}
}

quorum {
provider: corosync_votequorum
}

totem {
cluster_name: 1
config_version: 1
interface {
linknumber: 0
}
ip_version: ipv4-6
link_mode: passive
secauth: on
version: 2
}
 
Well from this output it seems that you have quorum. Did the node status change to being online now? If not, please share the output of systemctl statuscorosync.service pve-cluster.service pveproxy.service pvedaemon.service
 
Chris said:
Well from this output it seems that you have quorum. Did the node status change to being online now? If not, please share the output of systemctl statuscorosync.service pve-cluster.service pveproxy.service pvedaemon.service
Click to expand...

Thank you for your help, but unfortunately the status has not changed.

systemctl status corosync.service pve-cluster.service pveproxy.service pvedaemon.service
● corosync.service - Corosync Cluster Engine
Loaded: loaded (/lib/systemd/system/corosync.service; enabled; preset: enabled)
Active: active (running) since Wed 2023-11-15 12:08:21 MSK; 1 day 23h ago
Docs: man:corosync
man:corosync.conf
man:corosync_overview
Main PID: 977 (corosync)
Tasks: 9 (limit: 18945)
Memory: 130.3M
CPU: 15min 15.594s
CGroup: /system.slice/corosync.service
└─977 /usr/sbin/corosync -f

Nov 15 12:08:21 pve1 corosync[977]: [QB ] server name: quorum
Nov 15 12:08:21 pve1 corosync[977]: [TOTEM ] Configuring link 0
Nov 15 12:08:21 pve1 corosync[977]: [TOTEM ] Configured link number 0: local addr: 192.168.100.153, port=5405
Nov 15 12:08:21 pve1 corosync[977]: [KNET ] link: Resetting MTU for link 0 because host 1 joined
Nov 15 12:08:21 pve1 corosync[977]: [QUORUM] Sync members[1]: 1
Nov 15 12:08:21 pve1 corosync[977]: [QUORUM] Sync joined[1]: 1
Nov 15 12:08:21 pve1 corosync[977]: [TOTEM ] A new membership (1.19) was formed. Members joined: 1
Nov 15 12:08:21 pve1 corosync[977]: [QUORUM] Members[1]: 1
Nov 15 12:08:21 pve1 corosync[977]: [MAIN ] Completed service synchronization, ready to provide service.
Nov 15 12:08:21 pve1 systemd[1]: Started corosync.service - Corosync Cluster Engine.

● pve-cluster.service - The Proxmox VE cluster filesystem
Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; preset: enabled)
Active: active (running) since Wed 2023-11-15 12:08:21 MSK; 1 day 23h ago
Process: 869 ExecStart=/usr/bin/pmxcfs (code=exited, status=0/SUCCESS)
Main PID: 873 (pmxcfs)
Tasks: 7 (limit: 18945)
Memory: 65.6M
CPU: 1min 1.553s
CGroup: /system.slice/pve-cluster.service
└─873 /usr/bin/pmxcfs

Nov 17 02:08:20 pve1 pmxcfs[873]: [dcdb] notice: data verification successful
Nov 17 03:08:20 pve1 pmxcfs[873]: [dcdb] notice: data verification successful
Nov 17 04:08:20 pve1 pmxcfs[873]: [dcdb] notice: data verification successful
Nov 17 05:08:20 pve1 pmxcfs[873]: [dcdb] notice: data verification successful
Nov 17 06:08:20 pve1 pmxcfs[873]: [dcdb] notice: data verification successful
Nov 17 07:08:20 pve1 pmxcfs[873]: [dcdb] notice: data verification successful
Nov 17 08:08:20 pve1 pmxcfs[873]: [dcdb] notice: data verification successful
Nov 17 09:08:20 pve1 pmxcfs[873]: [dcdb] notice: data verification successful
Nov 17 10:08:20 pve1 pmxcfs[873]: [dcdb] notice: data verification successful
Nov 17 11:08:20 pve1 pmxcfs[873]: [dcdb] notice: data verification successful

● pveproxy.service - PVE API Proxy Server
Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; preset: enabled)
Active: active (running) since Wed 2023-11-15 12:08:28 MSK; 1 day 23h ago
Process: 1030 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)
Process: 1036 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
Process: 79004 ExecReload=/usr/bin/pveproxy restart (code=exited, status=0/SUCCESS)
Main PID: 1037 (pveproxy)
Tasks: 4 (limit: 18945)
Memory: 181.3M
CPU: 36.881s
CGroup: /system.slice/pveproxy.service
├─ 1037 pveproxy
├─79046 "pveproxy worker"
├─79047 "pveproxy worker"
└─79048 "pveproxy worker"

Nov 16 00:00:02 pve1 pveproxy[1037]: starting 3 worker(s)
Nov 16 00:00:02 pve1 pveproxy[1037]: worker 79046 started
Nov 16 00:00:02 pve1 pveproxy[1037]: worker 79047 started
Nov 16 00:00:02 pve1 pveproxy[1037]: worker 79048 started
Nov 16 00:00:07 pve1 pveproxy[34496]: worker exit
Nov 16 00:00:07 pve1 pveproxy[28529]: worker exit
Nov 16 00:00:07 pve1 pveproxy[39332]: worker exit
Nov 16 00:00:07 pve1 pveproxy[1037]: worker 34496 finished
Nov 16 00:00:07 pve1 pveproxy[1037]: worker 39332 finished
Nov 16 00:00:07 pve1 pveproxy[1037]: worker 28529 finished

● pvedaemon.service - PVE API Daemon
Loaded: loaded (/lib/systemd/system/pvedaemon.service; enabled; preset: enabled)
Active: active (running) since Wed 2023-11-15 12:08:22 MSK; 1 day 23h ago
Process: 996 ExecStart=/usr/bin/pvedaemon start (code=exited, status=0/SUCCESS)
Main PID: 1025 (pvedaemon)
Tasks: 4 (limit: 18945)
Memory: 220.6M
CPU: 9.789s
CGroup: /system.slice/pvedaemon.service
├─1025 pvedaemon
├─1026 "pvedaemon worker"
├─1027 "pvedaemon worker"
└─1028 "pvedaemon worker"

Nov 15 16:21:44 pve1 pvedaemon[1026]: <root@pam> successful auth for user 'root@pam'
Nov 15 16:37:44 pve1 pvedaemon[1028]: <root@pam> successful auth for user 'root@pam'
Nov 15 16:53:44 pve1 pvedaemon[1027]: <root@pam> successful auth for user 'root@pam'
Nov 15 17:09:44 pve1 pvedaemon[1026]: <root@pam> successful auth for user 'root@pam'
Nov 15 17:25:44 pve1 pvedaemon[1027]: <root@pam> successful auth for user 'root@pam'
Nov 15 17:41:44 pve1 pvedaemon[1027]: <root@pam> successful auth for user 'root@pam'
Nov 15 17:57:44 pve1 pvedaemon[1028]: <root@pam> successful auth for user 'root@pam'
Nov 15 18:13:44 pve1 pvedaemon[1027]: <root@pam> successful auth for user 'root@pam'
Nov 17 11:42:08 pve1 pvedaemon[1026]: auth key pair too old, rotating..
Nov 17 11:42:08 pve1 pvedaemon[1026]: <root@pam> successful auth for user 'root@pam'
 
Okay, so nothing obviously wrong. What does pvesh get /cluster/resources return?
 
So it seems that the data cannot be fetched from the rrd dump via the pmxcfs. Try restart all the services by running systemctl restart pve-cluster.service pvestatd.service pvedaemon.service pveproxy.service
 
Chris said:
So it seems that the data cannot be fetched from the rrd dump via the pmxcfs. Try restart all the services by running systemctl restart pve-cluster.service pvestatd.service pvedaemon.service pveproxy.service
Click to expand...
Did. Without changes.

root@pve1:~# systemctl restart pve-cluster.service pvestatd.service pvedaemon.service pveproxy.service
root@pve1:~# systemctl status pve-cluster.service pvestatd.service pvedaemon.service pveproxy.service
● pve-cluster.service - The Proxmox VE cluster filesystem
Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; preset: enabled)
Active: active (running) since Fri 2023-11-17 13:27:54 MSK; 28s ago
Process: 326213 ExecStart=/usr/bin/pmxcfs (code=exited, status=0/SUCCESS)
Main PID: 326214 (pmxcfs)
Tasks: 6 (limit: 18945)
Memory: 22.4M
CPU: 49ms
CGroup: /system.slice/pve-cluster.service
└─326214 /usr/bin/pmxcfs

Nov 17 13:27:53 pve1 systemd[1]: Starting pve-cluster.service - The Proxmox VE cluster filesystem...
Nov 17 13:27:53 pve1 pmxcfs[326213]: [main] notice: resolved node name 'pve1' to '192.168.100.153' for default node IP address
Nov 17 13:27:53 pve1 pmxcfs[326213]: [main] notice: resolved node name 'pve1' to '192.168.100.153' for default node IP address
Nov 17 13:27:53 pve1 pmxcfs[326214]: [status] notice: update cluster info (cluster name 1, version = 1)
Nov 17 13:27:53 pve1 pmxcfs[326214]: [status] notice: node has quorum
Nov 17 13:27:53 pve1 pmxcfs[326214]: [dcdb] notice: members: 1/326214
Nov 17 13:27:53 pve1 pmxcfs[326214]: [dcdb] notice: all data is up to date
Nov 17 13:27:53 pve1 pmxcfs[326214]: [status] notice: members: 1/326214
Nov 17 13:27:53 pve1 pmxcfs[326214]: [status] notice: all data is up to date
Nov 17 13:27:54 pve1 systemd[1]: Started pve-cluster.service - The Proxmox VE cluster filesystem.

● pvestatd.service - PVE Status Daemon
Loaded: loaded (/lib/systemd/system/pvestatd.service; enabled; preset: enabled)
Active: active (running) since Fri 2023-11-17 13:27:54 MSK; 28s ago
Process: 326239 ExecStart=/usr/bin/pvestatd start (code=exited, status=0/SUCCESS)
Main PID: 326240 (pvestatd)
Tasks: 3 (limit: 18945)
Memory: 91.2M
CPU: 18.840s
CGroup: /system.slice/pvestatd.service
├─326240 pvestatd
└─326272 /usr/bin/kvm --version

Nov 17 13:27:54 pve1 systemd[1]: Starting pvestatd.service - PVE Status Daemon...
Nov 17 13:27:54 pve1 pvestatd[326240]: starting server
Nov 17 13:27:54 pve1 systemd[1]: Started pvestatd.service - PVE Status Daemon.

● pvedaemon.service - PVE API Daemon
Loaded: loaded (/lib/systemd/system/pvedaemon.service; enabled; preset: enabled)
Active: active (running) since Fri 2023-11-17 13:27:54 MSK; 28s ago
Process: 326221 ExecStart=/usr/bin/pvedaemon start (code=exited, status=0/SUCCESS)
Main PID: 326241 (pvedaemon)
Tasks: 4 (limit: 18945)
Memory: 201.1M
CPU: 775ms
CGroup: /system.slice/pvedaemon.service
├─326241 pvedaemon
├─326242 "pvedaemon worker"
├─326243 "pvedaemon worker"
└─326244 "pvedaemon worker"

Nov 17 13:27:54 pve1 systemd[1]: Starting pvedaemon.service - PVE API Daemon...
Nov 17 13:27:54 pve1 pvedaemon[326241]: starting server
Nov 17 13:27:54 pve1 pvedaemon[326241]: starting 3 worker(s)
Nov 17 13:27:54 pve1 pvedaemon[326241]: worker 326242 started
Nov 17 13:27:54 pve1 pvedaemon[326241]: worker 326243 started
Nov 17 13:27:54 pve1 pvedaemon[326241]: worker 326244 started
Nov 17 13:27:54 pve1 systemd[1]: Started pvedaemon.service - PVE API Daemon.

● pveproxy.service - PVE API Proxy Server
Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; preset: enabled)
Active: active (running) since Fri 2023-11-17 13:27:55 MSK; 27s ago
Process: 326245 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)
Process: 326247 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
Main PID: 326248 (pveproxy)
Tasks: 4 (limit: 18945)
Memory: 146.6M
Процессор: 1.041с
CGroup: /system.slice/pveproxy.service
├─326248 pveproxy
├─326249 "pveproxy worker"
├─326250 "pveproxy worker"
└─326251 "pveproxy worker"

17 ноября 13:27:54 pve1 systemd[1]: Запуск прокси-сервера pveproxy.service - PVE API...
17 ноября 13:27:55 pve1 pveproxy[326248]: запуск сервера
17 ноября 13:27:55 pve1 pveproxy[326248]: запускаются 3 рабочих узла
17 ноября 13:27:55 pve1 pveproxy[326248]: запущен рабочий 326249
17 ноября 13:27:55 pve1 pveproxy[326248]: запущен рабочий 326250
17 ноября 13:27:55 pve1 pveproxy[326248]: запущен рабочий 326251
17 ноября 13:27:55 pve1 systemd[1]: Запущен pveproxy.service - прокси-сервер PVE API.

1700217063547.png
 
The only service remaining left to check and/or restart would be systemctl status rrdcached.service.
 
Chris said:
The only service remaining left to check and/or restart would be systemctl status rrdcached.service.
Click to expand...

root@pve1:~# systemctl status rrdcached.service
● rrdcached.service - LSB: start or stop rrdcached
Loaded: loaded (/etc/init.d/rrdcached; generated)
Active: active (running) since Wed 2023-11-15 12:08:20 MSK; 2 days ago
Docs: man:systemd-sysv-generator(8)
Process: 820 ExecStart=/etc/init.d/rrdcached start (code=exited, status=0/SUCCESS)
Tasks: 10 (limit: 18945)
Memory: 10.1M
CPU: 2.319s
CGroup: /system.slice/rrdcached.service
└─862 /usr/bin/rrdcached -B -b /var/lib/rrdcached/db/ -j /var/lib/rrdcached/journal/ -p /var/run/rrdcached.pid -l unix:/var/run/rrdcached.sock

Nov 15 12:08:20 pve1 systemd[1]: Starting rrdcached.service - LSB: start or stop rrdcached...
Nov 15 12:08:20 pve1 rrdcached[820]: rrdcached started.
Nov 15 12:08:20 pve1 systemd[1]: Started rrdcached.service - LSB: start or stop rrdcached.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom