Proxmox: Server reboots randomly

nash.burns

New Member
May 7, 2018
6
0
1
39
Hi Guys,

We are noticing random server reboots and there is nothing found in the logs.

The following are the Only PROXMOX errors in logs before reboot.
-----
Apr 22 05:21:24 cav-cwh-node3 pmxcfs[4095]: [status] crit: cpg_send_message failed: 9
Apr 22 05:21:24 cav-cwh-node3 pmxcfs[4095]: [status] crit: cpg_send_message failed: 9
Apr 22 05:21:24 cav-cwh-node3 pmxcfs[4095]: [status] crit: cpg_send_message failed: 9
Apr 22 05:21:24 cav-cwh-node3 pmxcfs[4095]: [status] crit: cpg_send_message failed: 9
Apr 22 05:21:24 cav-cwh-node3 pmxcfs[4095]: [status] crit: cpg_send_message failed: 9
Apr 22 05:21:24 cav-cwh-node3 pmxcfs[4095]: [status] crit: cpg_send_message failed: 9
Apr 22 05:21:25 cav-cwh-node3 snmpd[3932]: error on subcontainer 'ia_addr' insert (-1)
Apr 22 05:40:29 cav-cwh-node3 kernel: imklog 5.8.11, log source = /proc/kmsg started.
-----

Could someone shed some light on this or anybody experienced a similar scenario?

I found a similar thread discussing about this, but nothing helpful there as well.

https://forum.proxmox.com/threads/cpg_send_message-failed-9.13423/
 

Xahid

Active Member
Feb 13, 2014
76
10
28
You didn't post much information for developers to understand.
My guess, If you have created the Cluster, its not working well, which is causing the reboot.
 

nash.burns

New Member
May 7, 2018
6
0
1
39
Hi,

The cluster is working fine. However, the server gets rebooted randomly(not very frequently but like once in a month or something like that. It doesn't look like an OS issue since nothing related to OS is seen in the logs. Is there any specific details you require?
 

adamb

Renowned Member
Mar 1, 2012
1,209
52
68
Hi,

The cluster is working fine. However, the server gets rebooted randomly(not very frequently but like once in a month or something like that. It doesn't look like an OS issue since nothing related to OS is seen in the logs. Is there any specific details you require?

Typically its either a cluster network issue or a hardware issue. Chances of it being an OS issue are pretty slim. Do you have other nodes with the same hardware having the issue as well?
 

GadgetPig

Member
Apr 26, 2016
138
24
18
52
I also think it may be a hardware issue, or possibly your UPS battery may need replacing. Several years back I had an old PowerEdge T300 server rebooting with ProxMox. First thing I did was replace the power supply (and CPU/Case fans for good measure) and it's been stable since. Another time a different server was rebooting, but thru troubleshooting (connecting server to non UPS backup outlet) I found it was the UPS battery going bad. After replacing the UPS battery it's been stable since. You can also try testing server memory thru memtest for like a few hours or so. Usually bad memory will often show up within an hour of memtest.

Out of curiosity, is this node connected to an NFS server for storage? Any anomalies on the storage server? Are you using ZFS on linux local storage? And do you have at least 50% memory allocated to ZFS?
 

nash.burns

New Member
May 7, 2018
6
0
1
39
Typically its either a cluster network issue or a hardware issue. Chances of it being an OS issue are pretty slim. Do you have other nodes with the same hardware having the issue as well?
Nope, I don't see any issue on the other node yet.
 

nash.burns

New Member
May 7, 2018
6
0
1
39
I also think it may be a hardware issue, or possibly your UPS battery may need replacing. Several years back I had an old PowerEdge T300 server rebooting with ProxMox. First thing I did was replace the power supply (and CPU/Case fans for good measure) and it's been stable since. Another time a different server was rebooting, but thru troubleshooting (connecting server to non UPS backup outlet) I found it was the UPS battery going bad. After replacing the UPS battery it's been stable since. You can also try testing server memory thru memtest for like a few hours or so. Usually bad memory will often show up within an hour of memtest.

Out of curiosity, is this node connected to an NFS server for storage? Any anomalies on the storage server? Are you using ZFS on linux local storage? And do you have at least 50% memory allocated to ZFS?
I have confirmed that it is not a hardware issue. Also Plenty of RAM available for ZFS. Is it something with the Proxmox version? The current version is 3.4-6/102d4547
 

gkovacs

Well-Known Member
Dec 22, 2008
509
48
48
Budapest, Hungary
We are noticing random server reboots and there is nothing found in the logs.

Are you on ZFS? Is your system low on free memory? If your answer to both questions is yes, then your spontaneous reboot can be prevented by the following two steps:

1. ZFS ARC size
We aggressively limit the ZFS ARC size, as it has led to several spontaneous reboots in the past when left unlimited. Basically, we add up all the memory the system uses without caches and buffers (like all the KVM maximum RAM combined), subtract that from total host RAM, and set the ARC to something a bit less than that, so it has to compete with system cache only. For example: on a 32GB server the maximum RAM allocation of KVM guests is 25 GB, so we set the ARC to max out at 5GB (leaving 2GB for anything else). We also set a lower limit of 1GB to the ARC, as it has been reported that it helps performance.

To do that, you have add the following lines to /etc/modprobe.d/zfs.conf
Code:
options zfs zfs_arc_max=5368709120
options zfs zfs_arc_min=1073741824
and after that run:
Code:
# update-initramfs -u
and reboot.

Looking at the ARC of this very server with arc_summary.py you can see it stays between the limits:
Code:
ARC Size:                               30.72%  1.54    GiB
       Target Size: (Adaptive)         30.72%  1.54    GiB
       Min Size (Hard Limit):          20.00%  1.00    GiB
       Max Size (High Water):          5:1     5.00    GiB

ARC Size Breakdown:
       Recently Used Cache Size:       35.27%  554.85  MiB
       Frequently Used Cache Size:     64.73%  1018.10 MiB

2. SWAP on ZFS zvol

You also have to make sure that swap behaves well if it resides on a ZFS zvol (default Proxmox installation places it there). Most important is disabling ARC caching the swap volume, but the other tweaks are important as well (and endorsed by the ZFS on Linux community):
https://github.com/zfsonlinux/zfs/wiki/FAQ

Execute these commands in your shell (left out the # so you can copy all lines at once):
Code:
zfs set primarycache=metadata rpool/swap
zfs set secondarycache=metadata rpool/swap
zfs set compression=zle rpool/swap
zfs set checksum=off rpool/swap
zfs set sync=always rpool/swap
zfs set logbias=throughput rpool/swap

You can verify these settings by running:
Code:
# zfs get all rpool/swap
 
  • Like
Reactions: GadgetPig

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!