Hi Guys can anyone help me in this situation actually I am not able to debug this issue what causing my server keeps going down again and again. below are the errors I get from logs. also let me know if you need any logs to understand the actual problem I just facing this issue from last 1 month.
Code:
These are the errors in proxmox
---------------------------------------------------------
1 - Volume Group Scan Failing (vgscan exit code 5)
---------------------------------------------------------
Feb 25 10:37:48 stream4tech-proxmox-2 pvestatd[1664]: command '/sbin/vgscan --ignorelockingfailure --mknodes' failed: exit code 5
Feb 25 10:37:59 stream4tech-proxmox-2 pvestatd[1664]: command '/sbin/vgscan --ignorelockingfailure --mknodes' failed: exit code 5
Feb 25 10:38:09 stream4tech-proxmox-2 pvestatd[1664]: command '/sbin/vgscan --ignorelockingfailure --mknodes' failed: exit code 5
Feb 25 10:38:18 stream4tech-proxmox-2 pvestatd[1664]: command '/sbin/vgscan --ignorelockingfailure --mknodes' failed: exit code 5
Feb 25 10:38:29 stream4tech-proxmox-2 pvestatd[1664]: command '/sbin/vgscan --ignorelockingfailure --mknodes' failed: exit code 5
---------------------------------------------------------
2 - InfluxDB Connection Failure (Timeout)
---------------------------------------------------------
Feb 25 10:37:49 stream4tech-proxmox-2 pvestatd[1664]: metrics send error 'influxdb-local': 500 Can't connect to 192.168.13.105:8086 (Connection timed out)
Feb 25 10:37:56 stream4tech-proxmox-2 pvestatd[1664]: metrics send error 'influxdb-local': 500 Can't connect to 192.168.13.105:8086 (Connection timed out)
Feb 25 10:37:57 stream4tech-proxmox-2 pvestatd[1664]: metrics send error 'influxdb-local': 500 Can't connect to 192.168.13.105:8086 (Connection timed out)
---------------------------------------------------------
3 - Reboot Detected at 10:39:35
---------------------------------------------------------
-- Boot b0e1721286f5406db8134112fa4eb6db --
Feb 25 10:39:35 stream4tech-proxmox-2 kernel: Linux version 6.5.11-7-pve (build@proxmox) ...
---------------------------------------------------------
4 - Cluster Quorum Failure
---------------------------------------------------------
pmxcfs[1191]: [quorum] crit: quorum_initialize failed: 2
pmxcfs[1191]: [quorum] crit: can't initialize service
---------------------------------------------------------
5 - Python Segfault
---------------------------------------------------------
kernel: python3[2159]: segfault at 7f6a618b8870
Code:
root@stream-proxmox-2 ~ # pveversion
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
pve-manager/8.1.4/ec5affc9e41f1d79 (running kernel: 6.5.11-7-pve)
root@stream-proxmox-2 ~ # cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@stream-proxmox-2 ~ # uname -r
6.5.11-7-pve
root@stream-proxmox-2 ~ # pvecm status
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
Cluster information
-------------------
Name: s4t-hetzner
Config Version: 9
Transport: knet
Secure auth: on
Quorum information
------------------
Date: Thu Feb 27 07:57:42 2025
Quorum provider: corosync_votequorum
Nodes: 4
Node ID: 0x00000002
Ring ID: 1.ab6
Quorate: Yes
Votequorum information
----------------------
Expected votes: 4
Highest expected: 4
Total votes: 4
Quorum: 3
Flags: Quorate
Membership information
----------------------
Nodeid Votes Name
0x00000001 1 192.168.31.1
0x00000002 1 192.168.31.2 (local)
0x00000003 1 192.168.31.3
0x00000004 1 192.168.31.4
root@stream4-proxmox-2 ~ # pvecm nodes
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
Membership information
----------------------
Nodeid Votes Name
1 1 stream4tech-proxmox-1
2 1 stream4tech-proxmox-2 (local)
3 1 stream4tech-proxmox-3
4 1 stream4tech-proxmox-4
root@stream-proxmox-2 ~ # df -h
Filesystem Size Used Avail Use% Mounted on
udev 63G 0 63G 0% /dev
tmpfs 13G 2.0M 13G 1% /run
/dev/nvme0n1p4 1.7T 40G 1.6T 3% /
tmpfs 63G 66M 63G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
efivarfs 192K 162K 26K 87% /sys/firmware/efi/efivars
/dev/nvme0n1p3 975M 797M 128M 87% /boot
/dev/nvme0n1p1 256M 148K 256M 1% /boot/efi
/dev/fuse 128M 84K 128M 1% /etc/pve
tmpfs 13G 0 13G 0% /run/user/1000
root@stream-proxmox-2 ~ # cat /etc/network/interfaces
source /etc/network/interfaces.d/*
auto lo
iface lo inet loopback
iface lo inet6 loopback
auto enp5s0
iface enp5s0 inet manual
auto enp5s0.4000
iface enp5s0.4000 inet static
address 192.168.31.2
netmask 255.255.255.0
vlan-raw-device enp5s0
mtu 1400
auto vmbr99
iface vmbr99 inet static
address mypublicip/26
gateway 162.225.181.1
bridge-ports enp5s0
bridge-stp off
bridge-fd 0
auto vmbr0
iface vmbr0 inet static
bridge-ports none
bridge-stp off
bridge-fd 0
root@stream-proxmox-2 ~ # journalctl -u pve-cluster --no-pager
Feb 26 19:32:35 stream4tech-proxmox-2 pmxcfs[1238]: [dcdb] notice: data verification successful
Feb 26 20:32:35 stream4tech-proxmox-2 pmxcfs[1238]: [dcdb] notice: data verification successful
Feb 26 20:44:00 stream4tech-proxmox-2 pmxcfs[1238]: [dcdb] notice: members: 2/1238
Feb 26 20:44:00 stream4tech-proxmox-2 pmxcfs[1238]: [status] notice: node lost quorum
Feb 26 20:44:00 stream4tech-proxmox-2 pmxcfs[1238]: [status] notice: members: 2/1238
Feb 26 20:44:00 stream4tech-proxmox-2 pmxcfs[1238]: [dcdb] crit: received write while not quorate - trigger resync
Feb 26 20:44:00 stream4tech-proxmox-2 pmxcfs[1238]: [dcdb] crit: leaving CPG group
Feb 26 20:44:01 stream4tech-proxmox-2 pmxcfs[1238]: [dcdb] notice: start cluster connection
Feb 26 20:44:01 stream4tech-proxmox-2 pmxcfs[1238]: [dcdb] crit: cpg_join failed: 14
Feb 26 20:44:01 stream4tech-proxmox-2 pmxcfs[1238]: [dcdb] crit: can't initialize service
-- Boot 236074d0be394c0d8e40786b9fb1c956 --
Feb 27 07:01:14 stream4tech-proxmox-2 systemd[1]: Starting pve-cluster.service - The Proxmox VE cluster filesystem...
Feb 27 07:01:14 stream4tech-proxmox-2 pmxcfs[1277]: [main] notice: resolved node name 'stream4tech-proxmox-2' to 'my ip' for default node IP address
Feb 27 07:01:14 stream4tech-proxmox-2 pmxcfs[1277]: [main] notice: resolved node name 'stream4tech-proxmox-2' to 'my ip' for default node IP address
Feb 27 07:01:14 stream4tech-proxmox-2 pmxcfs[1280]: [quorum] crit: quorum_initialize failed: 2
Feb 27 07:01:14 stream4tech-proxmox-2 pmxcfs[1280]: [quorum] crit: can't initialize service
Feb 27 07:01:14 stream4tech-proxmox-2 pmxcfs[1280]: [confdb] crit: cmap_initialize failed: 2
Feb 27 07:01:14 stream4tech-proxmox-2 pmxcfs[1280]: [confdb] crit: can't initialize service
Feb 27 07:01:14 stream4tech-proxmox-2 pmxcfs[1280]: [dcdb] crit: cpg_initialize failed: 2
Feb 27 07:01:14 stream4tech-proxmox-2 pmxcfs[1280]: [dcdb] crit: can't initialize service
Feb 27 07:01:14 stream4tech-proxmox-2 pmxcfs[1280]: [status] crit: cpg_initialize failed: 2
Feb 27 07:01:14 stream4tech-proxmox-2 pmxcfs[1280]: [status] crit: can't initialize service
Feb 27 07:01:15 stream4tech-proxmox-2 systemd[1]: Started pve-cluster.service - The Proxmox VE cluster filesystem.
Feb 27 07:01:20 stream4tech-proxmox-2 pmxcfs[1280]: [status] notice: update cluster info (cluster name s4t-hetzner, version = 9)
Feb 27 07:01:20 stream4tech-proxmox-2 pmxcfs[1280]: [status] notice: node has quorum
Feb 27 07:01:20 stream4tech-proxmox-2 pmxcfs[1280]: [dcdb] notice: members: 1/1261, 2/1280, 3/1987146, 4/2435024
Feb 27 07:01:20 stream4tech-proxmox-2 pmxcfs[1280]: [dcdb] notice: starting data syncronisation
Feb 27 07:01:20 stream4tech-proxmox-2 pmxcfs[1280]: [dcdb] notice: received sync request (epoch 1/1261/00000029)
Feb 27 07:01:20 stream4tech-proxmox-2 pmxcfs[1280]: [status] notice: members: 1/1261, 2/1280, 3/1987146, 4/2435024
Feb 27 07:01:20 stream4tech-proxmox-2 pmxcfs[1280]: [status] notice: starting data syncronisation
Feb 27 07:01:20 stream4tech-proxmox-2 pmxcfs[1280]: [status] notice: received sync request (epoch 1/1261/00000025)
Feb 27 07:01:20 stream4tech-proxmox-2 pmxcfs[1280]: [dcdb] notice: received all states
Feb 27 07:01:20 stream4tech-proxmox-2 pmxcfs[1280]: [dcdb] notice: leader is 1/1261
Feb 27 07:01:20 stream4tech-proxmox-2 pmxcfs[1280]: [dcdb] notice: synced members: 1/1261, 3/1987146, 4/2435024
Feb 27 07:01:20 stream4tech-proxmox-2 pmxcfs[1280]: [dcdb] notice: waiting for updates from leader
Feb 27 07:01:20 stream4tech-proxmox-2 pmxcfs[1280]: [status] notice: received all states
Feb 27 07:01:20 stream4tech-proxmox-2 pmxcfs[1280]: [status] notice: all data is up to date
Feb 27 07:01:20 stream4tech-proxmox-2 pmxcfs[1280]: [dcdb] notice: update complete - trying to commit (got 60 inode updates)
Feb 27 07:01:20 stream4tech-proxmox-2 pmxcfs[1280]: [dcdb] notice: all data is up to date
-- Boot 9ec127e79cc14267ae6c92846944a705 --
Feb 27 07:14:05 stream4tech-proxmox-2 systemd[1]: Starting pve-cluster.service - The Proxmox VE cluster filesystem...
Feb 27 07:14:05 stream4tech-proxmox-2 pmxcfs[1299]: [main] notice: resolved node name 'stream4tech-proxmox-2' to 'my ip' for default node IP address
Feb 27 07:14:05 stream4tech-proxmox-2 pmxcfs[1299]: [main] notice: resolved node name 'stream4tech-proxmox-2' to 'my ip' for default node IP address
Feb 27 07:14:05 stream4tech-proxmox-2 pmxcfs[1306]: [quorum] crit: quorum_initialize failed: 2
Feb 27 07:14:05 stream4tech-proxmox-2 pmxcfs[1306]: [quorum] crit: can't initialize service
Feb 27 07:14:05 stream4tech-proxmox-2 pmxcfs[1306]: [confdb] crit: cmap_initialize failed: 2
Feb 27 07:14:05 stream4tech-proxmox-2 pmxcfs[1306]: [confdb] crit: can't initialize service
Feb 27 07:14:05 stream4tech-proxmox-2 pmxcfs[1306]: [dcdb] crit: cpg_initialize failed: 2
Feb 27 07:14:05 stream4tech-proxmox-2 pmxcfs[1306]: [dcdb] crit: can't initialize service
Feb 27 07:14:05 stream4tech-proxmox-2 pmxcfs[1306]: [status] crit: cpg_initialize failed: 2
Feb 27 07:14:05 stream4tech-proxmox-2 pmxcfs[1306]: [status] crit: can't initialize service
Feb 27 07:14:06 stream4tech-proxmox-2 systemd[1]: Started pve-cluster.service - The Proxmox VE cluster filesystem.
Feb 27 07:14:11 stream4tech-proxmox-2 pmxcfs[1306]: [status] notice: update cluster info (cluster name s4t-hetzner, version = 9)
Feb 27 07:14:11 stream4tech-proxmox-2 pmxcfs[1306]: [status] notice: node has quorum
Feb 27 07:14:11 stream4tech-proxmox-2 pmxcfs[1306]: [dcdb] notice: members: 1/1261, 2/1306, 3/1987146, 4/2435024
Feb 27 07:14:11 stream4tech-proxmox-2 pmxcfs[1306]: [dcdb] notice: starting data syncronisation
Feb 27 07:14:11 stream4tech-proxmox-2 pmxcfs[1306]: [dcdb] notice: received sync request (epoch 1/1261/0000002B)
Feb 27 07:14:11 stream4tech-proxmox-2 pmxcfs[1306]: [status] notice: members: 1/1261, 2/1306, 3/1987146, 4/2435024
Feb 27 07:14:11 stream4tech-proxmox-2 pmxcfs[1306]: [status] notice: starting data syncronisation
Feb 27 07:14:11 stream4tech-proxmox-2 pmxcfs[1306]: [status] notice: received sync request (epoch 1/1261/00000027)
Feb 27 07:14:11 stream4tech-proxmox-2 pmxcfs[1306]: [dcdb] notice: received all states
Feb 27 07:14:11 stream4tech-proxmox-2 pmxcfs[1306]: [dcdb] notice: leader is 1/1261
Feb 27 07:14:11 stream4tech-proxmox-2 pmxcfs[1306]: [dcdb] notice: synced members: 1/1261, 3/1987146, 4/2435024
Feb 27 07:14:11 stream4tech-proxmox-2 pmxcfs[1306]: [dcdb] notice: waiting for updates from leader
Feb 27 07:14:11 stream4tech-proxmox-2 pmxcfs[1306]: [status] notice: received all states
Feb 27 07:14:11 stream4tech-proxmox-2 pmxcfs[1306]: [status] notice: all data is up to date
Feb 27 07:14:11 stream4tech-proxmox-2 pmxcfs[1306]: [dcdb] notice: update complete - trying to commit (got 4 inode updates)
Feb 27 07:14:11 stream4tech-proxmox-2 pmxcfs[1306]: [dcdb] notice: all data is up to date
Feb 27 07:32:35 stream4tech-proxmox-2 pmxcfs[1306]: [dcdb] notice: data verification successful
Last edited: