Every weekend after a reboot, the server gets stuck during boot, and no login prompt appears. This issue happens consistently on weekends after automatic reboots."
root@irage172:~# pveversion -v
proxmox-ve: 8.1.0 (running kernel: 6.5.11-8-pve)
pve-manager: 8.1.4 (running version: 8.1.4/ec5affc9e41f1d79)
We use proxmox-ve: 8.1.0 this version. But every weekend system got stuck and unable to ping or ssh of that VM. So every time we need to hard restart the box.Here are I am sharing logs files and also images. Kindly help me to find out the issue.
Feb 9 05:05:13 alert kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-1160.el7.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet LANG=en_IN.UTF-8
Feb 9 05:05:13 alert kernel: Reserving 162MB of memory at 688MB for crashkernel (System RAM: 31999MB)
Feb 9 05:05:13 alert kernel: Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-1160.el7.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet LANG=en_IN.UTF-8
Feb 9 05:05:13 alert kernel: acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM
Feb 9 05:05:13 alert kernel: acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
Feb 9 05:05:13 alert kernel: crash memory driver: version 1.1
Feb 9 05:05:13 alert kernel: BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter.
Feb 9 05:05:21 alert augenrules: failure 1
Feb 9 05:05:21 alert augenrules: failure 1
Feb 9 05:05:21 alert systemd: lm_sensors.service: main process exited, code=exited, status=1/FAILURE
Feb 9 05:05:21 alert systemd: Failed to start Initialize hardware monitoring sensors.
Feb 9 05:05:21 alert systemd: Unit lm_sensors.service entered failed state.
Feb 9 05:05:21 alert systemd: lm_sensors.service failed.
Feb 9 05:05:30 alert systemd: Starting Crash recovery kernel arming...
Feb 9 05:05:30 alert rsyslogd: imjournal: fscanf on state file /var/lib/rsyslog/imjournal.state' failed [v8.24.0-55.el7 try http://www.rsyslog.com/e/2027 ]
Feb 9 05:05:32 alert systemd: Started Crash recovery kernel arming.
Feb 9 05:05:34 alert dnsmasq[1434]: reply failover-telemetry-in.tradingview.com is 104.238.216.204
Feb 9 05:05:43 alert dnsmasq[1434]: cached failover-telemetry-in.tradingview.com is 104.238.216.204
Feb 9 05:06:01 alert bash: raise RuntimeError("none of symservers are online", self.addrs)
Feb 9 05:06:01 alert bash: RuntimeError: ('none of symservers are online', ['tcp://172.20.1.199:5959', 'tcp://172.20.1.217:5959'])
Feb 9 05:07:01 alert service_crash_alert: ({'content-type': 'application/json; charset=UTF-8', 'vary': 'Origin, X-Origin, Referer', 'date': 'Sat, 08 Feb 2025 23:37:01 GMT', 'server': 'ESF', 'x-xss-protection': '0', 'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff', 'expires': 'Sat, 08 Feb 2025 23:37:01 GMT', 'cache-control': 'private', 'set-cookie': 'COMPASS=dynamite-integration=CgAQrfGfvQYaTQAJa4lXutbUFrinH-SlcDvyuekWEGQvbyOUlbtEjX4RnpEgtsNwhsgdU8DlamcdtkBnz6uGWCgI6SZvx0F8aYgkT4KFgRUSghI7ndCUMAE; expires=Tue, 18-Feb-2025 23:37:01 GMT; path=/; Secure; HttpOnly', 'alt-svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000', 'x-l2-request-path': 'l2-managed-13', 'transfer-encoding': 'chunked', 'status': '200', 'content-length': '1899', '-content-encoding': 'gzip'}, b'{\n "name": "spaces/AAAAMtI_xw0/messages/rXStlbjY-gk.rXStlbjY-gk",\n "text": "# backend_rms Crashed with ExecMainCode=0. \\n## With crash log from log_.out \\n b\\"02/09/25, 04:59:17 AM GMT+5:30 : [COMMON] | UI.js:0 | [ \'[MCX_FO] uiData length 1638\' ]\\nUPD_serialize: 39.338ms\\nUPD_zstdCompressData: 30.026ms\\n02/09/25, 04:59:17 AM GMT+5:30 : [COMMON] | UI.js:0 | [ \'sending data to ui on BASKET.RMS.V3.TEST.MCX.FO of length 46992\' ]\\n02/09/25, 04:59:17 AM GMT+5:30 : [COMMON] | UI.js:0 | [ \'[MCX_FO] sending data to ui rawLength=2468018, compressedLength=46992, serialize=true, compressGzip=false, compressZstd=true\' ]\\n[MCX_FO] UPD_sendUIData: 105.038ms\\n02/09/25, 04:59:18 AM GMT+5:30 : [COMMON] | UI.js:0 | [ \'[NSE_CO] uiData length 228\' ]\\nUPD_serialize: 3.856ms\\nUPD_zstdCompressData: 2.974ms\\n02/09/25, 04:59:18 AM GMT+5:30 : [COMMON] | UI.js:0 | [ \'sending data to ui on BASKET.RMS.V3.TEST.NSE.CO of length 5012\' ]\\n02/09/25, 04:59:18 AM GMT+5:30 : [COMMON] | UI.js:0 | [ \'[NSE_CO] sending data to ui rawLength=336376, compressedLength=5012, serialize=true, compressGzip=false, compressZstd=true\' ]\\n[NSE_CO] UPD_sendUIData: 12.259ms\\n02/09/25, 04:59:19 AM GMT+5:30 : [COMMON] | UI.js:0 | [ \'[NSE_FO] uiData length 14459\' ]\\nUPD_serialize: 346.978ms\\nUPD_zstdCompressData: 334.086ms\\n02/09/25, 04:59:20 AM GMT+5:30 : [COMMON] | UI.js:0 | [ \'sending data to ui on BASKET.RMS.V3.TEST.NSE.FO of length 705940\' ]\\n02/09/25, 04:59:20 AM GMT+5:30 : [COMMON] | UI.js:0 | [ \'[NSE_FO] sending data to ui rawLength=24501489, compressedLength=705940, serialize=true, compressGzip=false, compressZstd=true\' ]\\n[NSE_FO] UPD_sendUIData: 1124.135ms\\n02/09/25, 04:59:20 AM GMT+5:30 : [COMMON] | UI.js:0 | [ \'[NSE_CM] uiData length 57988\' ]\\nUPD_serialize: 1241.608ms\\n\\" \\n",\n "thread": {\n "name": "spaces/AAAAMtI_xw0/threads/rXStlbjY-gk"\n },\n "space": {\n "name": "spaces/AAAAMtI_xw0"\n }\n}\n')
Feb 9 05:07:01 alert systemd: Failed to start irage Backend of RMS_UI3.
Feb 9 05:07:01 alert systemd: Unit rms.service entered failed state.
Feb 9 05:07:01 alert systemd: rms.service failed.
Feb 9 05:07:32 alert bash: raise RuntimeError("none of symservers are online", self.addrs)
Feb 9 05:07:32 alert bash: RuntimeError: ('none of symservers are online', ['tcp://172.20.1.199:5959', 'tcp://172.20.1.217:5959'])
Feb 9 05:09:43 alert dnsmasq[1434]: reply failover-telemetry-in.tradingview.com is 104.238.216.204
Feb 9 05:19:43 alert dnsmasq[1434]: reply failover-telemetry-in.tradingview.com is 104.238.216.204
Feb 9 05:20:18 alert dnsmasq[1434]: reply failover-telemetry-in.tradingview.com is 104.238.216.204
Feb 9 05:20:18 alert dnsmasq[1434]: query[type=65] failover-telemetry-in.tradingview.com from 172.16.150.116
Feb 9 05:20:18 alert dnsmasq[1434]: forwarded failover-telemetry-in.tradingview.com to 172.20.1.251
Feb 9 05:20:18 alert dnsmasq[1434]: query[AAAA] failover-telemetry-in.tradingview.com from 172.16.150.116
Feb 9 05:20:18 alert dnsmasq[1434]: forwarded failover-telemetry-in.tradingview.com to 172.20.1.251
Feb 9 05:20:18 alert dnsmasq[1434]: reply failover-telemetry-in.tradingview.com is NODATA-IPv6
Feb 9 23:56:11 alert kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-1160.el7.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet LANG=en_IN.UTF-8
Feb 9 23:56:11 alert kernel: Reserving 162MB of memory at 688MB for crashkernel (System RAM: 31999MB)
Feb 9 23:56:11 alert kernel: Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-1160.el7.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet LANG=en_IN.UTF-8
Feb 9 23:56:11 alert kernel: acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM
Feb 9 23:56:11 alert kernel: acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
Feb 9 23:56:11 alert kernel: crash memory driver: version 1.1
Feb 9 23:56:11 alert kernel: BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter.
Feb 9 23:56:14 alert augenrules: failure 1
Feb 9 23:56:14 alert augenrules: failure 1
Feb 9 23:56:15 alert systemd: lm_sensors.service: main process exited, code=exited, status=1/FAILURE
Feb 9 23:56:15 alert systemd: Failed to start Initialize hardware monitoring sensors.
Feb 9 23:56:15 alert systemd: Unit lm_sensors.service entered failed state.
Feb 9 23:56:15 alert systemd: lm_sensors.service failed.
Feb 9 23:56:23 alert systemd: Starting Crash recovery kernel arming...
Feb 9 23:56:23 alert rsyslogd: imjournal: fscanf on state file /var/lib/rsyslog/imjournal.state' failed [v8.24.0-55.el7 try http://www.rsyslog.com/e/2027 ]
Feb 9 23:56:26 alert systemd: Started Crash recovery kernel arming.
root@irage172:~# pveversion -v
proxmox-ve: 8.1.0 (running kernel: 6.5.11-8-pve)
pve-manager: 8.1.4 (running version: 8.1.4/ec5affc9e41f1d79)
We use proxmox-ve: 8.1.0 this version. But every weekend system got stuck and unable to ping or ssh of that VM. So every time we need to hard restart the box.Here are I am sharing logs files and also images. Kindly help me to find out the issue.
Feb 9 05:05:13 alert kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-1160.el7.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet LANG=en_IN.UTF-8
Feb 9 05:05:13 alert kernel: Reserving 162MB of memory at 688MB for crashkernel (System RAM: 31999MB)
Feb 9 05:05:13 alert kernel: Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-1160.el7.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet LANG=en_IN.UTF-8
Feb 9 05:05:13 alert kernel: acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM
Feb 9 05:05:13 alert kernel: acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
Feb 9 05:05:13 alert kernel: crash memory driver: version 1.1
Feb 9 05:05:13 alert kernel: BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter.
Feb 9 05:05:21 alert augenrules: failure 1
Feb 9 05:05:21 alert augenrules: failure 1
Feb 9 05:05:21 alert systemd: lm_sensors.service: main process exited, code=exited, status=1/FAILURE
Feb 9 05:05:21 alert systemd: Failed to start Initialize hardware monitoring sensors.
Feb 9 05:05:21 alert systemd: Unit lm_sensors.service entered failed state.
Feb 9 05:05:21 alert systemd: lm_sensors.service failed.
Feb 9 05:05:30 alert systemd: Starting Crash recovery kernel arming...
Feb 9 05:05:30 alert rsyslogd: imjournal: fscanf on state file /var/lib/rsyslog/imjournal.state' failed [v8.24.0-55.el7 try http://www.rsyslog.com/e/2027 ]
Feb 9 05:05:32 alert systemd: Started Crash recovery kernel arming.
Feb 9 05:05:34 alert dnsmasq[1434]: reply failover-telemetry-in.tradingview.com is 104.238.216.204
Feb 9 05:05:43 alert dnsmasq[1434]: cached failover-telemetry-in.tradingview.com is 104.238.216.204
Feb 9 05:06:01 alert bash: raise RuntimeError("none of symservers are online", self.addrs)
Feb 9 05:06:01 alert bash: RuntimeError: ('none of symservers are online', ['tcp://172.20.1.199:5959', 'tcp://172.20.1.217:5959'])
Feb 9 05:07:01 alert service_crash_alert: ({'content-type': 'application/json; charset=UTF-8', 'vary': 'Origin, X-Origin, Referer', 'date': 'Sat, 08 Feb 2025 23:37:01 GMT', 'server': 'ESF', 'x-xss-protection': '0', 'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff', 'expires': 'Sat, 08 Feb 2025 23:37:01 GMT', 'cache-control': 'private', 'set-cookie': 'COMPASS=dynamite-integration=CgAQrfGfvQYaTQAJa4lXutbUFrinH-SlcDvyuekWEGQvbyOUlbtEjX4RnpEgtsNwhsgdU8DlamcdtkBnz6uGWCgI6SZvx0F8aYgkT4KFgRUSghI7ndCUMAE; expires=Tue, 18-Feb-2025 23:37:01 GMT; path=/; Secure; HttpOnly', 'alt-svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000', 'x-l2-request-path': 'l2-managed-13', 'transfer-encoding': 'chunked', 'status': '200', 'content-length': '1899', '-content-encoding': 'gzip'}, b'{\n "name": "spaces/AAAAMtI_xw0/messages/rXStlbjY-gk.rXStlbjY-gk",\n "text": "# backend_rms Crashed with ExecMainCode=0. \\n## With crash log from log_.out \\n b\\"02/09/25, 04:59:17 AM GMT+5:30 : [COMMON] | UI.js:0 | [ \'[MCX_FO] uiData length 1638\' ]\\nUPD_serialize: 39.338ms\\nUPD_zstdCompressData: 30.026ms\\n02/09/25, 04:59:17 AM GMT+5:30 : [COMMON] | UI.js:0 | [ \'sending data to ui on BASKET.RMS.V3.TEST.MCX.FO of length 46992\' ]\\n02/09/25, 04:59:17 AM GMT+5:30 : [COMMON] | UI.js:0 | [ \'[MCX_FO] sending data to ui rawLength=2468018, compressedLength=46992, serialize=true, compressGzip=false, compressZstd=true\' ]\\n[MCX_FO] UPD_sendUIData: 105.038ms\\n02/09/25, 04:59:18 AM GMT+5:30 : [COMMON] | UI.js:0 | [ \'[NSE_CO] uiData length 228\' ]\\nUPD_serialize: 3.856ms\\nUPD_zstdCompressData: 2.974ms\\n02/09/25, 04:59:18 AM GMT+5:30 : [COMMON] | UI.js:0 | [ \'sending data to ui on BASKET.RMS.V3.TEST.NSE.CO of length 5012\' ]\\n02/09/25, 04:59:18 AM GMT+5:30 : [COMMON] | UI.js:0 | [ \'[NSE_CO] sending data to ui rawLength=336376, compressedLength=5012, serialize=true, compressGzip=false, compressZstd=true\' ]\\n[NSE_CO] UPD_sendUIData: 12.259ms\\n02/09/25, 04:59:19 AM GMT+5:30 : [COMMON] | UI.js:0 | [ \'[NSE_FO] uiData length 14459\' ]\\nUPD_serialize: 346.978ms\\nUPD_zstdCompressData: 334.086ms\\n02/09/25, 04:59:20 AM GMT+5:30 : [COMMON] | UI.js:0 | [ \'sending data to ui on BASKET.RMS.V3.TEST.NSE.FO of length 705940\' ]\\n02/09/25, 04:59:20 AM GMT+5:30 : [COMMON] | UI.js:0 | [ \'[NSE_FO] sending data to ui rawLength=24501489, compressedLength=705940, serialize=true, compressGzip=false, compressZstd=true\' ]\\n[NSE_FO] UPD_sendUIData: 1124.135ms\\n02/09/25, 04:59:20 AM GMT+5:30 : [COMMON] | UI.js:0 | [ \'[NSE_CM] uiData length 57988\' ]\\nUPD_serialize: 1241.608ms\\n\\" \\n",\n "thread": {\n "name": "spaces/AAAAMtI_xw0/threads/rXStlbjY-gk"\n },\n "space": {\n "name": "spaces/AAAAMtI_xw0"\n }\n}\n')
Feb 9 05:07:01 alert systemd: Failed to start irage Backend of RMS_UI3.
Feb 9 05:07:01 alert systemd: Unit rms.service entered failed state.
Feb 9 05:07:01 alert systemd: rms.service failed.
Feb 9 05:07:32 alert bash: raise RuntimeError("none of symservers are online", self.addrs)
Feb 9 05:07:32 alert bash: RuntimeError: ('none of symservers are online', ['tcp://172.20.1.199:5959', 'tcp://172.20.1.217:5959'])
Feb 9 05:09:43 alert dnsmasq[1434]: reply failover-telemetry-in.tradingview.com is 104.238.216.204
Feb 9 05:19:43 alert dnsmasq[1434]: reply failover-telemetry-in.tradingview.com is 104.238.216.204
Feb 9 05:20:18 alert dnsmasq[1434]: reply failover-telemetry-in.tradingview.com is 104.238.216.204
Feb 9 05:20:18 alert dnsmasq[1434]: query[type=65] failover-telemetry-in.tradingview.com from 172.16.150.116
Feb 9 05:20:18 alert dnsmasq[1434]: forwarded failover-telemetry-in.tradingview.com to 172.20.1.251
Feb 9 05:20:18 alert dnsmasq[1434]: query[AAAA] failover-telemetry-in.tradingview.com from 172.16.150.116
Feb 9 05:20:18 alert dnsmasq[1434]: forwarded failover-telemetry-in.tradingview.com to 172.20.1.251
Feb 9 05:20:18 alert dnsmasq[1434]: reply failover-telemetry-in.tradingview.com is NODATA-IPv6
Feb 9 23:56:11 alert kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-1160.el7.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet LANG=en_IN.UTF-8
Feb 9 23:56:11 alert kernel: Reserving 162MB of memory at 688MB for crashkernel (System RAM: 31999MB)
Feb 9 23:56:11 alert kernel: Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-1160.el7.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet LANG=en_IN.UTF-8
Feb 9 23:56:11 alert kernel: acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM
Feb 9 23:56:11 alert kernel: acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
Feb 9 23:56:11 alert kernel: crash memory driver: version 1.1
Feb 9 23:56:11 alert kernel: BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter.
Feb 9 23:56:14 alert augenrules: failure 1
Feb 9 23:56:14 alert augenrules: failure 1
Feb 9 23:56:15 alert systemd: lm_sensors.service: main process exited, code=exited, status=1/FAILURE
Feb 9 23:56:15 alert systemd: Failed to start Initialize hardware monitoring sensors.
Feb 9 23:56:15 alert systemd: Unit lm_sensors.service entered failed state.
Feb 9 23:56:15 alert systemd: lm_sensors.service failed.
Feb 9 23:56:23 alert systemd: Starting Crash recovery kernel arming...
Feb 9 23:56:23 alert rsyslogd: imjournal: fscanf on state file /var/lib/rsyslog/imjournal.state' failed [v8.24.0-55.el7 try http://www.rsyslog.com/e/2027 ]
Feb 9 23:56:26 alert systemd: Started Crash recovery kernel arming.
