VE hang randomly every few days

[johnny21]

Hello,

First of all, I'm not expert.
I setup my Proxmox VE about year and a half ago on one HP workstation (tower PC) and have two VM's and three LXC's. I have daily backups setup (during the night) and Metric server to InfluxDB (one of LXC containers). VE version is 7.4-17.

The problem is that very randomly VE hang up and it's not accessible anymore (no ping, no ssh, no giu). I run Homeassistant and MQTT as two separated VM's and that mean all my home automations not working that time. I need to force shutdown machine, power up and then everything boot up normally. I'm checking syslog, but can't find the reason why VE environment stop responding. Last time happen today, please find few last syslog lines below (force manual shutdown and startup was done on Nov 21 06:57:52).

Nov 20 23:34:56 b38prox smartd[789]: Device: /dev/sdc [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 69 to 70
Nov 21 00:00:59 b38prox systemd[1]: Starting Rotate log files...
Nov 21 00:00:59 b38prox systemd[1]: Starting Daily man-db regeneration...
Nov 21 00:00:59 b38prox systemd[1]: Reloading PVE API Proxy Server.
Nov 21 00:01:02 b38prox systemd[1]: man-db.service: Succeeded.
Nov 21 00:01:02 b38prox systemd[1]: Finished Daily man-db regeneration.
Nov 21 00:01:03 b38prox pveproxy[1208869]: send HUP to 1164
Nov 21 00:01:03 b38prox pveproxy[1164]: received signal HUP
Nov 21 00:01:03 b38prox systemd[1]: Reloaded PVE API Proxy Server.
Nov 21 00:01:03 b38prox pveproxy[1164]: server closing
Nov 21 00:01:03 b38prox pveproxy[1164]: server shutdown (restart)
Nov 21 00:01:03 b38prox systemd[1]: Reloading PVE SPICE Proxy Server.
Nov 21 00:01:03 b38prox spiceproxy[1208906]: send HUP to 1170
Nov 21 00:01:03 b38prox systemd[1]: Reloaded PVE SPICE Proxy Server.
Nov 21 00:01:03 b38prox spiceproxy[1170]: received signal HUP
Nov 21 00:01:03 b38prox spiceproxy[1170]: server closing
Nov 21 00:01:03 b38prox spiceproxy[1170]: server shutdown (restart)
Nov 21 00:01:03 b38prox systemd[1]: Stopping Proxmox VE firewall logger...
Nov 21 00:01:03 b38prox pvefw-logger[564713]: received terminate request (signal)
Nov 21 00:01:03 b38prox pvefw-logger[564713]: stopping pvefw logger
Nov 21 00:01:03 b38prox systemd[1]: pvefw-logger.service: Succeeded.
Nov 21 00:01:03 b38prox systemd[1]: Stopped Proxmox VE firewall logger.
Nov 21 00:01:03 b38prox systemd[1]: pvefw-logger.service: Consumed 7.066s CPU time.
Nov 21 00:01:04 b38prox systemd[1]: Starting Proxmox VE firewall logger...
Nov 21 00:01:04 b38prox systemd[1]: Started Proxmox VE firewall logger.
Nov 21 00:01:04 b38prox pvefw-logger[1208916]: starting pvefw logger
Nov 21 00:01:04 b38prox systemd[1]: logrotate.service: Succeeded.
Nov 21 00:01:04 b38prox systemd[1]: Finished Rotate log files.
Nov 21 00:01:04 b38prox spiceproxy[1170]: restarting server
Nov 21 00:01:04 b38prox spiceproxy[1170]: starting 1 worker(s)
Nov 21 00:01:04 b38prox spiceproxy[1170]: worker 1208920 started
Nov 21 00:01:05 b38prox pveproxy[1164]: restarting server
Nov 21 00:01:05 b38prox pveproxy[1164]: starting 3 worker(s)
Nov 21 00:01:05 b38prox pveproxy[1164]: worker 1208941 started
Nov 21 00:01:05 b38prox pveproxy[1164]: worker 1208942 started
Nov 21 00:01:05 b38prox pveproxy[1164]: worker 1208943 started
Nov 21 00:01:09 b38prox spiceproxy[564717]: worker exit
Nov 21 00:01:09 b38prox spiceproxy[1170]: worker 564717 finished
Nov 21 00:01:10 b38prox pveproxy[564720]: worker exit
Nov 21 00:01:10 b38prox pveproxy[564719]: worker exit
Nov 21 00:01:10 b38prox pveproxy[564718]: worker exit
Nov 21 00:01:11 b38prox pveproxy[1164]: worker 564720 finished
Nov 21 00:01:11 b38prox pveproxy[1164]: worker 564719 finished
Nov 21 00:01:11 b38prox pveproxy[1164]: worker 564718 finished
Nov 21 00:17:01 b38prox CRON[1215974]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Nov 21 00:17:01 b38prox CRON[1215975]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Nov 21 00:17:01 b38prox CRON[1215974]: pam_unix(cron:session): session closed for user root
Nov 21 00:34:56 b38prox smartd[789]: Device: /dev/sdc [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 70 to 69
-- Reboot --
Nov 21 06:57:52 b38prox kernel: Linux version 5.15.126-1-pve (build@proxmox) (gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP PVE 5.15.126-1 (2023-10-03T17:24Z) ()
Nov 21 06:57:52 b38prox kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.126-1-pve root=/dev/mapper/pve-root ro quiet
Nov 21 06:57:52 b38prox kernel: KERNEL supported cpus:
Nov 21 06:57:52 b38prox kernel: Intel GenuineIntel
Nov 21 06:57:52 b38prox kernel: AMD AuthenticAMD
Nov 21 06:57:52 b38prox kernel: Hygon HygonGenuine
Nov 21 06:57:52 b38prox kernel: Centaur CentaurHauls
Nov 21 06:57:52 b38prox kernel: zhaoxin Shanghai
.
.
.

Can anyone help me find the reason why this happen? Could be SSD where VM is running maybe?

Thank you and best regards,
Johnny

 

[Moayad]

Hello,

Thank you for the log!

Nov 21 00:34:56 b38prox smartd[789]: Device: /dev/sdc [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 70 to 69

I would check the airflow and cooling/fans your server. Could you also post the output of smarctl of the /dev/sdc ?

 

[johnny21]

Hello,

Thank you for the log!

I would check the airflow and cooling/fans your server. Could you also post the output of smarctl of the /dev/sdc ?

Hi,

please find below smartctl results:

=== START OF INFORMATION SECTION ===
Model Family: Toshiba 2.5" HDD MK..65GSX
Device Model: TOSHIBA MK6465GSX
Serial Number: Y0ACS1DZS
LU WWN Device Id: 5 000039 2e5b81d8c
Firmware Version: GJ002C
User Capacity: 640,135,028,736 bytes [640 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 5400 rpm
Form Factor: 2.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 2.6, 3.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Wed Nov 22 14:19:00 2023 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 120) seconds.
Offline data collection
capabilities: (0x51) SMART execute Offline immediate.
No Auto Offline data collection support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 164) minutes.
SCT capabilities: (0x003f) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 100 100 050 Pre-fail Always - 0
2 Throughput_Performance 0x0007 100 100 050 Pre-fail Always - 0
3 Spin_Up_Time 0x0003 100 100 002 Pre-fail Always - 2587
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 3970
5 Reallocated_Sector_Ct 0x0033 038 038 010 Pre-fail Always - 1272
7 Seek_Error_Rate 0x000f 100 100 050 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 100 100 050 Pre-fail Offline - 0
9 Power_On_Hours 0x0032 050 050 000 Old_age Always - 20177
10 Spin_Retry_Count 0x0013 178 100 030 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 3950
183 Runtime_Bad_Block 0x0022 100 100 001 Old_age Always - 6
184 End-to-End_Error 0x0033 100 100 097 Pre-fail Always - 0
185 Unknown_Attribute 0x0032 100 100 001 Old_age Always - 65535
187 Reported_Uncorrect 0x0032 092 092 000 Old_age Always - 8
188 Command_Timeout 0x0032 100 098 000 Old_age Always - 53
189 High_Fly_Writes 0x003a 100 100 001 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 073 060 045 Old_age Always - 27 (Min/Max 24/34)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 73
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 4390979
193 Load_Cycle_Count 0x0032 076 076 000 Old_age Always - 243749
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 347
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 2

SMART Error Log Version: 1
ATA Error Count: 10 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 10 occurred at disk power-on lifetime: 12810 hours (533 days + 18 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 41 0a 72 90 4b 61 Error: WP at LBA = 0x014b9072 = 21729394

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
61 10 f8 f0 e2 5c 40 00 7d+18:51:04.451 WRITE FPDMA QUEUED
61 10 00 e0 6c 52 40 00 7d+18:51:04.450 WRITE FPDMA QUEUED
61 58 f0 00 c2 27 40 00 7d+18:51:04.450 WRITE FPDMA QUEUED
60 08 e0 a0 ac 6f 40 00 7d+18:51:04.450 READ FPDMA QUEUED
60 08 08 70 90 4b 40 00 7d+18:51:04.449 READ FPDMA QUEUED

Error 9 occurred at disk power-on lifetime: 12810 hours (533 days + 18 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 41 92 72 90 4b 61 Error: WP at LBA = 0x014b9072 = 21729394

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
61 30 f8 40 e2 5c 40 00 7d+18:50:55.186 WRITE FPDMA QUEUED
60 00 a8 e0 90 4b 40 00 7d+18:50:55.186 READ FPDMA QUEUED
61 10 a0 c8 ae 5b 40 00 7d+18:50:52.079 WRITE FPDMA QUEUED
61 20 98 68 c1 27 40 00 7d+18:50:52.079 WRITE FPDMA QUEUED
60 00 90 e0 8f 4b 40 00 7d+18:50:51.317 READ FPDMA QUEUED

Error 8 occurred at disk power-on lifetime: 12570 hours (523 days + 18 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 41 ba 38 4a b3 63 Error: UNC at LBA = 0x03b34a38 = 62081592

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 08 b8 38 4a b3 40 00 47d+11:55:41.308 READ FPDMA QUEUED
60 b0 b0 68 50 70 40 00 47d+11:55:41.308 READ FPDMA QUEUED
60 08 a8 58 50 70 40 00 47d+11:55:41.308 READ FPDMA QUEUED
61 08 a0 d8 5d 53 40 00 47d+11:55:41.308 WRITE FPDMA QUEUED
ea 00 00 00 00 00 a0 00 47d+11:55:37.510 FLUSH CACHE EXT

Error 7 occurred at disk power-on lifetime: 12570 hours (523 days + 18 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 41 aa 38 4a b3 63 Error: WP at LBA = 0x03b34a38 = 62081592

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
61 30 d0 30 5d 53 40 00 47d+11:55:28.835 WRITE FPDMA QUEUED
60 00 c8 c0 4a b3 40 00 47d+11:55:28.829 READ FPDMA QUEUED
61 10 c0 18 d7 12 40 00 47d+11:55:27.081 WRITE FPDMA QUEUED
61 c0 b8 58 d4 12 40 00 47d+11:55:27.081 WRITE FPDMA QUEUED
61 38 b0 20 ce 12 40 00 47d+11:55:27.081 WRITE FPDMA QUEUED

Error 6 occurred at disk power-on lifetime: 9007 hours (375 days + 7 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 41 a2 01 08 40 69 Error: UNC at LBA = 0x09400801 = 155191297

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 08 90 00 08 40 40 00 00:00:33.397 READ FPDMA QUEUED
60 08 10 00 08 00 40 00 00:00:33.391 READ FPDMA QUEUED
60 08 00 00 08 c0 40 00 00:00:33.386 READ FPDMA QUEUED
60 08 c8 00 08 80 40 00 00:00:33.381 READ FPDMA QUEUED
60 08 a8 00 08 40 40 00 00:00:33.375 READ FPDMA QUEUED

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 1456 -
# 2 Short offline Completed without error 00% 1454 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

___________________________________________________________________________________

Looks like I have HDD problems, right? This is 640GB HDD only for daily/monthly scheduled backups.

 

[Moayad]

Hi,

Thank you for the output!

Looks like I have HDD problems, right? This is 640GB HDD only for daily/monthly scheduled backups.

Yes, its old disk, since it has been powered on for over 20000 hours. This is quite a significant amount of time.

187 Reported_Uncorrect 0x0032 092 092 000 Old_age Always - 8

And here reported with 8 uncorrectable errors.


I would make sure that you have a valid backup from all data on that disk, and consider replacing this disk soon.

 

[smittyj]

I have also had recent lockups after the recent updates, my proxmox node is running two Ubuntu22.04 LTS VMs so nothing stressful. I am in the process of running a memtest and will do a BIOS update next.

 

[lemmy]

Hi,
i have the same overheating problem, and then freezing, after some automatic jobs.
it seems to happen randomly. yesterday it happened, and a couple of months before that a few times.
it has 4 lubuntu VM instances running with loads cpu/disk of capacity left.
The cabinet has forced air ventilation.
Proxmox is a single instance with no reducancy or cluster. Backups are written to NAS storage.


This is the smartctl output:

root@pve:~# smartctl -a /dev/sda
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.8.8-2-pve] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 2.5 5400
Device Model:     ST1000LM048-2E7172
Serial Number:    WKPSFQS7
LU WWN Device Id: 5 000c50 0e071cac8
Firmware Version: 0001
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      2.5 inches
TRIM Command:     Available
Device is:        In smartctl database 7.3/5319
ATA Version is:   ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sat Jul  6 08:10:54 2024 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (    0) seconds.
Offline data collection
capabilities:                    (0x71) SMART execute Offline immediate.
                                        No Auto Offline data collection support.
                                        Suspend Offline collection upon new
                                        command.
                                        No Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 163) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x3035) SCT Status supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   083   064   006    Pre-fail  Always       -       192362941
  3 Spin_Up_Time            0x0003   099   099   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       21
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   080   060   045    Pre-fail  Always       -       90862004
  9 Power_On_Hours          0x0032   078   078   000    Old_age   Always       -       19621 (69 58 0)
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       21
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   061   048   040    Old_age   Always       -       39 (Min/Max 38/41)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       1
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       8
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       65
194 Temperature_Celsius     0x0022   039   052   000    Old_age   Always       -       39 (0 19 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       19614 (45 210 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       2373415629
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       11490829353
254 Free_Fall_Sensor        0x0032   100   100   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@pve:~#

(i'm not sure how to interpret the SMART information)