/var/lib/vz contains a file system with errors

T

TechLineX

Active Member
Mar 2, 2015
213
5
38
2ZHGf.png
+

In the last weeks, the server went down all 2 days randomly without any reason. Every boot, a fsck is running.
 
Hi,

does the syslog has a hit why the node goes down?
Can it be that the disk get old and is broken?

What smartmon tells about this disk?
 
Hi wolfgang,

Nothing special in the syslog or daemon log.

Code: 
root@host16:~# smartctl -a -d ata /dev/sda
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.4.35-1-pve] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Hitachi/HGST Ultrastar 7K4000
Device Model:     HGST HUS724020ALA640
Serial Number:    PN2181P5GYLJVX
LU WWN Device Id: 5 000cca 24ecd745f
Firmware Version: MF6OABY0
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Dec 29 17:11:55 2016 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (   28) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 321) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   137   137   054    Pre-fail  Offline      -       79
  3 Spin_Up_Time            0x0007   158   158   024    Pre-fail  Always       -       412 (Average 377)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       67
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   142   142   020    Pre-fail  Offline      -       25
  9 Power_On_Hours          0x0012   099   099   000    Old_age   Always       -       11190
10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       51
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       137
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       137
194 Temperature_Celsius     0x0002   193   193   000    Old_age   Always       -       31 (Min/Max 20/47)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      8200         -
# 2  Short offline       Completed without error       00%      8192         -
# 3  Short offline       Completed without error       00%      8192         -
# 4  Short offline       Completed without error       00%      7596         -
# 5  Short offline       Completed without error       00%      7589         -
# 6  Short offline       Completed without error       00%      7588         -
# 7  Short offline       Completed without error       00%      7385         -
# 8  Short offline       Completed without error       00%      7384         -
# 9  Short offline       Completed without error       00%      7384         -
#10  Short offline       Completed without error       00%      7335         -
#11  Short offline       Completed without error       00%      7327         -
#12  Short offline       Completed without error       00%      7327         -
#13  Short offline       Completed without error       00%      7305         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
 
Code: 
root@host16:~# smartctl -a -d ata /dev/sdb
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.4.35-1-pve] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     HGST HUS726020ALA610
Serial Number:    N4G6W3XY
LU WWN Device Id: 5 000cca 245c3200e
Firmware Version: A5GNT7J7
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Dec 29 17:12:00 2016 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  113) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 288) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   135   135   054    Pre-fail  Offline      -       112
  3 Spin_Up_Time            0x0007   139   139   024    Pre-fail  Always       -       215 (Average 215)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       15
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   128   128   020    Pre-fail  Offline      -       18
  9 Power_On_Hours          0x0012   100   100   000    Old_age   Always       -       1871
10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       15
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       85
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       85
194 Temperature_Celsius     0x0002   200   200   000    Old_age   Always       -       30 (Min/Max 19/37)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
 
Code: 
root@host16:~# smartctl -a -d ata /dev/sdc
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.4.35-1-pve] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     HGST HUS726020ALA610
Serial Number:    N4G38V4Y
LU WWN Device Id: 5 000cca 245c17e4f
Firmware Version: A5GNT920
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Dec 29 17:12:03 2016 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  113) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 288) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   136   136   054    Pre-fail  Offline      -       108
  3 Spin_Up_Time            0x0007   134   134   024    Pre-fail  Always       -       223 (Average 223)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       57
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   128   128   020    Pre-fail  Offline      -       18
  9 Power_On_Hours          0x0012   100   100   000    Old_age   Always       -       5871
10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       57
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       275
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       275
194 Temperature_Celsius     0x0002   222   222   000    Old_age   Always       -       27 (Min/Max 17/50)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      2877         -
# 2  Short offline       Completed without error       00%      2869         -
# 3  Short offline       Completed without error       00%      2869         -
# 4  Short offline       Completed without error       00%      2274         -
# 5  Short offline       Completed without error       00%      2266         -
# 6  Short offline       Completed without error       00%      2266         -
# 7  Short offline       Completed without error       00%      2062         -
# 8  Short offline       Completed without error       00%      2061         -
# 9  Short offline       Completed without error       00%      2061         -
#10  Short offline       Completed without error       00%      2000         -
#11  Short offline       Completed without error       00%      1996         -
#12  Short offline       Completed without error       00%      1996         -
#13  Short offline       Completed without error       00%       196         -
#14  Short offline       Completed without error       00%       189         -
#15  Short offline       Completed without error       00%       189         -
#16  Short offline       Completed without error       00%       145         -
#17  Short offline       Completed without error       00%       145         -
#18  Short offline       Completed without error       00%       122         -
#19  Short offline       Completed without error       00%       121         -
#20  Short offline       Completed without error       00%        58         -
#21  Short offline       Completed without error       00%         1         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Regards!
 
Do you use a software raid5?

If so I would recommend you to change this because
raid5 and virtualization are no good combination.
Use instead a raid10.
 
wolfgang said:
Do you use a software raid5?

If so I would recommend you to change this because
raid5 and virtualization are no good combination.
Use instead a raid10.
Click to expand...
Hello wolfgang,
Is it only for "Software Raid5" or do you think Hardware Raid5 is also bad combination ?
Just want to make sure...
 
TechLineX said:
I use no RAID 5, use the same RAID-configuration as all other Servers in my Cluster
Click to expand...
and would you be so nice to tell us what this configuration is :-)
I have no clue how your disk layout looks like.

Xahid said:
Is it only for "Software Raid5" or do you think Hardware Raid5 is also bad combination ?
Click to expand...
Raid 5 is always a bad idea
first bit rot is very presents in this setup.
second has bad random write with parallel access (more then 2 Vm are parallel workload)
thirdly SW only cpu usage.
 
Configured as a raid 0:

root@host16:~# cat /proc/mdstat
Personalities : [raid1] [raid0]
md4 : active raid0 sda4[0] sdc4[2] sdb4[1]
5782310400 blocks 512k chunks

md2 : active raid1 sda2[0] sdc2[2] sdb2[1]
20478912 blocks [3/3] [UUU]

unused devices: <none>

Sorry for the late reply
 
Just my 50 cents, when running RAID0 all it takes is a single disk to "flap" wether due to a bad SATA cable or a failing disk and this will cause a Kernel Panic due to delayed I/O (Not easy to pickup in your logs). You also will also lose 100% of your data once a single disk does fully fail (which I am sure your aware)
 
We´re writing daily backups, that isn't a problem. But the problem with the flap is new for me.
 
Yes, nothing special.. - it looks like someone put out a cable out of the server...

I can't find any reason
 
TechLineX said:
Yes, nothing special.. - it looks like someone put out a cable out of the server...

I can't find any reason
Click to expand...

So the server is fully off (console says no video output) for example, how quickly are you able to check from loss of the server?
 
TechLineX said:
The Server is not offline, is directly restarting
Click to expand...

Have you checked ILO/Server logs encase there is an issue with the server / PSU itself causing the restart, which off the back of a hard power pull will cause a filesystem check.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom