errors on file system

[etaggart]

I've been having some problems with my filesystem and getting the following syslog message after booting.

pve kernel: EXT4-fs (dm-10): warning: mounting fs with errors, running e2fsck is recommended

I'm new to proxmox and LVM file systems so I'm not sure the best way to address this is.

I've tried booting into single user mode and running e2fsck -p /dev/sda3 but i'm told it's in use, however when I try mount /dev/sda3 i'm told it's not mounted.

Sorry for the newbie question, but any help would be appreciated.

 

[fireon]

easiest way:

touch /forcefsck
reboot

But the better question is why you have this error? Some powerlost or damage HDDs...

 

[etaggart]

Thanks. I tried that but I'm still getting the same error.

Nov 13 09:54:49 pve kernel: EXT4-fs (dm-9): warning: mounting fs with errors, running e2fsck is recommended
Nov 13 09:54:49 pve kernel: EXT4-fs (dm-9): mounted filesystem with ordered data mode. Opts: (null)
...
Nov 13 09:59:49 pve kernel: EXT4-fs (dm-9): error count since last fsck: 4
Nov 13 09:59:49 pve kernel: EXT4-fs (dm-9): initial error at time 1478905387: ext4_journal_check_start:56
Nov 13 09:59:49 pve kernel: EXT4-fs (dm-9): last error at time 1478908776: ext4_put_super:813

It's a pretty new SSD drive, not that that means anything...

root@pve:~# fdisk -l

Disk /dev/ram0: 64 MiB, 67108864 bytes, 131072 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram1: 64 MiB, 67108864 bytes, 131072 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram2: 64 MiB, 67108864 bytes, 131072 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram3: 64 MiB, 67108864 bytes, 131072 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram4: 64 MiB, 67108864 bytes, 131072 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram5: 64 MiB, 67108864 bytes, 131072 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram6: 64 MiB, 67108864 bytes, 131072 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram7: 64 MiB, 67108864 bytes, 131072 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram8: 64 MiB, 67108864 bytes, 131072 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram9: 64 MiB, 67108864 bytes, 131072 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram10: 64 MiB, 67108864 bytes, 131072 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram11: 64 MiB, 67108864 bytes, 131072 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram12: 64 MiB, 67108864 bytes, 131072 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram13: 64 MiB, 67108864 bytes, 131072 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram14: 64 MiB, 67108864 bytes, 131072 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram15: 64 MiB, 67108864 bytes, 131072 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/sda: 119.2 GiB, 128035676160 bytes, 250069680 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: F6E39634-6FBF-4BCE-ADDD-F30AB03DA489

Device Start End Sectors Size Type
/dev/sda1 34 2047 2014 1007K BIOS boot
/dev/sda2 2048 262143 260096 127M EFI System
/dev/sda3 262144 250069646 249807503 119.1G Linux LVM

Disk /dev/mapper/pve-root: 29.8 GiB, 31943819264 bytes, 62390272 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/pve-swap: 7 GiB, 7516192768 bytes, 14680064 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/pve-vm--100--state--Oct_2016: 4.5 GiB, 4819255296 bytes, 9412608 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disk /dev/mapper/pve-vm--100--state--Nov_2016: 4.5 GiB, 4819255296 bytes, 9412608 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disk /dev/mapper/pve-vm--102--disk--1: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disk /dev/mapper/pve-vm--101--disk--1: 18 GiB, 19327352832 bytes, 37748736 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disk /dev/mapper/pve-vm--103--disk--1: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disk /dev/mapper/pve-vm--100--disk--1: 32 GiB, 34359738368 bytes, 67108864 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: dos
Disk identifier: 0x0002290a

Device Boot Start End Sectors Size Id Type
/dev/mapper/pve-vm--100--disk--1p1 * 2048 718848 716801 350M 83 Linux
/dev/mapper/pve-vm--100--disk--1p2 718849 4913152 4194304 2G 82 Linux swap / Solaris
/dev/mapper/pve-vm--100--disk--1p3 4913153 7010304 2097152 1G 83 Linux
/dev/mapper/pve-vm--100--disk--1p4 7010305 67108863 60098559 28.7G f W95 Ext'd (LBA)
/dev/mapper/pve-vm--100--disk--1p5 7010306 27076608 20066303 9.6G 83 Linux
/dev/mapper/pve-vm--100--disk--1p6 27076610 38340608 11263999 5.4G 83 Linux
/dev/mapper/pve-vm--100--disk--1p7 38340610 64704512 26363903 12.6G 83 Linux
/dev/mapper/pve-vm--100--disk--1p8 64704514 66648064 1943551 949M 83 Linux

Partition 3 does not start on physical sector boundary.

Partition 4 does not start on physical sector boundary.

Partition 5 does not start on physical sector boundary.

Partition 6 does not start on physical sector boundary.

Partition 7 does not start on physical sector boundary.

Partition 8 does not start on physical sector boundary.

Partition 9 does not start on physical sector boundary.


Disk /dev/mapper/pve-vm--100--state--Nov_2016_2: 4.5 GiB, 4819255296 bytes, 9412608 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes

 

[fireon]

What says SMART? And please boot with an Livedvd and run fsck.ext4 manual. Was the filesystem checked when you touched the file?

But it looks likes the error is in the VM100. Did you have an ext4 on this VM? When yes, check this.

 

[etaggart]

VM100 is the one giving me trouble. It's a Sophos UTM and it is ext4 file system.

Here is the smart info for SDA below.. I'll try booting from a ubuntu liveCD and running e2fsck again.


root@pve:~# smartctl -a /dev/sda
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.4.6-1-pve] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model: C400-MTFDDAT128MAM
Serial Number: 0000000013060366B0C3
LU WWN Device Id: 5 00a075 10366b0c3
Firmware Version: 07MH
User Capacity: 128,035,676,160 bytes [128 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: < 1.8 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 6
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sun Nov 13 16:04:28 2016 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x80) Offline data collection activity
was never started.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 595) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 9) minutes.
Conveyance self-test routine
recommended polling time: ( 3) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 100 100 050 Pre-fail Always - 0
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
9 Power_On_Hours 0x0032 100 100 001 Old_age Always - 1745
12 Power_Cycle_Count 0x0032 100 100 001 Old_age Always - 90
170 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
171 Unknown_Attribute 0x0032 100 100 001 Old_age Always - 0
172 Unknown_Attribute 0x0032 100 100 001 Old_age Always - 0
173 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 11
174 Unknown_Attribute 0x0032 100 100 001 Old_age Always - 79
181 Program_Fail_Cnt_Total 0x0022 100 100 001 Old_age Always - 7670856418382
183 Runtime_Bad_Block 0x0032 100 100 001 Old_age Always - 0
184 End-to-End_Error 0x0033 100 100 050 Pre-fail Always - 0
187 Reported_Uncorrect 0x0032 100 100 001 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 001 Old_age Always - 0
189 Unknown_SSD_Attribute 0x000e 100 100 001 Old_age Always - 110
194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 0
195 Hardware_ECC_Recovered 0x003a 100 100 001 Old_age Always - 0
196 Reallocated_Event_Count 0x0032 100 100 001 Old_age Always - 0
197 Current_Pending_Sector 0x0032 100 100 001 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 001 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 100 100 001 Old_age Always - 0
202 Unknown_SSD_Attribute 0x0018 100 100 001 Old_age Offline - 0
206 Unknown_SSD_Attribute 0x000e 100 100 001 Old_age Always - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[fireon]

No Errors Logged

Looks good. Did you run fsck in your VM too? When not, run.