EXT4-fs Error with Raid 10

[roenoe]

Good day,

we have sudden problems with our Proxmox host systems.

Without any indication of failure, the main partition "/" is set to read-only. Obviously there is a data loss in parallel.

The error currently occurs on two systems with Proxmox 6.3-1. A software Raid 10 is installed on both systems.

Is this a known problem and is there a solution?

Thanks in advance.

pveversion -v

proxmox-ve: 6.3-1 (running kernel: 5.4.73-1-pve)
pve-manager: 6.3-2 (running version: 6.3-2/22f57405)
pve-kernel-5.4: 6.3-1
pve-kernel-helper: 6.3-1
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.73-1-pve: 5.4.73-1
pve-kernel-5.4.60-1-pve: 5.4.60-2
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
pve-kernel-5.3.18-3-pve: 5.3.18-3
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libproxmox-backup-qemu0: 1.0.2-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-6
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.3-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.5-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-3
pve-cluster: 6.2-1
pve-container: 3.3-1
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-7
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-1
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.5-pve1

cat /proc/mdstat

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md2 : active raid10 sda4[0] sdd4[3] sdc4[2] sdb4[1]
      933110784 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
      bitmap: 0/7 pages [0KB], 65536KB chunk


md1 : active raid10 sda3[0] sdd3[3] sdc3[2] sdb3[1]
      2093056 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]


md0 : active raid10 sda2[0] sdd2[3] sdc2[2] sdb2[1]
      2093056 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]


unused devices: <none>

dmesg

EXT4-fs error (device md2): ext4_find_extent:920: inode #46137352: comm kvm: pblk 73081648 bad header/extent: invalid magic - magic 0, entries 0, max 0(0), depth 0(0)

[543917.873857] show_signal_msg: 4 callbacks suppressed
[543917.873860] mail[125487]: segfault at c0 ip 00007f8fda306100 sp 00007ffdc1118790 error 4 in libmailutils.so.5.0.5[7f8fda281000+90000]
[543917.873870] Code: 48 89 7d c8 48 89 75 c0 48 89 55 b8 48 89 4d b0 64 48 8b 04 25 28 00 00 00 48 89 45 f8 31 c0 c7 45 d4 00 00 00 00 48 8b 45 c8 <48> 8b 80 c0 00 00 00 48 85 c0 74 4a 48 8b 45 c8 8b 80 c8 00 00 00
[543971.136478] mail[125798]: segfault at c0 ip 00007fcf1ecbd100 sp 00007fff77ca90d0 error 4 in libmailutils.so.5.0.5[7fcf1ec38000+90000]
[543971.136487] Code: 48 89 7d c8 48 89 75 c0 48 89 55 b8 48 89 4d b0 64 48 8b 04 25 28 00 00 00 48 89 45 f8 31 c0 c7 45 d4 00 00 00 00 48 8b 45 c8 <48> 8b 80 c0 00 00 00 48 85 c0 74 4a 48 8b 45 c8 8b 80 c8 00 00 00
[543971.404038] mail[125804]: segfault at c0 ip 00007f51a4487100 sp 00007ffded9d2c10 error 4 in libmailutils.so.5.0.5[7f51a4402000+90000]
[543971.404047] Code: 48 89 7d c8 48 89 75 c0 48 89 55 b8 48 89 4d b0 64 48 8b 04 25 28 00 00 00 48 89 45 f8 31 c0 c7 45 d4 00 00 00 00 48 8b 45 c8 <48> 8b 80 c0 00 00 00 48 85 c0 74 4a 48 8b 45 c8 8b 80 c8 00 00 00
[543974.948390] mail[125813]: segfault at c0 ip 00007f78eac11100 sp 00007ffc2e0f1b20 error 4 in libmailutils.so.5.0.5[7f78eab8c000+90000]
[543974.948398] Code: 48 89 7d c8 48 89 75 c0 48 89 55 b8 48 89 4d b0 64 48 8b 04 25 28 00 00 00 48 89 45 f8 31 c0 c7 45 d4 00 00 00 00 48 8b 45 c8 <48> 8b 80 c0 00 00 00 48 85 c0 74 4a 48 8b 45 c8 8b 80 c8 00 00 00
[544065.546130] mail[126245]: segfault at c0 ip 00007f620bbca100 sp 00007ffe8ced9ad0 error 4 in libmailutils.so.5.0.5[7f620bb45000+90000]
[544065.546139] Code: 48 89 7d c8 48 89 75 c0 48 89 55 b8 48 89 4d b0 64 48 8b 04 25 28 00 00 00 48 89 45 f8 31 c0 c7 45 d4 00 00 00 00 48 8b 45 c8 <48> 8b 80 c0 00 00 00 48 85 c0 74 4a 48 8b 45 c8 8b 80 c8 00 00 00
[544065.783804] mail[126251]: segfault at c0 ip 00007f70614e6100 sp 00007ffefac78480 error 4 in libmailutils.so.5.0.5[7f7061461000+90000]
[544065.783811] Code: 48 89 7d c8 48 89 75 c0 48 89 55 b8 48 89 4d b0 64 48 8b 04 25 28 00 00 00 48 89 45 f8 31 c0 c7 45 d4 00 00 00 00 48 8b 45 c8 <48> 8b 80 c0 00 00 00 48 85 c0 74 4a 48 8b 45 c8 8b 80 c8 00 00 00
[544069.236378] mail[126283]: segfault at c0 ip 00007f537630c100 sp 00007ffd699fd5a0 error 4 in libmailutils.so.5.0.5[7f5376287000+90000]
[544069.236387] Code: 48 89 7d c8 48 89 75 c0 48 89 55 b8 48 89 4d b0 64 48 8b 04 25 28 00 00 00 48 89 45 f8 31 c0 c7 45 d4 00 00 00 00 48 8b 45 c8 <48> 8b 80 c0 00 00 00 48 85 c0 74 4a 48 8b 45 c8 8b 80 c8 00 00 00
[544069.361385] mail[126289]: segfault at c0 ip 00007fdac9b0c100 sp 00007ffe8443cc30 error 4 in libmailutils.so.5.0.5[7fdac9a87000+90000]
[544069.361393] Code: 48 89 7d c8 48 89 75 c0 48 89 55 b8 48 89 4d b0 64 48 8b 04 25 28 00 00 00 48 89 45 f8 31 c0 c7 45 d4 00 00 00 00 48 8b 45 c8 <48> 8b 80 c0 00 00 00 48 85 c0 74 4a 48 8b 45 c8 8b 80 c8 00 00 00
[544069.493473] mail[126295]: segfault at c0 ip 00007fe74aaf4100 sp 00007fff2adeb1f0 error 4 in libmailutils.so.5.0.5[7fe74aa6f000+90000]
[544069.493480] Code: 48 89 7d c8 48 89 75 c0 48 89 55 b8 48 89 4d b0 64 48 8b 04 25 28 00 00 00 48 89 45 f8 31 c0 c7 45 d4 00 00 00 00 48 8b 45 c8 <48> 8b 80 c0 00 00 00 48 85 c0 74 4a 48 8b 45 c8 8b 80 c8 00 00 00
[544197.487696] mail[127035]: segfault at c0 ip 00007fdb86888100 sp 00007ffd941da750 error 4 in libmailutils.so.5.0.5[7fdb86803000+90000]
[544197.487704] Code: 48 89 7d c8 48 89 75 c0 48 89 55 b8 48 89 4d b0 64 48 8b 04 25 28 00 00 00 48 89 45 f8 31 c0 c7 45 d4 00 00 00 00 48 8b 45 c8 <48> 8b 80 c0 00 00 00 48 85 c0 74 4a 48 8b 45 c8 8b 80 c8 00 00 00
[544213.835012] mail[127231]: segfault at c0 ip 00007ffbf4c83100 sp 00007fffedca1870 error 4 in libmailutils.so.5.0.5[7ffbf4bfe000+90000]
[544213.835021] Code: 48 89 7d c8 48 89 75 c0 48 89 55 b8 48 89 4d b0 64 48 8b 04 25 28 00 00 00 48 89 45 f8 31 c0 c7 45 d4 00 00 00 00 48 8b 45 c8 <48> 8b 80 c0 00 00 00 48 85 c0 74 4a 48 8b 45 c8 8b 80 c8 00 00 00
[544214.419876] mail[127237]: segfault at c0 ip 00007fdb54f38100 sp 00007ffe94137df0 error 4 in libmailutils.so.5.0.5[7fdb54eb3000+90000]
[544214.419885] Code: 48 89 7d c8 48 89 75 c0 48 89 55 b8 48 89 4d b0 64 48 8b 04 25 28 00 00 00 48 89 45 f8 31 c0 c7 45 d4 00 00 00 00 48 8b 45 c8 <48> 8b 80 c0 00 00 00 48 85 c0 74 4a 48 8b 45 c8 8b 80 c8 00 00 00
[544218.417779] mail[127269]: segfault at c0 ip 00007fb1f018a100 sp 00007fff0f325e50 error 4 in libmailutils.so.5.0.5[7fb1f0105000+90000]
[544218.417787] Code: 48 89 7d c8 48 89 75 c0 48 89 55 b8 48 89 4d b0 64 48 8b 04 25 28 00 00 00 48 89 45 f8 31 c0 c7 45 d4 00 00 00 00 48 8b 45 c8 <48> 8b 80 c0 00 00 00 48 85 c0 74 4a 48 8b 45 c8 8b 80 c8 00 00 00

 

[mira]

Do you get errors similar to the ones in the following thread as well?
https://forum.proxmox.com/threads/sudden-filesystem-errors.79974/

 

[roenoe]

Hello mira,

thank you for your prompt reply.

We get similar errors in kern.log like:

EXT4-fs error (device md2): ext4_find_extent:920: inode #46137352: comm kvm: pblk 73081648 bad header/extent: invalid magic - magic 0, entries 0, max 0(0), depth 0(0)

How can we rescue the VMs without data loss and fix the problem in the long term?

Thanks in advance.

 

[Sascha72036]

Hello,

have the similar issue. Is there a recommendation from Proxmox how to rescue the VM data?

Best regards,
S. Gericke