EXT4-fs error (device dm-1):

L

lubbe5

New Member
Jun 4, 2026
2
0
1
Hello,


I'm really at a loss. Since yesterday, my Proxmox server has crashed, and after a reboot the server is no longer reachable.
I can no longer log in through the console either. The only way I can access a console is by booting from the USB stick (that contains the Proxmox installation files.)


Please see the error message below.


Can anyone help me? The files stored on the server are very important to me, and I'm worried about losing them.


Thank you in advance for any assistance.

----------------------------
Welcome to the Proxmox Virtual Environment. Please use your web browser to
configure this server - connect to:

https://192.168.1.xxx:8006/

Proxmox-Server login:

[ 18.419043] EXT4-fs error (device dm-1):
ext4_journal_check_start:87: comm systemd-journal:
Detected aborted journal

[ 18.419115] EXT4-fs (dm-1):
Remounting filesystem read-only
 
Check these
Bash: 
journalctl -kr
ls -l /dev/mapper/
lsblk -o+FSTYLE,LABEL,MODEL
If you care about your data back it up.
 
@Impact, is on the money, if you need it, back it up.

When you say files, do you mean your guests?

What you can do is attach an external storage device, mount it, and then do a

Code: 
cp -r /etc/pve <your mounted storage device>

(for example)

cp -r /etc/pve /mnt/external_storage

Then you need something like clonezilla, bootable usb, and clone the corrupt physical drive.

Once you have that drive cloned and the pve folder all backed up.

From that point on you then need to just try any number of routes that will result in getting back to a running system, first off a new drive is needed, keep the clone safe until you are happy that you got everything up and running.
 
Thanks for the help everybody,

I was installing a truenas at the proxmox, but too late with the backup :mad:
 
lubbe5 said:
I was installing a truenas at the proxmox, but too late with the backup :mad:
Click to expand...

It's never too late unless the drive is totally dead. Clone the drive the first chance you get, do NOT disable RO mode, it does that to protect the data, at this point you have no idea what could have caused the ext4 journalling issue, hence cloning it at this stage is paramount, so that you have a point you can keep going back to until you recover.

I've recovered from far worse situations, the main rule is to get a clone before you do anything else!
 
Impact said:
journalctl -kr
ls -l /dev/mapper/
lsblk -o+FSTYLE,LABEL,MODEL
Click to expand...
lsblk -o+FSTYPE,LABEL,MODEL

Please post the requested info.

Were on your Proxmox server are the files you are worried about?
Depending on your setup they may be on another filesystem than the one that is currently troubled.

After backing up you should check the filesystem on /dev/dm-1 when unmounted.
You can do this by booting rescue mode from the installer USB and issue:
fsck.ext4 -n /dev/dm-1
This performs a read-only check of the ext4 filesystem without making any changes or repairs (yet).
 
Got this problem after an update couple of days ago. Spent 12 hours trying to figure out what it was. Downgrading pve-firmware and loading an older kernel worked and everything was accessible again. Just downgrading one of these did not do anything. I could still chroot to pve-root from liveusb with all data and VMs being in mint condition. This had nothing to do with data corruption, the disk was clean.
The filesystem journal had errors because the kernel communicates incorrectly with the Kingston NVME (in my case) and causes this catastrophic failure, later it simply remounts the root file system as read-only and starts spamming journal log with inability to write to the journal. This is a bug.
 
Last edited:
gfngfn256 said:
Possibly. However since you have not provided detailed info. as to what HW & PVE version etc. you are using, no analysis can be made concerning a potential bug. This also makes your post mostly useless to any other users.
Click to expand...
I am very sorry, I will provide them now. The kernel version is Kernel 6.8.12-23-pve, PVE version 8.4.19. Upgrade to 6.8.12-29 caused the aforementioned error. pve-firmware version 3.15-3 is the one that is working, update to 3.16-3 caused the problem. Please tell me if anything else can be useful.
lshw -short
system VP4650
/0 bus VP4650
/0/0 memory 1MiB BIOS
/0/4 processor Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz
/0/4/6 memory 128KiB L1 cache
/0/4/7 memory 1MiB L2 cache
/0/4/8 memory 6MiB L3 cache
/0/5 memory 128KiB L1 cache
/0/9 memory 64GiB System Memory
/0/9/0 memory 32GiB SODIMM DDR4 Synchronous 2667 MHz (0.4 ns)
/0/9/1 memory 32GiB SODIMM DDR4 Synchronous 2667 MHz (0.4 ns)
/0/100 bridge Comet Lake-U v1 4c Host Bridge/DRAM Controller
/0/100/2 /dev/fb0 display Comet Lake UHD Graphics
/0/100/8 /dev/fb0 generic Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model
/0/100/12 generic Comet Lake Thermal Subsytem
/0/100/14 bus Comet Lake PCH-LP USB 3.1 xHCI Host Controller
/0/100/14/0 usb1 bus xHCI Host Controller
/0/100/14/1 usb2 bus xHCI Host Controller
/0/100/14.2 memory RAM memory
/0/100/17 scsi0 storage Comet Lake SATA AHCI Controller
/0/100/17/0.0.0 /dev/sda volume 223GiB KINGSTON SA400S3
/0/100/1a mmc0 bus Intel Corporation
/0/100/1a/1 generic AJTD4R
/0/100/1a/1/1 /dev/mmcblk0rpmb generic
/0/100/1a/1/2 /dev/mmcblk0 generic
/0/100/1c bridge Comet Lake PCI Express Root Port #5
/0/100/1c/0 enp1s0 network Ethernet Controller I225-V
/0/100/1c.5 bridge Intel Corporation
/0/100/1c.5/0 enp2s0 network Ethernet Controller I225-V
/0/100/1c.6 bridge Intel Corporation
/0/100/1c.6/0 network Ethernet Controller I225-V
/0/100/1c.7 bridge Intel Corporation
/0/100/1c.7/0 network Ethernet Controller I225-V
/0/100/1d bridge Comet Lake PCI Express Root Port #9
/0/100/1d/0 network Ethernet Controller I225-V
/0/100/1d.1 bridge Comet Lake PCI Express Root Port #10
/0/100/1d.1/0 enp6s0 network Ethernet Controller I225-V
/0/100/1d.3 bridge Comet Lake PCI Express Root Port #12
/0/100/1d.4 bridge Comet Lake PCI Express Root Port #13
/0/100/1d.4/0 /dev/nvme0 storage KINGSTON SNV2S500G
/0/100/1d.4/0/0 hwmon0 disk NVMe disk
/0/100/1d.4/0/2 /dev/ng0n1 disk NVMe disk
/0/100/1d.4/0/1 /dev/nvme0n1 disk 500GB NVMe disk
/0/100/1d.4/0/1/1 /dev/nvme0n1p1 volume 1006KiB BIOS Boot partition
/0/100/1d.4/0/1/2 /dev/nvme0n1p2 volume 1023MiB Windows FAT volume
/0/100/1d.4/0/1/3 /dev/nvme0n1p3 volume 464GiB LVM Physical Volume
/0/100/1f bridge Comet Lake PCH-LP LPC Premium Controller/eSPI Controller
/0/100/1f/0 system PnP device PNP0c02
/0/100/1f/1 system PnP device PNP0c02
/0/100/1f/2 system PnP device PNP0103
/0/100/1f/3 system PnP device PNP0c02
/0/100/1f/4 system PnP device PNP0b00
/0/100/1f/5 system PnP device PNP0c02
/0/100/1f.3 card0 multimedia Comet Lake PCH-LP cAVS
/0/100/1f.3/0 input5 input HDA Intel PCH HDMI/DP,pcm=3
/0/100/1f.3/1 input6 input HDA Intel PCH HDMI/DP,pcm=7
/0/100/1f.3/2 input7 input HDA Intel PCH HDMI/DP,pcm=8
/0/100/1f.4 bus Comet Lake PCH-LP SMBus Host Controller
/0/100/1f.5 bus Comet Lake SPI (flash) Controller
/1 input0 input Power Button

smartctl /dev/nvme0n1:

martctl 7.3 2022-02-28 r5338 [x86_64-linux-6.8.12-23-pve] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number: KINGSTON SNV2S500G
Serial Number: 50026B7785A2C72B
Firmware Version: EJFK3N.9
PCI Vendor/Subsystem ID: 0x2646
IEEE OUI Identifier: 0x0026b7
Controller ID: 0
NVMe Version: 1.4
Number of Namespaces: 1
Namespace 1 Size/Capacity: 500,107,862,016 [500 GB]
Namespace 1 Formatted LBA Size: 512
Namespace 1 IEEE EUI-64: 0026b7 785a2c72b5
Local Time is: Sun Jun 14 15:35:53 2026 CEST
Firmware Updates (0x12): 1 Slot, no Reset required
Optional Admin Commands (0x0017): Security Format Frmw_DL Self_Test
Optional NVM Commands (0x0057): Comp Wr_Unc DS_Mngmt Sav/Sel_Feat Timestmp
Log Page Attributes (0x1e): Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg Pers_Ev_Lg
Maximum Data Transfer Size: 64 Pages
Warning Comp. Temp. Threshold: 77 Celsius
Critical Comp. Temp. Threshold: 79 Celsius

Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 5.00W - - 0 0 0 0 0 0
1 + 5.00W - - 1 1 1 1 0 0
2 + 5.00W - - 2 2 2 2 0 0
3 - 5.00W - - 3 3 3 3 999000 999000
4 - 5.00W - - 4 4 4 4 999000 999000

Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 1
1 - 4096 0 0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 38 Celsius
Available Spare: 100%
Available Spare Threshold: 10%
Percentage Used: 15%
Data Units Read: 14,662,521 [7.50 TB]
Data Units Written: 49,201,447 [25.1 TB]
Host Read Commands: 175,681,502
Host Write Commands: 1,237,060,061
Controller Busy Time: 3,595
Power Cycles: 59
Power On Hours: 14,250
Unsafe Shutdowns: 14
Media and Data Integrity Errors: 0
Error Information Log Entries: 0
Warning Comp. Temperature Time: 0
Critical Comp. Temperature Time: 0
Temperature Sensor 2: 57 Celsius

Error Information (NVMe Log 0x01, 16 of 63 entries)
No Errors Logged

sda 8:0 0 223.6G 0 disk KINGSTON SA400S37240G
├─pve--extra-vm--107--disk--0 252:0 0 4M 0 lvm
├─pve--extra-vm--107--disk--1 252:2 0 122G 0 lvm
└─pve--extra-vm--101--disk--0 252:4 0 80G 0 lvm
mmcblk0 179:0 0 14.6G 0 disk
mmcblk0boot0 179:8 0 4M 1 disk
mmcblk0boot1 179:16 0 4M 1 disk
nvme0n1 259:0 0 465.8G 0 disk KINGSTON SNV2S500G
├─nvme0n1p1 259:1 0 1007K 0 part
├─nvme0n1p2 259:2 0 1G 0 part /boot/efi
└─nvme0n1p3 259:3 0 464.8G 0 part
├─pve-swap 252:1 0 8G 0 lvm [SWAP]
├─pve-root 252:3 0 96G 0 lvm /
├─pve-data_tmeta 252:5 0 3.4G 0 lvm
│ └─pve-data-tpool 252:7 0 337.9G 0 lvm
│ ├─pve-data 252:8 0 337.9G 1 lvm
│ ├─pve-vm--100--disk--0 252:9 0 119.2G 0 lvm
│ ├─pve-vm--102--disk--0 252:10 0 64G 0 lvm
│ ├─pve-vm--103--disk--0 252:11 0 80G 0 lvm
│ ├─pve-vm--104--disk--0 252:12 0 35G 0 lvm
│ ├─pve-vm--105--disk--1 252:13 0 4M 0 lvm
│ ├─pve-vm--105--disk--0 252:14 0 32G 0 lvm
│ ├─pve-vm--106--disk--0 252:15 0 35G 0 lvm
│ └─pve-vm--108--disk--0 252:16 0 15G 0 lvm
└─pve-data_tdata 252:6 0 337.9G 0 lvm
└─pve-data-tpool 252:7 0 337.9G 0 lvm
├─pve-data 252:8 0 337.9G 1 lvm
├─pve-vm--100--disk--0 252:9 0 119.2G 0 lvm
├─pve-vm--102--disk--0 252:10 0 64G 0 lvm
├─pve-vm--103--disk--0 252:11 0 80G 0 lvm
├─pve-vm--104--disk--0 252:12 0 35G 0 lvm
├─pve-vm--105--disk--1 252:13 0 4M 0 lvm
├─pve-vm--105--disk--0 252:14 0 32G 0 lvm
├─pve-vm--106--disk--0 252:15 0 35G 0 lvm
└─pve-vm--108--disk--0 252:16 0 15G 0 lvm
 
Last edited:
You must log in or register to reply here.
Top Bottom
Back