I/O errors, PVE not booting

riesinger

New Member
Aug 30, 2020
5
1
3
24
Hi, I had a PVE 6.2 install running for quite some time.

Now, after creating a new VM, my server is acting up and I don't know why and how to fix it (even after extensive googling).

When trying to boot the server, I get repeated messages of this form:

Code:
EXT4-fs (dm-1): I/O error while writing superblock
EXT4-fs error (device dm-1): __ext4_find_entry:1532: inode #2621443: comm pvestatd: reading directory lblock 0
Buffer I/O error on dev dm-1, logical block 0, lost sync page write

This is the only thing I get, repeating about every 2 seconds. The server is not responsive to keyboard inputs, SSH sessions, pings, etc.

After booting a live USB, I can confirm that dm-1 should be the /pve/root LVM LV, installed on an SSD. I ran a fsck on it via fsck /dev/pve/root, which reported no issues at all.

Is there something I can do to repair the LV? I would rather not lose that server, since I have over 6 TB of data on it...

Thanks for your help!
 

oguz

Proxmox Retired Staff
Retired Staff
Nov 19, 2018
5,207
695
118
hi,

when you boot the live USB, can you see the contents of the LV? what does lvs -a say?
 

riesinger

New Member
Aug 30, 2020
5
1
3
24
Hi,

sorry for not answering earlier. Yes, I can see all data when I mount the LV (mount /dev/pve/root /mnt).
Also, lvs -a does not show something out of the ordinary, as far as I can tell. I attached a photo of its output below.Titan-Screenshot-LVS.jpg

Thank you for helping!
 

riesinger

New Member
Aug 30, 2020
5
1
3
24
Hi, I found something new when rebooting today. I might just not have catched it before:

Code:
blk_update_request: I/O error, dev sde, sector 2049 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
Buffer I/O error on dev sde2, logical block 1, lost async  page write
EXT4-fs error (device dm-1): __ext4_find_entry:1532: inode #2621443: comm task UPID:titan reading directory lblock 0
EXT4-fs (dm-1): I/O error while writing superblock
Aborting journal on device dm-1-8
Buffer I/O error on dev dm-1, logical block 12615680, lost sync page write
JBD2: Error -5 detected when updating journal superblock for dm-1-8
EXT-fs (dm-1): Remounting filesystem read-only

...followed by the messages I posted before
 

t.lamprecht

Proxmox Staff Member
Staff member
Jul 28, 2015
5,513
1,761
164
South Tyrol/Italy
shop.proxmox.com
Seems like sde is partly corrupted and on the way to its grave.. I'd first try to mount it as read only from the PVE installers debug modes second stage, or the arch linux iso like you used, and backup all VM and other relevant data (e.g., /var/lib/pve-cluster/config.db*)

You can also try to e2fsck but even if that works I'd migrate to another working disk ASAP.
 

synthetic64

New Member
Oct 4, 2020
15
0
1
31
I had similar issue because of improper creation of VM with PCI passthrough. After proxmox booted I quickly deleted last VM (qm destroy <number>) and rebooted proxmox. After that proxmox booted fine without errors. Check your main drive PCI-ID with command "udevadm info -q all -n /dev/sda" and ensure you don't PCI-passthrough it =)
 
Last edited:

riesinger

New Member
Aug 30, 2020
5
1
3
24
Sorry for not answering here. I ended up replacing the SSD for a new one and everything is working fine. Seems like I actually managed to kill the SSD
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!