I/O errors, PVE not booting

[riesinger]

Hi, I had a PVE 6.2 install running for quite some time.

Now, after creating a new VM, my server is acting up and I don't know why and how to fix it (even after extensive googling).

When trying to boot the server, I get repeated messages of this form:

EXT4-fs (dm-1): I/O error while writing superblock
EXT4-fs error (device dm-1): __ext4_find_entry:1532: inode #2621443: comm pvestatd: reading directory lblock 0
Buffer I/O error on dev dm-1, logical block 0, lost sync page write

This is the only thing I get, repeating about every 2 seconds. The server is not responsive to keyboard inputs, SSH sessions, pings, etc.

After booting a live USB, I can confirm that dm-1 should be the /pve/root LVM LV, installed on an SSD. I ran a fsck on it via fsck /dev/pve/root, which reported no issues at all.

Is there something I can do to repair the LV? I would rather not lose that server, since I have over 6 TB of data on it...

Thanks for your help!

 

[oguz]

hi,

when you boot the live USB, can you see the contents of the LV? what does lvs -a say?

 

[riesinger]

Hi,

sorry for not answering earlier. Yes, I can see all data when I mount the LV (mount /dev/pve/root /mnt).
Also, lvs -a does not show something out of the ordinary, as far as I can tell. I attached a photo of its output below.

Thank you for helping!

 

[riesinger]

Hi, I found something new when rebooting today. I might just not have catched it before:

blk_update_request: I/O error, dev sde, sector 2049 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
Buffer I/O error on dev sde2, logical block 1, lost async  page write
EXT4-fs error (device dm-1): __ext4_find_entry:1532: inode #2621443: comm task UPID:titan reading directory lblock 0
EXT4-fs (dm-1): I/O error while writing superblock
Aborting journal on device dm-1-8
Buffer I/O error on dev dm-1, logical block 12615680, lost sync page write
JBD2: Error -5 detected when updating journal superblock for dm-1-8
EXT-fs (dm-1): Remounting filesystem read-only

...followed by the messages I posted before

 

[t.lamprecht]

Seems like sde is partly corrupted and on the way to its grave.. I'd first try to mount it as read only from the PVE installers debug modes second stage, or the arch linux iso like you used, and backup all VM and other relevant data (e.g., /var/lib/pve-cluster/config.db*)

You can also try to e2fsck but even if that works I'd migrate to another working disk ASAP.

 

[synthetic64]

I had similar issue because of improper creation of VM with PCI passthrough. After proxmox booted I quickly deleted last VM (qm destroy <number>) and rebooted proxmox. After that proxmox booted fine without errors. Check your main drive PCI-ID with command "udevadm info -q all -n /dev/sda" and ensure you don't PCI-passthrough it =)

 

[riesinger]

Sorry for not answering here. I ended up replacing the SSD for a new one and everything is working fine. Seems like I actually managed to kill the SSD

 

[Ramalama]

Was the old ssd a nvme drive?

 

[riesinger]

No, this was a SATA SSD (Samsung 860 Evo)