Cannot start VMs - cant' activate LV ...

[atinazzi]

Hi,

This morning all VMs across all 3 nodes were displaying a warning sign saying "io-error".

I started looking up on this forum but found nothing relevant/applicable.

I have forced a restart of a node and installed latest updates, including newest kernel.
After rebooting and installing updates VMs on such node started normally

I moved on another node and installed all updates including the kernel, then issue a "qm resume ID" and the VMs on such node resumed fine.

I tried repeating this on the third node but VMs are not resuming or starting.
I have then forced a reset of that node

When I try restarting the VMs on this node I get

can't activate LV '/dev/vg2/vm-110-disk-1': device-mapper: reload ioctl on (253:5) failed: Read-only file system

How do I fix this?

Thanks

 

[pdegagne]

I had some really odd behavior today. Same issue, I had an "io-error" message. I rebooted my server only to find none of my containers wouldn't start. The sub-vol directories all appeared to only contain a "dev" directory. I found that the zfs filesystems were not mounting because these directories were in the way. Here's how I got things working again.

List subvol contents

root@hv01:/rpool/data/subvol-111-disk-0# ls
dev

Check ZFS mount

root@hv01:/rpool/data/subvol-111-disk-0# zfs get mounted rpool/data/subvol-111-disk-0
NAME                          PROPERTY  VALUE    SOURCE
rpool/data/subvol-111-disk-0  mounted   no       -

Move empty folder and rename to .bak

root@hv01:/rpool/data# mv subvol-111-disk-0{,.bak}

Mount ZFS Filesystem

zfs mount rpool/data/subvol-111-disk-0

I was then able to start all the containers again.

 

[atinazzi]

I do not use ZFS
the affected VMs are on an external disk array connected via ISCSI
the content of the ISCSI target is visible from all nodes including the one having problems starting VMs
If I manually move the config file to another node

mv /etc/pve/nodes/NB/qemu-server/110.conf /etc/pve/nodes/NA/qemu-server/

then I can start the VM on "Node A"

 

[atinazzi]

I have just noticed that I cannot create new VMs on the ISCSI target, from any node.

at boots it says:

Volume group "pve" not found
Cannot process volume group pve
Volume group "pve" not found
Cannot process volume group pve
Volume group "pve" not found
Cannot process volume group pve



dmesg says:

[ 36.431471] scsi host15: iSCSI Initiator over TCP/IP
[ 36.458523] scsi 15:0:0:0: Direct-Access SYNOLOGY iSCSI Storage 4.0 PQ: 0 ANSI: 5
[ 36.459360] sd 15:0:0:0: Attached scsi generic sg3 type 0
[ 36.460801] sd 15:0:0:0: [sdb] 8589934592 512-byte logical blocks: (4.40 TB/4.00 TiB)
[ 36.461025] sd 15:0:0:0: [sdb] Write Protect is on
[ 36.461027] sd 15:0:0:0: [sdb] Mode Sense: 43 00 90 08
[ 36.461417] sd 15:0:0:0: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
[ 36.461746] sd 15:0:0:0: [sdb] Optimal transfer size 16384 logical blocks > dev_max (8192 logical blocks)
[ 36.467085] sd 15:0:0:0: [sdb] Attached SCSI disk
[ 36.536765] device-mapper: table: 253:5: linear: Device lookup failed
[ 36.536794] device-mapper: ioctl: error adding target to table

any idea on what is going on and how I can fix it?

 

[atinazzi]

Anybody?