NVME Reset

[toomanylogins]

Hello, I have twice attempted to move a disk from sata ssd raw format to nvme qcow2 and get the following error. Proxmox 7.0.11

qemu-img: error while writing at byte 199715880960: Input/output error
qemu-img: Failed to flush the L2 table cache: Read-only file system
qemu-img: Failed to flush the refcount block cache: Read-only file system
unlink '/mnt/pve/nvme/images/301/vm-301-disk-0.qcow2' failed - Read-only file system
TASK ERROR: storage migration failed: copy failed: command '/usr/bin/qemu-img convert -p -n -f raw -O qcow2 /mnt/pve/dev-sdb/images/301/vm-301-disk-0.raw zeroinit:/mnt/pve/nvme/images/301/vm-301-disk-0.qcow2' failed: exit code 1

After this the disk no longer appears in the disks list for the node. I have tried

sgdisk --zap-all /dev/nvme0n1

and init gpt wipe umount etc same error every time.

Is this a hardware issue ?
Thanks
Paul

 

[toomanylogins]

This is a new sabrent nvme ssd in a pcie adaptor card and tests fine using windows. Changing the storage type to Lvm-thin similar errors.

qemu-img: error while writing at byte 103393112064: Input/output error
Command failed with status code 5.
command '/sbin/vgscan --ignorelockingfailure --mknodes' failed: exit code 5
Volume group "nvme" not found
Cannot process volume group nvme
command '/sbin/lvs --separator : --noheadings --units b --unbuffered --nosuffix --config 'report/time_format="%s"' --options vg_name,lv_name,lv_size,lv_attr,pool_lv,data_percent,metadata_percent,snap_percent,uuid,tags,metadata_size,time nvme' failed: exit code 5
TASK ERROR: storage migration failed: copy failed: command '/usr/bin/qemu-img convert -p -n -f raw -O raw /mnt/pve/dev-sdb/images/301/vm-301-disk-0.raw zeroinit:/dev/nvme/vm-301-disk-0' failed: exit code 1

After the error the ssd is no longer listed under "disks".

Anyone have any idea ?
Thanks

 

[toomanylogins]

Bit more info it seems this is the issue from var/log/messages

Monitoring thin pool nvme-nvme.
May 29 10:56:57 rx300s7-1-pve lvm[631]: No longer monitoring thin pool nvme-nvme.
May 29 10:56:58 rx300s7-1-pve lvm[631]: Monitoring thin pool nvme-nvme-tpool.
May 29 10:59:39 rx300s7-1-pve kernel: [ 863.942912] nvme nvme0: I/O 0 QID 1 timeout, aborting
May 29 10:59:39 rx300s7-1-pve kernel: [ 863.942931] nvme nvme0: I/O 1 QID 1 timeout, aborting
May 29 10:59:39 rx300s7-1-pve kernel: [ 863.942944] nvme nvme0: I/O 2 QID 1 timeout, aborting
May 29 10:59:39 rx300s7-1-pve kernel: [ 863.942947] nvme nvme0: I/O 3 QID 1 timeout, aborting
May 29 11:00:10 rx300s7-1-pve kernel: [ 894.666563] nvme nvme0: I/O 0 QID 1 timeout, reset controller
May 29 11:00:41 rx300s7-1-pve kernel: [ 925.382206] nvme nvme0: I/O 8 QID 0 timeout, reset controller

Any idea what maybe the cause of the timeout ?
From google it seems increasing Set io_timeout may solve the problem but I dont know how to do this.

 

[toomanylogins]

Tried reformat nvme to ext4 and copy disk from sata ssd. Now getting this error on move disk.

qemu-img: error while writing at byte 58181287936: Read-only file system
qemu-img: Failed to flush the L2 table cache: Read-only file system
qemu-img: Failed to flush the refcount block cache: Read-only file system
unlink '/mnt/pve/nvme/images/114/vm-114-disk-1.qcow2' failed - Read-only file system
TASK ERROR: storage migration failed: copy failed: command '/usr/bin/qemu-img convert -p -n -f qcow2 -O qcow2 /mnt/pve/dev-sdb/images/114/vm-114-disk-0.qcow2 zeroinit:/mnt/pve/nvme/images/114/vm-114-disk-1.qcow2' failed: exit code 1

After the error on reboot the nvme is not mounted

Anybody have any idea what issue could be ?
Thanks
Paul

 

[_gabriel]

try disk-filltest under Windows to fully write disk