[SOLVED] Disk Moves GUI vs CLI differences

[adamb]

Hey all I am in the process of moving disks residing on Ceph over to iscsi based storage.

If I use the GUI to move the disk, all is well and its succesful.

If I use the CLI, it fails with the following.

root@ccscloud3:~# qm move_disk 119 scsi3 Cloud-Udarchive1
create full clone of drive scsi3 (Cloud-Ceph1:vm-119-disk-0)
storage migration failed: target storage is known to cause issues with aio=io_uring (used by current drive)

Almost all of my storage is iscsi based and all of my VM disks are using aio=io_uring, seems a bit odd. Ideally I need this to work at the CLI level as I have 500 disks to move.

 

[aaron]

What if you run the same via the API?

pvesh create nodes/{node}/qemu/{vmid}/move_disk --disk {disk config key} --storage {target storage}

 

[aaron]

Optional with the --delete 1 parameter

 

[adamb]

Looks like the same issue.

root@ccscloud3:~# pvesh create nodes/ccscloud3/qemu/173/move_disk --disk scsi2 --storage Cloud-Udarchive1
create full clone of drive scsi2 (Cloud-Ceph1:vm-173-disk-0)
storage migration failed: target storage is known to cause issues with aio=io_uring (used by current drive)
UPID:ccscloud3:0006314A:65082874:64EDFC62:qmmove:173:root@pam:

 

[aaron]

Hmm, does it still work in the GUI if you restart pveproxy and pvedaemon?

It could be possible that this did not happen after an update, and the API daemons are still having old code without the check loaded.

 

[adamb]

Can confirm, that restart pveproxy and pvedaemon made it so the GUI behaves like the API CLI.

Should I be looking to no longer use io_uring with iscsi based disks?

 

[adamb]

Just so I understand the issue better @aaron

Is io_uring ok ontop of iscsi+LVM type storage?

Or is the issue the disk mirror process when running between the two storages that can cause the bug?

I noticed that if the VM is powered down I can do the disk move and it still uses io_uring.

 

[fiona]

Hi,

Just so I understand the issue better @aaron

Is io_uring ok ontop of iscsi+LVM type storage?

Or is the issue the disk mirror process when running between the two storages that can cause the bug?

I noticed that if the VM is powered down I can do the disk move and it still uses io_uring.

if you explicitly set aio=io_uring, that will be used, but if you use the default setting, certain storage configurations (CIFS/LVM/KRBD+write{back,through} cache) won't use io_uring as the default (even if the GUI wrongly shows it), because io_uring caused hangs/crashes with those in the past. We will re-evaluate those in the future and allow io_uring once it's safe.

Currently, it's not implemented to switch the aio setting during drive mirror, so those storages are prohibited as targets when the disk currently uses io_uring.

 

[adamb]

Hi,

if you explicitly set aio=io_uring, that will be used, but if you use the default setting, certain storage configurations (CIFS/LVM/KRBD+write{back,through} cache) won't use io_uring as the default (even if the GUI wrongly shows it), because io_uring caused hangs/crashes with those in the past. We will re-evaluate those in the future and allow io_uring once it's safe.

Currently, it's not implemented to switch the aio setting during drive mirror, so those storages are prohibited as targets when the disk currently uses io_uring.

Is my only option to set all 600 VM's to native or threads, move the disks, then change them all back to io_urinng? This would require 2 sets of reboots for 600 VM's as well.

 

[fiona]

Is my only option to set all 600 VM's to native or threads, move the disks, then change them all back to io_urinng? This would require 2 sets of reboots for 600 VM's as well.

I wouldn't change it to explicitly using io_uring on the LVM storage. It's not clear if all the bugs have been fixed (there were still issues when using kernel 6.1 and AFAIK it wasn't evaluated with kernel 6.2 yet).

 

[adamb]

I wouldn't change it to explicitly using io_uring on the LVM storage. It's not clear if all the bugs have been fixed (there were still issues when using kernel 6.1 and AFAIK it wasn't evaluated with kernel 6.2 yet).

Does that mean I shouldn't use io_uring for Async IO at all? The setting in the proxmox GUI itself?

It seems my only real options are.

- Change all VM's from Async IO - Default io_uring to Native or Threads
- Shutdown and start all VM's
- Then I can disk mirror all the disks to the new storage
- Change all VM's from Native or Threads back to Default io_uring
- Shutdown and start all VM's

Or should I not be using the default setting at all and stick with native or threads?

 

[fiona]

Does that mean I shouldn't use io_uring for Async IO at all? The setting in the proxmox GUI itself?

It seems my only real options are.

- Change all VM's from Async IO - Default io_uring to Native or Threads
- Shutdown and start all VM's
- Then I can disk mirror all the disks to the new storage
- Change all VM's from Native or Threads back to Default io_uring

If your storage is LVM, the default is not actually io_uring, but native (with direct cache) or threads with other cache settings. The GUI doesn't know about the special default and states Default (io_uring) for all storages. We hope to be able to enable it for LVM at some point again, until then you don't gain anything from switching back. But if you want, you can switch back without shutting down the VMs and keeping it as a pending change until next time the VM is shut down anyways.