Storage migration failed io_uring known to cause issues

[brucexx]

I am getting this error when trying to move drive from Ceph storage to PVE local hard drive:

TASK ERROR: storage migration failed: target storage is known to cause issues with aio=io_uring (used by current drive)

This started to happen on 8.2.x/8.3.0 , I have a 7.3 and 7.4 cluster and that type of migration with io_uring set is not causing any issues, I have done hundreds of moves from Ceph to local drives and back, never experience an issue.

First question is why on that version it is a problem ? I saw other posts and I know that changing this setting will allow me to move it back to local drive but the change requires a shutdown power up which is a pain as I have about 300 VM that I need to do that for as I am migrating from 7.3.x PVE cluster to 8.3.0 PVE cluster. On restore from PBS there is no option to change hard drive Async IO so I need to restore and then power down/up again.

Second, what is the second-best Async IO option that will allow me to move back and forth between ceph and local storage, which sometimes is the case? Should I select second option , move , then change it back to io_uring ?

Also, is there an option to force it to move back to local drive ?

Thank you

 

[brucexx]

Anybody on this ?

I changed drive settings to IO thread "on" + Async IO: native and can move now from Ceph to local drive.

I am not that clear on what option I should use. I primary use an external Ceph storage but at times move VMs to local drive on as needed basis, local drives are usually hardware RAID1, they are used for maintenance but need to function well during that time which can be several days to 1-2 weeks.

Does anybody have experience if the "native" option is my best choice based on the use ? Perhaps is does not matter that much and the difference is not significant ? On 7.4.x PVE + external Ceph with io_uring worked perfectly fine and I had no problems with workloads.

Thank you

 

[fiona]

Hi,
what target storage do you use? LVM (not thin) should be the only local storage for which IO uring default is currently not allowed and that is because it caused issues in the past for some users: https://git.proxmox.com/?p=qemu-ser...4;hp=e83dd50a361191b464eabdb3b2c14d33c76cfc09 Note that this has been the case since around Proxmox VE 7.0, not a recent change.

It has been a while and it might be time to re-evaluate turning it back on, in the hope that new kernels fixed the issue.

 

[fiona]

Second, what is the second-best Async IO option that will allow me to move back and forth between ceph and local storage, which sometimes is the case?

AFAIK there is not too much difference between threads and native, but this might depend on workload and specific configuration.

Should I select second option , move , then change it back to io_uring ?

I don't think there is an easier workaround right now unfortunately.

Also, is there an option to force it to move back to local drive ?

Not right now.

 

[brucexx]

Thank you for your response, I think I should start with it but is the io_uring option best/optimal for Ceph and other storage, in general ?

BTW now it makes sense our other cluster uses directory for local hard drive storage and the new one is using LVM. Depending on your answer with the io_uring I might simply used directory and not LVM on the new Cluster.

Let me know what you think. Again Thx

 

[fiona]

Thank you for your response, I think I should start with it but is the io_uring option best/optimal for Ceph and other storage, in general ?

Yes, in general io_uring is the best choice, also with RBD. Of course, there is no guarantee that it will be for each and every workload, but it does simply have a better architecture then the older AIO implementations. Essentially, it was written because these were "not good enough" (for certain people).

BTW now it makes sense our other cluster uses directory for local hard drive storage and the new one is using LVM. Depending on your answer with the io_uring I might simply used directory and not LVM on the new Cluster.

Yes, not using LVM would be a workaround for your use case.

 

[brucexx]

Thx , for your advice I changed from LVM to directory. The local storage is only used as a backup/maintenance for Ceph.