Disk Move issue from NFS

[tri-dp]

Hi,

Recently I have migrated a disk from Azure to Proxmox. I use a NFS share to easily download the disk to Proxmox directly and then mount it to a newly created machine. Which works fine! After testing everything I wanted to move the disk to ceph. But this does not seem to work. Even migrating to local-lvm does not seem to work, only moving from NFS to NFS works. I get the following error: TASK ERROR: storage migration failed: block job (mirror) error: drive-scsi0: 'mirror' has been cancelled

I tried a several things including:
- Turning off the machine and then moving the disk
- Converting the disk from qcow2 to raw
- Moving the disk from the CLI instead of the webui

All of the above do not seem to work and I cannot find anything related on the Proxmox forum.

 

[mira]

Either use `qm importdisk` if you have it available locally, or if it is part of a VM already, start the VM and then use `Move Disk` online. Online `Move Disk` makes it possible to move a disk to RBD backed storages (Ceph).

 

[tri-dp]

Thank you for your reply.

I just tried executing the following commands but no luck:

qm importdisk 132 data.qcow2 vmdisks-hdd
qm move-disk 132 scsi0 vmdisks-hdd

both commands are giving the following message:

copy failed: command '/usr/bin/qemu-img convert -p -n -f qcow2 -O raw data.qcow2 'zeroinit:rbd:vmdisks-hdd/vm-132-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/vmdisks-hdd.keyring'' failed: exit code 1

The task error given in my initial post does not seem to occur anymore though, which is weird since I have not changed anything.

 

[mira]

As mentioned above (may not be clear enough), offline move disk to RBD storage does not work. That's only possible online and this includes `qm importdisk`. In this case first move it to a different storage where you then can start the VM and move the disk once the VM is running.

 

[tri-dp]

Okay, so that is why I got the new message about the keyrings. I am able to start the VM from the current storage. Then I ran the command qm move-disk 132 scsi1 vmdisks-hdd. But now I get the initial error again (storage migration failed: block job (mirror) error: drive-scsi1: 'mirror' has been cancelled). Same for moving the storage from the GUI.

The error given in the start of this post is whilst migrating online already. Sorry for the inconvenience!

 

[mira]

Oh sorry, my mistake then.

Could you provide the complete task log of the migration?

 

[tri-dp]

root@proxmox-01:/mnt/pve/TrueNAS/images/132# qm move-disk 132 scsi1 vmdisks-hdd
create full clone of drive scsi1 (TrueNAS:132/data.qcow2)
drive mirror is starting for drive-scsi1
drive-scsi1: Cancelling block job
drive-scsi1: Done.
Removing image: 1% complete...
Removing image: 2% complete...
Removing image: 3% complete...
...........
Removing image: 99% complete...
Removing image: 100% complete...done.
storage migration failed: block job (mirror) error: drive-scsi1: 'mirror' has been cancelled

 

[mira]

What's the output of pveversion -v and the storage config (cat /etc/pve/storage.cfg)?

 

[tri-dp]

Output pveversion -v:

root@proxmox-01:~# pveversion -v
proxmox-ve: 7.2-1 (running kernel: 5.15.64-1-pve)
pve-manager: 7.2-11 (running version: 7.2-11/b76d3178)
pve-kernel-5.15: 7.2-13
pve-kernel-helper: 7.2-13
pve-kernel-5.13: 7.1-9
pve-kernel-5.15.64-1-pve: 5.15.64-1
pve-kernel-5.15.60-1-pve: 5.15.60-1
pve-kernel-5.15.39-3-pve: 5.15.39-3
pve-kernel-5.15.35-2-pve: 5.15.35-5
pve-kernel-5.15.35-1-pve: 5.15.35-3
pve-kernel-5.15.30-2-pve: 5.15.30-3
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph: 16.2.9-pve1
ceph-fuse: 16.2.9-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-3
libpve-guest-common-perl: 4.1-4
libpve-http-server-perl: 4.1-4
libpve-storage-perl: 7.2-10
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.7-1
proxmox-backup-file-restore: 2.2.7-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-3
pve-docs: 7.2-2
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-6
pve-firmware: 3.5-6
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 7.0.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1

cat /etc/pve/storage.cfg

root@proxmox-01:~# cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content vztmpl,backup,iso
        shared 0

lvmthin: local-lvm
        thinpool data
        vgname pve
        content images,rootdir

rbd: vmdisks-hdd
        content rootdir,images
        krbd 0
        pool vmdisks-hdd

nfs: TrueNAS
        export /mnt/Migratie
        path /mnt/pve/TrueNAS
        server 10.100.20.2
        content images
        prune-backups keep-all=1

 

[mira]

Is there anything in the journal/syslog indicating an issue?
According to that log, something cancels the job and the new disk gets removed. Does it even copy data to the rbd image?

 

[tri-dp]

I only get the following logs in /var/log/* :

Nov 14 10:38:11 proxmox-01 pvedaemon[2253032]: <user> move disk VM 132: move --disk scsi0 --storage vmdisks-hdd
Nov 14 10:38:12 proxmox-01 pvedaemon[2253032]: VM 132 qmp command failed - VM 132 qmp command 'block-job-cancel' failed - Block job 'drive-scsi0' not found
Nov 14 10:38:13 proxmox-01 pmxcfs[1932]: [status] notice: received log
Nov 14 10:38:15 proxmox-01 pvedaemon[2253032]: storage migration failed: block job (mirror) error: drive-scsi0: 'mirror' has been cancelled
Nov 14 10:38:15 proxmox-01 pvedaemon[2202426]: <user> end task UPID:proxmox-01:....:qmmove:132:<user>: storage migration failed: block job (mirror) error: drive-scsi0: 'mirror' has been cancelled

I cannot find out if it is copy data to the rdb image, but since iit is removing the image immediately I think it is copying something atleast.

I just tried moving a lxc disk from local-lvm to vmdisks-hdd(rdb) which is just working fine. So I can confirm it seems to be specifically for nfs > rdb

 

[mira]

In the logs above it shows `scsi1`, but here it complains about `scsi0`.
Do you have an additional disk connected on scsi0? Please provide the VM config: qm config <VMID>

 

[tri-dp]

Thats because I have two disks attached. Both with different filetype and datasource. Hereby qm config:

root@proxmox-01:/var/log# qm config 132
agent: 1
boot: order=net0;scsi0;ide2
cores: 4
ide2: none,media=cdrom
memory: 32768
meta: creation-qemu=7.0.0,ctime=1667567123
name: debian11
net0: virtio=8A::::4A,bridge=vmbr0,tag=100
numa: 0
ostype: l26
scsi0: ReadyNAS-2120:132/vm-132-disk-0.raw,size=34359738880
scsi1: TrueNAS:132/wvb_data.qcow2,size=68719477248
scsihw: virtio-scsi-pci
smbios1: uuid=b506082d-..-50d55078dd96
sockets: 1
vmgenid: 7bbae292-c97c-445a-b748-6aa24a839ae1

 

[mira]

Could you try without the first disk for now? The VM only needs to be running, there's no need for an OS to be loaded.

 

[tri-dp]

Shut down the VM, detached the OS disk and booted up the machine. Still the same error whilst copying the disk.

 

[mira]

Sorry for the outdated knowledge, offline migration to 'rbd' storages should work.

Tried reproducing it here, but didn't run into the same issues. Can you start the VM normally and access the 2nd disk without any issues?

 

[tri-dp]

Good to know. But still a bit confusing since I get another error while migrating offline (the keyring error in my second post).

I can start the VM normally and I can access the 2nd disk. The first disk also works just fine. Nothing really weird is happening when starting the VM. The only thing not working is the migration to rdb.

 

[mira]

Do all files referenced in that line exist?

/etc/pve/ceph.conf
/etc/pve/priv/ceph/vmdisks-hdd.keyring

Do you use the default id 'admin' as referenced by it?

 

[tri-dp]

Both files exist on the Proxmox Node(s).

The admin default id is the only id given in vmdisks-hdd.keyring

[client.admin]

To me it looks like I am using the correct id? How can I validate?

 

[mira]

Are the permissions correct for both files?
Please provide the output of ls -l /etc/pve/ and ls -l /etc/pve/priv/ceph/