Help: PVE Cluster storage problem

[kobemtl]

I think I screwed up the cluster. Below the details about my cluster storage.
pve-dellr330:
$lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert data pve twi-aotz-- <63.38g 40.32 2.64 root pve -wi-ao---- 29.00g swap pve -wi-ao---- 8.00g ... ...
pve-dellr530:
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert dellr530-proxthin lvmthin twi-aotz-- 196.47g 54.27 2.27 root pve -wi-ao---- 9.75g swap pve -wi-ao---- <4.88g ... ...
So now if I need to migrate any VM or CT from node to node I get error about storage xxx is not available on the other node. If I set the storage local-lvm and dellr530-proxthin accessible from both nodes I get error like this
() 2022-06-23 11:10:55 starting migration of CT 100 to node 'pve-dellr330' (192.168.0.141) 2022-06-23 11:10:55 found local volume 'dellr530-proxthin:vm-100-disk-0' (in current VM config) 2022-06-23 11:10:56 Volume group "lvmthin" not found 2022-06-23 11:10:56 Cannot process volume group lvmthin 2022-06-23 11:10:57 command '/sbin/lvs --separator : --noheadings --units b --unbuffered --nosuffix --config 'report/time_format="%s"' --options vg_name,lv_name,lv_size,lv_attr,pool_lv,data_percent,metadata_percent,snap_percent,uuid,tags,metadata_size,time lvmthin' failed: exit code 5 send/receive failed, cleaning up snapshot(s).. 2022-06-23 11:10:58 ERROR: storage migration for 'dellr530-proxthin:vm-100-disk-0' to storage 'dellr530-proxthin' failed - command 'set -o pipefail && pvesm export dellr530-proxthin:vm-100-disk-0 raw+size - -with-snapshots 0 | /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=pve-dellr330' root@192.168.0.141 -- pvesm import dellr530-proxthin:vm-100-disk-0 raw+size - -with-snapshots 0 -allow-rename 1' failed: exit code 5 2022-06-23 11:10:58 aborting phase 1 - cleanup resources 2022-06-23 11:10:58 ERROR: found stale volume copy 'dellr530-proxthin:vm-100-disk-0' on node 'pve-dellr330' 2022-06-23 11:10:58 start final cleanup 2022-06-23 11:10:58 ERROR: migration aborted (duration 00:00:03): storage migration for 'dellr530-proxthin:vm-100-disk-0' to storage 'dellr530-proxthin' failed - command 'set -o pipefail && pvesm export dellr530-proxthin:vm-100-disk-0 raw+size - -with-snapshots 0 | /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=pve-dellr330' root@192.168.0.141 -- pvesm import dellr530-proxthin:vm-100-disk-0 raw+size - -with-snapshots 0 -allow-rename 1' failed: exit code 5 TASK ERROR: migration aborted
Is it anyway to fix this? Or after the cluster nodes created nothing can be changed?
Thanks.

 

[bbgeek17]

You can carefully rename one of the sides to match volume group names, ie dellr530-proxthin>data, or vice versa.
You will also need to update the storage.cfg file, and may be vm.cfg files. Best to shut down VM/CTs as do this maintenance.
Dont rush into things - understand what the outcome you want and what the commands you plan to run will affect.
Good luck

Another option as you migrate the VM , I believe you can pick a different storage target.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[kobemtl]

Thanks a lot for the suggestion.

When I do the migration there is no option to choose the storage target. And I remember I had that kind of choice before. It was before when I had another node in the cluster. Any idea or document where can I have that option?

 

[bbgeek17]

you can try to use CLI:

qm migrate <vmid> <target> [OPTIONS]

       Migrate virtual machine. Creates a new migration task.

       <vmid>: <integer> (1 - N)
           The (unique) ID of the VM.

       <target>: <string>
           Target node.

       --bwlimit <integer> (0 - N) (default = migrate limit from datacenter or storage config)
           Override I/O bandwidth limit (in KiB/s).

       --force <boolean>
           Allow to migrate VMs which use local devices. Only root may use this option.

       --migration_network <string>
           CIDR of the (sub) network that is used for migration.

       --migration_type <insecure | secure>
           Migration traffic is encrypted using an SSH tunnel by default. On secure, completely private networks this can be disabled to increase performance.

       --online <boolean>
           Use online/live migration if VM is running. Ignored if VM is stopped.

       --targetstorage <string>
           Mapping from source to target storages. Providing only a single storage ID maps all source storages to that storage. Providing the special value 1 will map each source storage to itself.

       --with-local-disks <boolean>
           Enable live storage migration for local disk

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[kobemtl]

I am so sorry. I think I made a mistake. All the error and the missing option is for LXC migration NOT VM.

 

[bbgeek17]

the container functionality is the same, just different CLI tool:

pct migrate <vmid> <target> [OPTIONS]

       Migrate the container to another node. Creates a new migration task.

       <vmid>: <integer> (1 - N)
           The (unique) ID of the VM.

       <target>: <string>
           Target node.

       --bwlimit <number> (0 - N) (default = migrate limit from datacenter or storage config)
           Override I/O bandwidth limit (in KiB/s).

       --online <boolean>
           Use online/live migration.

       --restart <boolean>
           Use restart migration

       --target-storage <string>
           Mapping from source to target storages. Providing only a single storage ID maps all source storages to that storage. Providing the special value 1 will map each source storage to itself.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[kobemtl]

command line works! Damn it. I shall use command line.

Thanks a lot bbgeek17