Migration failure between two nodes

[StanM]

I have two nodes, say A and B. I upgraded both from Proxmox 6.4 to 7 a couple of weeks ago. The upgrade on A went well. However, after the upgrade on B, virtualization didn't work. When I tried to open an Ubuntu VM, I got a message that virtualization wasn't working and a suggestion to turn off qemu on the machine or change the bios. Node B is a Proctectli 6-port i3 machine. It has two SSDs, the second named Common, which is also the case on Node A.

When I disabled qemu, the VM would run, but so slowly as to be useless. In the VM BIOS setting I tried both the default SeaBIOS and the OVMF (UEFI), but the result was the same. The Protectli hardware BIOS was Coreboot, so I was unable to check the virtualization setting on the hardware. Eventually, I changed the Protectli BIOS back to it's basic BIOS so that I could check the virtualization setting. After the BIOS change, the virtualization was enabled. I destroyed the cluster on node A, reinstalled Proxmox on node B (the Protectli) and created a new cluster.

During the reinstallation of Proxmox on the Protectli, I accepted the storage defaults. I didn't notice any settings for the second SSD. Before setting up the new cluster, I formatted the second SSD as ZFS and named it "Common". The second SSD on node A is also formatted as ZFS and named "Common". That mirrors the setup I had before upgrading Proxmox, which worked.

Before wiping out BIOS on node B, I had migrated the VMs and a container there to node A. When I tried to migrate them back to node B, two were successfully done, a container with a Unifi controller and a VM that houses my Home Assistant Core and Supervisor. However, the Ubuntu desktop VM won't migrate. I get the message: ERROR: migration aborted (duration 00:00:00): ISO_NAS:iso/ubuntu-20.04.2.0-desktop-amd64.iso: content type 'iso' is not available on storage 'Common'.

ISO_NAS is the name of my ISO storage on a remote NAS. I've tried the migration with the CD/DVD drive attached to the Ubuntu VM and with the drive removed, but still get the same error message. The /etc/pve/storage.cfg files are identical on the two nodes.

Maybe I just need to enable the ISO storage type on the second SSD if that's a setting for a partition, but don't know how to do that. Can anyone help with that? Or is the cause of the problem different?

 

[dietmar]

Just go to the storage configuration for storage "Common", and enable content type "iso" there.

 

[StanM]

Dietmar,

Thanks for the quick response. The GUI for "Common" shows content type "Disk image, Container". I don't see a way of enabling "iso" in the GUI. Is there a way to do that in the GUI?

Using the Shell, I added "iso" to the storage types for "Common" in /etc/pve/storage.cfg. I then shutdown the node and started it again. I still get the same error message when attempting to migrate the VM. Also the GUI continues to show just "Disk image, Container" as the content types for "Common".

How do I enable content type "iso" for "Common"?

 

[StanM]

Supplemental information:

On the GUI under Datacenter, Storage Common is listed as having only Disk image and Container. Edit shows it available on both nodes. When I click on content, there are no other options for storage. If I limit the nodes to only node B or node A, there is still only Disk image and Container content available.

Is there a Corosync file that controls storage to which I can add "Iso" using the Shell?

 

[fiona]

Hi,
are you specifying a target storage when migrating? There currently is a buggy check that wrongly assumes that the ISO image would also be migrated to that target storage even if it's shared. Assuming that's the issue, there's a patch, but it's yet to be applied/released. As a workaround, detaching the ISO and re-attaching it after migration should work.

 

[StanM]

I have tried detaching the ISO, and I've tried removing the CD/DVD drive entirely. I still get the same result. Is there a way (Corosync file?) to add ISO as an enabled storage type? I've done that in the /etc/pve/storage.cfg file on the node in question, but that didn't have any effect.

 

[fiona]

I have tried detaching the ISO, and I've tried removing the CD/DVD drive entirely. I still get the same result. Is there a way (Corosync file?) to add ISO as an enabled storage type? I've done that in the /etc/pve/storage.cfg file on the node in question, but that didn't have any effect.

It really shouldn't be the same error if the ISO drive that was triggering the error isn't present at all. Could you post the VM config file, the storage config and the full migration log?

 

[StanM]

I now have compound errors and can't get back to the one described above. The new error is "Can't connect to destination address using public key." when I try to migrate the VM in question from node A to node B. A week or so ago, I put a different VM (on node B running Home Assistant) on a separate port in order to match it with the VLAN of my IoT devices. I had newly added the interface to node B. After that, I was unable to access the console of node B from node A, getting a similar error. I fixed that with "ssh-keygen -f "/etc/ssh/ssh_known_hosts" -R "[ip address of node B]" in the console for node B. When I saw the above error in the migration process, I tried running the command again, but that hasn't fixed the error I'm still getting in the migration process.

I'm beginning to think that the easiest way out of this is to destroy the cluster completely and establish a new cluster between nodes A and B. Would you recommend against doing that?

 

[fiona]

I now have compound errors and can't get back to the one described above. The new error is "Can't connect to destination address using public key." when I try to migrate the VM in question from node A to node B. A week or so ago, I put a different VM (on node B running Home Assistant) on a separate port in order to match it with the VLAN of my IoT devices. I had newly added the interface to node B. After that, I was unable to access the console of node B from node A, getting a similar error. I fixed that with "ssh-keygen -f "/etc/ssh/ssh_known_hosts" -R "[ip address of node B]" in the console for node B. When I saw the above error in the migration process, I tried running the command again, but that hasn't fixed the error I'm still getting in the migration process.

Please try running pvecm updatecerts. Do you have a migration network configured in /etc/pve/datacenter.cfg?

I'm beginning to think that the easiest way out of this is to destroy the cluster completely and establish a new cluster between nodes A and B. Would you recommend against doing that?

I mean that's a lot of work (requires re-installing one of the nodes) and I'd do that only as a last resort.

 

[StanM]

With the "pvecm updatecerts" command run on both nodes, I'm back to the initial problem. I do not have a migration network configured.

I've attached a file with the storage.cfg contents for both nodes, and what I hope are the VM config files for both nodes. I don't know where to find the full migration log. If you suggest where I can find that, I'll send it along too.

The VM that can't be migrated is "Ubuntu2004Dell" on the Dell3060 (node A). There is no indication of connection to my ISO storage on a NAS for that VM. But I see also on that same node four entries for a VM named "Ubuntu2004", which doesn't exist on that (or the other) node anymore. Each of the four entries shows "ide2: ISO_NAS:..." I wonder if I should just delete those four entries. But I'll wait to do anything untill I hear back.

 

[StanM]

The migration problem has been solved. Once I realized that the VM configuration file was for the VM number and that there were five instances of VMs with that number, I decided to delete the lines for the four instances of "Ubuntu2004". I'm guessing I reused the number at some point (but not four times). Once I'd done that, I re-encountered the "Can't connect to destination address using public key" error again. I ran "pvecm updatecerts" again on both nodes, and then the migration worked.

Thanks much for your help. I consider this issue to be closed.