New Import Wizard Available for Migrating VMware ESXi Based Virtual Machines

[versed.perception]

Fully Licensed VMware ESXi, 6.7.0, 15160138 which will be consistent over all my nodes.

Sorry this was at the Dev team, not our User group here. I am working on setting up a Lab to test this with licensed, unlicensed-trial, and free licensed hosts to see what errors i can get.

 

[Scarfula]

Perfect
Also perfect timing i just migrated 120+ VM last week from VMware to Proxmox 8.1.4

 

[tuxis]

Very, very cool. Kudos!

 

[obwielnls]

Yes.
I will give some more details later.
Perhaps this is happes because it's a lab env, and the ESXi is installed as regular PVE VM.
I am planning to install ESXi on real hardware and will keep you posted.
Thanks

I have this same error.

restore-scsi9: stream-job finished
restore-drive jobs finished successfully, removing all tracking block devices
An error occurred during live-restore: VM 117 qmp command 'blockdev-del' failed - Node 'drive-scsi0-restore' is busy: node is used as backing hd of '#block3269'

TASK ERROR: live-restore failed

retried with the machine powered off and got.

transferred 12.0 GiB of 48.0 GiB (25.02%)
transferred 12.5 GiB of 48.0 GiB (26.03%)
qemu-img: error while reading at byte 13690205184: Function not implemented
TASK ERROR: unable to create VM 117 - cannot import from 'esxi:ha-datacenter/SDDm2/VcenterNCS/VcenterNCS.vmdk' - copy failed: command '/usr/bin/qemu-img convert -p -n -t none -f vmdk -O raw /run/pve/import/esxi/esxi/mnt/ha-datacenter/SDDm2/VcenterNCS/VcenterNCS.vmdk zeroinit:/dev/zvol/HP1ZFS/vm-117-disk-0' failed: exit code 1

 

[buckweet1980]

While I think this is a great add, correct me if wrong, it still doesn't bring a WEB GUI method for importing OVA/OVF files correct?

That is one of the bigger shortfalls I see with proxmox for folks coming from the ESXi world.

 

[ofelbar]

I have this same error.

restore-scsi9: stream-job finished
restore-drive jobs finished successfully, removing all tracking block devices
An error occurred during live-restore: VM 117 qmp command 'blockdev-del' failed - Node 'drive-scsi0-restore' is busy: node is used as backing hd of '#block3269'

TASK ERROR: live-restore failed

retried with the machine powered off and got.

transferred 12.0 GiB of 48.0 GiB (25.02%)
transferred 12.5 GiB of 48.0 GiB (26.03%)
qemu-img: error while reading at byte 13690205184: Function not implemented
TASK ERROR: unable to create VM 117 - cannot import from 'esxi:ha-datacenter/SDDm2/VcenterNCS/VcenterNCS.vmdk' - copy failed: command '/usr/bin/qemu-img convert -p -n -t none -f vmdk -O raw /run/pve/import/esxi/esxi/mnt/ha-datacenter/SDDm2/VcenterNCS/VcenterNCS.vmdk zeroinit:/dev/zvol/HP1ZFS/vm-117-disk-0' failed: exit code 1

Same error for me.

<snip>
transferred 9.0 GiB of 64.0 GiB (14.01%)
qemu-img: error while reading at byte 10032773120: Function not implemented
TASK ERROR: unable to create VM 202 - cannot import from 'Zadeo-ESXi:ha-datacenter/sas3T/ZadeoMGMT/ZadeoMGMT.vmdk' - copy failed: command '/usr/bin/qemu-img convert -p -n -f vmdk -O qcow2 /run/pve/import/esxi/Zadeo-ESXi/mnt/ha-datacenter/sas3T/ZadeoMGMT/ZadeoMGMT.vmdk zeroinit:/mnt/pve/NFS/images/202/vm-202-disk-0.qcow2' failed: exit code 1

Update: This is towards
VMWare ESX 6.7.0 Update 3 (Build 16075168) vSphere 6 Enterprise Plus

 

[Gilberto Ferreira]

Same error for me.

<snip>
transferred 9.0 GiB of 64.0 GiB (14.01%)
qemu-img: error while reading at byte 10032773120: Function not implemented
TASK ERROR: unable to create VM 202 - cannot import from 'Zadeo-ESXi:ha-datacenter/sas3T/ZadeoMGMT/ZadeoMGMT.vmdk' - copy failed: command '/usr/bin/qemu-img convert -p -n -f vmdk -O qcow2 /run/pve/import/esxi/Zadeo-ESXi/mnt/ha-datacenter/sas3T/ZadeoMGMT/ZadeoMGMT.vmdk zeroinit:/mnt/pve/NFS/images/202/vm-202-disk-0.qcow2' failed: exit code 1

Update: This is towards
VMWare ESX 6.7.0 Update 3 (Build 16075168) vSphere 6 Enterprise Plus

Well... It's a tech-preview.
Expected bugs
Hope this all fixed in next release.

 

[tuxis]

While I think this is a great add, correct me if wrong, it still doesn't bring a WEB GUI method for importing OVA/OVF files correct?

That is one of the bigger shortfalls I see with proxmox for folks coming from the ESXi world.

"We plan to integrate our OVF/OVA import tools into this new stack in the future. Currently, integrating additional import sources is not on our roadmap, but will be re-evaluated periodically"

 

[obwielnls]

Same error for me.

<snip>
transferred 9.0 GiB of 64.0 GiB (14.01%)
qemu-img: error while reading at byte 10032773120: Function not implemented
TASK ERROR: unable to create VM 202 - cannot import from 'Zadeo-ESXi:ha-datacenter/sas3T/ZadeoMGMT/ZadeoMGMT.vmdk' - copy failed: command '/usr/bin/qemu-img convert -p -n -f vmdk -O qcow2 /run/pve/import/esxi/Zadeo-ESXi/mnt/ha-datacenter/sas3T/ZadeoMGMT/ZadeoMGMT.vmdk zeroinit:/mnt/pve/NFS/images/202/vm-202-disk-0.qcow2' failed: exit code 1

Update: This is towards
VMWare ESX 6.7.0 Update 3 (Build 16075168) vSphere 6 Enterprise Plus

This is looking like rate limiting. sadly I have like 70 machines on this node. I wonder if that makes a difference.

 

[obwielnls]

More retries when hitting the rate limit?

Mar 27 14:51:12 HPG9208 esxi-folder-fus[1578590]: HPG9208 esxi-folder-fuse[1578590]: rate limited, retrying (1 of 5)...
Mar 27 14:51:22 HPG9208 esxi-folder-fus[1578590]: HPG9208 esxi-folder-fuse[1578590]: rate limited, retrying (2 of 5)...
Mar 27 14:51:32 HPG9208 esxi-folder-fus[1578590]: HPG9208 esxi-folder-fuse[1578590]: rate limited, retrying (3 of 5)...
Mar 27 14:51:42 HPG9208 esxi-folder-fus[1578590]: HPG9208 esxi-folder-fuse[1578590]: rate limited, retrying (4 of 5)...
Mar 27 14:51:52 HPG9208 esxi-folder-fus[1578590]: HPG9208 esxi-folder-fuse[1578590]: rate limited => Response { status: 503, version: HTTP/1.1, headers: {"date": "Wed, 27 Mar 2024 19:54:09 GMT", "connection": "close", "content-security-policy": "block-all-mixed-content", "content-type": "text/plain; charset=utf-8", "strict-transport-security": "max-age=31536000", "x-content-type-options": "nosniff", "x-frame-options": "DENY", "x-xss-protection": "1", "content-length": "0"}, body: Body(Empty) }
Mar 27 14:51:52 HPG9208 esxi-folder-fus[1578590]: HPG9208 esxi-folder-fuse[1578590]: error handling request: cached read failed: rate limited

 

[ofelbar]

This is looking like rate limiting. sadly I have like 70 machines on this node. I wonder if that makes a difference.

I get this error sometimes now also.


(vim.fault.HostConnectFault) { dynamicType = <unset>, dynamicProperty = (vmodl.DynamicProperty) [], msg = '503 Service Unavailable', faultCause = <unset>, faultMessage = (vmodl.LocalizableMessage) []} (500)

 

[versed.perception]

Ok, I could not get authentication to work on directly to ESXi with the free license present. While I could authenticate (no errors) with an Enterprise+ key present on the host it would not read any of the VMs. I deployed VCSA and created a Datacenter>Cluster and changed the PVE mapping to vCenter and it all worked. Tested on ESXi 7.0.3, 20328353, 7.0U3d-19482537 and 6.7.0u3b-15160138. Currently am doing a ESXi->PVE migration through vCenter on 6 testing VMs.

I really like how PVE knows that VMware SCSI has to be converted to SATA

 

[undone]

Beautiful. Now we need one for Hyper V too, I have so many to migrate from.

 

[obwielnls]

Ok, I could not get authentication to work on directly to ESXi with the free license present. While I could authenticate (no errors) with an Enterprise+ key present on the host it would not read any of the VMs. I deployed VCSA and created a Datacenter>Cluster and changed the PVE mapping to vCenter and it all worked. Tested on ESXi 7.0.3, 20328353, 7.0U3d-19482537 and 6.7.0u3b-15160138. Currently am doing a ESXi->PVE migration through vCenter on 6 testing VMs.

I really like how PVE knows that VMware SCSI has to be converted to SATA

Connecting to vsphere gets me constant rate limit errors in the log.
( I do have 3 v7 vcenters linked. )

Mar 27 15:56:19 PVIntel systemd[1]: run-pve-import-esxi-vspherencs-mnt.mount: Deactivated successfully.
Mar 27 15:57:05 PVIntel esxi-folder-fus[2114953]: PVIntel esxi-folder-fuse[2114953]: rate limited, retrying (1 of 5)...
Mar 27 15:57:15 PVIntel esxi-folder-fus[2114953]: PVIntel esxi-folder-fuse[2114953]: rate limited, retrying (2 of 5)...
Mar 27 15:57:25 PVIntel esxi-folder-fus[2114953]: PVIntel esxi-folder-fuse[2114953]: rate limited, retrying (3 of 5)...
Mar 27 15:57:35 PVIntel esxi-folder-fus[2114953]: PVIntel esxi-folder-fuse[2114953]: rate limited, retrying (4 of 5)...
Mar 27 15:57:45 PVIntel esxi-folder-fus[2114953]: PVIntel esxi-folder-fuse[2114953]: rate limited => Response { status: 503, version: HTTP/1.1, headers: {"date": "Wed, 27 Mar 2024 21:00:02 GMT", "set-cookie": "vmware_soap_session=\"9f051beaf60e715f6fe55684cad0f6a3c60bc6a7\"; Path=/; HttpOnly; Secure;", "content-security-policy": "block-all-mixed-content", "content-type": "text/plain; charset=utf-8", "strict-transport-security": "max-age=31536000", "x-content-type-options": "nosniff", "x-frame-options": "DENY", "x-xss-protection": "1", "content-length": "0", "x-envoy-upstream-service-time": "58", "server": "envoy"}, body: Body(Empty) }
Mar 27 15:57:45 PVIntel systemd[1]: pve-esxi-fuse-vspherencs.scope: Deactivated successfully.
Mar 27 15:57:45 PVIntel pvestatd[1734]: failed to spawn fuse mount, process exited with status 256
Mar 27 15:57:46 PVIntel pvestatd[1734]: status update time (90.986 seconds)
Mar 27 15:57:46 PVIntel systemd[1]: Started pve-esxi-fuse-vspherencs.scope.
Mar 27 15:57:46 PVIntel systemd[1]: run-pve-import-esxi-vspherencs-mnt.mount: Deactivated successfully.
Mar 27 15:58:08 PVIntel esxi-folder-fus[2115266]: PVIntel esxi-folder-fuse[2115266]: rate limited, retrying (1 of 5)...
Mar 27 15:58:18 PVIntel esxi-folder-fus[2115266]: PVIntel esxi-folder-fuse[2115266]: rate limited, retrying (2 of 5)...
Mar 27 15:58:28 PVIntel esxi-folder-fus[2115266]: PVIntel esxi-folder-fuse[2115266]: rate limited, retrying (3 of 5)...
Mar 27 15:58:38 PVIntel esxi-folder-fus[2115266]: PVIntel esxi-folder-fuse[2115266]: rate limited, retrying (4 of 5)...
Mar 27 15:58:48 PVIntel esxi-folder-fus[2115266]: PVIntel esxi-folder-fuse[2115266]: rate limited => Response { status: 503, version: HTTP/1.1, headers: {"date": "Wed, 27 Mar 2024 21:01:05 GMT", "set-cookie": "vmware_soap_session=\"73ba1cfd7cfe413b95416e2fe2ddaf2cd569e199\"; Path=/; HttpOnly; Secure;", "content-security-policy": "block-all-mixed-content", "content-type": "text/plain; charset=utf-8", "strict-transport-security": "max-age=31536000", "x-content-type-options": "nosniff", "x-frame-options": "DENY", "x-xss-protection": "1", "content-length": "0", "x-envoy-upstream-service-time": "51", "server": "envoy"}, body: Body(Empty) }
Mar 27 15:58:48 PVIntel systemd[1]: pve-esxi-fuse-vspherencs.scope: Deactivated successfully.
Mar 27 15:58:48 PVIntel pvestatd[1734]: failed to spawn fuse mount, process exited with status 256
Mar 27 15:58:48 PVIntel pvestatd[1734]: status update time (62.642 seconds)
Mar 27 15:58:53 PVIntel systemd[1]: Started pve-esxi-fuse-vspherencs.scope.
Mar 27 15:58:53 PVIntel systemd[1]: run-pve-import-esxi-vspherencs-mnt.mount: Deactivated successfully.
Mar 27 15:58:58 PVIntel esxi-folder-fus[2115732]: PVIntel esxi-folder-fuse[2115732]: rate limited, retrying (1 of 5)...
Mar 27 15:59:09 PVIntel esxi-folder-fus[2115732]: PVIntel esxi-folder-fuse[2115732]: rate limited, retrying (2 of 5)...

 

[tuxis]

Just tested from a trial esxi-box, worked like a charm.

But now:

Mar 27 22:03:19 pve1 pvedaemon[1498]: (vim.fault.HostConnectFault) {
                                         dynamicType = <unset>,
                                         dynamicProperty = (vmodl.DynamicProperty) [],
                                         msg = '503 Service Unavailable',
                                         faultCause = <unset>,
                                         faultMessage = (vmodl.LocalizableMessage) []
                                      }

Which seems to be ratelimiting as well.

After that, I got 'username or password incorrect', which I 'solved' be resetting the password on the esxi-side (to the same password) ...

 

[Falk R.]

Very good work.
This saves a lot of time when creating the VMs, only the copying is a little slower than my previous method via NFS.
If someone has an idea how to integrate the Virtio SCSI driver into Windows so that you don't have to rebuild the disk afterwards, then that's the game changer.

 

[spirit]

a rust fuse to mount a esx datastore ? You are crazy guys ! Congrats !

 

[spirit]

Very good work.
This saves a lot of time when creating the VMs, only the copying is a little slower than my previous method via NFS.
If someone has an idea how to integrate the Virtio SCSI driver into Windows so that you don't have to rebuild the disk afterwards, then that's the game changer.

personnally, for my last migrations, I have simply install the drivers with virtio-win iso, when vm was still running on esx, and the vm was able to boot directly on virtio-scsi.

 

[tuxis]

Very good work.
This saves a lot of time when creating the VMs, only the copying is a little slower than my previous method via NFS.
If someone has an idea how to integrate the Virtio SCSI driver into Windows so that you don't have to rebuild the disk afterwards, then that's the game changer.

You can use this as a basis. Issue is that Windows needs to boot once with the virtio-driver needed, and only after that, you can boot from Virtio-scsi. (As far as I know, at least: https://support.tuxis.nl/en/knowledgebase/article/convert-windows-boot-disk-from-virtio-to-scsi)

 

[Falk R.]

You can use this as a basis. Issue is that Windows needs to boot once with the virtio-driver needed, and only after that, you can boot from Virtio-scsi. (As far as I know, at least: https://support.tuxis.nl/en/knowledgebase/article/convert-windows-boot-disk-from-virtio-to-scsi)

That has been my approach in recent years

personnally, for my last migrations, I have simply install the drivers with virtio-win iso, when vm was still running on esx, and the vm was able to boot directly on virtio-scsi.

Then tell me what trick you used to teach Windows to use the driver for booting. I have already set the bootable flag in the registry, but it is gone again after a reboot.