Struggling to Migrate Windows guests from XEN to Proxmox

[BeDazzler]

Hi All,

I'm relatively new to Promox, having come from XEN and ovirt platforms, but enjoying Proxmox!

I'm struggling to migrate Windows 2012 R2 STD guest VMs from XEN 7 to Proxmox 6.2. They come across and launch into a BSOD loop (something went wrong...).

I've read a number of forum posts and techniques for achieving VM migrations, however none of them have worked for me.

My process is:

1) Follow the steps to remove Citrix agent software, etc as per https://www.reddit.com/r/Proxmox/comments/anhyn0/xenserver_to_proxmox_windows_server_migration/
2) Install QEMU agent support
3) Install iostor, netKVM, VirtSCSI driver support on Windows VM.
4) Shutdown Windows VM on XEN.
5) Use http and the xva-conf.sh script to bring VMs across. This works fine. Yes I rename the Ref file, etc.
6) Create a new VM in Proxmox with default values, adjusting RAM/CPU/Disk to match the XEN guest parameters.
7) qm importdisk ID bla.raw ZFS-FileSystem-bla. That works.
8) I then edit the VM hardware configuration and switch the original disk with the migrated disk.

Then, on boot I hit the BSOD loop and the VM never starts. The only thing I can do is enter repair mode.
If I go back to the XEN host and start the VM again, it boots fine, albeit without XEN guest tools.

I've tried changing the hard disk controller type in Proxmox and the boot order is correct.
I can get into Repair mode, logon to the NTFS partition and see the C: drive.

I would really like to hear how others have overcome this challenge.

Many thanks

Darren.

 

[ness1602]

You didn't say what windows version are you trying to migrate?
Also why don't you use export ova file from Xen?

 

[BeDazzler]

Hi @ness1602
Windows 2012 R2 STD. I have updated the original info.
I have tried ova export previously without success. After exporting then converting to a raw image using qemu-img and then bringing the image into Proxmox, it boots with the same problem.

The disk and data are all in tact, I can access via Windows Repair mode, however despite anything I try using diskpart and Windows boot repair tools it refuses to boot into Windows and goes to BSOD after Windows logo.

D.

 

[BeDazzler]

Anyone ???

Surely someone here has migrated Windows 2012 HVM VMs from XEN to Proxmox.

 

[ness1602]

Maybe if you can screenshot the error on BSOD?

 

[Petr Svacina]

Migration from ANY version Xenserver to ANYWHERE (proxmox, vmware) is PURE HELL.

You must follow exactly this:

https://support.storagecraft.com/ar...m-thread-exception-not-handled?language=en_US

Migrate LVM, VHD from xen to proxmox. This should be okay.
Start VM and you will get BSOD. That is OK. Boot to recovery, in recovery mode:

1. Do this:

Use administrative command prompt and navigate to C:\windows\system32 directory.
DIR Xen*.*
DEL Xen*.*

Navigate to C:\windows\system32\drivers directory.
DIR Xen*.*
DEL Xen*.*

2. Do this:

You are in the Windows PE based Recovery Environment.
Open a command prompt by going to Tools > Command Shell
Load the Windows System registry hive from the boot volume.

reg load HKLM\restore c:\Windows\System32\config\SYSTEM

Run regedit.exe from the command prompt.

Browse to following registry locations. There maybe multiple keys with that name. Look through them to find UpperFilters keys with the value XENFLT: (THIS IS MOST IMPORTANT OF ALL THE MIGRATION !!!)

Examples:
HKEY_LOCAL_MACHINE\restore\ControlSet001\Control\Class\{4d36e96a-e325-11ce-bfc1-08002be10318}
HKEY_LOCAL_MACHINE\restore\ControlSet001\Control\Class\{4d36e97d-e325-11ce-bfc1-08002be10318}

Right-click the UpperFilters key with the XENFLT entry and click Delete.
For the remainder of the keys, search the registry (Ctrl+F) and search for UpperFilters -> XENFLT.
Remove any keys mentioning XenFLT. Use F3 to search for the next instance.

Remove competely:

HKEY_LOCAL_MACHINE\restore\ControlSet001\Services\XEN
HKEY_LOCAL_MACHINE\restore\ControlSet001\Services\xenbus
HKEY_LOCAL_MACHINE\restore\ControlSet001\Services\xendisk
HKEY_LOCAL_MACHINE\restore\ControlSet001\Services\xenfilt
HKEY_LOCAL_MACHINE\restore\ControlSet001\Services\xeniface
HKEY_LOCAL_MACHINE\restore\ControlSet001\Services\xenlite
HKEY_LOCAL_MACHINE\restore\ControlSet001\Services\xennet
HKEY_LOCAL_MACHINE\restore\ControlSet001\Services\XenSvc
HKEY_LOCAL_MACHINE\restore\ControlSet001\Services\xenvbd
HKEY_LOCAL_MACHINE\restore\ControlSet001\Services\xenvif

Close Registry Editor and unload the hive:

reg unload HKLM\restore

Reboot. Your Windows 7/8/10/2008/2012/2016/2019 will normaly boot ....

3. Do this: Uninstall xentools / agent from the add/remove programs ....

 

[Petr Svacina]

PS: Example successful migration from Xenserver: https://techblog.jeppson.org/2018/01/migrate-xenserver-proxmox/

But of course, after migration, you will get BSOD and you must follow upper post

 

[BeDazzler]

PS: Example successful migration from Xenserver: https://techblog.jeppson.org/2018/01/migrate-xenserver-proxmox/

But of course, after migration, you will get BSOD and you must follow upper post

Thank you @Petr Svacina for taking the time to reply with the above information.
I have previously done pretty much everything except regedit in PE mode to remove the XEN entries so I tried your suggested steps, however that didn't work either.

I also tried converting the Win2K12 HVM VM to PV mode before exporting from XEN to Proxmox which made no difference.

I also tried the dism /image steps to remove XEN OEM drivers in addition to your steps, but that also made no difference.

I can see and access the filesystem in Windows repair mode, which does boot from the same VM, so it has to be something related to disk driver or boot parameters for the 2nd partition where Windows resides since the primary (recovery) partition boots fine.

I'm open to suggestions.

D.

 

[ness1602]

https://www.drivereasy.com/knowledge/fix-bad-system-config-info-error-windows-10/

https://answers.microsoft.com/en-us...ations-0/52359f87-de4a-41dc-b0c3-cc275e1d9fbf

Delete all xen devices,even hidden.
Delete xen* files from system32 folder.
Try all that and see if it is working

 

[Petr Svacina]

I forget to mention, that after conversion first boot MUST be done with hardrisk attached to IDE. SO do not use Virtio, or SCSI, LSI, but pure IDE.

Delete all xen devices, even hidden. - He is not able to boot, so this can not be done.
I have migrated hundred of the Xen / Xenserver VMs , and problem always was the Citrix shit drivers ... If you uninstall them first, registry remain untouched and this bug leads to the BOSD.

PS: Stick with HVM

PS2:

You have loaded hive like this: reg load HKLM\restore c:\Windows\System32\config\SYSTEM (suggesting c: is VM disk, not recovery part !)

How many UpperFilters XENFLT entries did you deleted ? This entries are in 2 main trees:

HKEY_LOCAL_MACHINE\restore\ControlSet001
HKEY_LOCAL_MACHINE\restore\ControlSet002

Did you delete them all ? Are you sure ?

Picture attached show up Upperfilter which MUST be deleted .....
I count them 6 entries to delete ....
ATTENTION : Do not delete any orher UpperFIlter (like PartMgr), or you will also not boot

 

[Petr Svacina]

Paste here vm config in PVE (ie.: 100.conf)

Last question: VM You are migrating is BIOS (MBR) or UEFI (GPT) type of installation ? Do not mix them, better post even old Xen vm config ...

 

[BeDazzler]

Paste here vm config in PVE (ie.: 100.conf)

Last question: VM You are migrating is BIOS (MBR) or UEFI (GPT) type of installation ? Do not mix them, better post even old Xen vm config ...

Hi @Petr Svacina

Thanks again for the additional information.

Yes I agree the XEN Guest drivers are an absolute headache!

I can confirm that I am using only IDE for the VM disk, even though a SCSI adapter appears when creating new VMs there is nothing attached to it.

I've exported HVM from XEN every time except once to test if PV made any difference, which it did not.

On Proxmox, the VM boot order is set to CDROM (which contains the Win2K12 R2 CD) then IDE (imported disk from XEN img export).

The migrated disk is set to IDE0.

The method I use for XEN export is:
wget --http-user=root --http-password=bla http://xenhostIP/export?uuid=6408e583-1e92-7521-0e2e-d8b015495816 -O - | tar --to-command=./xva-conv.sh -xf

This is done from /root on the Promox host which contains the script xva-conv.sh with chmod +x.

I then run qm importdisk 101 imagename.raw ZFS-R1 which completes fine and is shown in Proxmox admin as a new disk.

Then I goto the VM, detach the default created disk and attach the newly migrated disk and boot the VM.

It usually shows the Windows logo, then the dots appear whilst it thinks about booting, usually followed by a BSOD.

Then I boot from the CDROM drive which contains the Win2K12 R2 ISO. I select troubleshoot, command prompt.

When I edit the registry I do a reg load HKLM\restore c:\Windows\System32\config\SYSTEM then I navigate to the /restore hive and being my search/replace from there, filtering on "XEN" and manually reviewing each entry before I delete either the entry or the key + entry.

I've just performed a fresh VM qm importdisk onto a newly create VM in Proxmox.

I booted using the Windows 2012 R2 CD, selecting Repair mode.

From there I entered the command prompt and can see the recovery partiton as C: with OS and files as D:

I then do del D:\Windows\System32\xen*.* which finds the files and removes them.
Same for D:\Windows\System32\drivers\xen*.*

Then, reg load HKLM\restore D:\Windows\System32\config\SYSTEM

regedit, navigate to restore hive and begin search for "XENFILT"

There are 2 entries which I delete:
HKLM\restore\CurrentControlSet001\Control\Class\{numbers}
HKLM\restore\CurrentControlSet001\Control\Class\{numbers}

Then I search for UpperFilters and remove the line containing XENFILT, of which there were 6. To confirm I am deleting only the key entry (right hand side) and not the entire key (from the left side).

Then I navigate to the restore hive, scroll down to ControlSet001\Services to find XEN entries near the bottom, which I delete starting with XEN and ending with xenif.

Then I close regedit and key reg unload HKLM/restore which completes OK.

After that I reboot the VM, expecting it to load Windows.

Windows starts, logo appears, dots start circling for about 20 seconds, then BSOD and automatic reboot which repeats until repair mode starts.

And that's where I'm up to.

To answer your other question, the Windows 2012 R2 VM runs in BIOS boot mode on XEN.

There must be a step or process I am missing here.

D.

 

[mbosma]

I've used the same script to migrate hundreds of vm's from Xen to Proxmox.
Even a Windows 2003 vm worked after changing some settings from our default vm template:

acpi: 1
agent: 0
bios: seabios
boot: cdn
bootdisk: ide0
cores: 4
ide0: local-lvm:vm-XXX-disk-0,size=100G
ide2: none,media=cdrom
memory: 4096
name: XXXXXXX
net0: e1000=XXXXXXX,bridge=vmbr0,tag=XXX
numa: 0
onboot: 1
parent: preboot
scsihw: virtio-scsi-pci
smbios1: uuid=XXXXXXX
sockets: 1
vmgenid: XXXXXXXXXXXXXX

Could you try to boot the vm in safe mode (without the windows logo), this way you can see which driver gets loaded while the vm crashes.

 

[Petr Svacina]

Even can try safe mode with no automatic reboot upon crash to see what is exactly crashing ....

 

[aaron]

Are you trying it with the same VM all the time? Maybe something else is wrong/misconfigured in the VM that causes all these troubles.

Migrating another VM might work "quite well".

 

[BeDazzler]

I've used the same script to migrate hundreds of vm's from Xen to Proxmox.
Even a Windows 2003 vm worked after changing some settings from our default vm template:

acpi: 1
agent: 0
bios: seabios
boot: cdn
bootdisk: ide0
cores: 4
ide0: local-lvm:vm-XXX-disk-0,size=100G
ide2: none,media=cdrom
memory: 4096
name: XXXXXXX
net0: e1000=XXXXXXX,bridge=vmbr0,tag=XXX
numa: 0
onboot: 1
parent: preboot
scsihw: virtio-scsi-pci
smbios1: uuid=XXXXXXX
sockets: 1
vmgenid: XXXXXXXXXXXXXX

Could you try to boot the vm in safe mode (without the windows logo), this way you can see which driver gets loaded while the vm crashes.

My VM config was very similar, except mine had VGA as the standard display adapter and no acpi. I changed to match yours but no difference.

 

[BeDazzler]

Are you trying it with the same VM all the time? Maybe something else is wrong/misconfigured in the VM that causes all these troubles.

Migrating another VM might work "quite well".

Hi @aaron this is a standard Windows 2012 build with 3 basic 64 bit apps installed that don't run until you launch them. It doesn't get any more basic/vanilla than this.

 

[BeDazzler]

Even can try safe mode with no automatic reboot upon crash to see what is exactly crashing ....

Good idea @Petr Svacina . I've done that and the error is: INACCESSIBLE_BOOT_DEVICE.

That tells me it's driver related.
I'm booting in IDE mode, so I assume Windows would have support for at least that and not require anything special.

 

[Petr Svacina]

Oki. I remember a similar problem, once ...
I was not able to migrate Windows 7 VM, same error like you. Now I can't remember what exactly was partition table in original VM, but I remember, that I have suspected, tak there is something wrong / differect with booting of the original VM, so after migration I was not able to boot...

But I solved it a different way:

1. Boot original XEN VM.
2. Stop all the running services to "suspend the VM not to do anything"
3. Run this app: https://docs.microsoft.com/en-us/sysinternals/downloads/disk2vhd
4. Convert Xen VM to VHD by disk2vhd.
5. convert vhd by qemu-img to raw, qcow2, LVM
6. Boot the vm, if necessarily, follow the step to clean up registry and other "xen left drivers"

PS: Your config is OK, steps you are doing are OK. So I'm running out of ideas :-D

 

[mbosma]

I've had issues with the "INACCESSIBLE_BOOT_DEVICE" message when transfering a hw machine to a vm. The issue was resolved by deleting Intels Rapid storage.
I suspect something in the boot process is still looking for a Xen hdd in specific instead of a common (ide) drive.

disk2vhd works fine for migrating vm's but I don't think this will solve the issue since you're still creating a backup of the same disk with Citrix drivers.
Maybe try running the guest agent cleaning tool (on a copy of the vm ofrourse) before migrating it to proxmox:
https://xcp-ng.org/forum/topic/1049/windows-guest-tools-cleaner-tool