Migrating VM from TrueNAS Core's bhyve (which uses KVM too) to PVE?

[Dunuin]

Hi,

Got a OPNsense VM on TrueNAS Core 13.0-U3 that I want to migrate to PVE 7.2.
PVE also uses ZFS, so I shutdown the VM on TrueNAS, created a snapshot "migate" of the two zvols the VM is using (one zvol for root filesystem, one zvol for swap partition) and then used replication over ssh to move the zvols from the TrueNAS pool to the PVE pool:

# On TrueNAS
zfs snapshot tank/encrypted/rootdisk@migrate
zfs snapshot tank/encrypted/swapdisk@migrate
zfs send tank/encrypted/rootdisk@migrate | ssh root@IP.OF.MY.PVE zfs recv -o encryption=on tank2/encrypted/rootdisk
zfs send tank/encrypted/swapdisk@migrate | ssh root@IP.OF.MY.PVE zfs recv -o encryption=on tank2/encrypted/swapdisk
zfs destroy tank/encrypted/rootdisk@migrate
zfs destroy tank/encrypted/swapdisk@migrate

Then I created a new VM with VMID 141 with same RAM, same vCPU count, without a disk and OVMF as my VM on TrueNAS was set to use UEFI. TrueNAS VM was using 9 virtio NICs so I added 9 virtio NICs in the same order attached to the same bridges/vlans. So OPNsense should see the same NIC names using the same subnets.

Then I ran this:

# on PVE:
zfs destroy tank2/encrypted/rootdisk@migrate
zfs destroy tank2/encrypted/swapdisk@migrate
zfs rename tank2/encrypted/rootdisk tank2/encrypted/vm-141-disk-0
zfs rename tank2/encrypted/swapdisk tank2/encrypted/vm-141-disk-1
qm rescan

The VM now found the disks so I attached vm-141-disk-0 as scsi0 and vm-141-disk-1 as scsi1. Also changed the boot order so that the VM only tries to boot from scsi0. And I added a EFI disk without pre-enrolled keys and changed the disk controller to virtio SCSI single. In TrueNAS the disks were using "virtio" but not sure if that is virtio SCSI, virtio block or whatever.

Then I tried to boot that VM but it looks like it won't do that. No matter if I try i440fx or Q35, all I see when booting the VM is this:


Maybe the sector size is a problem as virtio disks of TrueNAS use 4K/4K as logical/physical sectors and PVe is using 512B/512B?

Anyone got an idea how to get the VM booting?

Config of my newly created PVE VM:

agent: 1
bios: ovmf
boot: order=scsi0
cores: 4
cpu: host,flags=+pcid;+spec-ctrl;+ssbd;+aes
efidisk0: VMpool_VLT_VM:vm-141-disk-2,efitype=4m,size=1M
ide2: none,media=cdrom
machine: q35
memory: 2048
meta: creation-qemu=7.0.0,ctime=1667917913
name: OPNsense2
net0: virtio=42:74:12:XX:XX:XX,bridge=vmbr4,firewall=1
net1: virtio=22:6C:02:XX:XX:XX,bridge=vmbr2,firewall=1
net2: virtio=5A:31:0F:XX:XX:XX,bridge=vmbr41,firewall=1
net3: virtio=CE:CB:FE:XX:XX:XX,bridge=vmbr42,firewall=1
net4: virtio=62:13:54:XX:XX:XX,bridge=vmbr43,firewall=1
net5: virtio=3A:DD:51:XX:XX:XX,bridge=vmbr44,firewall=1
net6: virtio=F2:F4:47:XX:XX:XX,bridge=vmbr46,firewall=1
net7: virtio=A6:EF:15:XX:XX:XX,bridge=vmbr47,firewall=1
net8: virtio=0A:1A:8B:XX:XX:XX,bridge=vmbr50,firewall=1
numa: 0
ostype: other
scsi0: VMpool_VLT_VM:vm-141-disk-0,discard=on,iothread=1,size=33554496K,ssd=1
scsi1: VMpool_VLT_VM:vm-141-disk-1,size=4194368K,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=554a7cc0-c71d-4ee6-bd84-31a3976569dd
sockets: 1
startup: order=1
vmgenid: f19c7a52-42e5-47de-9f6b-889efa1891c2

Wasn't able to find any bhyve config files on TrueNAS. I will show some screenshots of the webUI:

logical_sectorsize: 4096
physical_sectorsize: 4096
type: VIRTIO
path: /dev/zvol/tank/encrypted/rootdisk

logical_sectorsize: 4096
physical_sectorsize: 4096
type: VIRTIO
path: /dev/zvol/tank/encrypted/swapdisk

 

[Dunuin]

Any ideas?

Tried it today with dd and qm import but still no success. Also tried SCSI, SATA, IDE and virtio block as protocols for the virtual disk. EFI disk with and without pre-enrolled keys.