Running a single guest with different OVMF Firmware

[AndreasAC]

Hi there,

I want to run a guest operating system which requires slightly patched OVMF Firmware. As I don't want to use this firmware for all VMs, I'm looking for a way to substitute it for just a single VM, instead of just replacing the pve-edk2-firmware package, which seems to be what other people are currently doing.

I read the beginning of https://www.linux-kvm.org/downloads/lersek/ovmf-whitepaper-c770f8c.txt and gather from it that it would be possible in theory.

If there is no simple way to do this currently via a config file change, how difficult would you estimate developing a patch would be for an average software developer completely unfamiliar with your code base?

Happy easter and thank you very much

AndreasAC

 

[t.lamprecht]

Hi,

so there are various ways to address this, from simple workaround to more elaborate inclusion.

If this is a local VM which will stay on this node you should be able to just hack it in by omitting the "OVMF" option completely and manually add the additional KVM/QEMU arguments:

-drive 'if=pflash,unit=0,format=raw,readonly,file=/path/to/OVMF_CODE.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,size=131072,file=/path/to/OVMF_VARS.df'

Adapt path and size respectively to a local saved copy of your extra build OVMF code and efivars images.

You can tell PVE to add this to a VM via:

qm set 100 --args "-drive 'if=pflash,unit=0,format=raw,readonly,file=/path/to/OVMF_CODE.fd' -drive if=pflash,unit=1,format=raw,id=drive-efidisk0,size=131072,file=/path/to/OVMF_VARS.df'"

So that would be one quick way, but naturally you can also patch this in the code.

One method would be to change the get_ovmf_files method in the PVE::QemuServer module, it could be extended to get the VMID and then respective to that VMID calue decide if the normal one should be returned.
https://git.proxmox.com/?p=qemu-ser...78c674e221ed93634e2fcbfeced7e7e;hb=HEAD#l2802

You could either build a Debian packet with those changes yourself, or edit the live one under //usr/share/perl5/PVE/QemuServer.pm + restart pveproxy and pvedaemon services. The latter will get overwritten on updates of the qemu-server package though.

More fancier, which we possibly also would accept to upstream, would be extending the "bios" property to be a format and have additional have a optional options, like firmware code path or the like.
https://git.proxmox.com/?p=qemu-ser...578c674e221ed93634e2fcbfeced7e7e;hb=HEAD#l619

How to do that sanely and in a backwards compatible manner is a bit much for this forum, if you decide to really dig into it and go that way then head over to the pve-devel mailinglist

 

[AndreasAC]

Thank you very much for the quick and detailed reply! I will first go for the hack to verify if the guest works like that and will then decide on how to proceed ​

 

[FormerVMW]

Hi,

so there are various ways to address this, from simple workaround to more elaborate inclusion.

If this is a local VM which will stay on this node you should be able to just hack it in by omitting the "OVMF" option completely and manually add the additional KVM/QEMU arguments:

-drive 'if=pflash,unit=0,format=raw,readonly,file=/path/to/OVMF_CODE.fd'
-drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,size=131072,file=/path/to/OVMF_VARS.df'

Adapt path and size respectively to a local saved copy of your extra build OVMF code and efivars images.

You can tell PVE to add this to a VM via:

qm set 100 --args "-drive 'if=pflash,unit=0,format=raw,readonly,file=/path/to/OVMF_CODE.fd' -drive if=pflash,unit=1,format=raw,id=drive-efidisk0,size=131072,file=/path/to/OVMF_VARS.df'"

So that would be one quick way, but naturally you can also patch this in the code.

One method would be to change the get_ovmf_files method in the PVE::QemuServer module, it could be extended to get the VMID and then respective to that VMID calue decide if the normal one should be returned.
https://git.proxmox.com/?p=qemu-ser...78c674e221ed93634e2fcbfeced7e7e;hb=HEAD#l2802

You could either build a Debian packet with those changes yourself, or edit the live one under //usr/share/perl5/PVE/QemuServer.pm + restart pveproxy and pvedaemon services. The latter will get overwritten on updates of the qemu-server package though.

More fancier, which we possibly also would accept to upstream, would be extending the "bios" property to be a format and have additional have a optional options, like firmware code path or the like.
https://git.proxmox.com/?p=qemu-ser...578c674e221ed93634e2fcbfeced7e7e;hb=HEAD#l619

How to do that sanely and in a backwards compatible manner is a bit much for this forum, if you decide to really dig into it and go that way then head over to the pve-devel mailinglist

Is there a way to do what you describe that actually works? SonicWall, and other vendors, are now providing their own OMVF files. This obviously works fine under KVM/VirtManager because the files can be specified. Under Proxmox it's not so joyous...

The below is directly from SonicWall's latest install instructions:

Deploy SonicCoreX NSv KVM image on virt-manager

Provision /etc/qemu/firmware directory on virt-manager hostCreate /etc/qemu/firmware directory on virt-manager host if not present
copy 10-sonicwall-x86_64-secure-enrolled.json (JSON file) to /etc/qemu/firmware
sudo systemctl reload libvirtd


Provision /usr/share/OVMF directory on virt-manager hostCreate /usr/share/OVMF directory on virt-manager host if not present
Copy OVMF_CODE.sw.fd and OVMF_VARS.sw.fd to /usr/share/OVMF directory

Further in the instructions you choose the firmware to be used and point to the proper file (in Virt-Manager). I post the above only to demonstrate this is becoming more common and I'm not seeing an easy solution to deploy these types of VMs on Proxmox.


I added the args to the conf file of the VM, but Proxmox ignores it (puts it at the end of the kvm command) and proceeds to boot the default OVMF files. It then errors out and states bus0,unit0 is already loaded.

The error message:

kvm: -drive if=pflash,unit=0,format=raw,readonly=on,file=/var/lib/vz/images/148/OVMF_CODE.sw.fd: drive with bus=0, unit=0 (index=-1) exists
TASK ERROR: start failed: QEMU exited with code 1

Using just the args in the conf file without trying to trick it by putting the VARS as an EFI disk, it produces the below error. Still pointing to the issue of Proxmox forcing the loading of the default UEFI files. Note that unit0 and unit1 are already in use so unit2 and unit3 are used:

kvm: -drive if=pflash,format=raw,readonly=on,file=/var/lib/vz/images/148/OVMF_CODE.sw.fd: machine type does not support if=pflash,bus=0,unit=2
kvm: -drive if=pflash,format=raw,file=/var/lib/vz/images/148/OVMF_VARS.sw.fd: machine type does not support if=pflash,bus=0,unit=3

This is the command being used by proxmox to start the VM (via ps -ax):

/usr/bin/kvm -id 148 -name NSv270-NEW,debug-threads=on -no-shutdown -chardev socket,id=qmp,path=/var/run/qemu-server/148.qmp,server=on,wait=off -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/148.pid -daemonize -smbios type=1,uuid=f15919f0-bff3-4038-b224-b763b8861567 -drive if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd -drive if=pflash,unit=1,id=drive-efidisk0,format=raw,file=/tmp/148-ovmf.fd,size=131072 -smp 4,sockets=2,cores=2,maxcpus=4 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/148.vnc,password=on -cpu qemu64,+abm,+aes,+avx,+avx2,+bmi1,+bmi2,enforce,+f16c,+fma,+kvm_pv_eoi,+kvm_pv_unhalt,+movbe,+pni,+popcnt,+sse4.1,+sse4.2,+ssse3,+xsave -m 8192 -object iothread,id=iothread-virtioscsi3 -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg -device vmgenid,guid=0037074e-3336-4053-a437-067e94ba4372 -device usb-tablet,id=tablet,bus=ehci.0,port=1 -chardev socket,id=serial0,path=/var/run/qemu-server/148.serial0,server=on,wait=off -device isa-serial,chardev=serial0 -device VGA,id=vga,bus=pcie.0,addr=0x1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on -iscsi initiator-name=iqn.1993-08.org.debian:01:df4293a3366a -drive if=none,id=drive-ide2,media=cdrom,aio=io_uring -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=101 -device virtio-scsi-pci,id=virtioscsi3,bus=pci.3,addr=0x4,iothread=iothread-virtioscsi3 -drive file=/var/lib/vz/images/148/vm-148-disk-2.qcow2,if=none,id=drive-scsi3,format=qcow2,cache=none,aio=io_uring,detect-zeroes=on -device scsi-hd,bus=virtioscsi3.0,channel=0,scsi-id=0,lun=3,drive=drive-scsi3,id=scsi3,bootindex=100 -netdev type=tap,id=net0,ifname=tap148i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=BC:24:11:1B:C5:C3,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=102 -netdev type=tap,id=net1,ifname=tap148i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=BC:24:11:65:46:15,netdev=net1,bus=pci.0,addr=0x13,id=net1,rx_queue_size=1024,tx_queue_size=256 -netdev type=tap,id=net2,ifname=tap148i2,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=BC:24:11:78:7D:E2,netdev=net2,bus=pci.0,addr=0x14,id=net2,rx_queue_size=1024,tx_queue_size=256 -netdev type=tap,id=net3,ifname=tap148i3,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=BC:24:11:AF:AF:99,netdev=net3,bus=pci.0,addr=0x15,id=net3,rx_queue_size=1024,tx_queue_size=256 -netdev type=tap,id=net4,ifname=tap148i4,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=BC:24:11:1A:96:04,netdev=net4,bus=pci.0,addr=0x16,id=net4,rx_queue_size=1024,tx_queue_size=256 -netdev type=tap,id=net5,ifname=tap148i5,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=BC:24:11:EA:4E:CD,netdev=net5,bus=pci.0,addr=0x17,id=net5,rx_queue_size=1024,tx_queue_size=256 -netdev type=tap,id=net6,ifname=tap148i6,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=BC:24:11:29:5B:E0,netdev=net6,bus=pci.1,addr=0x1,id=net6,rx_queue_size=1024,tx_queue_size=256 -netdev type=tap,id=net7,ifname=tap148i7,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=BC:24:11:26:E9:5A,netdev=net7,bus=pci.1,addr=0x2,id=net7,rx_queue_size=1024,tx_queue_size=256 -machine type=q35+pve0 -drive if=pflash,format=raw,readonly=on,file=/var/lib/vz/images/148/OVMF_CODE.sw.fd -drive if=pflash,format=raw,file=/var/lib/vz/images/148/OVMF_VARS.sw.fd

If I manually remove the default -drive args that Proxmox forces into the startup and replace them with the args I want, the VM starts.

 

[t.lamprecht]

Is there a way to do what you describe that actually works? SonicWall, and other vendors, are now providing their own OMVF files. This obviously works fine under KVM/VirtManager because the files can be specified. Under Proxmox it's not so joyous...

I'm currently a bit swamped for looking into this closely here on the community forum, but having deeper integration for vendor-specific OVMF files can make sense. So if you do not mind, you can open an enhancement request for deviating from the standard paths over at our Bugzilla instance (link to this thread) to better keep track of this.

How it exactly will be implemented needs some thoughts though, in the long term we ideally have such files backed by the Proxmox storage plugin system, as then they backup and migration comes mostly for free.
That might be a bit more work though, as such an intermediate stop-gap might be nice, e.g. where one can just set a sub-property of the VM config's bios property which then acts as override to where PVE loads the OVMF/EFI file from. Which can be also nice to have in general for quickly testing some OVMF/EFI related things.

 

[FormerVMW]

Done: https://bugzilla.proxmox.com/show_bug.cgi?id=5898