Support for SEV-SNP in guest VMs

P

Pegah

New Member
Dec 18, 2024
1
0
1
Hi,

As far as I can see only SEV and SEV-ES are supported by proxmox, is there anyway (non-officail) to use AMD sev-snp in guest VMs now? what is the roadmap to add support for that?


Regards,
Pegah
 
Hi,

Yes, we plan to support AMD SEV-SNP when we get the required hardware.

The "unofficial" way would be to get the qemu command with qm showcmd <vmid> --pretty and modify it to enable SEV-SNP:
https://www.qemu.org/docs/master/system/i386/amd-memory-encryption.html#launching-sev-snp

This file contains the values for cbitpos and reduced-phys-bits for your CPU:
cat /run/qemu-server/host-hw-capabilities.json

If you can get a VM to run with SEV-SNP enabled, we would be very grateful if you could describe how you did it and what the limitations are.
 
Last edited:
Hi @MarkusF,

adding the following to the QEMU command

Bash: 
  -machine memory-encryption=sev0,vmport=off \
  -object sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=5,policy=0x30000 \
  -D ./log.txt \
  -bios '/root/OVMF.fd'

and changing the CPU from qemu64 to EPYC-v4 allows us to boot an SNP CVM. Note that this requires QEMU v9.2; other than that, so far we did not see any limitations specific to SNP.

How can we help prioritise this so that SNP gets mainline support in Proxmox sooner than later?
 
Last edited:
Hello,

good to hear that it does work so well.
We are planning to get SEV-SNP supported hardware soon and I will be happy to work on support for SEV-SNP on PVE as soon as I can.

All you can do to help at the moment is to collect information about requirements (kernel command line arguments, supported CPUs, etc.), limitations (for example: live-migration, snapshots), helpful links/resources and other things that would make development and documentation easier for us.

You can also get involved in the development process if you like.
More information here:
https://pve.proxmox.com/wiki/Developer_Documentation
 
nico_ said:
Hi @MarkusF,

adding the following to the QEMU command

Bash: 
  -machine memory-encryption=sev0,vmport=off \
  -object sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=5,policy=0x30000 \
  -D ./log.txt \
  -bios '/root/OVMF.fd'

and changing the CPU from qemu64 to EPYC-v4 allows us to boot an SNP CVM. Note that this requires QEMU v9.2; other than that, so far we did not see any limitations specific to SNP.

How can we help prioritise this so that SNP gets mainline support in Proxmox sooner than later?
Click to expand...
Hi @nico_ ,
how did you upgrade the qemu server to version 9.2 on proxmox?
I've installed the latest version of proxmox (8.3.0) but the qemu server is at the 8.3.3.
 
piers said:
Hi @nico_ ,
how did you upgrade the qemu server to version 9.2 on proxmox?
I've installed the latest version of proxmox (8.3.0) but the qemu server is at the 8.3.3.
Click to expand...
qemu-server is not the same as qemu.

qemu-system-x86_64 --version or apt info pve-qemu-kvm will show the current qemu version.

SEV-SNP support should also work with QEMU-9.1.
 
MarkusF said:
qemu-server is not the same as qemu.

qemu-system-x86_64 --version or apt info pve-qemu-kvm will show the current qemu version.

SEV-SNP support should also work with QEMU-9.1.
Click to expand...
Thanks Markus for the quick response and sorry for the confusion, the pve-qemu-kvm: 9.0.2-4 is the one provided in the current proxmox.
I am checking the repository (https://github.com/proxmox/pve-qemu) and seems that the QEMU 9.1 is being added last month. Is there a procedure to update this module on proxmox?
 
MarkusF said:
Edit: Either you wait until it is in your repository or you try to build it yourself (do at your own risk):
https://pve.proxmox.com/wiki/Developer_Documentation
Click to expand...
Hi Markus, I've followed the provided guide to build the pve-common successfully.
Then I tried to build the pve-qemu 9.1 from the official repo https://git.proxmox.com/git/pve-qemu.git and faced several issues:

  • First I solved the edk2 subhook submodule missing referred by this issue https://gitlab.com/qemu-project/qemu/-/issues/2660 substituting the repo with the mirrored one https://github.com/tianocore/edk2-subhook
  • The make dinstall return this error when I try to run it inside the pve-qemu folder:
    Bash: 
    ERROR: missing subprojects
--disable-download specified but subprojects were not
checked out.  Please invoke "meson subprojects download"
before configuring QEMU, or remove --disable-download
from the command line.

make[1]: *** [debian/rules:34: build/config.status] Error 1
make[1]: Leaving directory '/root/proxmox/pve-qemu/pve-qemu-kvm-9.1.2'
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
make: *** [Makefile:61: pve-qemu-kvm_9.1.2-1_amd64.deb] Error 2
    I've also tried to run meson subprojects download without resolving the error.
  • Here the error inside /root/proxmox/pve-qemu/pve-qemu-kvm-9.1.2/build/config.log:
    Code: 
    # QEMU configure log Fri Jan 17 05:16:53 PM CET 2025# Configured with: '/root/proxmox/pve-qemu/pve-qemu-kvm-9.1.2/configure' '--disable-download' '--docdir=/usr/share/doc/pve-qemu-kvm' '--localstatedir=/var' '--prefix=/usr' '--sysconfdir=/etc' '--target-list=x86_64-softmmu,aarch64-softmmu' '--with-suffix=kvm' '--with-pkgversion=pve-qemu-kvm_9.1.2-1' '--audio-drv-list=alsa' '--datadir=/usr/share' '--libexecdir=/usr/lib/kvm' '--disable-capstone' '--disable-gtk' '--disable-guest-agent' '--disable-guest-agent-msi' '--disable-libnfs' '--disable-libssh' '--disable-sdl' '--disable-smartcard' '--disable-strip' '--disable-xen' '--enable-curl' '--enable-docs' '--enable-glusterfs' '--enable-gnutls' '--enable-libiscsi' '--enable-libusb' '--enable-linux-aio' '--enable-linux-io-uring' '--enable-numa' '--enable-opengl' '--enable-rbd' '--enable-seccomp' '--enable-slirp' '--enable-spice' '--enable-usb-redir' '--enable-virglrenderer' '--enable-virtfs' '--enable-zstd'

#
cc -g -O2 -ffile-prefix-map=/root/proxmox/pve-qemu/pve-qemu-kvm-9.1.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wall -c -o config-temp/qemu-conf.o config-temp/qemu-conf.c
cc -g -O2 -ffile-prefix-map=/root/proxmox/pve-qemu/pve-qemu-kvm-9.1.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wall -c -o config-temp/qemu-conf.o config-temp/qemu-conf.c
config-temp/qemu-conf.c:2:2: error: #error __i386__ not defined
    2 | #error __i386__ not defined
      |  ^~~~~
cc -g -O2 -ffile-prefix-map=/root/proxmox/pve-qemu/pve-qemu-kvm-9.1.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wall -c -o config-temp/qemu-conf.o config-temp/qemu-conf.c
cc -g -O2 -ffile-prefix-map=/root/proxmox/pve-qemu/pve-qemu-kvm-9.1.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wall -c -o config-temp/qemu-conf.o config-temp/qemu-conf.c
config-temp/qemu-conf.c:2:2: error: #error __ILP32__ not defined
    2 | #error __ILP32__ not defined
      |  ^~~~~
Do you have any idea on how to go forward on that?
Thanks in advance.
 
I think it might work if you remove the --disable-download line in debian/rules and than run make clean deb
 
Last edited:
Hi,

we haven't used QEMU 9.2 but rather the latest checkout of pve-qemu (9.1.2-1).

@piers Running `meson subprojects download` from pve-qemu/qemu fixed the issue for me. Have you ran `git submodule sync` after updating the subhook reference?
 
pipo said:
Hi,

we haven't used QEMU 9.2 but rather the latest checkout of pve-qemu (9.1.2-1).

@piers Running `meson subprojects download` from pve-qemu/qemu fixed the issue for me. Have you ran `git submodule sync` after updating the subhook reference?
Click to expand...
Thank you @pipo! I was running the meson command in the wrong folder... I was able to install the pve-qemu 9.1 version.


I tried to add the same options as @_nico suggested, but I'm failing to start the VM:

Bash: 
root@:/usr/share/pve-edk2-firmware# qm start 100
kvm: pflash with kvm requires KVM readonly memory support
start failed: QEMU exited with code 1
root@:/usr/share/pve-edk2-firmware# qm showcmd 100 --pretty
/usr/bin/kvm \
  -id 100 \
  -name 'test-snp01,debug-threads=on' \
  -no-shutdown \
  -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server=on,wait=off' \
  -mon 'chardev=qmp,mode=control' \
  -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' \
  -mon 'chardev=qmp-event,mode=control' \
  -pidfile /var/run/qemu-server/100.pid \
  -daemonize \
  -smbios 'type=1,uuid=f6aa99df-bbc2-4a19-93e1-5ae418f3f683' \
  -drive 'if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CODE_4M.secboot.fd' \
  -drive 'if=pflash,unit=1,id=drive-efidisk0,format=raw,file=/dev/pve/vm-100-disk-0,size=540672' \
  -global 'ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off' \
  -smp '4,sockets=4,cores=1,maxcpus=4' \
  -nodefaults \
  -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' \
  -vnc 'unix:/var/run/qemu-server/100.vnc,password=on' \
  -cpu 'EPYC-Rome-v4,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,vendor=AuthenticAMD' \
  -m 4096 \
  -object 'iothread,id=iothread-virtioscsi0' \
  -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg \
  -device 'vmgenid,guid=0bee9fce-8d09-4d87-8524-08485cb5db9f' \
  -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' \
  -device 'VGA,id=vga,bus=pcie.0,addr=0x1' \
  -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
  -iscsi 'initiator-name=iqn.1993-08.org.debian:01:d69f83d64b64' \
  -drive 'file=/var/lib/vz/template/iso/ubuntu-22.04.5-live-server-amd64.iso,if=none,id=drive-ide2,media=cdrom,format=raw,aio=io_uring' \
  -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=101' \
  -device 'virtio-scsi-pci,id=virtioscsi0,bus=pci.3,addr=0x1,iothread=iothread-virtioscsi0' \
  -drive 'file=/dev/pve/vm-100-disk-1,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on' \
  -device 'scsi-hd,bus=virtioscsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' \
  -machine 'type=q35+pve0' \
  -machine 'memory-encryption=sev0,vmport=off' \
  -object 'sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=6,policy=0x30000'

Am I missing something?
 
Maybe the two -machine lines are the problem, combine them to one:
Bash: 
-machine 'type=q35+pve0,memory-encryption=sev0,vmport=off' \
 
Last edited:
Hi @piers,

do you have any output that could help with identifying the problem?

Still, a couple of ideas to consider:
  • Are you using a kernel that supports SNP? The kernel shipped with Ubuntu 22.04 (version 5.15) does not include SNP support. SNP support was introduced in the upstream kernel starting from version 6.11. To enable SNP, you would need to either patch the kernel or use a newer Ubuntu release, such as 24.10, which includes the 6.11 kernel with SNP support.
  • I am not sure whether the PVE fork of edk2 has upstream support for SNP. Maybe try using the original edk2 tree instead. While not recommended for production, you could test with the original tree by including OVMF.fd via the -bios option.
  • Have you ran SNP VMs on that machine before? To verify that the machine is correctly configured for SNP, you can use the scripts provided in https://github.com/AMDESE/AMDSEV/tree/snp-latest and also check with ./snphost ok (https://github.com/virtee/snphost).
 
Hi @pipo ,
the problem was related to your second point. I built the OVMF.fd from edk2 and booted the VM adding the -bios flag as you suggested.

In my understanding the OVMF.fd is a bios without secureboot, were you able to run a VM with a bios with secureboot enabled?
 
Hi @MarkusF,

I am looking into adding support for SEV-SNP in Proxmox myself, building upon the existing code for SEV(-ES). While I’ve made some progress, I’ve encountered an issue with OVMF that I’m unsure how to approach.

Currently, Proxmox uses OVMF_CODE.fd and OVMF_VARS.fd with the -drive if=pflash option. However, I haven’t been able to make this setup work with SNP. Reference scripts for SNP utilize the -bios option instead (https://github.com/AMDESE/AMDSEV/issues/218).

Would you happen to have any suggestions on this? Specifically, would it be fine to use the -bios option when working with SNP?

Thanks in advance for your guidance!
 
pipo said:
Hi @MarkusF,

I am looking into adding support for SEV-SNP in Proxmox myself, building upon the existing code for SEV(-ES). While I’ve made some progress, I’ve encountered an issue with OVMF that I’m unsure how to approach.

Currently, Proxmox uses OVMF_CODE.fd and OVMF_VARS.fd with the -drive if=pflash option. However, I haven’t been able to make this setup work with SNP. Reference scripts for SNP utilize the -bios option instead (https://github.com/AMDESE/AMDSEV/issues/218).

Would you happen to have any suggestions on this? Specifically, would it be fine to use the -bios option when working with SNP?

Thanks in advance for your guidance!
Click to expand...
Hi @pipo, I had the same problem, the bios: ovmf configuration in the <vmid>.conf file was the issue. In my case I resolved it by removing it and adding the -bios option as below:
Code: 
boot: order=scsi0;ide2;net0
cores: 1
cpu: EPYC-Rome-v4
efidisk0: local-lvm:vm-103-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
ide2: local:iso/ubuntu-24.04.1-live-server-amd64.iso,media=cdrom,size=2708862K
memory: 4096
meta: creation-qemu=9.1.2,ctime=1737633630
name: test-snp3-nino
net0: virtio=BC:24:11:46:E2:AE,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local-lvm:vm-103-disk-1,iothread=1,size=32G
scsihw: virtio-scsi-single
smbios1: uuid=2e0364bc-d77c-4bfe-9fac-31fe119c3d2c
sockets: 4
vmgenid: 5c70992e-8383-410c-808c-687285d91b0e
args: -bios '/root/OVMF.fd' -machine memory-encryption=sev0,vmport=off -object sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=6,policy=0x30000

I had to build the OVMF.fd like this:
Bash: 
git clone https://github.com/tianocore/edk2.git
cd edk2
git submodule update --init

make -C BaseTools
export EDK_TOOLS_PATH=/root/tianocore-edk2-git/edk2/BaseTools
. edksetup.sh BaseTools

nano Config/target.txt
'''
ACTIVE_PLATFORM       = OvmfPkg/OvmfPkgX64.dsc
TARGET_ARCH           = X64
TOOL_CHAIN_TAG        = GCC5
'''

build

The OVMF_CODE and _VARS seems not supported by qemu with SNP at the moment.
Hope it helps.
 
Hi @pipo,

If you want to contribute be sure you read this:
https://pve.proxmox.com/wiki/Developer_Documentation#Software_License_and_Copyright

I don't think it should be a problem if you replace `-drive if=pflash` with `-bios` when sev-snp is enabled.
You would need to check that this does not break anything else.
I would also add something like `FIXME: remove this when issue https://github.com/AMDESE/AMDSEV/issues/218 is resolved`.

We would also need to update pve-edk2-firmware to a version newer than 202405.
https://github.com/tianocore/edk2/releases/tag/edk2-stable202405
 
Last edited:
Hi @MarkusF,

I am working on patches to support SEV-SNP.

During development, I initially attempted to build OVMF using edk2-stable202308 to run SEV-SNP VMs but was unsuccessful. While, as you pointed out, edk2-stable202405 mentions support for SNP under SVSM, I do not think this is the actual requirement since that is specifically for an environment using SVSM. In any case, I opted to proceed with 202411, which allowed me to run SNP VMs—provided that OVMF was compiled without SECURE_BOOT_ENABLE and SMM_REQUIRE. The same applied when running SEV-ES VMs.

This also aligns with the following document: https://lenovopress.lenovo.com/lp15...crypted-virtualization-encrypted-state-sev-es.
The following features are not supported when SEV-ES is enabled:
  • UEFI Secure Boot
Click to expand...
Given the available OVMF targets in pve-edk2-firmware, what would you recommend? Would it make sense to introduce a new target that builds OVMF without these flags specifically for SEV-ES and SEV-SNP?

Looking forward to your thoughts!
 
You must log in or register to reply here.
Top Bottom