SEV-ES setup and de-bugging help - CPUs are not resettable

Loci

New Member
Feb 6, 2025
1
0
1
Hello,

The edge-case I bring this fine forum today is that any VM with SEV-ES enabled is failing to start. SEV works perfectly, but the -ES is unhappy.

AMD EPYC 7532 - AsRock Rack ROMED6U-2L2T

I know this is a "very experimental" option but hopefully there's others out there with it working who can help me!

With:
YAML:
acpi: 1
agent: 0
args: -object sev-guest,id=sev0,cbitpos=47,reduced-phys-bits=1,policy=0x5 -machine memory-encryption=sev0,confidential-guest-support=sev0,smm=off,kernel-irqchip=on
balloon: 0
bios: ovmf
boot: order=scsi0
cores: 4
cpu: EPYC
efidisk0: local-lvm:vm-101-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
ide2: none,media=cdrom
kvm: 1
machine: q35,viommu=virtio
memory: 8096
meta: creation-qemu=9.0.2,ctime=1737707576
name: SEVES
net0: virtio=BC:24:11:FA:99:E8,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
protection: 1
rng0: source=/dev/urandom
scsi0: local-lvm:vm-101-disk-1,iothread=1,size=32G
scsihw: virtio-scsi-single
smbios1: uuid=74591c21-6dd6-4644-a8a8-32c866270f1a
sockets: 1
vmgenid: cb8fe509-072d-42fc-9a17-3e71b95114c5
watchdog: model=i6300esb,action=pause

I receive a:
Code:
proxmox QEMU[66906]: kvm: cpus are not resettable, terminating

It's implied that QEMU should recognise that SEV-ES is enabled by reading the policy value (0x5) and thus not allow CPU resetting, but this doesn't seem to be the case here.

I've tried putting many different things in the -machine setting, anything past the -memory-encryption has been tried with and without.

Is there anything else I can do to get more verbose debug logs? Or is there an obvious setting I have in here which isn't compatible?

Here's the PVE host SEV related dmesg:
Code:
[    0.199165] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.8.12-8-pve root=/dev/mapper/pve-root ro quiet iommu=pt mem_encrypt=on kvm_amd.sev=1 kvm_amd.sev_es=1 swiotlb=262144 vfio_pci.ids=1002:66af
[    5.616670] ccp 0000:47:00.1: sev enabled
[    5.660976] ccp 0000:47:00.1: SEV API:0.24 build:18
[    5.793531] kvm_amd: SEV enabled (ASIDs 3 - 253)
[    5.793534] kvm_amd: SEV-ES enabled (ASIDs 1 - 2)

I don't know what I'm doing. :)

Many thanks,
Loci
 
Hello,

it is most likely related to the edk2 firmware.
You cannot boot SEV-ES enabled machines when the edk2 firmware was compiled with the flags SMM_REQUIRE and SECURE_BOOT_ENABLE:
https://github.com/tianocore/edk2/pull/6285

There is a patch series for amd sev-snp that fixes this issue:
https://lore.proxmox.com/pve-devel/.../T/#m3303cd43eab4ce3b614159b98b88449ff04f352c