Proxmox 5.4/6 Threadripper not working with EPYC/host profile

[Admiral]

Hi guys ,

The following issue (It works on Fedora 30 with KVM though) :
On both Proxmox 5.4 and 6 i can't get VM's to boot properly with the EPYC or host-cpuprofile
With KVM64 or Opteron G3 it works
My setup :

# pveversion --verbose proxmox-ve: 6.0-1 (running kernel: 5.0.15-1-pve) pve-manager: 6.0-1 (running version: 6.0-1/c148050a) pve-kernel-5.0: 6.0-2 pve-kernel-4.15: 5.3-3 pve-kernel-5.0.15-1-pve: 5.0.15-1 pve-kernel-4.15.18-12-pve: 4.15.18-35 ceph-fuse: 12.2.11+dfsg1-2.1 corosync: 3.0.2-pve2 criu: 3.11-3 glusterfs-client: 5.5-3 ksm-control-daemon: 1.3-1 libjs-extjs: 6.0.1-10 libknet1: 1.10-pve1 libpve-access-control: 6.0-2 libpve-apiclient-perl: 3.0-1 libpve-common-perl: 6.0-2 libpve-guest-common-perl: 3.0-1 libpve-http-server-perl: 3.0-1 libpve-storage-perl: 6.0-4 libqb0: 1.0.5-1 lvm2: 2.03.02-pve1 lxc-pve: 3.1.0-60 lxcfs: 3.0.3-pve60 novnc-pve: 1.0.0-60 proxmox-mini-journalreader: 1.1-1 proxmox-widget-toolkit: 2.0-4 pve-cluster: 6.0-4 pve-container: 3.0-2 pve-docs: 6.0-2 pve-edk2-firmware: 2.20190614-1 pve-firewall: 4.0-3 pve-firmware: 3.0-2 pve-ha-manager: 3.0-1 pve-i18n: 2.0-1 pve-qemu-kvm: 4.0.0-2 pve-xtermjs: 3.13.2-1 qemu-server: 6.0-4 smartmontools: 7.0-pve2 spiceterm: 3.1-1 vncterm: 1.6-1 zfsutils-linux: 0.8.1-pve1

# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 43 bits physical, 48 bits virtual CPU(s): 64 On-line CPU(s) list: 0-63 Thread(s) per core: 2 Core(s) per socket: 32 Socket(s): 1 NUMA node(s): 4 Vendor ID: AuthenticAMD CPU family: 23 Model: 8 Model name: AMD Ryzen Threadripper 2990WX 32-Core Processor Stepping: 2 CPU MHz: 1709.899 CPU max MHz: 3000.0000 CPU min MHz: 2200.0000 BogoMIPS: 5999.37 Virtualization: AMD-V

Conf of 1 of the VM's :
bootdisk: virtio0
cores: 2
cpu: host
ide2: local:iso/ubuntu-18.04.2-desktop-amd64.iso,media=cdrom
machine: q35
memory: 4096
name: test
net0: virtio=FE:01:9E:0A:33:4E,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=252d608c-f8ea-4ba4-914b-2fd26f2ce553
sockets: 1
virtio0: local-lvm:vm-101-disk-0,size=32G
vmgenid: f8a84b36-c90c-4931-ae39-397b7f468ef0

 

[t.lamprecht]

On both Proxmox 5.4 and 6 i can't get VM's to boot properly with the EPYC or host-cpuprofile

Can you add a bit more details regarding what you mean or experience with "boot poorly"?

It would be also great if you could use "code" tags for command output, much easier to read with that, e.g., use:
[code]command and it's output go here[/code]

We have some Ryzen, Threadripper and Epyc platforms here with no bigger issues known, at least besides some with MSR register on older kernels and windows VMs (which current 5.4 or 6.0 Beta do not have anymore)..
Maybe also post the VM config of one of the problematic VMs.

 

[Admiral]

Reformatted the code , i mean with booting poorly : They either hang or crash

 

[t.lamprecht]

Reformatted the code , i mean with booting poorly : They either hang or crash

Better, thanks! What's happening in the syslog/journal or dmesg during a problematic start? Any error messages?

 

[Admiral]

This hangs with the EPYC/host profile (it just hangs forever on the USB, while the generic kvm64 works)

 

[t.lamprecht]

And on the Proxmox VE host itself, is there anything in the syslog (you should be able to check it out through the WebUI: Datacenter -> pve -> System -> Syslog)?

 

[Admiral]

After switching it from opteron G3 to EPYC (So i could make the screenshot)


Jul 08 19:14:30 pve pvedaemon[1721]: <root@pam> update VM 100: -cores 1 -cpu EPYC -delete vcpus,cpulimit,cpuunits -numa 0 -sockets 1
Jul 08 19:14:30 pve pvedaemon[1721]: cannot delete 'vcpus' - not set in current configuration!
Jul 08 19:14:30 pve pvedaemon[1721]: cannot delete 'cpulimit' - not set in current configuration!
Jul 08 19:14:30 pve pvedaemon[1721]: cannot delete 'cpuunits' - not set in current configuration!

 

[t.lamprecht]

That's just a warning, and is not problematic..

On a VM start something like the following example - from a VM with id 101 on my host, should be logged in the hosts syslog:

Jul 02 09:02:03 nina pve-guests[5665]: <root@pam> starting task UPID:nina:00001622:00002755:5D1B016B:qmstart:101:root@pam:
Jul 02 09:02:03 nina pve-guests[5666]: start VM 101: UPID:nina:00001622:00002755:5D1B016B:qmstart:101:root@pam:
Jul 02 09:02:03 nina systemd[1]: Created slice qemu.slice.
Jul 02 09:02:03 nina systemd[1]: Started 101.scope.
Jul 02 09:02:04 nina kernel: device tap101i0 entered promiscuous mode
Jul 02 09:02:04 nina kernel: vmbr0: port 2(tap101i0) entered blocking state
Jul 02 09:02:04 nina kernel: vmbr0: port 2(tap101i0) entered disabled state
Jul 02 09:02:04 nina kernel: device eno1 entered promiscuous mode
...

Often, if the VM crashes or fails to boot there are some errors logged during that startup time period, could you try to look at that and see if that is the case with your setup?
Also a VM configuration would be still nice, you could get one via the CLI/Terminal with:

qm config VMID

 

[Admiral]

qm config 100
bootdisk: virtio0
cores: 1
cpu: EPYC
hotplug: disk,network
memory: 8192
name: BSD
net0: virtio=FE:87:95:5E:29:09,bridge=vmbr0,firewall=1
numa: 0
ostype: other
sata0: local:iso/FreeBSD-12.0-RELEASE-amd64-disc1.iso,media=cdrom,size=871550K
scsihw: virtio-scsi-single
smbios1: uuid=ec0f81e7-6db8-4e06-b7f1-2c11912e74de
sockets: 1
tablet: 0
virtio0: local-lvm:vm-100-disk-0,size=32G
vmgenid: e9b5573b-8ee1-48fc-bbd5-810b1626f5c8

No errors in the logs of the host

 

[t.lamprecht]

sata0: local:iso/FreeBSD-12.0-RELEASE-amd64-disc1.iso,media=cdrom,size=871550K

Hmm, that's freeBSD, which is known to make some problems on certain HW, the threadripper CPU topology may just not be fully supported there, at least in a virtual KVM context.
Can you reproduce such issues by booting a common Linux Distro (like, for example, Debian)?

Else, I'd just recommend to keep using a bit filtered CPU type for BSD variants.

 

[Admiral]

Well , this FreeBSD worked flawless with the EPYC cputype on Fedora 30 though
And i see the same weird behaviour with Debian , Ubuntu and even Arch

 

[t.lamprecht]

Interesting. Could you by any chance provide the QEMU/KVM command line used in fedora? If you put it together manually then just paste that, if you configured it with virt-manager, or similar, you could see if the can produce a output like our "qm showcmd VMID -p" can, what always should work is

pgrep -fa 'kvm|qemu'

would be interesting to see what they specifically do different in your case, maybe also just a different QEMU version?

 

[Admiral]

That is impossible unfortunately , the Fedora box got reinstalled with proxmox ...

 

[t.lamprecht]

Would you be willing to use Fedora in a Proxmox VE backed VM to try it out? You could temporarily enable nested virtualization for this experiment to be a bit faster.

 

[lhorace]

Just FYI, had similar issues with Freebsd and AMD processors and a fix did land in the most RECENT Freebsd kernel (11.2 I think?) to address my problems. Though, this could be a new and separate issue than what I had. Might want to have a look at FreeBSB Bugzilla for reports.

 

[lhorace]

See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213155

 

[Admiral]

Just FYI, had similar issues with Freebsd and AMD processors and a fix did land in the most RECENT Freebsd kernel (11.2 I think?) to address my problems. Though, this could be a new and separate issue than what I had. Might want to have a look at FreeBSB Bugzilla for reports.

I was running 12..

 

[t.lamprecht]

I was running 12..

Does not necessarily means that the kernel bug was fixed there too, if it was just a stable fix for the 11.2 kernel. At least a colleague here reported that he cannot reproduce this of a 11.2 based (freenas/pfsense (not to sure anymore)) but on a 12 based plain installation.

 

[Admiral]

Does not necessarily means that the kernel bug was fixed there too, if it was just a stable fix for the 11.2 kernel. At least a colleague here reported that he cannot reproduce this of a 11.2 based (freenas/pfsense (not to sure anymore)) but on a 12 based plain installation.

Fair enough , but 12 did work on the Fedora , i am reproducing it in a vm with Fedora now
Results will we posted later on

 

[Admiral]

Update :
FreeBSD 11 does work , FreeBSD 12 hangs with the EPYC or host-cpuprofile
Fedora 30 works
Running Fedora 30 nested : FreeBSD 12 works as well
Qemuversion on that : 3.1.0-9.fc30

pgrep -fa 'kvm|qemu'
2686 /usr/bin/qemu-system-x86_64 -name guest=freebsd12.0,debug-threads=on -S -ob                                                                                                                                                             ject secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-3-freebsd                                                                                                                                                             12.0/master-key.aes -machine pc-i440fx-3.1,accel=kvm,usb=off,vmport=off,dump-gue                                                                                                                                                             st-core=off -cpu EPYC-IBPB,x2apic=on,tsc-deadline=on,hypervisor=on,tsc_adjust=on                                                                                                                                                             ,cmp_legacy=on,perfctr_core=on,virt-ssbd=on,monitor=off -m 2048 -realtime mlock=                                                                                                                                                             off -smp 2,sockets=2,cores=1,threads=1 -uuid b89a9a9c-3aab-4936-8e0e-e41705ff938                                                                                                                                                             5 -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=30,server,nowait                                                                                                                                                              -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -g                                                                                                                                                             lobal kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disa                                                                                                                                                             ble_s3=1 -global PIIX4_PM.disable_s4=1 -boot menu=on,strict=on -device ich9-usb-                                                                                                                                                             ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,first                                                                                                                                                             port=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.                                                                                                                                                             0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firs                                                                                                                                                             tport=4,bus=pci.0,addr=0x5.0x2 -device ahci,id=sata0,bus=pci.0,addr=0x6 -drive f                                                                                                                                                             ile=/var/lib/libvirt/images/freebsd12.0.qcow2,format=qcow2,if=none,id=drive-virt                                                                                                                                                             io-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-d                                                                                                                                                             isk0,id=virtio-disk0,bootindex=2 -drive file=/home/blaat/Downloads/FreeBSD-12.0-                                                                                                                                                             RELEASE-amd64-disc1.iso,format=raw,if=none,id=drive-sata0-0-0,readonly=on -devic                                                                                                                                                             e ide-cd,bus=sata0.0,drive=drive-sata0-0-0,id=sata0-0-0,bootindex=1 -netdev tap,                                                                                                                                                             fd=32,id=hostnet0,vhost=on,vhostfd=33 -device virtio-net-pci,netdev=hostnet0,id=                                                                                                                                                             net0,mac=52:54:00:4b:f7:33,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -devic                                                                                                                                                             e isa-serial,chardev=charserial0,id=serial0 -spice port=5900,addr=127.0.0.1,disa                                                                                                                                                             ble-ticketing,image-compression=off,seamless-migration=on -device qxl-vga,id=vid                                                                                                                                                             eo0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outpu                                                                                                                                                             ts=1,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device h                                                                                                                                                             da-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,na                                                                                                                                                             me=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=1 -cha                                                                                                                                                             rdev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,i                                                                                                                                                             d=redir1,bus=usb.0,port=2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=                                                                                                                                                             0x8 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=                                                                                                                                                             deny -msg timestamp=on
2719 kvm-pit/2686