Hot adding vCPU's bug in Proxmox/Debian Jessie?

[wosp]

When hot adding vCPU's to a VM running Debian Jessie in Proxmox 4.1-22 the console of the VM gives the following error one time for each vCPU that is added:

platform microcode: firmware: failed to load intel-ucode/0f-06-01 (-2)

The VM gives the following error logs:

/var/log/syslog:

Apr 10 15:50:14 jessie-vm kernel: [ 6469.941989] tsc: Marking TSC unstable due to check_tsc_sync_source failed
Apr 10 15:50:14 jessie-vm kernel: [ 6469.942666] platform microcode: firmware: failed to load intel-ucode/0f-06-01 (-2)
Apr 10 15:50:14 jessie-vm kernel: [ 6469.960685] platform microcode: firmware: failed to load intel-ucode/0f-06-01 (-2)
Apr 10 15:50:14 jessie-vm kernel: [ 6469.978737] platform microcode: firmware: failed to load intel-ucode/0f-06-01 (-2)
Apr 10 15:50:14 jessie-vm kernel: [ 6469.997528] platform microcode: firmware: failed to load intel-ucode/0f-06-01 (-2)
Apr 10 15:50:14 jessie-vm kernel: [ 6470.018695] platform microcode: firmware: failed to load intel-ucode/0f-06-01 (-2)
Apr 10 15:50:14 jessie-vm kernel: [ 6470.046282] platform microcode: firmware: failed to load intel-ucode/0f-06-01 (-2)

/var/log/messages:

Apr 10 15:50:14 jessie-vm kernel: [ 6469.648335] CPU4 has been hot-added
Apr 10 15:50:14 jessie-vm kernel: [ 6469.653117] smpboot: Booting Node 0 Processor 4 APIC 0x4
Apr 10 15:50:14 jessie-vm kernel: [ 0.008000] kvm-clock: cpu 4, msr 0:3ffd3101, secondary cpu clock
Apr 10 15:50:14 jessie-vm kernel: [ 6469.941971] TSC synchronization [CPU#1 -> CPU#4]:
Apr 10 15:50:14 jessie-vm kernel: [ 6469.941975] Measured 17237596227296 cycles TSC warp between CPUs, turning off TSC clock.
Apr 10 15:50:14 jessie-vm kernel: [ 6469.941989] tsc: Marking TSC unstable due to check_tsc_sync_source failed
Apr 10 15:50:14 jessie-vm kernel: [ 6469.942471] KVM setup async PF for cpu 4
Apr 10 15:50:14 jessie-vm kernel: [ 6469.942481] kvm-stealtime: cpu 4, msr 3fc8d080
Apr 10 15:50:14 jessie-vm kernel: [ 6469.942597] microcode: CPU4 sig=0xf61, pf=0x1, revision=0x1
Apr 10 15:50:14 jessie-vm kernel: [ 6469.943185] Will online and init hotplugged CPU: 4
Apr 10 15:50:14 jessie-vm kernel: [ 6469.945217] CPU5 has been hot-added
Apr 10 15:50:14 jessie-vm kernel: [ 6469.945508] CPU6 has been hot-added
Apr 10 15:50:14 jessie-vm kernel: [ 6469.945777] CPU7 has been hot-added
Apr 10 15:50:14 jessie-vm kernel: [ 6469.946054] CPU8 has been hot-added
Apr 10 15:50:14 jessie-vm kernel: [ 6469.946319] CPU9 has been hot-added
Apr 10 15:50:14 jessie-vm kernel: [ 6469.948693] smpboot: Booting Node 0 Processor 5 APIC 0x5
Apr 10 15:50:14 jessie-vm kernel: [ 0.008000] kvm-clock: cpu 5, msr 0:3ffd3141, secondary cpu clock
Apr 10 15:50:14 jessie-vm kernel: [ 6469.960557] KVM setup async PF for cpu 5
Apr 10 15:50:14 jessie-vm kernel: [ 6469.960568] kvm-stealtime: cpu 5, msr 3fcad080
Apr 10 15:50:14 jessie-vm kernel: [ 6469.960657] microcode: CPU5 sig=0xf61, pf=0x1, revision=0x1
Apr 10 15:50:14 jessie-vm kernel: [ 6469.961145] Will online and init hotplugged CPU: 5
Apr 10 15:50:14 jessie-vm kernel: [ 6469.966584] smpboot: Booting Node 0 Processor 8 APIC 0x8
Apr 10 15:50:14 jessie-vm kernel: [ 0.008000] kvm-clock: cpu 8, msr 0:3ffd3201, secondary cpu clock
Apr 10 15:50:14 jessie-vm kernel: [ 6469.978581] KVM setup async PF for cpu 8
Apr 10 15:50:14 jessie-vm kernel: [ 6469.978592] kvm-stealtime: cpu 8, msr 3fd0d080
Apr 10 15:50:14 jessie-vm kernel: [ 6469.978720] microcode: CPU8 sig=0xf61, pf=0x1, revision=0x1
Apr 10 15:50:14 jessie-vm kernel: [ 6469.979200] Will online and init hotplugged CPU: 8
Apr 10 15:50:14 jessie-vm kernel: [ 6469.985464] smpboot: Booting Node 0 Processor 9 APIC 0x9
Apr 10 15:50:14 jessie-vm kernel: [ 0.008000] kvm-clock: cpu 9, msr 0:3ffd3241, secondary cpu clock
Apr 10 15:50:14 jessie-vm kernel: [ 6469.997410] KVM setup async PF for cpu 9
Apr 10 15:50:14 jessie-vm kernel: [ 6469.997422] kvm-stealtime: cpu 9, msr 3fd2d080
Apr 10 15:50:14 jessie-vm kernel: [ 6469.997505] microcode: CPU9 sig=0xf61, pf=0x1, revision=0x1
Apr 10 15:50:14 jessie-vm kernel: [ 6469.998015] Will online and init hotplugged CPU: 9
Apr 10 15:50:14 jessie-vm kernel: [ 6470.006056] smpboot: Booting Node 0 Processor 6 APIC 0x6
Apr 10 15:50:14 jessie-vm kernel: [ 0.008000] kvm-clock: cpu 6, msr 0:3ffd3181, secondary cpu clock
Apr 10 15:50:14 jessie-vm kernel: [ 6470.018587] KVM setup async PF for cpu 6
Apr 10 15:50:14 jessie-vm kernel: [ 6470.018598] kvm-stealtime: cpu 6, msr 3fccd080
Apr 10 15:50:14 jessie-vm kernel: [ 6470.018678] microcode: CPU6 sig=0xf61, pf=0x1, revision=0x1
Apr 10 15:50:14 jessie-vm kernel: [ 6470.019277] Will online and init hotplugged CPU: 6
Apr 10 15:50:14 jessie-vm kernel: [ 6470.034169] smpboot: Booting Node 0 Processor 7 APIC 0x7
Apr 10 15:50:14 jessie-vm kernel: [ 0.008000] kvm-clock: cpu 7, msr 0:3ffd31c1, secondary cpu clock
Apr 10 15:50:14 jessie-vm kernel: [ 6470.046147] KVM setup async PF for cpu 7
Apr 10 15:50:14 jessie-vm kernel: [ 6470.046176] kvm-stealtime: cpu 7, msr 3fced080
Apr 10 15:50:14 jessie-vm kernel: [ 6470.046265] microcode: CPU7 sig=0xf61, pf=0x1, revision=0x1
Apr 10 15:50:14 jessie-vm kernel: [ 6470.046716] Will online and init hotplugged CPU: 7

However, the extra vCPU's are added and working. Google didn't help me on this. I tried to install intel-microcode en uicode-tool (from contrib and non-free repo's) in the VM, but didn't make a difference. Any ideas? Debian Wheezy doesn't give the "platform microcode" errors, but also have the "TSC unstable" error in logfiles when hot adding vCPU's.

 

[t.lamprecht]

Hi,

have you added a respective udev rule to allow hot plugging?

See https://pve.proxmox.com/wiki/Hotplug_(qemu_disk,nic,cpu,memory)#cpu_.26.26_memory_hotplug if not.

 

[wosp]

Yes, I did. Otherwise the hotplugging would not work at all.
Any other ideas?

 

[t.lamprecht]

Can you post the config from the respective VM?

 

[wosp]

Sure, here it is:

boot: dc
bootdisk: virtio0
cores: 12
hotplug: disk,network,usb,memory,cpu
ide2: none,media=cdrom
memory: 10240
name: jessie-vm
net0: virtio=3A:3D:20:B9:E2:D7,bridge=vmbr0100
numa: 1
onboot: 1
ostype: l26
protection: 1
scsihw: virtio-scsi-pci
smbios1: uuid=23d37b05-10dc-41b7-b7b1-0ab2fbbc94be
sockets: 1
vcpus: 4
virtio0: Ceph:vm-100-disk-1,size=100G

 

[t.lamprecht]

Can you try to change CPU type to "host"?

 

[wosp]

Almost the same result, but error message is a bit different (intel-ucode/06-2c-02 instead of intel-ucode/0f-06-01):

Apr 12 17:13:08 jessie-vm kernel: [ 64.194384] CPU4 has been hot-added
Apr 12 17:13:08 jessie-vm kernel: [ 64.200509] CPU5 has been hot-added
Apr 12 17:13:08 jessie-vm kernel: [ 64.203526] smpboot: Booting Node 0 Processor 4 APIC 0x4
Apr 12 17:13:08 jessie-vm kernel: [ 0.008000] kvm-clock: cpu 4, msr 0:3ffd3101, secondary cpu clock
Apr 12 17:13:08 jessie-vm kernel: [ 64.217697] TSC synchronization [CPU#2 -> CPU#4]:
Apr 12 17:13:08 jessie-vm kernel: [ 64.217697] Measured 200033228006 cycles TSC warp between CPUs, turning off TSC clock.
Apr 12 17:13:08 jessie-vm kernel: [ 64.217697] tsc: Marking TSC unstable due to check_tsc_sync_source failed
Apr 12 17:13:08 jessie-vm kernel: [ 64.221021] KVM setup async PF for cpu 4
Apr 12 17:13:08 jessie-vm kernel: [ 64.221032] kvm-stealtime: cpu 4, msr 3fc8d080
Apr 12 17:13:08 jessie-vm kernel: [ 64.221180] microcode: CPU4 sig=0x206c2, pf=0x1, revision=0x1
Apr 12 17:13:08 jessie-vm kernel: [ 64.233268] platform microcode: firmware: failed to load intel-ucode/06-2c-02 (-2)
Apr 12 17:13:08 jessie-vm kernel: [ 64.233875] Will online and init hotplugged CPU: 4
Apr 12 17:13:08 jessie-vm kernel: [ 64.237328] smpboot: Booting Node 0 Processor 5 APIC 0x5
Apr 12 17:13:08 jessie-vm kernel: [ 0.008000] kvm-clock: cpu 5, msr 0:3ffd3141, secondary cpu clock
Apr 12 17:13:08 jessie-vm kernel: [ 64.249209] KVM setup async PF for cpu 5
Apr 12 17:13:08 jessie-vm kernel: [ 64.249219] kvm-stealtime: cpu 5, msr 3fcad080
Apr 12 17:13:08 jessie-vm kernel: [ 64.249311] microcode: CPU5 sig=0x206c2, pf=0x1, revision=0x1
Apr 12 17:13:08 jessie-vm kernel: [ 64.249334] platform microcode: firmware: failed to load intel-ucode/06-2c-02 (-2)
Apr 12 17:13:08 jessie-vm kernel: [ 64.249825] Will online and init hotplugged CPU: 5

 

[t.lamprecht]

The host microcode is up to date?
It could (could, I'm fishing a bit in the dark here, sorry...) be a BIOS issue. Whats your underlying hardware, motherboard and CPU?

And if its working but you get those message then its not the best result to expect but better than not working at all, I'd guess

 

[wosp]

With host microcode you mean just the intel-microcode and iucode-tool in the VM?
Underlying hardware is a Dell PowerEdge R610 with dual Xeon X5650 CPU's. BIOS version running is 6.4.0, which is the latest.

And if its working but you get those message then its not the best result to expect but better than not working at all, I'd guess

Absolutely true, but offcourse I don't like errors like these in my logs so would be nice if it can be solved. ;-)

 

[t.lamprecht]

With host microcode you mean just the intel-microcode and iucode-tool in the VM?

Yeah sort of, I mean those but not in the VM but rather the host.

Absolutely true, but offcourse I don't like errors like these in my logs so would be nice if it can be solved. ;-)

I can relate to you, but have no other solution for now only fishing in the dark and that is normally just demotivating.

Does QEMU/KVM on the host says/logs anything, I guess not but maybe we have luck?

 

[wosp]

Nothing in logfiles on the Proxmox VE node, only this entry in kern.log, messages and syslog:

Apr 13 10:51:41 node1 pvedaemon[4550]: <wosp@pve> update VM 100: -vcpus 9 -delete cpulimit,cpuunits

On the physical server intel-microcode wasn't installed, this is a basic Proxmox VE installation, but even installing this on the physical server don't change a thing and I think I know why. The code is a referral to the CPU used. For example intel-ucode/06-2c-02 is for the Intel(R) Xeon(R) CPU X5650 @ 2.67GHz (fam: 06, model: 2c, stepping: 02). Apparently 0f-06-01 is for kvm64.

According to https://packages.debian.org/nl/jessie/amd64/intel-microcode/filelist 06-2c-02 and 0f-06-01 are both not in the current intel-microcode package. Tested also with qemu64 (06-06-03), but that one is also not in the package, so no luck for me.