[SOLVED] [Warning] Latest Patch just broke all my WINDOWS vms. (6.3-4) [Patch Inside] (See #53)

t.lamprecht

Proxmox Staff Member
Staff member
Jul 28, 2015
4,550
1,202
164
South Tyrol/Italy
shop.proxmox.com
Same here!
That was a very bad update. Nobody tested it before the release?
Again, use the pve-enterprise repository if you need production grade stability.

We tested it and all our hundreds of different Linux VM workloads run fine here, as did the Windows 10 test instances we have here, that's why we moved the update along.

We're evaluating this to see what specific set of properties there need to be for it to happen, also in this thread people have some VMs where it does not happen, so it's not universal. Do you actually have anything to add for actually helping to narrow it down?
 
  • Like
Reactions: xqlusive

xqlusive

Member
May 28, 2015
3
0
21
Thomas, thanks for looking into it. I am not sure how to find out with which qemu version my windows vm's were installed initially. I would say it was about 5 years when I converted them from bare metal. Happy to give you one vm backup if you need it for testing. Will do more upgrades soon on a cluster with newer vm's and let you know if the issue persists.
 

Whatever

Active Member
Nov 19, 2012
263
13
38
Maybe, we have still a hard time reproducing this, not meaning we don't believe it is not an issue, it just seems we're missing a feature-bit which needs to be toggled for this to happen and are currently trying and evaluating differences (the long time Windows VMs I have for testing were unaffected by this, so something needs to be different)

Did you try the referenced possible workaround by setting the machine type?

Further, do you have a rough idea under what QEMU version those VMs got created and thus Windows installed?
Stating a PVE version and year would already help a lot.

Unfortunately, all the affected VMs were from production environments and had to be fixed ASAP.

Hopefully, I've managed to reproduce this issue on one of our client's test system and here the information I've collected so far:

PVE host:
Code:
root@pve:~# dumpe2fs $(mount | grep 'on \/ ' | awk '{print $1}') | grep 'Filesystem created:'
dumpe2fs 1.44.5 (15-Dec-2018)
Filesystem created:       Mon Jun 18 12:30:29 2018

Windows Server 2016 Standard (with QEMU 5.1)
Code:
C:\Users\Администратор.WIN-REJFV9JJIVT>systeminfo

Имя узла:                         PDC
Название ОС:                      Майкрософт Windows Server 2016 Standard
Версия ОС:                        10.0.14393 Н/Д построение 14393
Изготовитель ОС:                  Microsoft Corporation
Параметры ОС:                     Основной контроллер домена
Построение ОС:                    Multiprocessor Free
Зарегистрированный владелец:      Пользователь Windows
Зарегистрированная организация:
Код продукта:                     00377-60000-00000-AA934
Дата установки:                   19.06.2018, 16:25:31
Время загрузки системы:           12.02.2021, 12:50:06
Изготовитель системы:             QEMU
Модель системы:                   Standard PC (i440FX + PIIX, 1996)
Тип системы:                      x64-based PC
Процессор(ы):                     Число процессоров - 1.
                                  [01]: Intel64 Family 6 Model 58 Stepping 9 GenuineIntel ~3300 МГц
Версия BIOS:                      SeaBIOS rel-1.13.0-48-gd9c812dda519-prebuilt.qemu.org, 01.04.2014
Папка Windows:                    C:\Windows
Системная папка:                  C:\Windows\system32
Устройство загрузки:              \Device\HarddiskVolume1
Язык системы:                     ru;Русский
Язык ввода:                       ru;Русский
Часовой пояс:                     (UTC+03:00) Москва, Санкт-Петербург
Полный объем физической памяти:   2 047 МБ
Доступная физическая память:      553 МБ
Виртуальная память: Макс. размер: 2 431 МБ
Виртуальная память: Доступна:     983 МБ
Виртуальная память: Используется: 1 448 МБ
Расположение файла подкачки:      C:\pagefile.sys
Домен:                            sunlife.local
Сервер входа в сеть:              \\PDC
Исправление(я):                   Число установленных исправлений - 7.
                                  [01]: KB3192137
                                  [02]: KB4049065
                                  [03]: KB4093137
                                  [04]: KB4132216
                                  [05]: KB4485447
                                  [06]: KB4503537
                                  [07]: KB4493470
Сетевые адаптеры:                 Число сетевых адаптеров - 1.
                                  [01]: Red Hat VirtIO Ethernet Adapter
                                        Имя подключения: Ethernet
                                        DHCP включен:    Нет
                                        IP-адрес
                                        [01]: 10.30.0.240
                                        [02]: fe80::8cb8:45f1:52d3:686b
Требования Hyper-V:               Обнаружена низкоуровневая оболочка. Функции, необходимые для Hyper-V, отображены не будут.

Unfortunately system is localized (Russian edition) but hope it's still readable

Windows Server 2016 Standard (after QEMU 5.2 update)
Code:
C:\Users\Администратор.WIN-REJFV9JJIVT>systeminfo


Имя узла:                         PDC
Название ОС:                      Майкрософт Windows Server 2016 Standard
Версия ОС:                        10.0.14393 Н/Д построение 14393
Изготовитель ОС:                  Microsoft Corporation
Параметры ОС:                     Основной контроллер домена
Построение ОС:                    Multiprocessor Free
Зарегистрированный владелец:      Пользователь Windows
Зарегистрированная организация:
Код продукта:                     00377-60000-00000-AA934
Дата установки:                   19.06.2018, 16:25:31
Время загрузки системы:           27.02.2021, 16:46:00
Изготовитель системы:             QEMU
Модель системы:                   Standard PC (i440FX + PIIX, 1996)
Тип системы:                      x64-based PC
Процессор(ы):                     Число процессоров - 1.
                                  [01]: Intel64 Family 6 Model 58 Stepping 9 GenuineIntel ~3300 МГц
Версия BIOS:                      SeaBIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org, 01.04.2014
Папка Windows:                    C:\Windows
Системная папка:                  C:\Windows\system32
Устройство загрузки:              \Device\HarddiskVolume1
Язык системы:                     ru;Русский
Язык ввода:                       ru;Русский
Часовой пояс:                     (UTC+03:00) Москва, Санкт-Петербург
Полный объем физической памяти:   2 047 МБ
Доступная физическая память:      235 МБ
Виртуальная память: Макс. размер: 4 898 МБ
Виртуальная память: Доступна:     896 МБ
Виртуальная память: Используется: 4 002 МБ
Расположение файла подкачки:      C:\pagefile.sys
Домен:                            sunlife.local
Сервер входа в сеть:              \\PDC
Исправление(я):                   Число установленных исправлений - 7.
                                  [01]: KB3192137
                                  [02]: KB4049065
                                  [03]: KB4093137
                                  [04]: KB4132216
                                  [05]: KB4485447
                                  [06]: KB4503537
                                  [07]: KB4493470
Сетевые адаптеры:                 Число сетевых адаптеров - 1.
                                  [01]: Red Hat VirtIO Ethernet Adapter
                                        Имя подключения: Ethernet 2
                                        DHCP включен:    Да
                                        DHCP-сервер:     10.30.0.1
                                        IP-адрес
                                        [01]: 10.30.0.135
                                        [02]: fe80::30f7:8847:981b:ba93
Требования Hyper-V:               Обнаружена низкоуровневая оболочка. Функции, необходимые для Hyper-V, отображены не будут.


Setting "qm set 100 -machine pc-i440fx-5.1" (this VM had pc-i440 chipset from the beggining) DOES NOT restore "old" network adapter (settings)

Hope this will help)

P.S. I made a backup of this VM right before PVE upgrade and probably could share it with you if needed
 
Last edited:

tom

Proxmox Staff Member
Staff member
Aug 29, 2006
15,250
807
163

Whatever

Active Member
Nov 19, 2012
263
13
38
We can see the

If your run production environments, you should use the enterprise repository, recommended for production workloads.

See https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_package_repositories

Seems you ignored this.

We are already working on a fix for this and will update this forum thread as soon as we have more details.

Seems you ignored quoted text and cut my response to your colleague from the context. I was replying him and saying that I was not able to check his suggestion and workaround at that moment
 
Last edited:

t.lamprecht

Proxmox Staff Member
Staff member
Jul 28, 2015
4,550
1,202
164
South Tyrol/Italy
shop.proxmox.com
We could reproduce it now with a Windows Server 2016 Version, we had to install it on PVE 5.4 with QEMU 3.0 and then move it to a host with PVE 6.3 with QEMU 5.2 to trigger it. Pulling it up from an on QEMU 5.1 or even 4.0 installed version to 5.2 seems to not trigger the issue.

The PCIe address stays the same on all three version, so that is not the problem. There's something else which changed too much for making Windows update from QEMU 3.0 -> 5.2 go bonkers but not enough to make it go bonkers from QEMU 4.0 to 5.2 update, which works just fine... fun stuff.
In any case, we're still checking this out more closely. Hold off updates to QEMU 5.2 if you run Windows VMs which got created/installed before PVE 6.0.
 
Last edited:

fragpic

New Member
Feb 7, 2021
12
0
1
31
There's something else which changed too much for making Windows update from QEMU 3.0 -> 5.2 go bonkers but not enough to make it go bonkers from QEMU 4.0 to 5.2 update
This not true in my case. I have 3 windows VMs created on PVE 6.2 and they too had the issue.
 

Whatever

Active Member
Nov 19, 2012
263
13
38
We could reproduce it now with a Windows Server 2016 Version, we had to install it on PVE 5.4 with QEMU 3.0 and then move it to a host with PVE 6.3 with QEMU 5.2 to trigger it. Pulling it up from an on QEMU 5.1 or even 4.0 installed version to 5.2 seems to not trigger the issue.

The PCIe address stays the same on all three version, so that is not the problem. There's something else which changed too much for making Windows update from QEMU 3.0 -> 5.2 go bonkers but not enough to make it go bonkers from QEMU 4.0 to 5.2 update, which works just fine... fun stuff.
In any case, we're still checking this out more closely. Hold off updates to QEMU 5.2 if you run Windows VMs which got created/installed before PVE 6.0.

Thanks for clarifying
 

t.lamprecht

Proxmox Staff Member
Staff member
Jul 28, 2015
4,550
1,202
164
South Tyrol/Italy
shop.proxmox.com
This not true in my case. I have 3 windows VMs created on PVE 6.2 and they too had the issue.
That'd be an upgrade from QEMU 5.0 -> 5.2 then, I have not tried that specific tuple out yet...
Which Windows version and can you post the configuration? (just to be sure we gather as much information as possible).
But I def. have Windows VMs created with PVE 6.0 which boot OK here.

I mean, the change is not binary, as else any version upgrade would trigger an issue from 5.1 to 5.2 too, which it seems to not too.

Currently, I'm bisecting between 5.1.0 and 5.2.0 and seeing if the reproducer boots correct again, that should help to narrow it down.
 

fragpic

New Member
Feb 7, 2021
12
0
1
31
Which Windows version and can you post the configuration? (just to be sure we gather as much information as possible).
Windows Server 2019

Here's the conf
Bash:
agent: 1
balloon: 2048
bios: ovmf
bootdisk: scsi0
cores: 4
cpu: host
efidisk0: SSD1:vm-101-disk-1,size=4M
machine: q35
memory: 4096
name: DC
net0: virtio=4A:A6:28:0B:4A:A9,bridge=vmbr0
numa: 0
onboot: 1
ostype: win10
scsi0: SSD1:vm-101-disk-0,cache=writeback,discard=on,size=30G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=fbf3c4d8-f589-4626-b367-8e055093fd54
sockets: 1
 

t.lamprecht

Proxmox Staff Member
Staff member
Jul 28, 2015
4,550
1,202
164
South Tyrol/Italy
shop.proxmox.com
FYI: I found the problematic commit, which is sadly not a real bug but a correction to ACPI devices UIDs to make QEMU behave more standard conform. As its late Saturday and such things are not those which can be decided alone I cannot make any final solution, rather I've informed the original author of the patch and some other QEMU and EDK2 maintainers with good in-depth knowledge of all pieces involved (which is mostly firmware like SeaBIOS/OVMF and QEMU) so that we can get to a sane, future proof solution, if there's any, or a workaround/better upgrade to this behaviour in a more defined manner.

Until then, if you want to stay on QEMU 5.2 you can use the following package I build, it has only that commit reverted and makes my reproducer happy again (note, the other new "ghost" device won't go away, but the original Ethernet adapter will be in use again, from Windows POV).

You can download and install that build by doing:

Bash:
wget http://download.proxmox.com/temp/pve-qemu-5.2-with-acpi-af1b80ae56-reverted/pve-qemu-kvm_5.2.0-2%2B1~windevfix_amd64.deb

# verify check sum:
sha256sum pve-qemu-kvm_5.2.0-2+1\~windevfix_amd64.deb
33e8ce10b5a4005160c68f79c979d53b1a84a1d79436adbd00c48ec93d3bf1de  pve-qemu-kvm_5.2.0-2+1~windevfix_amd64.deb

# install it
apt install ./pve-qemu-kvm_5.2.0-2+1~windevfix_amd64.deb

After installation do a fresh boot of the VM, if unsure shut it down completely and then start it through the Proxmox VE webinterface again.
 

Whatever

Active Member
Nov 19, 2012
263
13
38
FYI: I found the problematic commit, which is sadly not a real bug but a correction to ACPI devices UIDs to make QEMU behave more standard conform. As its late Saturday and such things are not those which can be decided alone I cannot make any final solution, rather I've informed the original author of the patch and some other QEMU and EDK2 maintainers with good in-depth knowledge of all pieces involved (which is mostly firmware like SeaBIOS/OVMF and QEMU) so that we can get to a sane, future proof solution, if there's any, or a workaround/better upgrade to this behaviour in a more defined manner.

Until then, if you want to stay on QEMU 5.2 you can use the following package I build, it has only that commit reverted and makes my reproducer happy again (note, the other new "ghost" device won't go away, but the original Ethernet adapter will be in use again, from Windows POV).

You can download and install that build by doing:

Bash:
wget http://download.proxmox.com/temp/pve-qemu-5.2-with-acpi-af1b80ae56-reverted/pve-qemu-kvm_5.2.0-2%2B1~windevfix_amd64.deb

# verify check sum:
sha256sum pve-qemu-kvm_5.2.0-2+1\~windevfix_amd64.deb
33e8ce10b5a4005160c68f79c979d53b1a84a1d79436adbd00c48ec93d3bf1de  pve-qemu-kvm_5.2.0-2+1~windevfix_amd64.deb

# install it
apt install ./pve-qemu-kvm_5.2.0-2+1~windevfix_amd64.deb

After installation do a fresh boot of the VM, if unsure shut it down completely and then start it through the Proxmox VE webinterface again.

Checked. Works as expected. Thanks
 
Last edited:

ggzengel

New Member
Jul 8, 2019
14
0
1
Nice and good solution. Thank you.
I have always fixed my VMs and removed the unused device from device manager. I was to late to find this thread.
If it's possible, I would like to have a switch for changing the ACPI behavior.
I think in a later release (Proxmox 7?) there will be breaking changes and this fix will be standard?!
 

speedlnx

Active Member
Feb 7, 2016
36
1
28
41
Too late :eek: I just reconfigure all of my windows server... hope it doesn't happen again after the next update!
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!