[SOLVED] [Warning] Latest Patch just broke all my WINDOWS vms. (6.3-4) [Patch Inside] (See #53)
- Thread starter XoCluTch
- Start date
Again, use the pve-enterprise repository if you need production grade stability.
We tested it and all our hundreds of different Linux VM workloads run fine here, as did the Windows 10 test instances we have here, that's why we moved the update along.
We're evaluating this to see what specific set of properties there need to be for it to happen, also in this thread people have some VMs where it does not happen, so it's not universal. Do you actually have anything to add for actually helping to narrow it down?
Thought first, but https://forum.proxmox.com/threads/w...e-all-my-windows-vms-6-3-4.84915/#post-373042 has both, q35 and pc
Thomas, thanks for looking into it. I am not sure how to find out with which qemu version my windows vm's were installed initially. I would say it was about 5 years when I converted them from bare metal. Happy to give you one vm backup if you need it for testing. Will do more upgrades soon on a cluster with newer vm's and let you know if the issue persists.
Unfortunately, all the affected VMs were from production environments and had to be fixed ASAP.
Hopefully, I've managed to reproduce this issue on one of our client's test system and here the information I've collected so far:
PVE host:
Code:
root@pve:~# dumpe2fs $(mount | grep 'on \/ ' | awk '{print $1}') | grep 'Filesystem created:'
dumpe2fs 1.44.5 (15-Dec-2018)
Filesystem created: Mon Jun 18 12:30:29 2018Windows Server 2016 Standard (with QEMU 5.1)
Code:
C:\Users\Администратор.WIN-REJFV9JJIVT>systeminfo
Имя узла: PDC
Название ОС: Майкрософт Windows Server 2016 Standard
Версия ОС: 10.0.14393 Н/Д построение 14393
Изготовитель ОС: Microsoft Corporation
Параметры ОС: Основной контроллер домена
Построение ОС: Multiprocessor Free
Зарегистрированный владелец: Пользователь Windows
Зарегистрированная организация:
Код продукта: 00377-60000-00000-AA934
Дата установки: 19.06.2018, 16:25:31
Время загрузки системы: 12.02.2021, 12:50:06
Изготовитель системы: QEMU
Модель системы: Standard PC (i440FX + PIIX, 1996)
Тип системы: x64-based PC
Процессор(ы): Число процессоров - 1.
[01]: Intel64 Family 6 Model 58 Stepping 9 GenuineIntel ~3300 МГц
Версия BIOS: SeaBIOS rel-1.13.0-48-gd9c812dda519-prebuilt.qemu.org, 01.04.2014
Папка Windows: C:\Windows
Системная папка: C:\Windows\system32
Устройство загрузки: \Device\HarddiskVolume1
Язык системы: ru;Русский
Язык ввода: ru;Русский
Часовой пояс: (UTC+03:00) Москва, Санкт-Петербург
Полный объем физической памяти: 2 047 МБ
Доступная физическая память: 553 МБ
Виртуальная память: Макс. размер: 2 431 МБ
Виртуальная память: Доступна: 983 МБ
Виртуальная память: Используется: 1 448 МБ
Расположение файла подкачки: C:\pagefile.sys
Домен: sunlife.local
Сервер входа в сеть: \\PDC
Исправление(я): Число установленных исправлений - 7.
[01]: KB3192137
[02]: KB4049065
[03]: KB4093137
[04]: KB4132216
[05]: KB4485447
[06]: KB4503537
[07]: KB4493470
Сетевые адаптеры: Число сетевых адаптеров - 1.
[01]: Red Hat VirtIO Ethernet Adapter
Имя подключения: Ethernet
DHCP включен: Нет
IP-адрес
[01]: 10.30.0.240
[02]: fe80::8cb8:45f1:52d3:686b
Требования Hyper-V: Обнаружена низкоуровневая оболочка. Функции, необходимые для Hyper-V, отображены не будут.Unfortunately system is localized (Russian edition) but hope it's still readable
Windows Server 2016 Standard (after QEMU 5.2 update)
Code:
C:\Users\Администратор.WIN-REJFV9JJIVT>systeminfo
Имя узла: PDC
Название ОС: Майкрософт Windows Server 2016 Standard
Версия ОС: 10.0.14393 Н/Д построение 14393
Изготовитель ОС: Microsoft Corporation
Параметры ОС: Основной контроллер домена
Построение ОС: Multiprocessor Free
Зарегистрированный владелец: Пользователь Windows
Зарегистрированная организация:
Код продукта: 00377-60000-00000-AA934
Дата установки: 19.06.2018, 16:25:31
Время загрузки системы: 27.02.2021, 16:46:00
Изготовитель системы: QEMU
Модель системы: Standard PC (i440FX + PIIX, 1996)
Тип системы: x64-based PC
Процессор(ы): Число процессоров - 1.
[01]: Intel64 Family 6 Model 58 Stepping 9 GenuineIntel ~3300 МГц
Версия BIOS: SeaBIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org, 01.04.2014
Папка Windows: C:\Windows
Системная папка: C:\Windows\system32
Устройство загрузки: \Device\HarddiskVolume1
Язык системы: ru;Русский
Язык ввода: ru;Русский
Часовой пояс: (UTC+03:00) Москва, Санкт-Петербург
Полный объем физической памяти: 2 047 МБ
Доступная физическая память: 235 МБ
Виртуальная память: Макс. размер: 4 898 МБ
Виртуальная память: Доступна: 896 МБ
Виртуальная память: Используется: 4 002 МБ
Расположение файла подкачки: C:\pagefile.sys
Домен: sunlife.local
Сервер входа в сеть: \\PDC
Исправление(я): Число установленных исправлений - 7.
[01]: KB3192137
[02]: KB4049065
[03]: KB4093137
[04]: KB4132216
[05]: KB4485447
[06]: KB4503537
[07]: KB4493470
Сетевые адаптеры: Число сетевых адаптеров - 1.
[01]: Red Hat VirtIO Ethernet Adapter
Имя подключения: Ethernet 2
DHCP включен: Да
DHCP-сервер: 10.30.0.1
IP-адрес
[01]: 10.30.0.135
[02]: fe80::30f7:8847:981b:ba93
Требования Hyper-V: Обнаружена низкоуровневая оболочка. Функции, необходимые для Hyper-V, отображены не будут.Setting "qm set 100 -machine pc-i440fx-5.1" (this VM had pc-i440 chipset from the beggining) DOES NOT restore "old" network adapter (settings)
Hope this will help)
P.S. I made a backup of this VM right before PVE upgrade and probably could share it with you if needed
Last edited:
We can see the
If your run production environments, you should use the enterprise repository, recommended for production workloads.
See https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_package_repositories
Seems you ignored this.
We are already working on a fix for this and will update this forum thread as soon as we have more details.
If your run production environments, you should use the enterprise repository, recommended for production workloads.
See https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_package_repositories
Seems you ignored this.
We are already working on a fix for this and will update this forum thread as soon as we have more details.
We could reproduce it now with a Windows Server 2016 Version, we had to install it on PVE 5.4 with QEMU 3.0 and then move it to a host with PVE 6.3 with QEMU 5.2 to trigger it. Pulling it up from an on QEMU 5.1 or even 4.0 installed version to 5.2 seems to not trigger the issue.
The PCIe address stays the same on all three version, so that is not the problem. There's something else which changed too much for making Windows update from QEMU 3.0 -> 5.2 go bonkers but not enough to make it go bonkers from QEMU 4.0 to 5.2 update, which works just fine... fun stuff.
In any case, we're still checking this out more closely. Hold off updates to QEMU 5.2 if you run Windows VMs which got created/installed before PVE 6.0.
The PCIe address stays the same on all three version, so that is not the problem. There's something else which changed too much for making Windows update from QEMU 3.0 -> 5.2 go bonkers but not enough to make it go bonkers from QEMU 4.0 to 5.2 update, which works just fine... fun stuff.
In any case, we're still checking this out more closely. Hold off updates to QEMU 5.2 if you run Windows VMs which got created/installed before PVE 6.0.
Last edited:
@t.lamprecht and @tom ... thank you very much for your hard work on a Saturday ... I will subscribe now for my two private testing machines ...
This not true in my case. I have 3 windows VMs created on PVE 6.2 and they too had the issue.
That'd be an upgrade from QEMU 5.0 -> 5.2 then, I have not tried that specific tuple out yet...
Which Windows version and can you post the configuration? (just to be sure we gather as much information as possible).
But I def. have Windows VMs created with PVE 6.0 which boot OK here.
I mean, the change is not binary, as else any version upgrade would trigger an issue from 5.1 to 5.2 too, which it seems to not too.
Currently, I'm bisecting between 5.1.0 and 5.2.0 and seeing if the reproducer boots correct again, that should help to narrow it down.
Windows Server 2019
Here's the conf
Bash:
agent: 1
balloon: 2048
bios: ovmf
bootdisk: scsi0
cores: 4
cpu: host
efidisk0: SSD1:vm-101-disk-1,size=4M
machine: q35
memory: 4096
name: DC
net0: virtio=4A:A6:28:0B:4A:A9,bridge=vmbr0
numa: 0
onboot: 1
ostype: win10
scsi0: SSD1:vm-101-disk-0,cache=writeback,discard=on,size=30G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=fbf3c4d8-f589-4626-b367-8e055093fd54
sockets: 1FYI: I found the problematic commit, which is sadly not a real bug but a correction to ACPI devices UIDs to make QEMU behave more standard conform. As its late Saturday and such things are not those which can be decided alone I cannot make any final solution, rather I've informed the original author of the patch and some other QEMU and EDK2 maintainers with good in-depth knowledge of all pieces involved (which is mostly firmware like SeaBIOS/OVMF and QEMU) so that we can get to a sane, future proof solution, if there's any, or a workaround/better upgrade to this behaviour in a more defined manner.
Until then, if you want to stay on QEMU 5.2 you can use the following package I build, it has only that commit reverted and makes my reproducer happy again (note, the other new "ghost" device won't go away, but the original Ethernet adapter will be in use again, from Windows POV).
You can download and install that build by doing:
After installation do a fresh boot of the VM, if unsure shut it down completely and then start it through the Proxmox VE webinterface again.
Until then, if you want to stay on QEMU 5.2 you can use the following package I build, it has only that commit reverted and makes my reproducer happy again (note, the other new "ghost" device won't go away, but the original Ethernet adapter will be in use again, from Windows POV).
You can download and install that build by doing:
Bash:
wget http://download.proxmox.com/temp/pve-qemu-5.2-with-acpi-af1b80ae56-reverted/pve-qemu-kvm_5.2.0-2%2B1~windevfix_amd64.deb
# verify check sum:
sha256sum pve-qemu-kvm_5.2.0-2+1\~windevfix_amd64.deb
33e8ce10b5a4005160c68f79c979d53b1a84a1d79436adbd00c48ec93d3bf1de pve-qemu-kvm_5.2.0-2+1~windevfix_amd64.deb
# install it
apt install ./pve-qemu-kvm_5.2.0-2+1~windevfix_amd64.debAfter installation do a fresh boot of the VM, if unsure shut it down completely and then start it through the Proxmox VE webinterface again.
Checked. Works as expected. Thanks
Last edited:
Nice and good solution. Thank you.
I have always fixed my VMs and removed the unused device from device manager. I was to late to find this thread.
If it's possible, I would like to have a switch for changing the ACPI behavior.
I think in a later release (Proxmox 7?) there will be breaking changes and this fix will be standard?!
I have always fixed my VMs and removed the unused device from device manager. I was to late to find this thread.
If it's possible, I would like to have a switch for changing the ACPI behavior.
I think in a later release (Proxmox 7?) there will be breaking changes and this fix will be standard?!