Win2019 Server PCie passthrough boot problems since last Proxmox update to 7.2 - \boot\bcd 0xc00000e9

S

SimonR

Active Member
Proxmox Subscriber
Jan 2, 2020
35
6
28
Hi all,

since the last Proxmox Updates we had problems with a Win2019 Server using an Intel Optane PCIe-passthrough-device as boot volume. First we thought, it was an Windows-Update-Error, but it's not. A fresh install on a new VM of Win2019 Server runs normal till it reboots. The reboot ends in error, like the attached screenshot. Proxmox is well configured to use PCIe-passthrough. We hope, anyone has a solution for this. Find all other config-information here, we can't find any error in the config, why windows can't boot from the Intel Optane.

We have connected the Intel Optane Memory to another machine for testing and can't find any hardware error, e.g. a Windows volume restore to the PCIe device was successful without showing any error restoring 3 partitions with data in it.

The above reported new initial WinServer installation was made with complete deleting all existing partitions on the PCIe-storage card, ending in the 0x00000e9 error after the first reboot in installation process.


proxmox-ve: 7.2-1 (running kernel: 5.15.35-1-pve)
pve-manager: 7.2-3 (running version: 7.2-3/c743d6c1)
pve-kernel-5.15: 7.2-3
pve-kernel-helper: 7.2-3
pve-kernel-5.13: 7.1-9
pve-kernel-5.11: 7.0-10
pve-kernel-5.3: 6.1-6
pve-kernel-5.15.35-1-pve: 5.15.35-2
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-3-pve: 5.13.19-7
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: 0.8.36+pve1
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-8
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-6
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.2-2
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.12-1
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.1.8-1
proxmox-backup-file-restore: 2.1.8-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-10
pve-cluster: 7.2-1
pve-container: 4.2-1
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.4-2
pve-ha-manager: 3.3-4
pve-i18n: 2.7-1
pve-qemu-kvm: 6.2.0-5
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-2
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.4-pve1



0000:18:00.0 Non-Volatile memory controller [0108]: Intel Corporation NVMe Datacenter SSD [Optane] [8086:4140]




GNU nano 5.4 /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd



dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[ 0.017069] ACPI: DMAR 0x000000006D202EC8 000150 (v01 ALASKA A M I 00000001 INTL 20091013)
[ 0.017121] ACPI: Reserving DMAR table memory at [mem 0x6d202ec8-0x6d203017]
[ 0.143561] DMAR: IOMMU enabled
[ 0.365350] DMAR: Host address width 46
[ 0.365352] DMAR: DRHD base: 0x000000c5ffc000 flags: 0x0
[ 0.365358] DMAR: dmar0: reg_base_addr c5ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[ 0.365362] DMAR: DRHD base: 0x000000e0ffc000 flags: 0x0
[ 0.365366] DMAR: dmar1: reg_base_addr e0ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[ 0.365369] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[ 0.365373] DMAR: dmar2: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[ 0.365376] DMAR: DRHD base: 0x000000aaffc000 flags: 0x1
[ 0.365380] DMAR: dmar3: reg_base_addr aaffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[ 0.365383] DMAR: RMRR base: 0x0000006f5f7000 end: 0x0000006f607fff
[ 0.365385] DMAR: ATSR flags: 0x0
[ 0.365387] DMAR: RHSA base: 0x000000aaffc000 proximity domain: 0x0
[ 0.365389] DMAR: RHSA base: 0x000000c5ffc000 proximity domain: 0x0
[ 0.365391] DMAR: RHSA base: 0x000000e0ffc000 proximity domain: 0x0
[ 0.365392] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x0
[ 0.365395] DMAR-IR: IOAPIC id 12 under DRHD base 0xfbffc000 IOMMU 2
[ 0.365397] DMAR-IR: IOAPIC id 11 under DRHD base 0xe0ffc000 IOMMU 1
[ 0.365399] DMAR-IR: IOAPIC id 10 under DRHD base 0xc5ffc000 IOMMU 0
[ 0.365401] DMAR-IR: IOAPIC id 8 under DRHD base 0xaaffc000 IOMMU 3
[ 0.365403] DMAR-IR: IOAPIC id 9 under DRHD base 0xaaffc000 IOMMU 3
[ 0.365405] DMAR-IR: HPET id 0 under DRHD base 0xaaffc000
[ 0.365407] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.366287] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 0.809882] DMAR: No SATC found
[ 0.809885] DMAR: dmar2: Using Queued invalidation
[ 0.809889] DMAR: dmar0: Using Queued invalidation
[ 0.809894] DMAR: dmar3: Using Queued invalidation
[ 0.825431] DMAR: Intel(R) Virtualization Technology for Directed I/O



root@pve1:~# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/55/devices/0000:64:0b.2
/sys/kernel/iommu_groups/83/devices/0000:b2:15.0
/sys/kernel/iommu_groups/17/devices/0000:00:16.4
/sys/kernel/iommu_groups/17/devices/0000:00:16.0
/sys/kernel/iommu_groups/17/devices/0000:00:16.1
/sys/kernel/iommu_groups/45/devices/0000:64:0a.0
/sys/kernel/iommu_groups/73/devices/10000:00:01.0
/sys/kernel/iommu_groups/73/devices/10000:00:00.0
/sys/kernel/iommu_groups/73/devices/0000:b2:05.5
/sys/kernel/iommu_groups/35/devices/0000:16:11.2
/sys/kernel/iommu_groups/35/devices/0000:16:11.0
/sys/kernel/iommu_groups/35/devices/0000:16:11.3
/sys/kernel/iommu_groups/35/devices/0000:16:11.1
/sys/kernel/iommu_groups/7/devices/0000:00:04.6
/sys/kernel/iommu_groups/63/devices/0000:64:0c.6
/sys/kernel/iommu_groups/25/devices/0000:16:05.0
/sys/kernel/iommu_groups/53/devices/0000:64:0b.0
/sys/kernel/iommu_groups/81/devices/0000:b2:12.5
/sys/kernel/iommu_groups/81/devices/0000:b2:12.1
/sys/kernel/iommu_groups/81/devices/0000:b2:12.2
/sys/kernel/iommu_groups/15/devices/0000:00:11.1
/sys/kernel/iommu_groups/15/devices/0000:00:11.0
/sys/kernel/iommu_groups/43/devices/0000:64:08.0
/sys/kernel/iommu_groups/71/devices/0000:b2:05.2
/sys/kernel/iommu_groups/33/devices/0000:16:0f.5
/sys/kernel/iommu_groups/33/devices/0000:16:0f.3
/sys/kernel/iommu_groups/33/devices/0000:16:0f.1
/sys/kernel/iommu_groups/33/devices/0000:16:0f.6
/sys/kernel/iommu_groups/33/devices/0000:16:0f.4
/sys/kernel/iommu_groups/33/devices/0000:16:0f.2
/sys/kernel/iommu_groups/33/devices/0000:16:0f.0
/sys/kernel/iommu_groups/33/devices/0000:16:0f.7
/sys/kernel/iommu_groups/5/devices/0000:00:04.4
/sys/kernel/iommu_groups/61/devices/0000:64:0c.4
/sys/kernel/iommu_groups/23/devices/0000:16:00.0
/sys/kernel/iommu_groups/51/devices/0000:64:0a.6
/sys/kernel/iommu_groups/13/devices/0000:00:08.1
/sys/kernel/iommu_groups/41/devices/0000:64:05.2
/sys/kernel/iommu_groups/31/devices/0000:16:0b.0
/sys/kernel/iommu_groups/31/devices/0000:16:0b.3
/sys/kernel/iommu_groups/31/devices/0000:16:0b.1
/sys/kernel/iommu_groups/31/devices/0000:16:0b.2
/sys/kernel/iommu_groups/3/devices/0000:00:04.2
/sys/kernel/iommu_groups/88/devices/0000:b5:00.0
/sys/kernel/iommu_groups/21/devices/0000:01:00.0
/sys/kernel/iommu_groups/78/devices/0000:b2:10.0
/sys/kernel/iommu_groups/11/devices/0000:00:05.4
/sys/kernel/iommu_groups/68/devices/0000:64:0d.3
/sys/kernel/iommu_groups/1/devices/0000:00:04.0
/sys/kernel/iommu_groups/58/devices/0000:64:0c.1
/sys/kernel/iommu_groups/86/devices/0000:b3:00.0
/sys/kernel/iommu_groups/48/devices/0000:64:0a.3
/sys/kernel/iommu_groups/76/devices/0000:b2:0f.0
/sys/kernel/iommu_groups/38/devices/0000:17:00.0
/sys/kernel/iommu_groups/66/devices/0000:64:0d.1
/sys/kernel/iommu_groups/28/devices/0000:16:08.6
/sys/kernel/iommu_groups/28/devices/0000:16:08.4
/sys/kernel/iommu_groups/28/devices/0000:16:08.2
/sys/kernel/iommu_groups/28/devices/0000:16:08.0
/sys/kernel/iommu_groups/28/devices/0000:16:08.7
/sys/kernel/iommu_groups/28/devices/0000:16:08.5
/sys/kernel/iommu_groups/28/devices/0000:16:08.3
/sys/kernel/iommu_groups/28/devices/0000:16:08.1
/sys/kernel/iommu_groups/56/devices/0000:64:0b.3
/sys/kernel/iommu_groups/84/devices/0000:b2:16.4
/sys/kernel/iommu_groups/84/devices/0000:b2:16.0
/sys/kernel/iommu_groups/18/devices/0000:00:1c.0
/sys/kernel/iommu_groups/46/devices/0000:64:0a.1
/sys/kernel/iommu_groups/74/devices/0000:b2:0e.0
/sys/kernel/iommu_groups/36/devices/0000:16:1d.0
/sys/kernel/iommu_groups/36/devices/0000:16:1d.3
/sys/kernel/iommu_groups/36/devices/0000:16:1d.1
/sys/kernel/iommu_groups/36/devices/0000:16:1d.2
/sys/kernel/iommu_groups/8/devices/0000:00:04.7
/sys/kernel/iommu_groups/64/devices/0000:64:0c.7
/sys/kernel/iommu_groups/26/devices/0000:16:05.2
/sys/kernel/iommu_groups/54/devices/0000:64:0b.1
/sys/kernel/iommu_groups/82/devices/0000:b2:12.4
/sys/kernel/iommu_groups/16/devices/0000:00:14.2
/sys/kernel/iommu_groups/16/devices/0000:00:14.0
/sys/kernel/iommu_groups/44/devices/0000:64:09.0
/sys/kernel/iommu_groups/72/devices/0000:b2:05.4
/sys/kernel/iommu_groups/34/devices/0000:16:10.6
/sys/kernel/iommu_groups/34/devices/0000:16:10.4
/sys/kernel/iommu_groups/34/devices/0000:16:10.2
/sys/kernel/iommu_groups/34/devices/0000:16:10.0
/sys/kernel/iommu_groups/34/devices/0000:16:10.7
/sys/kernel/iommu_groups/34/devices/0000:16:10.5
/sys/kernel/iommu_groups/34/devices/0000:16:10.3
/sys/kernel/iommu_groups/34/devices/0000:16:10.1
/sys/kernel/iommu_groups/6/devices/0000:00:04.5
/sys/kernel/iommu_groups/62/devices/0000:64:0c.5
/sys/kernel/iommu_groups/24/devices/0000:16:02.0
/sys/kernel/iommu_groups/52/devices/0000:64:0a.7
/sys/kernel/iommu_groups/80/devices/0000:b2:12.0
/sys/kernel/iommu_groups/14/devices/0000:00:08.2
/sys/kernel/iommu_groups/42/devices/0000:64:05.4
/sys/kernel/iommu_groups/70/devices/0000:b2:05.0
/sys/kernel/iommu_groups/32/devices/0000:16:0e.2
/sys/kernel/iommu_groups/32/devices/0000:16:0e.0
/sys/kernel/iommu_groups/32/devices/0000:16:0e.7
/sys/kernel/iommu_groups/32/devices/0000:16:0e.5
/sys/kernel/iommu_groups/32/devices/0000:16:0e.3
/sys/kernel/iommu_groups/32/devices/0000:16:0e.1
/sys/kernel/iommu_groups/32/devices/0000:16:0e.6
/sys/kernel/iommu_groups/32/devices/0000:16:0e.4
/sys/kernel/iommu_groups/4/devices/0000:00:04.3
/sys/kernel/iommu_groups/60/devices/0000:64:0c.3
/sys/kernel/iommu_groups/89/devices/0000:b5:00.1
/sys/kernel/iommu_groups/22/devices/0000:03:00.0
/sys/kernel/iommu_groups/22/devices/0000:02:00.0
/sys/kernel/iommu_groups/50/devices/0000:64:0a.5
/sys/kernel/iommu_groups/79/devices/0000:b2:10.1
/sys/kernel/iommu_groups/12/devices/0000:00:08.0
/sys/kernel/iommu_groups/40/devices/0000:64:05.0
/sys/kernel/iommu_groups/69/devices/0000:b2:02.0
/sys/kernel/iommu_groups/30/devices/0000:16:0a.6
/sys/kernel/iommu_groups/30/devices/0000:16:0a.4
/sys/kernel/iommu_groups/30/devices/0000:16:0a.2
/sys/kernel/iommu_groups/30/devices/0000:16:0a.0
/sys/kernel/iommu_groups/30/devices/0000:16:0a.7
/sys/kernel/iommu_groups/30/devices/0000:16:0a.5
/sys/kernel/iommu_groups/30/devices/0000:16:0a.3
/sys/kernel/iommu_groups/30/devices/0000:16:0a.1
/sys/kernel/iommu_groups/2/devices/0000:00:04.1
/sys/kernel/iommu_groups/59/devices/0000:64:0c.2
/sys/kernel/iommu_groups/87/devices/0000:b4:03.0
/sys/kernel/iommu_groups/20/devices/0000:00:1f.2
/sys/kernel/iommu_groups/20/devices/0000:00:1f.0
/sys/kernel/iommu_groups/20/devices/0000:00:1f.5
/sys/kernel/iommu_groups/20/devices/0000:00:1f.4
/sys/kernel/iommu_groups/49/devices/0000:64:0a.4
/sys/kernel/iommu_groups/77/devices/0000:b2:0f.1
/sys/kernel/iommu_groups/10/devices/0000:00:05.2
/sys/kernel/iommu_groups/39/devices/0000:18:00.0
/sys/kernel/iommu_groups/67/devices/0000:64:0d.2
/sys/kernel/iommu_groups/29/devices/0000:16:09.0
/sys/kernel/iommu_groups/29/devices/0000:16:09.7
/sys/kernel/iommu_groups/29/devices/0000:16:09.5
/sys/kernel/iommu_groups/29/devices/0000:16:09.3
/sys/kernel/iommu_groups/29/devices/0000:16:09.1
/sys/kernel/iommu_groups/29/devices/0000:16:09.6
/sys/kernel/iommu_groups/29/devices/0000:16:09.4
/sys/kernel/iommu_groups/29/devices/0000:16:09.2
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/57/devices/0000:64:0c.0
/sys/kernel/iommu_groups/85/devices/0000:b2:17.0
/sys/kernel/iommu_groups/19/devices/0000:00:1c.5
/sys/kernel/iommu_groups/47/devices/0000:64:0a.2
/sys/kernel/iommu_groups/75/devices/0000:b2:0e.1
/sys/kernel/iommu_groups/37/devices/0000:16:1e.5
/sys/kernel/iommu_groups/37/devices/0000:16:1e.3
/sys/kernel/iommu_groups/37/devices/0000:16:1e.1
/sys/kernel/iommu_groups/37/devices/0000:16:1e.6
/sys/kernel/iommu_groups/37/devices/0000:16:1e.4
/sys/kernel/iommu_groups/37/devices/0000:16:1e.2
/sys/kernel/iommu_groups/37/devices/0000:16:1e.0
/sys/kernel/iommu_groups/9/devices/0000:00:05.0
/sys/kernel/iommu_groups/65/devices/0000:64:0d.0
/sys/kernel/iommu_groups/27/devices/0000:16:05.4



GNU nano 5.4 106.conf
boot: order=ide2;hostpci0;net0
cores: 16
hostpci0: 0000:18:00,pcie=1,rombar=0
ide2: local:iso/de_windows_server_2019_x64_dvd_17559a5b.iso,media=cdrom,size=4778578K
machine: pc-q35-6.2
memory: 2048
meta: creation-qemu=6.2.0,ctime=1652333851
name: TESTPCIE
net0: virtio=EA:EB:A8:C9:8B:36,bridge=vmbr0
numa: 0
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=e2b2c7cb-3918-4aa0-bb01-002ce8811bbe
sockets: 1
vmgenid: 7662aeee-362d-4d67-8506-9f155332fd32
 

Attachments

  • boot_bcd_0x00000e9.png
    boot_bcd_0x00000e9.png
    177 KB · Views: 9
Last edited:
It seems kernel 5.15.35 creates a lot of problems with IOMMU. If you are using PCI-E passthrough I would advise you to rebuild the server with an older kernel.
I just spent a lot of time on my homelab trying to figure what happened after getting back to a completely broken host after updating... You will find other topics on this, just google "5.15.35 IOMMU". There seems to be fixes if you really need that passthrough, but for now it seems to be kind of janky.
 
is there anything visible in the syslog/journal/dmesg ? does it work when you install a linux vm ?
 
  • Like
Reactions: leesteken
dcsapak said:
is there anything visible in the syslog/journal/dmesg ? does it work when you install a linux vm ?
Click to expand...
We just kicked the plan to passthrough it again, I'm angry, my customer is more angry, but I warned him a few months before to not passthrough PCIe NVME. But it was everythink o.k. for more than 6 months. I don't know if a linux vm would run, cause we don't need a linux vm, and we will not spend any more time in the project. We changed the 400 GB passed-through NVME to a LVM-thin-volume, and restored a windows backup to the new VHD, the peformance is a bit slower, but still very fast. We will not change back, even if there is a solution. Do not worry about us, better take care to the other hundreds, not able to passthrough their GPUs anymore after the last kernel update.
 
Erlum said:
It seems kernel 5.15.35 creates a lot of problems with IOMMU. If you are using PCI-E passthrough I would advise you to rebuild the server with an older kernel.
I just spent a lot of time on my homelab trying to figure what happened after getting back to a completely broken host after updating... You will find other topics on this, just google "5.15.35 IOMMU". There seems to be fixes if you really need that passthrough, but for now it seems to be kind of janky.
Click to expand...
Rebuilding with older kernels? And living with security problems, or hoping, that the over-next kernel-update will solve this? We solved it with changing passthrough NVME to a Proxmox LVM-thin on this NVME.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom