[SOLVED] New to Proxmox - First issue

R

RudyZ

New Member
Dec 10, 2022
6
1
3
Hi everyone,

As the title says, I'm new to Proxmox and wanted to play with it as a weekend project.

First, this is the hardware I run:
Ryzen 3 3200
ASRock B450 Pro4 v2.0
16GB RAM
Proxmox (7.3.1 version) runs on a SATA enterprise SSD plugged directly to one of the sata port on the mobo
MSI RTX 3060TI
LSI SAS9211-8 in IT mode with a mini-sas to sata cable running 3 x 8TB Enterprise HGST hard disks

Proxmox installation went fine. I enabled IOMMU and VT-D (whatever is the AMD equivalent) in the motherboard BIOS and updated GRUB accordingly for AMD.
I installed a VM and it boots without problem but as soon as I try to boot the VM with my LSI card passed-through as a PCI device, the all system freezes and I have to hard reboot it.

This is what the shell output looks like when it happens:

[ 274.518805] Buffer I/D error on device dm-1, logical block 22480892
[ 275.186413] Buffer I/O error on device dm-1, logical block 13905
[ 275.186446] Buffer I/O error on device dm-1, logical block 14203
[ 275.186464] Buffer I/0 error on device dm-1, logical block 13907
[ 275.186486] Buffer I/D error on device dm-1, logical block 14202
[ 275.186505] Buffer I/D error on device dm-1, logical block 13909
[ 275.186524] Buffer I/D error on device dm-1, logical block 18909626
[ 275.186524] Buffer I/0 error on device dm-1, logical block 13910
[ 275.186565] Buffer I/D error on device dm-1, logical block 13914
[ 275.186567] Buffer I/D error on device dm-1, logical block 22446108
[ 278.195054] Aborting journal on device dm-1-8.
[ 278.195081] Buffer I/D error on dev dm-1, logical block 12615680, lost sync page urite
[ 278.195103] JBD2: Error -5 detected when updating journal superblock for dm-1-8.
[ 278.195110] EXT4-s error (device dm-1): ext_journal_check_start:83: comm pmcfs: Detected aborted journal
[ 278.1951811 Buffer I/D error on dev dm-1, logical block 0, lost sync page write
[ 278.195205] EXT4-fs (dm-1): I/0 error while writing superblock
[ 278.195221] EXT4-fs (dm-1): Remounting filesystem read-only

I also tried passing the LSI card without the 3 HDs but same issue.

Googled around but cannot find any solution.
Any help would be greatly appreciated.

Happy to provide any logs or any information needed.

Thank you
 
Last edited:
Please provide the full output from the PVE-host in code-tags of:
Bash: 
for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
 
  • Like
Reactions: leesteken
Your LSI-card is most likely in the big chipset IOMMU group, and you cannot share devices from the same group between VMs and/or the Proxmox host. Except for the X570 chipset, Ryzen system can only passthrough the first x16 PCIe slot (or two x8x8 in X470/X370) and the first M.2 slot (x4 PCIe) connected to the CPU. As soon as you start your VM with passthrough, all other devices in the same group (network, USB and SATA controllers) are no longer accessible by the Proxmox host and it crashes.
 
  • Like
Reactions: _gabriel and Colin 't Hart
Neobin said:
Please provide the full output from the PVE-host in code-tags of:
Bash: 
for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
Click to expand...

Thanks for your reply. Here's the output from that command. Looks like my LSI card shares group with other devices which cause the crash:

IOMMU group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 0 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 0 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 0 01:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
IOMMU group 0 02:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset USB 3.1 XHCI Controller [1022:43d5] (rev 01)
IOMMU group 0 02:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01)
IOMMU group 0 02:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge [1022:43c6] (rev 01)
IOMMU group 0 03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU group 0 03:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU group 0 03:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU group 0 03:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU group 0 03:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU group 0 03:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU group 0 08:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
IOMMU group 0 09:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU group 10 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440]
IOMMU group 10 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441]
IOMMU group 10 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442]
IOMMU group 10 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443]
IOMMU group 10 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444]
IOMMU group 10 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445]
IOMMU group 10 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446]
IOMMU group 10 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447]
IOMMU group 11 0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU group 12 0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU group 13 0c:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU group 14 0c:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU group 15 0c:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
IOMMU group 1 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 2 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 2 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 2 0a:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2489] (rev a1)
IOMMU group 2 0a:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller [10de:228b] (rev a1)
IOMMU group 3 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 4 00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 5 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 6 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU group 7 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 8 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU group 9 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
IOMMU group 9 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
 
Last edited:
leesteken said:
Your LSI-card is most likely in the big chipset IOMMU group, and you cannot share devices from the same group between VMs and/or the Proxmox host. Except for the X570 chipset, Ryzen system can only passthrough the first x16 PCIe slot (or two x8x8 in X470/X370) and the first M.2 slot (x4 PCIe) connected to the CPU. As soon as you start your VM with passthrough, all other devices in the same group (network, USB and SATA controllers) are no longer accessible by the Proxmox host and it crashes.
Click to expand...
Thanks for your reply. That makes a lot of sense.
 
It looks like you're encountering a problem with your storage device. The error messages indicate that there are issues with the data being written to and read from the device, specifically "Buffer I/D error" and "Buffer I/O error". This could be caused by a faulty cable, a problem with the hard disks, or even an issue with the LSI SAS9211-8 itself.
RudyZ said:
Thanks for your reply. Here's the output from that command. Looks like my LSI card shares group with other devices which cause the crash:

IOMMU group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 0 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 0 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 0 01:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
IOMMU group 0 02:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset USB 3.1 XHCI Controller [1022:43d5] (rev 01)
IOMMU group 0 02:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01)
IOMMU group 0 02:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge [1022:43c6] (rev 01)
IOMMU group 0 03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU group 0 03:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU group 0 03:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU group 0 03:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU group 0 03:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU group 0 03:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU group 0 08:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
IOMMU group 0 09:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU group 10 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440]
IOMMU group 10 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441]
IOMMU group 10 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442]
IOMMU group 10 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443]
IOMMU group 10 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444]
IOMMU group 10 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445]
IOMMU group 10 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446]
IOMMU group 10 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447]
IOMMU group 11 0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU group 12 0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU group 13 0c:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU group 14 0c:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU group 15 0c:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
IOMMU group 1 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 2 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 2 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 2 0a:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2489] (rev a1)
IOMMU group 2 0a:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller [10de:228b] (rev a1)
IOMMU group 3 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 4 00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 5 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 6 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU group 7 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 8 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU group 9 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
IOMMU group 9 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
Click to expand...

It's worth noting that the LSI SAS9211-8 is a hardware-level RAID controller, which means it is designed to manage multiple hard disks as a single storage device. In order to pass it through to a VM, you will need to disable the RAID functionality on the controller, as this is not supported in a virtualized environment.

Based on the error messages you provided, it seems like there may be a problem with the hard disks or the controller itself. It's worth trying to update the firmware on the controller and also replacing the cable to see if that resolves the issue. Additionally, you can try connecting the hard disks directly to the motherboard to see if the issue persists without the LSI SAS9211-8 in the mix. This will help you narrow down the source of the problem.

You might want to check this thread too: https://forum.proxmox.com/threads/proxmox-iommu-passthrough-crashes-host.108439/
You cannot share devices from the same IOMMU group between VMs and/or the Proxmox host.
 
rason said:
It looks like you're encountering a problem with your storage device. The error messages indicate that there are issues with the data being written to and read from the device, specifically "Buffer I/D error" and "Buffer I/O error". This could be caused by a faulty cable, a problem with the hard disks, or even an issue with the LSI SAS9211-8 itself.


It's worth noting that the LSI SAS9211-8 is a hardware-level RAID controller, which means it is designed to manage multiple hard disks as a single storage device. In order to pass it through to a VM, you will need to disable the RAID functionality on the controller, as this is not supported in a virtualized environment.

Based on the error messages you provided, it seems like there may be a problem with the hard disks or the controller itself. It's worth trying to update the firmware on the controller and also replacing the cable to see if that resolves the issue. Additionally, you can try connecting the hard disks directly to the motherboard to see if the issue persists without the LSI SAS9211-8 in the mix. This will help you narrow down the source of the problem.

You might want to check this thread too: https://forum.proxmox.com/threads/proxmox-iommu-passthrough-crashes-host.108439/
You cannot share devices from the same IOMMU group between VMs and/or the Proxmox host.
Click to expand...
Thanks for your reply.

The LSI is in IT mode so no longer a RAID card per se. The HDs and cable were tested right before I put together that setup so I know they're fine. The IOMMU group sharing seems to be the issue here. Thanks for the link you provided, will take a look at it.
 
So I went with the added pcie_acs_override=downstream,multifunction option and that made the trick.

Thank you all for the help. Much appreciated.
 
  • Like
Reactions: _gabriel
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back