Proxmox SEGFAULTS & Issues Running TrueNAS Scale in a VM

Z

zurz2

New Member
Oct 22, 2023
8
0
1
I am encountering persistent issues when trying to run TrueNAS Scale inside a virtual machine (VM). TrueNAS Core, running on the same hypervisor, works without major issues (aside from occasional python3.9 core dumps), but TrueNAS Scale fails to boot properly.

Help or notes in any way would help me a lot! I have no idea what to check and have just been having issues with TrueNAS VMs for a while. Thanks!

Steps to Reproduce
  1. Create a new VM for TrueNAS Scale.
  2. Use default configurations (detailed below).
  3. Boot the VM.
Expected Result
The TrueNAS Scale VM boots successfully without errors.

Actual Result
The TrueNAS Scale VM fails to boot, with multiple errors, including:
  • Middleware (middlewared.service) failing to initialize.
  • Disk cache table and ZFS pool import issues (ix-syndisks.service and ix-zfs.service).
Environment
  • Hypervisor: Proxmox
  • TrueNAS Core VM Configuration:
    • Memory: 16 GB
    • CPUs: 8 cores
    • BIOS: SeaBIOS
    • Storage: VirtIO SCSI Disk (32 GB)
    • Network: VirtIO
      (Works fine apart from occasional core dumps.)
  • TrueNAS Scale VM Configuration:
    • Memory: 8 GB
    • CPUs: 8 cores
    • BIOS: SeaBIOS
    • Storage: VirtIO SCSI Disk (32 GB)
    • Network: VirtIO
      (Fails to boot with middleware and ZFS errors.)
Logs and Errors
  1. Middleware Error Logs:
Code: 
Traceback (most recent call last):
     File "/usr/bin/middlewared", line 33, in <module>
       sys.exit(load_entry_point('middlewared==0.0.0', 'console_scripts', 'middlewared')())
     File "/usr/lib/python3/dist-packages/middlewared/main.py", line 2220, in main
       ).run()
     File "/usr/lib/python3/dist-packages/middlewared/main.py", line 2034, in run
       self.loop.run_until_complete(self.__initialize())
     File "/usr/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
       return future.result()
     File "/usr/lib/python3/dist-packages/middlewared/schema/patch.py", line 24, in resolve
       cp.attrs[name] = attr.copy()
   AttributeError: 'tuple' object has no attribute '__dict__'
  1. Service Failures:
    • middlewared.service fails to initialize TrueNAS middleware.
    • ix-syndisks.service fails to sync the disk cache table.
    • ix-zfs.service fails to import ZFS pools.
  2. Console Logs:
    Code: 
    Failed to start middlewared.service - TrueNAS Middleware.
Failed to import ZFS pools. See "systemctl status ix-zfs.service" for details.
Additional screenshots of the errors during the boot process:
  1. Middleware Errors:
  2. ZFS and Disk Cache Errors:
  1. Verified the ISO file integrity using the SHA256 checksum.
  2. Tested default VM configurations for TrueNAS Scale.
  3. Compared VM settings with TrueNAS Core, which works fine. (EXCEPT core dumps lol)


3
31530×963 232 KB
2
21531×960 263 KB
1
11473×946 210 KB
Screenshot 2025-01-01 125234
Screenshot 2025-01-01 125234897×498 33.5 KB
Screenshot 2025-01-01 125244
Screenshot 2025-01-01 125244867×482 27.6 KB


  • Full logs and screenshots of the errors.
  • VM configuration details for both Core and Scale.
 
I run TrueNAS scale in the latest version of Proxmox without any issues. I am using UEFI as Kingneutron noted above. I gave it 16 GB of ram (non ballooning), 4 cores (type="host", without NUMA). My virtual drive is 64GB using VirtioSCSI single, cache=writeback, iothreads=1. I DO NOT have TPM enabled. I am passing through an ASmedia 1166 based M.2 to SATA adapter, and I modified /etc/kernel/cmdline and /etc/modules as per these instructions: https://pve.proxmox.com/wiki/PCI(e)_Passthrough

It works completely fine for me.
 
  • Like
Reactions: zurz2 and Kingneutron
Okay, looking at it my main problem for the TrueNAS scale was the uefi setup and I kept trying to turn on TPM on at the same time.
It works VIA these settings so far, (UEFI boot NO TPM)

Now I can get TrueNAS Scale to boot SOMETIMES, with a couple of retries, with it sometimes freezing at boot.
Screenshot_2025-01-04_at_4.04.22_PM.png
I still get these python errors throughout TrueNAS when trying to do anything such as press the upgrade pool button, or importing my keys.
Screenshot 2025-01-04 161640.png

May this be my IOMMU Groups? I do not have a pcie controller for SATA drives, I just passed through the onboard one. It looks like that onboard sata controller should be fine as its in its own group?

Here is my IOMMU setup and VM setup along with my IOMMU groups.

/etc/kernel/cmdline:
Code: 
root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on i915.force_probe=1 i915.enable_gvt=1 iommu=pt

/etc/default/grub:
Code: 
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on i915.force_probe=1 i915.enable_gvt=1 iommu=pt"
GRUB_CMDLINE_LINUX="


pci info:
Code: 
root@prox:~# lspci -nnk -s 0000:00:17.0
00:17.0 SATA controller [0106]: Intel Corporation Raptor Lake SATA AHCI Controller [8086:7a62] (rev 11)
        DeviceName: Onboard - SATA
        Subsystem: Micro-Star International Co., Ltd. [MSI] Raptor Lake SATA AHCI Controller [1462:7e07]
        Kernel driver in use: vfio-pci
        Kernel modules: ahci
root@prox:~# ls -l /sys/kernel/iommu_groups/8/devices/
total 0
lrwxrwxrwx 1 root root 0 Jan  4 16:46 0000:00:17.0 -> ../../../../devices/pci0000:00/0000:00:17.0

Screenshot 2025-01-04 163829.png
Screenshot 2025-01-04 163731.png
 
zurz2 said:
Okay, looking at it my main problem for the TrueNAS scale was the uefi setup and I kept trying to turn on TPM on at the same time.
It works VIA these settings so far, (UEFI boot NO TPM)

Now I can get TrueNAS Scale to boot SOMETIMES, with a couple of retries, with it sometimes freezing at boot.
View attachment 80201
I still get these python errors throughout TrueNAS when trying to do anything such as press the upgrade pool button, or importing my keys.
View attachment 80200

May this be my IOMMU Groups? I do not have a pcie controller for SATA drives, I just passed through the onboard one. It looks like that onboard sata controller should be fine as its in its own group?

Here is my IOMMU setup and VM setup along with my IOMMU groups.

/etc/kernel/cmdline:
Code: 
root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on i915.force_probe=1 i915.enable_gvt=1 iommu=pt

/etc/default/grub:
Code: 
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on i915.force_probe=1 i915.enable_gvt=1 iommu=pt"
GRUB_CMDLINE_LINUX="


pci info:
Code: 
root@prox:~# lspci -nnk -s 0000:00:17.0
00:17.0 SATA controller [0106]: Intel Corporation Raptor Lake SATA AHCI Controller [8086:7a62] (rev 11)
        DeviceName: Onboard - SATA
        Subsystem: Micro-Star International Co., Ltd. [MSI] Raptor Lake SATA AHCI Controller [1462:7e07]
        Kernel driver in use: vfio-pci
        Kernel modules: ahci
root@prox:~# ls -l /sys/kernel/iommu_groups/8/devices/
total 0
lrwxrwxrwx 1 root root 0 Jan  4 16:46 0000:00:17.0 -> ../../../../devices/pci0000:00/0000:00:17.0

View attachment 80198
Code: 
Jan  4 16:58:43 truenas kernel: middlewared (wo[5844]: segfault at 18 ip 0000000000420f0a sp 00007ffcd77692e0 error 4 in python3.11[41f000+2b6000] likely on CPU 2 (core 2, socket 0)
Jan  4 16:58:43 truenas kernel: Code: 31 ff e9 0b e3 07 00 48 89 ef e8 a1 91 0a 00 48 89 04 25 00 00 00 00 0f 0b 83 c8 ff e9 41 c1 0b 00 66 0f ef c0 c3 4c 89 6d 38 <48> 8b 04 25 18 00 00 00 0f 0b 48 8b 04 25 18 00 00 00 0f 0b 0f b6
Jan  4 18:47:39 truenas kernel: python3[8974]: segfault at 18 ip 0000000000420f0a sp 00007fffa92d7050 error 4 in python3.11[41f000+2b6000] likely on CPU 0 (core 0, socket 0)
Jan  4 18:47:39 truenas kernel: Code: 31 ff e9 0b e3 07 00 48 89 ef e8 a1 91 0a 00 48 89 04 25 00 00 00 00 0f 0b 83 c8 ff e9 41 c1 0b 00 66 0f ef c0 c3 4c 89 6d 38 <48> 8b 04 25 18 00 00 00 0f 0b 48 8b 04 25 18 00 00 00 0f 0b 0f b6
View attachment 80199
Click to expand...
update: couple of segfaults in the console as well.
Code: 
Jan  4 16:58:43 truenas kernel: middlewared (wo[5844]: segfault at 18 ip 0000000000420f0a sp 00007ffcd77692e0 error 4 in python3.11[41f000+2b6000] likely on CPU 2 (core 2, socket 0)
Jan  4 16:58:43 truenas kernel: Code: 31 ff e9 0b e3 07 00 48 89 ef e8 a1 91 0a 00 48 89 04 25 00 00 00 00 0f 0b 83 c8 ff e9 41 c1 0b 00 66 0f ef c0 c3 4c 89 6d 38 <48> 8b 04 25 18 00 00 00 0f 0b 48 8b 04 25 18 00 00 00 0f 0b 0f b6
Jan  4 18:47:39 truenas kernel: python3[8974]: segfault at 18 ip 0000000000420f0a sp 00007fffa92d7050 error 4 in python3.11[41f000+2b6000] likely on CPU 0 (core 0, socket 0)
Jan  4 18:47:39 truenas kernel: Code: 31 ff e9 0b e3 07 00 48 89 ef e8 a1 91 0a 00 48 89 04 25 00 00 00 00 0f 0b 83 c8 ff e9 41 c1 0b 00 66 0f ef c0 c3 4c 89 6d 38 <48> 8b 04 25 18 00 00 00 0f 0b 48 8b 04 25 18 00 00 00 0f 0b 0f b6
 
What are the two PCI devices that you are passing through to TrueNAS? Also, it may not make a difference, but try turning off the firewall in the VM
 
louie1961 said:
What are the two PCI devices that you are passing through to TrueNAS? Also, it may not make a difference, but try turning off the firewall in the VM
Click to expand...
Shown in the other images. here are the devices.

Onboard - SATA Intel Corporation Raptor Lake SATA AHCI Controller -> 0000:00:17.0
NVME SSD p2 -> 0000:04:00.0


also firewall is already off
 
louie1961 said:
Did you edit /etc/modules to load these modules?
vfio
vfio_iommu_type1
vfio_pci



View attachment 80209
Click to expand...

Yes, its definitely on.

Code: 
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Jan  4 16:18:47 2025 from 192.168.50.59
root@prox:~# lsmod | grep vfio
vfio_pci               16384  2
vfio_pci_core          86016  1 vfio_pci
vfio_iommu_type1       49152  1
vfio                   65536  11 vfio_pci_core,vfio_iommu_type1,vfio_pci
iommufd                94208  1 vfio
irqbypass              12288  8 vfio_pci_core,kvm
 
I would re-install TrueNAS to make sure you didn't corrupt something the first time. Fresh ISO and everything.
 
louie1961 said:
I would re-install TrueNAS to make sure you didn't corrupt something the first time. Fresh ISO and everything.
Click to expand...
SHA256 hash ensures it’s a good iso, and it’s not the first time i’ve installed TrueNAS scale lol. It’s been multiple times i’ve reinstalled it
 
Last edited:
update. same issues, i have to reboot Truenas scale multiple times to get it to eventually work, and I will still get core dumps. eliminated possible issue of onboard SATA controller, as I have now switched to a SAS HBA LSI 9211-8I controller installed and passed in instead.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom