macOS 10.13 Ventura hackintosh VM makes passthroughed FL1100 USB controller unusable among kernel 5.13~6.1

[fty1777]

My PC has a B660 chipset and there is only one Intel USB controller which is in the same IOMMU group as System RAM and lacks available reset mechanism. Actually I could passthrough the intel USB controller and use it with all basic USB 2.0 functions work as expected. But USB 3.0 could only work on the first boot of any VM with the controller passthroughed. Once the VM was shutdown or rebooted, the USB 3.0 would no longer work until I reboot the host. Which is quite annoying.

I thought a seperate USB controller connected through PCI-e would help so I bought a FL1100 PCI-e 1x USB controller with 4 USB type-A ports. It works perfect when passthoughed to a Windows 11 vm, all the functions including USB 2.0 and 3.0 works like a charm. I can reboot the windows VM as many times as I want and the controller just keep working correctly and fluently, and can even be "removed and rescanned" to be used in Proxmox VE host to connect USB storage devices to the host -- I can put the controller anywhere (used inside VM or by host) at anytime as I want and switch it back and forth without any error. No driver blacklist or vfio related modprobe conf modification is needed and it just works.

However, if I passthrough it to a macOS 10.13 VM, it works fine the first time the VM starts until the macOS is shutdown or rebooted. Then the controller can not be passed-through to any VM anymore. It looks fine in lspci -nnv at this point:

08:00.0 USB controller [0c03]: Fresco Logic FL1100 USB 3.0 Host Controller [1b73:1100] (rev 10) (prog-if 30 [XHCI])
        Subsystem: Fresco Logic FL1100 USB 3.0 Host Controller [1b73:1100]
        Flags: fast devsel, IRQ 18, IOMMU group 21
        Memory at 97300000 (64-bit, non-prefetchable) [size=64K]
        Memory at 97311000 (64-bit, non-prefetchable) [size=4K]
        Memory at 97310000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/8 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [b0] MSI-X: Enable- Count=8 Masked-
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: xhci_pci

At this point I can still remove and rescan by

echo 1 > /sys/bus/pci/devices/0000\:08\:00.0/remove
echo 1 > /sys/bus/pci/rescan

and it is brought back to the host and xhci driver will take over, as shown in lspci -nnv:

08:00.0 USB controller [0c03]: Fresco Logic FL1100 USB 3.0 Host Controller [1b73:1100] (rev 10) (prog-if 30 [XHCI])
        Subsystem: Fresco Logic FL1100 USB 3.0 Host Controller [1b73:1100]
        Flags: bus master, fast devsel, latency 0, IRQ 18, IOMMU group 21
        Memory at 97300000 (64-bit, non-prefetchable) [size=64K]
        Memory at 97310000 (64-bit, non-prefetchable) [size=4K]
        Memory at 97311000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/8 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [b0] MSI-X: Enable+ Count=8 Masked-
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci

In dmesg I can see that my attached USB devices are enumerated correctly:

[49281.081635] pci 0000:08:00.0: [1b73:1100] type 00 class 0x0c0330
[49281.081666] pci 0000:08:00.0: reg 0x10: [mem 0x97300000-0x9730ffff 64bit]
[49281.081685] pci 0000:08:00.0: reg 0x18: [mem 0x97311000-0x97311fff 64bit]
[49281.081703] pci 0000:08:00.0: reg 0x20: [mem 0x97310000-0x97310fff 64bit]
[49281.081808] pci 0000:08:00.0: supports D1
[49281.081810] pci 0000:08:00.0: PME# supported from D0 D1 D3hot
[49281.081954] pci 0000:08:00.0: Adding to iommu group 21
[49281.082052] pci 0000:08:00.0: BAR 0: assigned [mem 0x97300000-0x9730ffff 64bit]
[49281.082073] pci 0000:08:00.0: BAR 2: assigned [mem 0x97310000-0x97310fff 64bit]
[49281.082086] pci 0000:08:00.0: BAR 4: assigned [mem 0x97311000-0x97311fff 64bit]
[49281.082361] xhci_hcd 0000:08:00.0: xHCI Host Controller
[49281.082366] xhci_hcd 0000:08:00.0: new USB bus registered, assigned bus number 3
[49281.084497] xhci_hcd 0000:08:00.0: hcc params 0x200071e1 hci version 0x100 quirks 0x0000000000000410
[49281.084804] xhci_hcd 0000:08:00.0: xHCI Host Controller
[49281.084807] xhci_hcd 0000:08:00.0: new USB bus registered, assigned bus number 4
[49281.084809] xhci_hcd 0000:08:00.0: Host supports USB 3.0 SuperSpeed
[49281.084897] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.15
[49281.084899] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[49281.084900] usb usb3: Product: xHCI Host Controller
...
[49284.441812] scsi 8:0:0:0: Direct-Access     SanDisk  Extreme          0001 PQ: 0 ANSI: 6
[49284.441965] sd 8:0:0:0: Attached scsi generic sg0 type 0
[49284.552965] usb 4-2.4: New USB device found, idVendor=0b95, idProduct=1790, bcdDevice= 2.00
[49284.552969] usb 4-2.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[49284.552971] usb 4-2.4: Product: AX88179A
[49284.552972] usb 4-2.4: Manufacturer: ASIX
[49284.552973] usb 4-2.4: SerialNumber: 006AB2A0
[49284.617407] sd 8:0:0:0: [sda] 61282631 512-byte logical blocks: (31.4 GB/29.2 GiB)
[49284.617686] sd 8:0:0:0: [sda] Write Protect is off
[49284.617688] sd 8:0:0:0: [sda] Mode Sense: 33 00 00 08
[49284.617969] sd 8:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[49284.631967]  sda: sda1 sda2
[49284.632802] sd 8:0:0:0: [sda] Attached SCSI disk
[49284.649708] usb 3-3.2: USB disconnect, device number 7
[49284.900706] x86/split lock detection: #AC: CPU 5/KVM/107893 took a split_lock trap at address: 0x754c6793
[49284.903790] x86/split lock detection: #AC: CPU 5/KVM/107893 took a split_lock trap at address: 0x754c6793
[49285.188228] ax88179_178a 4-2.4:1.0 (unnamed net_device) (uninitialized): Failed to read reg index 0x0040: -32
[49285.499955] ax88179_178a 4-2.4:1.0 eth0: register 'ax88179_178a' at usb-0000:08:00.0-2.4, ASIX AX88179 USB 3.0 Gigabit Ethernet, f8:e4:3b:6a:b2:a0
[49285.501534] ax88179_178a 4-2.4:1.0 enxf8e43b6ab2a0: renamed from eth0

If i insert a flash drive or USB disk, I can see it from fdisk -l and lsusb. I can also mount a flash drive volume to access the filesystem on it without any problem.

HOWEVER, once I passthrough the FL1100 controller to any VM and start the VM, it immediately becomes unusable. With dmesg:

[49581.418760] vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
[49581.439709] vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
[49581.455727] vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
[49581.582410] vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
[49581.582695] vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
[49581.582714] vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
[49581.582832] vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
[49581.600759] vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
[49581.616518] vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
[49581.616689] vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs
[49581.741303] vfio-pci 0000:08:00.0: vfio_bar_restore: reset recovery - restoring BARs

and lspci -nnv shows:

08:00.0 USB controller [0c03]: Fresco Logic FL1100 USB 3.0 Host Controller [1b73:1100] (rev ff) (prog-if ff)
        !!! Unknown header type 7f
        Kernel driver in use: vfio-pci
        Kernel modules: xhci_pci

Then I can never use it again until:

• the host is rebooted, which is quite unacceptable since I have a ubuntu VM running services, or
• the host is SUSPENDED then woken up by pressing the power button, which suprised me and is the workaround solution I'm using now. The controller will be automatically rescanned and usable in host. I can passthrough it to VM and it works great until it suffers a "macOS running" -- once I use it in the macOS VM, it will again becomes unusable if:
  • the VM is shutdown and the controller is passed-through to any VM and the VM starts (including the same macOS VM), or
  • the VM is rebooted, which I think is equivalent to shutting down the macOS VM and then start it.

In conclusion:

• the FL1100 USB controller "loses its ability to be passed-through to any VM again" at the time it is used in the macOS VM(to be more precise, at the time the macOS OS is booted. Just starting the VM, entering the OpenCore bootloader or BIOS and shutting down the VM without booting macOS OS will not break the controller).
• Before its passed-through again (including rebooting the VM), it works great in the macOS, and even works in proxmox ve host if I "shutdown the VM, remove the PCIe device and rescan" to let xhci_hcd to take control. It works perfect if I do not start any VM with it passed-through.
• Both rebooting the host and "suspending the host and waking it up" give back the controller its ability to be passed-through. But it will be broken again by booting macOS.

I conclude the controller to have 3 states (I am not sure, just according to its behavior):

• Normal: Before used in macOS, after host reboot or after suspending and waking up the host. Everything works like a charm.
• Broken: After used in macOS. It still works in macOS, and works in pve host if I shutdown the macOS, remove the controller and rescan to load the xhci_hcd driver.
• Down: If the controller is "broken" and passed-through again, it goes "down". dmesg shows vfio_bar_restore: reset recovery - restoring BARs and lspci -nnv shows 08:00.0 USB controller [0c03]: Fresco Logic FL1100 USB 3.0 Host Controller [1b73:1100] (rev ff) (prog-if ff) !!! Unknown header type 7f

The state transition: Normal -----used in macOS -----> Broken -----passed-through again-----> Down -----reboot/suspend+wake-----> Normal

Should I buy another USB PCIe card with FL1100 or other controller IC?
I have tried to upgrade/downgrade the kernel (tried 5.13/5.15/5.19/6.1) but the problem persists.

I haved searched for days without any progress. New to this forum and if any additional information is needed I am glad to collect and provide.
Any solution or idea would be really appreciated.

 

[fty1777]

UPDATE: Killing the kvm process of the macOS VM will prevent the FL1100 controller from becoming "broken". Seems like something during the shutdown process of macOS breaks something about the controller's state.

However, killing the process every time I want to shutdown the VM afaik is not safe and may cause data loss or something else so it's not a good solution.

 

[izatzman]

Did you ever figure out a fix for this?
I just ran into the exact same issue.