I hope there is a better way but I currently worked around this by adding all devices from a group to vfio
IOMMU Group 10:
-e 0a:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Raven/Raven2/Fenghuang HDMI/DP Audio Controller [1002:15de]
-e 0a:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor [1022:15df]
-e 0a:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Raven USB 3.1 [1022:15e0]
-e 0a:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Raven USB 3.1 [1022:15e1]
-e 0a:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller [1022:15e3]
and then unbind and bind the driver in /etc/rc.local
echo '0000:0a:00.3' > /sys/bus/pci/drivers/xhci_hcd/unbind
echo '0000:0a:00.3' > /sys/bus/pci/drivers/vfio-pci/bind
and then add the pci device to the VM.