I hope there is a better way but I currently worked around this by adding all devices from a group to vfio
IOMMU Group 10:
-e 0a:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Raven/Raven2/Fenghuang HDMI/DP Audio Controller [1002:15de]
-e 0a:00.2 Encryption controller...