setting up nvidia drivers in host vs client issues?

[basing]

I've tried PCIe passthrough on Proxmox 7.2, and was working somewhat OK, but not reliably. I have 3x RTX 3070 connected, and I need them form encoding videos. The problem I was experiencing was that 1st card didn't really want to passthrough well, or it did, but host would froze. I've used info from tutorial here and checked other resources and so I've done:
in the host:

## substitue /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off,efifb:off"

update-grub

#add following lines to /etc/modules
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf
echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf
echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf
echo "options vfio-pci ids=10de:2482,10de:228b disable_vga=1"> /etc/modprobe.d/vfio.conf

update-initramfs -u
reboot

On VM
Modified Ubuntu 20 VM to use OMVF bios, went into the Bios and disabled secure boot. Added PCIe devices under Hardware tab in Proxmox website setup

#remove nvidia drivers
apt remove nvidia-drivers-510
apt autoremove
apt-get --purge remove xserver-xorg-video-nouveau
apt install build-essential libglvnd-dev pkg-config

#wget Nvidia driver from their website and run it

and it was working, apart from 1st GPU giving me issues.
So I've tried to install driver directly on host and disable PCIe passthrough on VM, by:

## substitue /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
update-grub

#Remove following lines to /etc/modules
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

rm -f /etc/modprobe.d/iommu_unsafe_interrupts.conf
rm -f /etc/modprobe.d/kvm.conf
echo "blacklist nouveau" > /etc/modprobe.d/blacklist.conf
rm -f /etc/modprobe.d/vfio.conf

apt install pve-headers python3 python3-pip git build-essential dkms jq --fix-missing

update-initramfs -u
reboot
#wget Nvidia driver from their website and run it

Nvidia driver loaded properly, but when I've tried using ffmpeg with nvenc it would freeze and had to do hard reset. Or it would take very long to even show lspci |grep VGA, and would also freeze process when I would do nvidia-smi.

Looks like it was still locked by VM or something wasn't setup properly. So I reverted back, but now even VM freezes as soon as I issue nvidia-smi

What could be the issue?