I did the following
apt remove nvidia-driver
apt purge *nvidia*
apt autoremove
apt clean
Then the nvidia installer https://www.nvidia.com/Download/driverResults.aspx/224022/en-us
I have the same issue. So far the only way I have gotten around it is by waiting for everything to start up, then downing the physical nic for the wan connection and then starting it up again, then renew dhcp wan lease on pfsense. It seems to be a timing issue of sorts, but I haven't figured out...