Feels like there should be some closure.
I tried 4 different kernel, 2 different DIMMs (both of them 0 issues in multiple memtests), FW update for SSD, microcode update, disabling IOMMU/passthrough for pfSense/converting all NICs to virtual bridges, switching drivers from paravirtualized to E1000, fresh installs, running only pfSense VM without any passthrough and honestly bunch of other things I can't even remember since I've been fighting with this for last 6 months.
None of it worked, ended up getting crashes 3 times a day. So I gave up.
What finally did the trick for me was switch to vSphere. I replicated same setup I had on Proxmox - no crash so far (15 days and counting).
I still think crashes were somehow related to virtual bridges, but how exactly is beyond my expertise.