Your monitor input select might be set to "auto"
selecting diplayport input on your monitor instead of auto might prevent it to "fall back" to your proxmox
In my case, "master node " hosted the vm disks ( exposed via NFS) so without it, cluster would be down anyway.
But yes, this setup is not very compatible with the spirit of multi master qorum
https://forum.proxmox.com/threads/homelab-3-nodes-cluster-2-diskless-and-one-exposing-storage-via-nfs.68587/#post-307776
more or less the same kind of idea. changing the number of votes worked very well IN MY HOMELAB
My experience is the oposite:
unmanaged switches don't dare to look at vlan tags and act as all ports being "hybrid" ports. ( forward tagged and untagged packets "everywhere" )
This is from memory ( I've no access to ESXI anymore).... and my memory is slow ( saw your post yesterday and trying since then to remember the word "promiscous"..)
Glad you could solve your issue !
EC