Since proxmox doesn't even install an X server out of the box, this sounds like a "you" problem.
-You- need to be more careful and make sure you're dialed into the right box. Maybe use a laptop or tablet instead of a small-screen phone.
Proxmox does support clustering and HA, maybe look into that so you don't have a single-server point of failure.
