Ahhhh, this makes so much sense then. Heh. I am a dumbass. I switched my SSH to be on a different port and that's when my problems began. Of course now I know why.
Thanks for that! I shall investigate how to switch systemd socket to look at a different port instead, if it's possible