SMTP you can find in the search, many people tried different profiles. In my Advancing PMG Thread I just filter two cases, killmasta93 also block some more using redmail fail2ban samples. SSH is shipped in the default settings of fail2ban, you just need to activate, GUI you may find some help in the search for Proxmox VE GUI (which is technically similar). Honestly instead of banning all the script kiddies out there, I would set up a firewall and VPN and allow only connections to SSH and GUI via VPN, for SMTP as well as closing down with VPN SSH and GUI you can find all information in my Advancing PMG Thread.