Yes, for security reasons you should prefer setting up a VPN server so your PVE server isn't public and attackable.
Have a look at the manual of your router. They often can act as a VPN server. If your router is too crappy that it doesn't support such features you could also roll out a proper router in a VM like pfsense/OPNsense.