I ran the following commands via monitor:
change vnc 0.0.0.0:100,password
set_password vnc foobar1
Also I tried different vnc port (>100) and my own password instead of foobar1 .
When I ran these commands at Proxmox 5.1 ,I connected successfully to all VMs using TightVNC. After I upgraded Proxmox to 5.3 I ran these commands and I got the message "VNC Authentication Failed" to specific VMs.