If you use Linux PAM standard authentication, then you have to create a system user manually, e.g. with `useradd`. Proxmox VE does not do that. You can then create a user in Proxmox VE for the GUI on top of it.
Could you provide the exact steps that you tried? This might be easier with command line.
Please try this:
When you add a user it should appear in /etc/pve/user.cfg. You can use the following commands to create a user named "testuser" in Proxmox VE authentication server and verify that it was really created. The second command should print "user:testuser@pve:1:0::::::"
Code:
pveum user add testuser@pve
cat /etc/pve/user.cfg | grep testuser