Some more informations:
With PVE 7, if I use a wrong client ID or secret, I am redirected to keycloak where i got a error, which is the expected behavior.
I suppose this should be the same with PBS.
So, I guess this is not a config mistake on keycloak side.