Problem with TOTP after PVE 8 upgrade

[Quasar90]

Hello,

i have a Problem with the TOTP after upgrading to PVE 8.
After the upgrade to PVE 8 the TOTP for the GUI Users has been queried on every Login, but only for about 24 hours. After that the Login was possible without the query of the TOTP. In the TwoFactor settings page in the GUI the TOTP was still configured and set to "enabled". I have recreate the TOTP for my User and again the TOTP was queried for about 24 hours at every Login, but not more after the 24 hours.

Has anyone an idea why this happened or is this a Bug in the PVE 8 ?

 

[wbumiller]

Do you happen to be using LDAP with realm sync? If so, can you post the settings (ideally the the section from `/etc/pve/domains.cfg`) and compare `/etc/pve/user.cfg` between working and non-working?

If not, can you please answer all the following:
- Describe your setup a bit more (how many nodes, what services you run on the host in addition to PVE, ...)
- How did you install PVE
- Compare the contents of `/etc/pve/priv/tfa.cfg` when it's working and after it stopped working? (any small change can matter)
- Compare the contents of `/etc/pve/user.cfg` when it's working and after it stopped working? (any small change can matter)
- Check the logs while logging in when it's working and when it's not working?
- Tell us if there are multiple users and whether all experience the same issue or just specific ones
- and what kind of users they are (what *realm* (PVE/PAM/ldap/...) the are using, and how it is configured)
- And finally: do the users have multiple 2nd factors or only TOTP?

 

[Quasar90]

Do you happen to be using LDAP with realm sync? If so, can you post the settings (ideally the the section from `/etc/pve/domains.cfg`) and compare `/etc/pve/user.cfg` between working and non-working?

If not, can you please answer all the following:
- Describe your setup a bit more (how many nodes, what services you run on the host in addition to PVE, ...)
- How did you install PVE
- Compare the contents of `/etc/pve/priv/tfa.cfg` when it's working and after it stopped working? (any small change can matter)
- Compare the contents of `/etc/pve/user.cfg` when it's working and after it stopped working? (any small change can matter)
- Check the logs while logging in when it's working and when it's not working?
- Tell us if there are multiple users and whether all experience the same issue or just specific ones
- and what kind of users they are (what *realm* (PVE/PAM/ldap/...) the are using, and how it is configured)
- And finally: do the users have multiple 2nd factors or only TOTP?

Hi,

the answers for your questions:
-it`s a 2 node Cluster with an additional Debian 11 QDevice, one of the Nodes has a Nvidia GPU and on this node we installed the NVIDIA Driver 535.54.03 for GPU usage in LXC,
-there is no further Software or Service running beyond the PVE
-orinally the Nodes were installed with PVE 7 from CD and boot with systemd-boot
-the problem exist for all User where the TOTP ist configured and all of this users are AD-Realm user
-yes the AD-Sync since PVE 8 is active
-the TOTP is the only active 2nd factor for the users

The Check of the configs when it`s working and not and an intensive Log Check is ongoing, but i hope the answered question already help.

kind regards

 

[Quasar90]

Hi,

the answers for your questions:
-it`s a 2 node Cluster with an additional Debian 11 QDevice, one of the Nodes has a Nvidia GPU and on this node we installed the NVIDIA Driver 535.54.03 for GPU usage in LXC,
-there is no further Software or Service running beyond the PVE
-orinally the Nodes were installed with PVE 7 from CD and boot with systemd-boot
-the problem exist for all User where the TOTP ist configured and all of this users are AD-Realm user
-yes the AD-Sync since PVE 8 is active
-the TOTP is the only active 2nd factor for the users

The Check of the configs when it`s working and not and an intensive Log Check is ongoing, but i hope the answered question already help.

kind regards

I have found out, that the realm sync job ist the problem. I recreat the TOTP for my user and saw in the "user.cfg" that my user has an "x" between the last to "::" and after the realm sync the little "x" is gone and no TOTP query happened on login. I also tested to manually set this "x" and from that on the TOTP is queried again at login.

I also belive that i found a solution. In the realm sync setting under the "remove vanished options" the option "Remove vanished properties from synced users." is removing this little "x" from the users in the "user.cfg". In my opinion this options is a little miss explained. The related option has not vanished but added. This option overrides the existing properties for all synced users and groups. If this option for the sync is working right then the declaration is wrong otherwise the feature has a bug.

 

[wbumiller]

Thanks for confirming.

Yeah this is a regression and the sync code needs fixing (or the changes to it better documented, as in theory this is the 'keys' property which could be set explicitly, but this way of doing TFA is very old and not flexible and not really editable by the users themselves...)

Furthermore, the `x` part is mostly a leftover from the conversion from that old way where could only have tiny tfa keys of a single type directly in the user config, with the 'x' indicating the real data is somewhere else. These days we should just ignore whether that 'x' is there and load the actual tfa data as single source of truth.