Make a backup and do not use the nvidia hook until you uninstall the manually installed driver.Fantastic, I'll give that a try in a fresh container.
Btw. do you have a suggestion on how to clean up what was already installed in the existing ones (with --no-kernel-modules)? I don't want to risk moving everything, but I don't want to keep doing this double install when I update drivers either
Run the installer and uninstall the drivers, then you "should" be able to use the bind mound options in the config, delete the card permissions as well. You don't need to handle the card permissions either, as the nvidia hook handles all that for you automatically.
However, my recommendation would be to spin up a new container in case something gets leftover.