[SOLVED] Fingerprint suddenly not valid

[William Edwards]

Both my PMG clusters are in a 'strange' state. I have two completely separate PMG clusters and they are both complaining about the same fingerprint:

fingerprint 'C2:DB:59:C8:1F:C4:C4:8E:50:1D:6F:9C:4B:B1:4F:CF:8A:B5:9B:BA:46:78:E4:EB:B5:5E:38:AC:62:90:7D:9A' not verified, abort!

I do not know where this fingerprint comes from. I grepped through /root, /etc/ssh and /etc/pmg, but this fingerprint is nowhere to be found. It is not a fingerprint associated with any of my nodes either.

I emptied `/etc/ssh/ssh_known_hosts2` and `/root/.ssh/known_hosts`, re-added host keys and am able to SSH from and to both nodes.

Why is this happening?

 

[Stoiko Ivanov]

did you change the/some certificate of PMG inside the cluster?

where exactly do you see the messages?
I think the messages relates most likely to the pinned fingerprint in pmg's cluster.conf ('/etc/pmg/cluster.conf')

it should not have anything to do with ssh host keys

I hope this helps!

 

[William Edwards]

did you change the/some certificate of PMG inside the cluster?

Yes. Certificates (/etc/pmg/pmg-api.pem and /etc/pmg/pmg-tls.pem) have been renewed and updated.

where exactly do you see the messages?

Upon logging in.

I think the messages relates most likely to the pinned fingerprint in pmg's cluster.conf ('/etc/pmg/cluster.conf')

I do not see this fingerprint in /etc/pmg/cluster.conf . Where should I change the fingerprint?

Also, does this mean manual intervention is required every single time the SSL certificate used by PMG is updated?

 

[Stoiko Ivanov]

Upon logging in.

via ssh? or via GUI?
(could you post a session-log or a screenshot)?

 

[William Edwards]

via ssh? or via GUI?
(could you post a session-log or a screenshot)?

Via GUI. See attachment.

And in `pmgcm status` as well:

proot@mgw0-0:~# pmgcm status
NAME(CID)--------------IPADDRESS----ROLE-STATE---------UPTIME---LOAD----MEM---DISK
mgw0-0(1)            $ipremoved master S   57 days 12:40   0.17    62%    38%
mgw0-1(2)            $ipremoved node   ERROR: fingerprint 'C2:DB:59:C8:1F:C4:C4:8E:50:1D:6F:9C:4B:B1:4F:CF:8A:B5:9B:BA:46:78:E4:EB:B5:5E:38:AC:62:90:7D:9A' not verified, abort!                -      -     -%     -%

 

[Stoiko Ivanov]

hmm - I think that's the fingerprint of your current /etc/pmg/pmg-api.pem
check with: `openssl x509 -in /etc/pmg/pmg-api.pem -fingerprint -noout -sha256`

if yes - you need to add that to the cluster.conf

 

[William Edwards]

hmm - I think that's the fingerprint of your current /etc/pmg/pmg-api.pem
check with: `openssl x509 -in /etc/pmg/pmg-api.pem -fingerprint -noout -sha256`

if yes - you need to add that to the cluster.conf

Yes, that was the issue. I fixed it. Thank you.

What should I do differently when I replace SSL certificates in the future? Should I always manually update `cluster.conf`?

 

[Stoiko Ivanov]

What should I do differently when I replace SSL certificates in the future? Should I always manually update `cluster.conf`?

currently this is the way to go - it will always need to be a dedicated manual step - since the cluster.conf is the trust-anchor for clustercommunication.

 

[William Edwards]

currently this is the way to go - it will always need to be a dedicated manual step - since the cluster.conf is the trust-anchor for clustercommunication.

Clear, thank you. I have internally documented this behaviour.

 

[Stoiko Ivanov]

Glad that works out for you - please mark the thread as 'SOLVED' - it will help others with a similar issue

Thanks!

 

[Gege09]

@wiliam halo william would you like to tell me how to fix them, I have some issue with fingerprint on my pmg.

 

[linux]

Hi again, we've seen this fingerprint issue recur for the first time (in the last week) since it was fixed in PVE 8.0 (we run PMG this way).

https://forum.proxmox.com/threads/fingerprint-error.126816/#post-593864 (to show the history/changelog/etc we're talking about)

Quite strange, as we'd come to enjoy the peace-of-mind that it wasn't a problem since October 2023 when we applied PVE 8.0 to all.