PMG certificates for joining cluster

Xela · Jan 30, 2023

Dear Forum

As I can see when joining a cluster the certificates are in RSA (/etc/pmg/cluster.conf). Is there a way to use ed25519 instead of RSA? Is there a way to regenerate new RSA certificates for joining the cluster?

I always create after a fresh install new certificates (ed25519). In this PMG setup also. Now when joining the cluster I see that there are still RSA pubkeys instead of the new generated ed25519.

Best regards

Stoiko Ivanov · Jan 31, 2023

currently rsa as key algorithm is hardcoded in a few places - e.g.:
https://git.proxmox.com/?p=pmg-api....7d7e1ec524a27c844c110f05391c8b50;hb=HEAD#l231

as this still is considered secure we did not rush to adapt the code.

I hope this explains it!

Xela · Jan 31, 2023

Dear Stoiko,

thank you for your support. Is it possible to create new RSA certs in case the initiated ones heve been deleted? Or is it necessary to start with a new fresh install?

Best regards

Stoiko Ivanov · Jan 31, 2023

Xela said:
thank you for your support. Is it possible to create new RSA certs in case the initiated ones heve been deleted? Or is it necessary to start with a new fresh install?

the "certs" are ssh-keys...
one is the ssh-host-key - if you deleted that see e.g. https://serverfault.com/questions/471327/how-to-change-a-ssh-host-key (or any of the many other tutorials)
the other is an ssh-key of the root user - and if it's not present it should get generated when running pmgcm create or join

Xela · Jan 31, 2023

Please excuse me, of course ssh-keys (RSA) not certs. I will try to delete the keys and will see if new ones will be generated.

Stoiko Ivanov · Jan 31, 2023

Xela said:
Please excuse me,

Nothing to apologize for! I just wrote it to not cause confusion to future readers!
Let us know if it works out as expected!
Thanks!

Xela · Jan 31, 2023

Dear Stoiko,

I deleted all ssh-keys in /root/.ssh; created on master (mx1) cluster and a new RSA SSH-key was generated. After that I deleted also all ssh-keys on 2nd node and tried to join (mx2) to the cluster with given IP, password and fingerprint. Here also a new RSA SSH-key was generated but no success to join the cluster:

Code:

@mx1:/etc/pmg# pmgcm status
NAME(CID)------------IPADDRESS----ROLE-STATE---------UPTIME---LOAD----MEM---DISK
mx1(1)               IPmx1.IPmx1.IPmx1.IPmx1  master S           00:24   0.08    60%    21%
mx2(2)               IPmx2.IPmx2.IPmx2.IPmx2  node   ERROR: 401 permission denied - invalid PMG ticket                -      -     -%     -%

Code:

@mx2:/etc/pmg# pmgcm status
NAME(CID)------------IPADDRESS----ROLE-STATE---------UPTIME---LOAD----MEM---DISK
mx1(1)               IPmx1.IPmx1.IPmx1.IPmx1  master ERROR: 401 permission denied - invalid PMG ticket                -      -     -%     -%
mx2(2)               IPmx2.IPmx2.IPmx2.IPmx2  node   S           00:24   0.00    60%    21%

The logs show:

Code:

Jan 31 14:56:57 mx1 pmgdaemon[1111]: successful auth for user 'root@pam'
Jan 31 14:57:05 mx1 pmgtunnel[1653]: starting tunnel 1709 IPmx1.IPmx1.IPmx1.IPmx1
Jan 31 14:57:05 mx1 pmgtunnel[1653]: tunnel finished 1709 IPmx1.IPmx1.IPmx1.IPmx1
Jan 31 14:57:24 mx1 pmgtunnel[1653]: restarting crashed tunnel 1716 IPmx1.IPmx1.IPmx1.IPmx1
Jan 31 14:57:24 mx1 pmgtunnel[1653]: tunnel finished 1716 IPmx1.IPmx1.IPmx1.IPmx1
Jan 31 14:57:43 mx1 pmgtunnel[1653]: restarting crashed tunnel 1722 IPmx1.IPmx1.IPmx1.IPmx1
Jan 31 14:57:43 mx1 pmgtunnel[1653]: tunnel finished 1722 IPmx1.IPmx1.IPmx1.IPmx1
Jan 31 14:57:47 mx1 pmgmirror[1659]: starting cluster synchronization
Jan 31 14:57:47 mx1 pmgmirror[1659]: database sync 'mx2' failed - DBI connect('dbname=Proxmox_ruledb;host=/run/pmgtunnel;port=2;','root',...) failed: could not connect to server: No such file or directory#012#011Is the server running locally and accepting#012#011connections on Unix domain socket "/run/pmgtunnel/.s.PGSQL.2"? at /usr/share/perl5/PMG/DBTools.pm line 66.
Jan 31 14:57:47 mx1 pmgmirror[1659]: cluster synchronization finished  (1 errors, 0.06 seconds (files 0.00, database 0.06, config 0.00))
Jan 31 14:57:49 mx1 pmgpolicy[1664]: starting policy database maintenance (greylist, rbl)
Jan 31 14:57:49 mx1 pmgpolicy[1664]: end policy database maintenance (22 ms, 6 ms)
Jan 31 14:58:03 mx1 pmgtunnel[1653]: restarting crashed tunnel 1731 IPmx1.IPmx1.IPmx1.IPmx1
Jan 31 14:58:03 mx1 pmgtunnel[1653]: tunnel finished 1731 IPmx1.IPmx1.IPmx1.IPmx1

=====

Jan 31 14:56:57 mx2 pmgdaemon[1149]: starting task UPID:mx2:0000069A:0000E65F:63D91E29:clusterjoin::root@pam:
Jan 31 14:56:57 mx2 systemd[1]: Stopping Proxmox Mail Gateway Policy Daemon...
Jan 31 14:56:57 mx2 pmgpolicy[1123]: 2023/01/31-14:56:57 Server closing!
Jan 31 14:56:57 mx2 systemd[1]: pmgpolicy.service: Succeeded.
Jan 31 14:56:57 mx2 systemd[1]: Stopped Proxmox Mail Gateway Policy Daemon.
Jan 31 14:56:57 mx2 systemd[1]: pmgpolicy.service: Consumed 2.529s CPU time.
Jan 31 14:56:57 mx2 systemd[1]: Stopping Proxmox Mail Gateway Database Mirror Daemon...
Jan 31 14:56:59 mx2 pmgmirror[1124]: received signal TERM
Jan 31 14:56:59 mx2 pmgmirror[1124]: server closing
Jan 31 14:56:59 mx2 pmgmirror[1124]: server stopped
Jan 31 14:57:00 mx2 systemd[1]: pmgmirror.service: Succeeded.
Jan 31 14:57:00 mx2 systemd[1]: Stopped Proxmox Mail Gateway Database Mirror Daemon.
Jan 31 14:57:00 mx2 systemd[1]: pmgmirror.service: Consumed 3.153s CPU time.
Jan 31 14:57:00 mx2 systemd[1]: Stopping Proxmox Mail Gateway Cluster Tunnel Daemon...
Jan 31 14:57:01 mx2 pmgtunnel[712]: received signal TERM
Jan 31 14:57:01 mx2 pmgtunnel[712]: server closing
Jan 31 14:57:01 mx2 pmgtunnel[712]: server stopped
Jan 31 14:57:02 mx2 systemd[1]: pmgtunnel.service: Succeeded.
Jan 31 14:57:02 mx2 systemd[1]: Stopped Proxmox Mail Gateway Cluster Tunnel Daemon.
Jan 31 14:57:02 mx2 systemd[1]: pmgtunnel.service: Consumed 3.271s CPU time.
Jan 31 14:57:02 mx2 pmgdaemon[1149]: successful auth for user 'root@pam'
Jan 31 14:57:02 mx2 systemd[1]: Stopping Proxmox SMTP Filter Daemon...
Jan 31 14:57:03 mx2 pmg-smtp-filter[1135]: 2023/01/31-14:57:03 Server closing!
Jan 31 14:57:03 mx2 systemd[1]: pmg-smtp-filter.service: Succeeded.
Jan 31 14:57:03 mx2 systemd[1]: Stopped Proxmox SMTP Filter Daemon.
Jan 31 14:57:03 mx2 systemd[1]: pmg-smtp-filter.service: Consumed 7.270s CPU time.
Jan 31 14:57:03 mx2 systemd[1]: Starting Proxmox SMTP Filter Daemon...
Jan 31 14:57:05 mx2 pmg-smtp-filter[1714]: Process Backgrounded
Jan 31 14:57:05 mx2 pmg-smtp-filter[1714]: 2023/01/31-14:57:05 main (type Net::Server::PreFork) starting! pid(1714)
Jan 31 14:57:05 mx2 pmg-smtp-filter[1714]: Binding to TCP port 10023 on host 127.0.0.1 with IPv4
Jan 31 14:57:05 mx2 pmg-smtp-filter[1714]: Binding to TCP port 10024 on host 127.0.0.1 with IPv4
Jan 31 14:57:05 mx2 pmg-smtp-filter[1714]: Group Not Defined.  Defaulting to EGID '0'
Jan 31 14:57:05 mx2 pmg-smtp-filter[1714]: User Not Defined.  Defaulting to EUID '0'
Jan 31 14:57:05 mx2 pmg-smtp-filter[1714]: Setting up serialization via flock
Jan 31 14:57:05 mx2 pmg-smtp-filter[1714]: Filter daemon (re)started (max. 33 processes)
Jan 31 14:57:05 mx2 systemd[1]: Started Proxmox SMTP Filter Daemon.
Jan 31 14:57:05 mx2 systemd[1]: Starting Proxmox Mail Gateway Cluster Tunnel Daemon...
Jan 31 14:57:06 mx2 pmgtunnel[1721]: starting server
Jan 31 14:57:06 mx2 pmgtunnel[1721]: starting tunnel 1723 IPmx2.IPmx2.IPmx2.IPmx2
Jan 31 14:57:06 mx2 systemd[1]: Started Proxmox Mail Gateway Cluster Tunnel Daemon.
Jan 31 14:57:06 mx2 systemd[1]: Starting Proxmox Mail Gateway Database Mirror Daemon...
Jan 31 14:57:06 mx2 pmgtunnel[1721]: tunnel finished 1723 IPmx2.IPmx2.IPmx2.IPmx2
Jan 31 14:57:08 mx2 pmgmirror[1728]: starting server
Jan 31 14:57:08 mx2 systemd[1]: Started Proxmox Mail Gateway Database Mirror Daemon.
Jan 31 14:57:08 mx2 systemd[1]: Starting Proxmox Mail Gateway Policy Daemon...
Jan 31 14:57:09 mx2 pmgpolicy[1733]: Process Backgrounded
Jan 31 14:57:09 mx2 pmgpolicy[1733]: 2023/01/31-14:57:09 main (type Net::Server::PreForkSimple) starting! pid(1733)
Jan 31 14:57:09 mx2 pmgpolicy[1733]: Binding to TCP port 10022 on host 127.0.0.1 with IPv4
Jan 31 14:57:09 mx2 pmgpolicy[1733]: Group Not Defined.  Defaulting to EGID '0'
Jan 31 14:57:09 mx2 pmgpolicy[1733]: User Not Defined.  Defaulting to EUID '0'
Jan 31 14:57:09 mx2 pmgpolicy[1733]: Setting up serialization via flock
Jan 31 14:57:09 mx2 pmgpolicy[1733]: Policy daemon (re)started
Jan 31 14:57:09 mx2 pmgpolicy[1733]: Beginning prefork (5 processes)
Jan 31 14:57:09 mx2 pmgpolicy[1733]: Starting "5" children
Jan 31 14:57:09 mx2 systemd[1]: Started Proxmox Mail Gateway Policy Daemon.
Jan 31 14:57:09 mx2 pmgdaemon[1690]: syncing master configuration from 'IPmx2.IPmx2.IPmx2.IPmx2' failed: rsync error: unexplained error (code 255) at io.c(228) [Receiver=3.2.3]
Jan 31 14:57:09 mx2 pmgdaemon[1149]: end task UPID:mx2:0000069A:0000E65F:63D91E29:clusterjoin::root@pam: syncing master configuration from 'IPmx2.IPmx2.IPmx2.IPmx2' failed: rsync error: unexplained error (code 255) at io.c(228) [Receiver=3.2.3]
Jan 31 14:57:09 mx2 pmg-smtp-filter[1714]: Beginning prefork (2 processes)
Jan 31 14:57:09 mx2 pmg-smtp-filter[1714]: Starting "2" children
Jan 31 14:57:25 mx2 pmgtunnel[1721]: restarting crashed tunnel 1764 IPmx2.IPmx2.IPmx2.IPmx2
Jan 31 14:57:25 mx2 pmgtunnel[1721]: tunnel finished 1764 IPmx2.IPmx2.IPmx2.IPmx2
Jan 31 14:57:39 mx2 pmg-smtp-filter[1714]: starting database maintenance
Jan 31 14:57:39 mx2 pmg-smtp-filter[1714]: end database maintenance (68 ms)
Jan 31 14:57:44 mx2 pmgtunnel[1721]: restarting crashed tunnel 1774 IPmx2.IPmx2.IPmx2.IPmx2
Jan 31 14:57:44 mx2 pmgtunnel[1721]: tunnel finished 1774 IPmx2.IPmx2.IPmx2.IPmx2
Jan 31 14:58:03 mx2 pmgtunnel[1721]: restarting crashed tunnel 1780 IPmx2.IPmx2.IPmx2.IPmx2
Jan 31 14:58:03 mx2 pmgtunnel[1721]: tunnel finished 1780 IPmx2.IPmx2.IPmx2.IPmx2
Jan 31 14:58:22 mx2 pmgtunnel[1721]: restarting crashed tunnel 1788 IPmx2.IPmx2.IPmx2.IPmx2
Jan 31 14:58:22 mx2 pmgtunnel[1721]: tunnel finished 1788 IPmx2.IPmx2.IPmx2.IPmx2

Stoiko Ivanov · Jan 31, 2023

Xela said:
I deleted all ssh-keys in /root/.ssh; created on master (mx1) cluster and a new RSA SSH-key was generated. After that I tried to join (mx2) to the cluster with given IP, password and fingerprint. No success:

was mx2 part of that cluster before?
if yes you need to manually adapt the cluster.conf and /root/.ssh/authorized_keys on both hosts

Xela · Jan 31, 2023

Yes and I checked that before and now again. SSH as root from one to the other node works:

Code:

@mx1:~/.ssh# ssh -i /root/.ssh/id_rsa root@IPmx2.IPmx2.IPmx2.IPmx2
Linux mx2 6.1.2-1-pve #1 SMP PREEMPT_DYNAMIC PVE 6.1.2-1 (2023-01-10T00:00Z) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Jan 31 14:54:25 2023 from IPmx1.IPmx1.IPmx1.IPmx1


@mx2:~/.ssh# ssh -i /root/.ssh/id_rsa root@IPmx1.IPmx1.IPmx1.IPmx1
Linux mx2 6.1.2-1-pve #1 SMP PREEMPT_DYNAMIC PVE 6.1.2-1 (2023-01-10T00:00Z) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Jan 31 14:55:28 2023 from IPmx2.IPmx2.IPmx2.IPmx2

Stoiko Ivanov · Jan 31, 2023

did you reboot the nodes after changing the keys?
(or at least restart pmgtunnel and pmgmirror)?

Xela · Jan 31, 2023

Both nodes have been rebooted and cluster.conf has been checked - ok. Still: ERROR: 401 permission denied - invalid PMG ticket

Stoiko Ivanov · Jan 31, 2023

Xela said:
Both nodes have been rebooted and cluster.conf has been checked - ok. Still: ERROR: 401 permission denied - invalid PMG ticket

any messages from pmgtunnel/pmgmirror which point to the issue?

check that /etc/pmg/pmg-authkey* match on both nodes:
https://pmg.proxmox.com/pmg-docs/pmg-admin-guide.html#_keys_and_certificates

else - you could also just delete mx2 and rejoin it afterwards

Xela · Jan 31, 2023

Code:

Jan 31 17:26:39 mx1 pmgtunnel[3454]: tunnel finished 4932 IPmx2.IPmx2.IPmx2.IPmx2
Jan 31 17:27:13 mx1 pmgmirror[3465]: starting cluster synchronization
Jan 31 17:27:13 mx1 pmgmirror[3465]: database sync 'mx2' failed - DBI connect('dbname=Proxmox_ruledb;host=/run/pmgtunnel;port=2;','root',...) failed: could not connect to server: No such file or directory#012#011Is the server running locally and accepting#012#011connections on Unix domain socket "/run/pmgtunnel/.s.PGSQL.2"? at /usr/share/perl5/PMG/DBTools.pm line 66.
Jan 31 17:27:13 mx1 pmgmirror[3465]: cluster synchronization finished  (1 errors, 0.01 seconds (files 0.00, database 0.01, config 0.00))
Jan 31 17:27:48 mx1 pmgtunnel[3454]: restarting crashed tunnel 4940 IPmx2.IPmx2.IPmx2.IPmx2
Jan 31 17:27:48 mx1 pmgtunnel[3454]: tunnel finished 4940 IPmx2.IPmx2.IPmx2.IPmx2


Jan 31 17:27:05 mx2 pmgtunnel[5734]: tunnel finished 5894 IPmx1.IPmx1.IPmx1.IPmx1
Jan 31 17:27:54 mx2 pmgmirror[4545]: starting cluster synchronization
Jan 31 17:27:54 mx2 pmgmirror[4545]: sync error: syncing master configuration from 'IPmx1.IPmx1.IPmx1.IPmx1' failed: rsync error: unexplained error (code 255) at io.c(228) [Receiver=3.2.3]
Jan 31 17:28:06 mx2 pmgpolicy[4389]: starting policy database maintenance (greylist, rbl)
Jan 31 17:28:06 mx2 pmgpolicy[4389]: end policy database maintenance (22 ms, 6 ms)
Jan 31 17:28:14 mx2 pmgtunnel[5734]: restarting crashed tunnel 5905 IPmx1.IPmx1.IPmx1.IPmx1
Jan 31 17:28:14 mx2 pmgtunnel[5734]: tunnel finished 5905 IPmx1.IPmx1.IPmx1.IPmx1

Search

Search

PMG certificates for joining cluster

Xela

Well-Known Member

Stoiko Ivanov

Proxmox Staff Member

Xela

Well-Known Member

Stoiko Ivanov

Proxmox Staff Member

Xela

Well-Known Member

Stoiko Ivanov

Proxmox Staff Member

Xela

Well-Known Member

Stoiko Ivanov

Proxmox Staff Member

Xela

Well-Known Member

Stoiko Ivanov

Proxmox Staff Member

Xela

Well-Known Member

Stoiko Ivanov

Proxmox Staff Member

Xela

Well-Known Member