Syncing error

[cglmicro]

Hi guys.

I have a master (proxmoxmg1.legardeur.net / 51.222.184.12) that run for about 2 months, and I just add a slave (proxmoxmg2.legardeur.net / 158.69.109.57) to test cluster.
I did tried to reboot both servers.

On my slave I receive this:

Nov 06 19:53:55 proxmoxmg2 pmgmirror[381]: starting cluster synchronization
Nov 06 19:54:30 proxmoxmg2 pmg-smtp-filter[402]: starting database maintenance
Nov 06 19:54:30 proxmoxmg2 pmg-smtp-filter[402]: end database maintenance (2 ms)
Nov 06 19:54:36 proxmoxmg2 pmgmirror[381]: database sync 'proxmoxmg1' failed - command 'rsync '--rsh=ssh -l root -o BatchMode=yes -o HostKeyAlias=proxmoxmg1' -q --timeout 10 '[51.222.184.12]:/var/spool/pmg' /var/spool/pmg --files-from /tmp/quarantinefilelist.381' failed: exit code 30
Nov 06 19:54:36 proxmoxmg2 pmgmirror[381]: Use of uninitialized value $domain in concatenation (.) or string at /usr/share/perl5/PMG/Utils.pm line 629.
Nov 06 19:54:36 proxmoxmg2 pmgmirror[381]: cluster synchronization finished  (1 errors, 40.83 seconds (files 0.00, database 15.53, config 25.30))

What should I try first?

Thanks.

 

[Pifouney]

Hi,

First of all, check you FQDN declaration, your host file declaration, and check if all is OK.

SEcond think, Proxmox VE not running good with 2nodes cluster. You need at least a 3 node for clustering commodly.

 

[cglmicro]

Hi Pifourney.

I'm not sure what to look for your FQDN question, but here is something to start with.

From my master :

root@proxmoxmg1:/etc/fail2ban# cat /etc/hosts
127.0.0.1       localhost
::1             localhost ip6-localhost ip6-loopback
ff02::1         ip6-allnodes
ff02::2         ip6-allrouters
# --- BEGIN PVE ---
51.222.184.12 proxmoxmg1.legardeur.net proxmoxmg1
# --- END PVE ---

root@proxmoxmg1:/etc/fail2ban# cat /proc/sys/kernel/hostname
proxmoxmg1

From my slave :

root@proxmoxmg2:/etc/fail2ban# cat /etc/hosts
127.0.0.1       localhost
::1             localhost ip6-localhost ip6-loopback
ff02::1         ip6-allnodes
ff02::2         ip6-allrouters
# --- BEGIN PVE ---
158.69.109.57 proxmoxmg2.legardeur.net proxmoxmg2
# --- END PVE ---

root@proxmoxmg2:/etc/fail2ban# cat /proc/sys/kernel/hostname
proxmoxmg2

Regarding the 2 or 3 nodes setups, Proxmox VE is suggestion 3 nodes at least to get quorum. But for PROXMOX MAIL GATEWAY, 2 nodes is OK since I don't think they use quorum on this MASTER/SLAVE configurations.

Thanks.

 

[Pifouney]

Sorry for the first declaration, misstake of PMG, and not PVE ^^

I see that in your logfile:

Nov 06 19:54:36 proxmoxmg2 pmgmirror[381]: Use of uninitialized value $domain in concatenation (.) or string at /usr/share/perl5/PMG/Utils.pm line 629.

How is declared your hostname (/etc/hostname)?
If he's declared like "proxmoxmg2", try to replace by his FQDN like "proxmoxmg2.legardeur.net"

In your case, don't know if reboot of clusters nodes required

 

[cglmicro]

In my MASTER honame file, the server was renamed from "proxmoxmg1" to "proxmoxmg1.legadeur.net" and the SLAVE from "proxmoxmg2" to "proxmoxmg2.legardeur.net". I rebooted both hosts. Same synching issue.

I tought that the problem came with the fact that the hostname was changed (in the documentation they say it can't be changed), so I've "pmgcm delete 2" on my master, and I "rm /etc/pmg/cluster.conf" on the slave. I rebooted both server.
I joined again my slave to my master, and same synching issue.

I can force the synching to receive this error if it help :

root@proxmoxmg2:~# pmgcm sync
syncing master configuration from '51.222.184.12'
Use of uninitialized value $domain in concatenation (.) or string at /usr/share/perl5/PMG/Utils.pm line 629.
could not change directory to "/root": Permission denied
could not change directory to "/root": Permission denied
could not change directory to "/root": Permission denied
could not change directory to "/root": Permission denied
could not change directory to "/root": Permission denied
could not change directory to "/root": Permission denied
could not change directory to "/root": Permission denied
could not change directory to "/root": Permission denied
could not change directory to "/root": Permission denied
could not change directory to "/root": Permission denied
could not change directory to "/root": Permission denied
could not change directory to "/root": Permission denied

Thanks.

 

[cglmicro]

I don't know if it help, but I notice that every time I changer the content of /etc/hostname and I reboot, it revert to the original name "proxmoxmg1" or "proxmoxmg2", the change is not permanent.
I tried many method to have it change permanent, but I don't have sysctl nor hostnamectl.
This CT was installed from a CT image on a Proxmox VE.

EDIT: I've checked in the PVE for the file /etc/pve/lxc/1111.conf (my VM is 1111), and it contain the real hostname "proxmoxmg2.legardeur.net". Must not be a problem of hostname, must be something regarding the PERMISSION DENIED shown on my last post on this forum?

 

[Stoiko Ivanov]

Use of uninitialized value $domain in concatenation (.) or string at /usr/share/perl5/PMG/Utils.pm line 629

make sure that your '/etc/resolv.conf' contains a search-domain
if this is a LXC deployment on PVE you need to add the search domain to the container's config

see:
https://pmg.proxmox.com/pmg-docs/pmg-admin-guide.html#_configuration_files_overview

could not change directory to "/root": Permission denied

this warning in this context is usually harmless (you ran the command in '/root', the command switches to other users for certain operations (e.g. postgres) - and postgres cannot access /root) - if you don't want the message simply `cd /tmp` before running the command

I hope this helps!

 

[cglmicro]

Hi Stoiko.

I still have my problem, but here is where I'm at now:

You are right, my resolv.conf wasn't receiving the search domain since it was not populate in my ProxmoxVE... now it's set to cdns.ovh.net (it'S OVH recommendation) and I rebooted the 2 nodes.

You are also right for the permission denied; if I cd to /tmp/ and run the pmgcm sync, I don't receive these errors... but still not working for me.

Here is what my SLAVE > ADMINISTRATION > SERVICES > PMGMIRROR > SYSLOG tell me:

Nov 08 14:27:56 proxmoxmg2 pmgmirror[405]: starting cluster synchronization
Nov 08 14:28:32 proxmoxmg2 pmgmirror[405]: database sync 'proxmoxmg1' failed - command 'rsync '--rsh=ssh -l root -o BatchMode=yes -o HostKeyAlias=proxmoxmg1' -q --timeout 10 '[51.222.184.12]:/var/spool/pmg' /var/spool/pmg --files-from /tmp/quarantinefilelist.405' failed: exit code 255
Nov 08 14:28:32 proxmoxmg2 pmgmirror[405]: cluster synchronization finished  (1 errors, 36.74 seconds (files 0.00, database 11.45, config 25.29))
Nov 08 14:29:56 proxmoxmg2 pmgmirror[405]: starting cluster synchronization
Nov 08 14:30:37 proxmoxmg2 pmgmirror[405]: database sync 'proxmoxmg1' failed - command 'rsync '--rsh=ssh -l root -o BatchMode=yes -o HostKeyAlias=proxmoxmg1' -q --timeout 10 '[51.222.184.12]:/var/spool/pmg' /var/spool/pmg --files-from /tmp/quarantinefilelist.405' failed: exit code 30
Nov 08 14:30:37 proxmoxmg2 pmgmirror[405]: cluster synchronization finished  (1 errors, 40.84 seconds (files 0.00, database 15.54, config 25.30))

Thanks for your help, and what next?

 

[Pifouney]

Hi Stoiko.

I still have my problem, but here is where I'm at now:

You are right, my resolv.conf wasn't receiving the search domain since it was not populate in my ProxmoxVE... now it's set to cdns.ovh.net (it'S OVH recommendation) and I rebooted the 2 nodes.

You are also right for the permission denied; if I cd to /tmp/ and run the pmgcm sync, I don't receive these errors... but still not working for me.

Here is what my SLAVE > ADMINISTRATION > SERVICES > PMGMIRROR > SYSLOG tell me:

Nov 08 14:27:56 proxmoxmg2 pmgmirror[405]: starting cluster synchronization
Nov 08 14:28:32 proxmoxmg2 pmgmirror[405]: database sync 'proxmoxmg1' failed - command 'rsync '--rsh=ssh -l root -o BatchMode=yes -o HostKeyAlias=proxmoxmg1' -q --timeout 10 '[51.222.184.12]:/var/spool/pmg' /var/spool/pmg --files-from /tmp/quarantinefilelist.405' failed: exit code 255
Nov 08 14:28:32 proxmoxmg2 pmgmirror[405]: cluster synchronization finished  (1 errors, 36.74 seconds (files 0.00, database 11.45, config 25.29))
Nov 08 14:29:56 proxmoxmg2 pmgmirror[405]: starting cluster synchronization
Nov 08 14:30:37 proxmoxmg2 pmgmirror[405]: database sync 'proxmoxmg1' failed - command 'rsync '--rsh=ssh -l root -o BatchMode=yes -o HostKeyAlias=proxmoxmg1' -q --timeout 10 '[51.222.184.12]:/var/spool/pmg' /var/spool/pmg --files-from /tmp/quarantinefilelist.405' failed: exit code 30
Nov 08 14:30:37 proxmoxmg2 pmgmirror[405]: cluster synchronization finished  (1 errors, 40.84 seconds (files 0.00, database 15.54, config 25.30))

Thanks for your help, and what next?

Hi
Can you check if you've in correct time mode on both servers ? Time is really important for this kind of synchronisation

 

[cglmicro]

Hi.
Both are at the same time, on the same timezone:


I suspect something about the fact that the master was a 6.x on buster that I upgraded to 7.x on bullseye.
I started over again with 2 brand new servers and redid all my config and it work fine now.
Thanks for your help, I learned many things troubleshooting with all of you.