Problem with pmgcm sync between nodes

[YavuzGM]

Hi,

I have a Proxmail Mail Gateway cluster with 1 master and 2 nodes.

Master and first node can sync successfully pmg configuration but second node always returns an error like this:

root@gw2:~# pmgcm sync
syncing master configuration from '77.2XX.XX.XX'
file '/etc/pmg/master/transport' too long - aborting

Transport file size is 1.4MB.

pmgcm status Command always shows "S / syncing" for second node:

root@gw2:~# pmgcm status
NAME(CID)--------------IPADDRESS----ROLE-STATE---------UPTIME---LOAD----MEM---DISK
mx1(1)               77.XX.XX.XX   master A           02:13   1.88    27%    40%
gw1(2)               77.XX.XX.XX   node   A           02:13   3.03    33%    41%
gw2(4)               77.XX.XX.XX   node   S           01:32   1.99    32%    37%

All of the master and nodes has same settings but i can't figure out what's the root cause.

Thanks in advance.

 

[Stoiko Ivanov]

please post `pmgversion -v` from all nodes
Thanks!

 

[YavuzGM]

Hi Stoiko,

Thanks for the answer.

Version details from master and 2 nodes:

Master:

root@mx1:~# pmgversion -v
proxmox-mailgateway: 7.3-1 (API: 7.3-3/a3d66da0, running kernel: 5.15.104-1-pve)
pmg-api: 7.3-3
pmg-gui: 3.3-2
pve-kernel-5.15: 7.4-1
pve-kernel-helper: 7.3-8
pve-kernel-5.13: 7.1-9
pve-kernel-5.15.104-1-pve: 5.15.104-1
pve-kernel-5.15.102-1-pve: 5.15.102-1
pve-kernel-5.15.74-1-pve: 5.15.74-1
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-1-pve: 5.13.19-3
clamav-daemon: 0.103.8+dfsg-0+deb11u1
ifupdown2: 3.1.0-1+pmx3
libarchive-perl: 3.4.0-1
libjs-extjs: 7.0.0-1
libjs-framework7: 4.4.7-1
libproxmox-acme-perl: 1.4.4
libproxmox-acme-plugins: 1.4.4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-4
libpve-http-server-perl: 4.2-1
libxdgmime-perl: 1.0-1
lvm2: 2.03.11-2.1
pmg-docs: 7.3-2
pmg-i18n: 2.12-1
pmg-log-tracker: 2.3.2-1
postgresql-13: 13.9-0+deb11u1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.1-1
proxmox-spamassassin: 4.0.0-2
proxmox-widget-toolkit: 3.6.5
pve-firmware: 3.6-4
pve-xtermjs: 4.16.0-1
zfsutils-linux: 2.1.9-pve1

Node 1

root@gw1:~# pmgversion -v
proxmox-mailgateway: 7.3-1 (API: 7.3-3/a3d66da0, running kernel: 5.15.102-1-pve)
pmg-api: 7.3-3
pmg-gui: 3.3-2
pve-kernel-helper: 7.3-8
pve-kernel-5.15: 7.3-3
pve-kernel-5.13: 7.1-9
pve-kernel-5.15.102-1-pve: 5.15.102-1
pve-kernel-5.15.74-1-pve: 5.15.74-1
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-1-pve: 5.13.19-3
clamav-daemon: 0.103.8+dfsg-0+deb11u1
ifupdown2: 3.1.0-1+pmx3
libarchive-perl: 3.4.0-1
libjs-extjs: 7.0.0-1
libjs-framework7: 4.4.7-1
libproxmox-acme-perl: 1.4.4
libproxmox-acme-plugins: 1.4.4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-4
libpve-http-server-perl: 4.2-1
libxdgmime-perl: 1.0-1
lvm2: 2.03.11-2.1
pmg-docs: 7.3-2
pmg-i18n: 2.12-1
pmg-log-tracker: 2.3.2-1
postgresql-13: 13.9-0+deb11u1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.1-1
proxmox-spamassassin: 4.0.0-2
proxmox-widget-toolkit: 3.6.4
pve-firmware: 3.6-4
pve-xtermjs: 4.16.0-1
zfsutils-linux: 2.1.9-pve1

GW2

root@gw2:~# pmgversion -v
proxmox-mailgateway: 7.3-1 (API: 7.3-3/a3d66da0, running kernel: 5.15.107-2-pve)
pmg-api: 7.3-3
pmg-gui: 3.3-2
pve-kernel-5.15: 7.4-3
pve-kernel-helper: 7.3-8
pve-kernel-5.13: 7.1-9
pve-kernel-5.15.107-2-pve: 5.15.107-2
pve-kernel-5.15.104-1-pve: 5.15.104-2
pve-kernel-5.15.102-1-pve: 5.15.102-1
pve-kernel-5.15.74-1-pve: 5.15.74-1
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-1-pve: 5.13.19-3
clamav-daemon: 0.103.8+dfsg-0+deb11u1
ifupdown2: 3.1.0-1+pmx4
libarchive-perl: 3.4.0-1
libjs-extjs: 7.0.0-1
libjs-framework7: 4.4.7-1
libproxmox-acme-perl: 1.4.4
libproxmox-acme-plugins: 1.4.4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.4-1
libpve-http-server-perl: 4.2-3
libxdgmime-perl: 1.0-1
lvm2: 2.03.11-2.1
pmg-docs: 7.3-2
pmg-i18n: 2.12-1
pmg-log-tracker: 2.3.2-1
postgresql-13: 13.11-0+deb11u1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.1-1
proxmox-spamassassin: 4.0.0-2
proxmox-widget-toolkit: 3.7.0
pve-firmware: 3.6-5
pve-xtermjs: 4.16.0-1
zfsutils-linux: 2.1.11-pve1

 

[Stoiko Ivanov]

Thanks - the odd thing here is that it works with one of the 2 nodes...
the limit for the config-files is 1MB (which really should be enough for config files)

Why do you have such a large transports file?
How does it look?

 

[YavuzGM]

There are many records in the file, all of which are authenticated and necessary.

Node1 can sync this 1.4MB transport file without any error, that's exactly what I'm surprised about.

Is it possible to minimize or split this file and use it with multiple files? Am I the only one with a transport file larger than 1MB?

 

[Stoiko Ivanov]

Is it possible to minimize or split this file and use it with multiple files? Am I the only one with a transport file larger than 1MB?

I suppose so - at least we did not get any other report about this yet.
and no - splitting is not possible.

There are many records in the file, all of which are authenticated and necessary.

* how many domains does your PMG serve?
* if many of the domains have the same down-stream server - consider setting a default-relay and removing those domains from the explicit transport file

Node1 can sync this 1.4MB transport file without any error, that's exactly what I'm surprised about.

the code has been with the 1MB limit for a long while (at least since 2017) - did you maybe modify the sources?
(or did the file grow so large recently?)

please check and verify content and size of:
`/etc/pmg/transport` and `/etc/pmg/master/transport` on all nodes
as this is odd

 

[YavuzGM]

Hi,

Master and nodes has same file sizes:

1.4M    transport
2.1M    transport.db

Is it possible to increase the limits?

 

[Stoiko Ivanov]

Is it possible to increase the limits?

This has to be done in the code:
https://git.proxmox.com/?p=pmg-api....dbf750ee06b7cd345beb4db512ba23cb;hb=HEAD#l271

however I still wonder why this should be necessary in almost all installations - how many domains do you serve? could you share the file - so I can see why this is really needed?

Master and nodes has same file sizes:

which of the 2 paths?:
`/etc/pmg/transport` and `/etc/pmg/master/transport`

 

[YavuzGM]

Hi,

I only have /etc/pmg/transport, there is no "master" folder in /etc/pmg/

I have 29096 lines in transport file. I attached the copy of my transport file from master node but community also blocked the attachment about file size (1.4MB TXT file)

Could you please give more details about code change for this file:

https://git.proxmox.com/?p=pmg-api....dbf750ee06b7cd345beb4db512ba23cb;hb=HEAD#l271

Which row or variable do I need to edit?

 

[Stoiko Ivanov]

I have 29096 lines in transport file. I attached the copy of my transport file from master node but community also blocked the attachment about file size

do you have 29096 domains which you handle through your pmg?
do all of these domains really have a downstream server of their own?

please zip the file and attach it here

 

[YavuzGM]

Zip attached.

do you have 29096 domains which you handle through your pmg?
do all of these domains really have a downstream server of their own?

There may be records that shouldn't exist and that no longer work for us. However in general, the content of this file is correct and valid.

I also wish there were less records here and I wouldn't have any problems about syncing

 

[Stoiko Ivanov]

Many of the domains listed in the file do not exist in the public internet?
(checked with `dig ns <domain>` - many return NXDOMAIN)
so I assume you have them only internally?!
* all of the entries are '<domain> smtp:[<domain>]:25' ?

The question remains:
* why have explicit transport entries
* especially if you have entered the domains in your local DNS (only) - you can just as easy add an MX-record for each domain (pointing to mail.<domain>) and then there is no need for any of the transport entries!

I deleted the zip you attached - since it contains sensitive data...

 

[YavuzGM]

I'll remove expired and unused domains for minify the content.

I have common MX addresses for domains in the transport list. E-mails come to Proxmox PMG Server first for filter the spam. After the filter applied, Proxmox redirect e-mails to the real mail destinations with transport details in the file.

I know this is the way it has to be because of the way it works.

So i couldn't understand this or there is another way to use that I missed.

* why have explicit transport entries
* especially if you have entered the domains in your local DNS (only) - you can just as easy add an MX-record for each domain (pointing to mail.<domain>) and then there is no need for any of the transport entries!