Pmgmirror could not create socket: Too many open files

[AnNH]

Dear team,

We are facing pmgmirror error on Mail Proxmox Gateway as follows:

root@pmg1:~# journalctl -u pmgmirror | tail -20
Mar 24 17:40:25 pmg8 pmgmirror[1203]: database sync 'pmg2' failed - DBI connect('dbname=Proxmox_ruledb;host=/run/pmgtunnel;port=9;','root',...) failed: could not create socket: Too many open files at /usr/share/perl5/PMG/DBTools.pm line 66.
Mar 24 17:40:25 pmg8 pmgmirror[1203]: database sync 'pmg3' failed - DBI connect('dbname=Proxmox_ruledb;host=/run/pmgtunnel;port=13;','root',...) failed: could not create socket: Too many open files at /usr/share/perl5/PMG/DBTools.pm line 66.
Mar 24 17:40:25 pmg8 pmgmirror[1203]: database sync 'pmg4' failed - DBI connect('dbname=Proxmox_ruledb;host=/run/pmgtunnel;port=10;','root',...) failed: could not create socket: Too many open files at /usr/share/perl5/PMG/DBTools.pm line 66.
Mar 24 17:40:25 pmg8 pmgmirror[1203]: database sync 'pmg5' failed - DBI connect('dbname=Proxmox_ruledb;host=/run/pmgtunnel;port=11;','root',...) failed: could not create socket: Too many open files at /usr/share/perl5/PMG/DBTools.pm line 66.
Mar 24 17:40:25 pmg8 pmgmirror[1203]: database sync 'pmg6' failed - DBI connect('dbname=Proxmox_ruledb;host=/run/pmgtunnel;port=26;','root',...) failed: could not create socket: Too many open files at /usr/share/perl5/PMG/DBTools.pm line 66.
Mar 24 17:40:25 pmg8 pmgmirror[1203]: database sync 'pmg7' failed - DBI connect('dbname=Proxmox_ruledb;host=/run/pmgtunnel;port=3;','root',...) failed: could not create socket: Too many open files at /usr/share/perl5/PMG/DBTools.pm line 66.
Mar 24 17:40:25 pmg8 pmgmirror[1203]: database sync 'pmg8' failed - DBI connect('dbname=Proxmox_ruledb;host=/run/pmgtunnel;port=32;','root',...) failed: could not create socket: Too many open files at /usr/share/perl5/PMG/DBTools.pm line 66.
Mar 24 17:40:25 pmg8 pmgmirror[1203]: database sync 'pmg9' failed - DBI connect('dbname=Proxmox_ruledb;host=/run/pmgtunnel;port=5;','root',...) failed: could not create socket: Too many open files at /usr/share/perl5/PMG/DBTools.pm line 66.
Mar 24 17:40:25 pmg8 pmgmirror[1203]: database sync 'pmg10' failed - DBI connect('dbname=Proxmox_ruledb;host=/run/pmgtunnel;port=12;','root',...) failed: could not create socket: Too many open files at /usr/share/perl5/PMG/DBTools.pm line 66.
Mar 24 17:40:25 pmg8 pmgmirror[1203]: database sync 'pmg11' failed - DBI connect('dbname=Proxmox_ruledb;host=/run/pmgtunnel;port=28;','root',...) failed: could not create socket: Too many open files at /usr/share/perl5/PMG/DBTools.pm line 66.
Mar 24 17:40:25 pmg8 pmgmirror[1203]: database sync 'pmg12' failed - DBI connect('dbname=Proxmox_ruledb;host=/run/pmgtunnel;port=17;','root',...) failed: could not create socket: Too many open files at /usr/share/perl5/PMG/DBTools.pm line 66.
Mar 24 17:40:25 pmg8 pmgmirror[1203]: database sync 'pmg13' failed - DBI connect('dbname=Proxmox_ruledb;host=/run/pmgtunnel;port=14;','root',...) failed: could not create socket: Too many open files at /usr/share/perl5/PMG/DBTools.pm line 66.
Mar 24 17:40:25 pmg8 pmgmirror[1203]: database sync 'pmg14' failed - DBI connect('dbname=Proxmox_ruledb;host=/run/pmgtunnel;port=6;','root',...) failed: could not create socket: Too many open files at /usr/share/perl5/PMG/DBTools.pm line 66.
......................
Mar 24 17:40:25 pmg8 pmgmirror[1203]: cluster synchronization finished  (31 errors, 0.01 seconds (files 0.00, database 0.01, config 0.00))

We have check log in /var/log/postgresql/postgresql-13-main.log only see this log when we change setting in pgsql config file in /etc/pmg/templates/, max connection, shared_buffer, max_files_per_process and restart services.

2022-03-24 16:49:01.440 +07 [30468] LOG:  starting PostgreSQL 13.3 (Debian 13.3-1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2022-03-24 16:49:01.441 +07 [30468] LOG:  listening on IPv6 address "::1", port 5432
2022-03-24 16:49:01.441 +07 [30468] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2022-03-24 16:49:01.444 +07 [30468] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-03-24 16:49:01.456 +07 [30469] LOG:  database system was shut down at 2022-03-24 16:49:01 +07
2022-03-24 16:49:01.469 +07 [30468] LOG:  database system is ready to accept connections

My version:

root@pmg1:/etc/pmg/templates# pmgversion -v
proxmox-mailgateway-container: 7.0-1 (API: 7.0-8/36f2537c, running kernel: 5.4.157-1-pve)
pmg-gui: 3.0-5
clamav-daemon: 0.103.3+dfsg-0+deb11u1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
libarchive-perl: 3.4.0-1
libjs-extjs: 7.0.0-1
libjs-framework7: 4.4.7-1
libproxmox-acme-perl: 1.3.0
libproxmox-acme-plugins: 1.3.0
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-10
libpve-http-server-perl: 4.0-2
libxdgmime-perl: 1.0-1
lvm2: not correctly installed
pmg-docs: 7.0-2
pmg-i18n: 2.5-1
pmg-log-tracker: 2.2.0-1
postgresql-13: 13.3-1
proxmox-mini-journalreader: 1.2-1
proxmox-spamassassin: 3.4.6-3
proxmox-widget-toolkit: 3.3-6
pve-xtermjs: 4.12.0-1

 

[dcsapak]

how many nodes are in your cluster? the mirroring does not really scale that well for many nodes. it's designed for a couple at max, not >10

 

[AnNH]

Hi,

Thanks for your information,

Currently we have 32 nodes on the system.

Best regards !

 

[dcsapak]

as i said, running clusters this big is not really recommended or tested...
that said, you could try to increase the open file limit for that service

with

systemctl edit pmgmirror.service

an editor opens, and you have to add

[Service]
LimitNOFileSoft=4096

for 4 times the open file limit (the default is 1024)
you can ofc try to set it even higher...

after that you have to restart the service