Cluster non syncing: command 'ip -details -json link show' failed: failed to execute

Z

zenaconsult

New Member
Jan 31, 2024
2
3
3
After a Debian update, my PMG cluster is no longer synching. Main node is active and the two nodes are always in 'synching' state. Syslog keeps on showing on both nodes:

Code: 
pmgtunnel[4515]: status update error: command 'ip -details -json link show' failed: failed to execute

Service is running:

Bash: 
systemctl status pmgtunnel
● pmgtunnel.service - Proxmox Mail Gateway Cluster Tunnel Daemon
     Loaded: loaded (/lib/systemd/system/pmgtunnel.service; enabled; preset: enabled)
     Active: active (running) since Sat 2025-08-16 11:43:59 CEST; 4min 3s ago
    Process: 4512 ExecStart=/usr/bin/pmgtunnel start (code=exited, status=0/SUCCESS)
   Main PID: 4515 (pmgtunnel)
      Tasks: 1 (limit: 4579)
     Memory: 75.1M
        CPU: 927ms
     CGroup: /system.slice/pmgtunnel.service
             └─4515 pmgtunnel

I've removed the nodes, readded them successfully with pmgcm join etc. but the situation stays the same. The command runs fine one the command line, paths are available for perl. What's wrong ?

- update - The servers ARE actually synching so that's good. Just the states are not updated.
 

Attachments

  • Screenshot 2025-08-16 at 12.47.38.png
    Screenshot 2025-08-16 at 12.47.38.png
    21.9 KB · Views: 1
Last edited:
  • Like
Reactions: Stoiko Ivanov
Please provide the output of `pmgversion -v`

else:
zenaconsult said:
pmgtunnel[4515]: status update error: command 'ip -details -json link show' failed: failed to execute
Click to expand...
what's the output of `ip -details -json link show` ?

EDIT: did you reboot/restart any services after the upgrade? - if not - does restarting `pmgtunnel` make the warnings go away?
EDIT2: please provide the output from all your clusternodes...

thanks!
 
Last edited:
I have the same problem here. Occured after debian update. I restarted the both nodes after the upgrade twice. The sync is working, both nodes show "syncing", but the error is showing up in syslog.

In syslog the same error shows up on both nodes:

Code: 
Aug 18 12:21:54 mx02.xxx.de pmgtunnel[1028]: status update error: command 'ip -details -json link show' failed: failed to execute
Aug 18 12:22:04 mx02.xxx.de pmgtunnel[1028]: status update error: command 'ip -details -json link show' failed: failed to execute
Aug 18 12:22:12 mx02.xxx.de pmgmirror[1314]: starting cluster synchronization
Aug 18 12:22:14 mx02.xxx.de pmgmirror[1314]: database sync 'mx01' failed - DBI connect('dbname=Proxmox_ruledb;host=/run/pmgtunnel;port=1;','root',...) failed: connection to server on socket "/run/pmgtunnel/.s.PGSQL.1" failed: No such file or directory
    Is the server running locally and accepting connections on that socket? at /usr/share/perl5/PMG/DBTools.pm line 66.
Aug 18 12:22:15 mx02.xxx.de pmgtunnel[1028]: status update error: command 'ip -details -json link show' failed: failed to execute

Code: 
Aug 18 12:26:41 mx01.xxx.de pmgmirror[1371]: starting cluster synchronization
Aug 18 12:26:41 mx01.xxx.de pmgmirror[1371]: database sync 'mx02' failed - DBI connect('dbname=Proxmox_ruledb;host=/run/pmgtunnel;port=2;','root',...) failed: connection to server on socket "/run/pmgtunnel/.s.PGSQL.2" failed: No such file or directory
    Is the server running locally and accepting connections on that socket? at /usr/share/perl5/PMG/DBTools.pm line 66.
Aug 18 12:26:41 mx01.xxx.de pmgmirror[1371]: cluster synchronization finished  (1 errors, 0.02 seconds (files 0.00, database 0.02, config 0.00))
Aug 18 12:26:41 mx01.xxx.de pmgtunnel[1062]: status update error: command 'ip -details -json link show' failed: failed to execute

Code: 
root@mx01:~# pmgversion -v
proxmox-mailgateway: 8.2.0 (API: 8.2.1/5f1bd658e7ca, running kernel: 6.8.12-13-pve)
pmg-api: 8.2.1
pmg-gui: 4.2.0
proxmox-kernel-helper: 8.1.4
proxmox-kernel-6.8.12-13-pve-signed: 6.8.12-13
proxmox-kernel-6.8: 6.8.12-13
proxmox-kernel-6.8.12-8-pve-signed: 6.8.12-8
proxmox-kernel-6.8.12-5-pve-signed: 6.8.12-5
proxmox-kernel-6.8.12-2-pve-signed: 6.8.12-2
proxmox-kernel-6.5.13-6-pve-signed: 6.5.13-6
proxmox-kernel-6.5: 6.5.13-6
clamav-daemon: 1.0.7+dfsg-1~deb12u1
ifupdown: 0.8.41
libarchive-perl: 3.6.2
libjs-extjs: 7.0.0-5
libjs-framework7: 4.4.7-2
libproxmox-acme-perl: 1.6.0
libproxmox-acme-plugins: 1.6.0
libpve-apiclient-perl: 3.3.2
libpve-common-perl: 8.3.4
libpve-http-server-perl: 5.2.2
libxdgmime-perl: 1.1.0
lvm2: 2.03.16-2
pmg-docs: 8.2.0
pmg-i18n: 3.4.5
pmg-log-tracker: 2.5.0
proxmox-mini-journalreader: 1.5
proxmox-spamassassin: 4.0.1-2
proxmox-widget-toolkit: 4.3.13
pve-firmware: 3.16-3
pve-xtermjs: 5.5.0-2

Code: 
root@mx02:~# pmgversion -v
proxmox-mailgateway: 8.2.0 (API: 8.2.1/5f1bd658e7ca, running kernel: 6.8.12-13-pve)
pmg-api: 8.2.1
pmg-gui: 4.2.0
proxmox-kernel-helper: 8.1.4
proxmox-kernel-6.8.12-13-pve-signed: 6.8.12-13
proxmox-kernel-6.8: 6.8.12-13
proxmox-kernel-6.8.12-8-pve-signed: 6.8.12-8
proxmox-kernel-6.8.12-5-pve-signed: 6.8.12-5
proxmox-kernel-6.8.12-2-pve-signed: 6.8.12-2
proxmox-kernel-6.5.13-6-pve-signed: 6.5.13-6
proxmox-kernel-6.5: 6.5.13-6
clamav-daemon: 1.0.7+dfsg-1~deb12u1
ifupdown: 0.8.41
libarchive-perl: 3.6.2
libjs-extjs: 7.0.0-5
libjs-framework7: 4.4.7-2
libproxmox-acme-perl: 1.6.0
libproxmox-acme-plugins: 1.6.0
libpve-apiclient-perl: 3.3.2
libpve-common-perl: 8.3.4
libpve-http-server-perl: 5.2.2
libxdgmime-perl: 1.1.0
lvm2: 2.03.16-2
pmg-docs: 8.2.0
pmg-i18n: 3.4.5
pmg-log-tracker: 2.5.0
proxmox-mini-journalreader: 1.5
proxmox-spamassassin: 4.0.1-2
proxmox-widget-toolkit: 4.3.13
pve-firmware: 3.16-3
pve-xtermjs: 5.5.0-2


Code: 
root@mx01:~# ip -details -json link show
[{"ifindex":1,"ifname":"lo","flags":["LOOPBACK","UP","LOWER_UP"],"mtu":65536,"qdisc":"noqueue","operstate":"UNKNOWN","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"loopback","address":"00:00:00:00:00:00","broadcast":"00:00:00:00:00:00","promiscuity":0,"allmulti":0,"min_mtu":0,"max_mtu":0,"inet6_addr_gen_mode":"eui64","num_tx_queues":1,"num_rx_queues":1,"gso_max_size":65536,"gso_max_segs":65535,"tso_max_size":524280,"tso_max_segs":65535,"gro_max_size":65536},{"ifindex":2,"ifname":"eth0","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1500,"qdisc":"pfifo_fast","operstate":"UP","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"96:00:01:73:fc:74","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"allmulti":0,"min_mtu":68,"max_mtu":65535,"inet6_addr_gen_mode":"eui64","num_tx_queues":1,"num_rx_queues":1,"gso_max_size":65536,"gso_max_segs":65535,"tso_max_size":65536,"tso_max_segs":65535,"gro_max_size":65536,"parentbus":"virtio","parentdev":"virtio0","altnames":["enp0s3","ens3"]}]

Code: 
root@mx02:~# ip -details -json link show
[{"ifindex":1,"ifname":"lo","flags":["LOOPBACK","UP","LOWER_UP"],"mtu":65536,"qdisc":"noqueue","operstate":"UNKNOWN","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"loopback","address":"00:00:00:00:00:00","broadcast":"00:00:00:00:00:00","promiscuity":0,"allmulti":0,"min_mtu":0,"max_mtu":0,"inet6_addr_gen_mode":"eui64","num_tx_queues":1,"num_rx_queues":1,"gso_max_size":65536,"gso_max_segs":65535,"tso_max_size":524280,"tso_max_segs":65535,"gro_max_size":65536},{"ifindex":2,"ifname":"eth0","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1500,"qdisc":"pfifo_fast","operstate":"UP","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"96:00:01:73:fc:7b","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"allmulti":0,"min_mtu":68,"max_mtu":65535,"inet6_addr_gen_mode":"eui64","num_tx_queues":1,"num_rx_queues":1,"gso_max_size":65536,"gso_max_segs":65535,"tso_max_size":65536,"tso_max_segs":65535,"gro_max_size":65536,"parentbus":"virtio","parentdev":"virtio0","altnames":["enp0s3","ens3"]}]
 
DerSeekin said:
I have the same problem here. Occured after debian update. I restarted the both nodes after the upgrade twice. The sync is working, both nodes show "syncing", but the error is showing up in syslog.

In syslog the same error shows up on both nodes:
Click to expand...
Thanks for adding the output and reporting that you're affected as well!
hm - did not manage to reproduce the issue here locally...

Could you additionally provide:
* the complete output of `pmgcm status`
* the contents of `/etc/pmg/cluster.conf`
* the contents of `/etc/hosts`
* the output of `ping -c4 $(uname -n)`

from all machines in your cluster

How were the PMG instances setup? (using the PMG ISO, on top of Debian?)
Thanks!
 
The instances were setup on top of Debian. These are Cloud VMs at Hetzner if this matters. Running since at least 2,5 years without any problems until now.

Code: 
root@mx01:~# pmgcm status
NAME(CID)--------------IPADDRESS----ROLE-STATE---------UPTIME---LOAD----MEM---DISK
mx01(1)              x.x.x.x      master     S           01:38   0.11    75%    35%
mx02(2)              x.x.x.x       node      S           01:36   0.01    70%    33%

Code: 
root@mx02:~# pmgcm status
NAME(CID)--------------IPADDRESS----ROLE-STATE---------UPTIME---LOAD----MEM---DISK
mx01(1)              x.x.x.x       master S           01:38   0.05    72%    35%
mx02(2)              x.x.x.x       node   S           01:37   0.00    72%    33%

Code: 
root@mx01:~# cat /etc/pmg/cluster.conf
master: 1
        fingerprint BA:E8:xxxxxx
        hostrsapubkey xxx
        ip x.x.x.x
        maxcid 2
        name mx01
        rootrsapubkey xxx
node: 2
        fingerprint 3E:F5:xxxxxx
        hostrsapubkey xxx
        ip x.x.x.x
        name mx02
        rootrsapubkey xxx

Code: 
root@mx02:~# cat /etc/pmg/cluster.conf
master: 1
        fingerprint BA:E8:xxxxxx
        hostrsapubkey xxx
        ip x.x.x.x
        maxcid 2
        name mx01
        rootrsapubkey xxx

node: 2
        fingerprint 3E:F5:xxxxxx
        hostrsapubkey xxx
        ip x.x.x.x
        name mx02
        rootrsapubkey xxx

Code: 
root@mx01:~# cat /etc/hosts
# Your system has configured 'manage_etc_hosts' as True.
# As a result, if you wish for changes to this file to persist
# then you will need to either
# a.) make changes to the master file in /etc/cloud/templates/hosts.debian.tmpl
# b.) change or remove the value of 'manage_etc_hosts' in
#     /etc/cloud/cloud.cfg or cloud-config from user-data
#
127.0.1.1 mx01.xxx.de mx01
127.0.0.1 localhost
127.0.0.1 mx01.xxx.de
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

Code: 
root@mx02:~# cat /etc/hosts
# Your system has configured 'manage_etc_hosts' as True.
# As a result, if you wish for changes to this file to persist
# then you will need to either
# a.) make changes to the master file in /etc/cloud/templates/hosts.debian.tmpl
# b.) change or remove the value of 'manage_etc_hosts' in
#     /etc/cloud/cloud.cfg or cloud-config from user-data
#
127.0.1.1 mx02.xxx.de mx02
127.0.0.1 localhost
127.0.0.1 mx02.xxx.de
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

Code: 
root@mx01:~# ping -c4 $(uname -n)
PING mx01.xxx.de (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.050 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.045 ms
^C
--- mx01.xxx.de ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1026ms
rtt min/avg/max/mdev = 0.045/0.047/0.050/0.002 ms

Code: 
root@mx02:~# ping -c4 $(uname -n)
PING mx02.xxx.de (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.058 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.053 ms
64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.042 ms
^C
--- mx02.xxx.de ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2031ms
rtt min/avg/max/mdev = 0.042/0.051/0.058/0.006 ms
 
  • Like
Reactions: Stoiko Ivanov
You must log in or register to reply here.
Top Bottom