Single Node in Cluster Grayed Out - Logs reflect issue with /usr/lib/x86_64-linux-gnu/perl-base/IO/Select.pm line 60

[jsalas424]

A single node is grayed out, this has happened before and succesfully cleared with

systemctl restart pvestatd

but it keeps recurring so I checked my logs:

There was an uptick around 13:00 so I honed in on that to find that these error messages suddenly started appearing over and over:



Full messages:

status update error: command 'ebtables-save' failed: Modification of a read-only value attempted at /usr/lib/x86_64-linux-gnu/perl-base/IO/Select.pm line 60.

status update error: command 'iptables-save' failed: Modification of a read-only value attempted at /usr/lib/x86_64-linux-gnu/perl-base/IO/Select.pm line 60.

message repeated 3 times: [ status update error: command 'iptables-save' failed: Modification of a read-only value attempted at /usr/lib/x86_64-linux-gnu/perl-base/IO/Select.pm line 60.]

etc.

PVE 7.4-16

 

[floh8]

seems something want update your firewall rule. Maybe u are hacked.

 

[jsalas424]

seems something want update your firewall rule. Maybe u are hacked.

I rebooted the node and the firewall errors stopped, there was definitely something odd.

Although always possible it’s an intrusion, that feels very unlikely given that the only 2 open ports are an SSL port to a 2FA guarded Nextcloud instance and a wireguard VPN.

I also only have default firewall values on Proxmox, I handle everything else on pfsense.

Nonetheless, is there anything else I can look out for? I keeep lots of logs.

 

[jsalas424]

This has recurred and was once more quickly fixed with `systemctl restart pvestatd`. No firewall log issues so I think that was a red herring, but I did find some more enlightening logs:

Aug 29 13:04:48 ServerB pvestatd[2470]: lxc console cleanup error: IO::Handle: bad open mode:  at /usr/share/perl5/PVE/Tools.pm line 293.
Aug 29 13:04:48 ServerB pvestatd[2470]: ERROR: panic: attempt to copy freed scalar 5597854c76e0 to 559784dd2378 at /usr/lib/x86_64-linux-gnu/perl-base/IO/File.pm line 53.
Aug 29 13:04:48 ServerB pvestatd[2470]: server closing
Aug 29 13:04:48 ServerB pvestatd[2470]: server stopped
Aug 29 13:04:48 ServerB kernel: pvestatd[2470]: segfault at ff0000000e ip 000055978462e7f8 sp 00007ffcba1e6550 error 4 in perl[559784560000+185000]
Aug 29 13:04:48 ServerB kernel: Code: 48 8b 43 10 48 85 c0 74 c9 83 c2 01 89 53 08 48 8b 30 48 85 f6 74 0a f6 46 0e 10 0f 85 c1 02 00 00 48 8b 70 28 48 85 f6 74 0a <f6> 46 0e 10 0f 85 d6 02 00 00 48 8b 70 20 48 85 f6 74 0a f6 46 0e
Aug 29 13:04:48 ServerB systemd[1]: pvestatd.service: Main process exited, code=killed, status=11/SEGV
Aug 29 13:04:48 ServerB systemd[1]: pvestatd.service: Failed with result 'signal'.
Aug 29 13:04:48 ServerB systemd[1]: pvestatd.service: Consumed 7h 40min 46.201s CPU time.

Here's my corosync config:

root@ServerB:~# cat /etc/pve/corosync.conf
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: ServerA
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 10.0.0.3
    ring1_addr: 192.168.1.24
  }
  node {
    name: ServerB
    nodeid: 3
    quorum_votes: 1
    ring0_addr: 10.0.0.4
    ring1_addr: 192.168.1.25
  }
  node {
    name: Server
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 10.0.0.2
    ring1_addr: 192.168.1.129
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: NodeA
  config_version: 5
  interface {
    linknumber: 0
  }
  interface {
    linknumber: 1
  }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
  version: 2
}

 

[fiona]

Hi,
since you got errors in both IO/Select.pm and IO/File.pm, I'd verify that the Perl installation is correct. You can use apt install debsums and then run debsums -s libperl5.36 perl-base, or to verify all packages (can take a while) debsums -s. If that doesn't show anything wrong, I'd run a memory test just to be sure.

 

[jsalas424]

Hi,
since you got errors in both IO/Select.pm and IO/File.pm, I'd verify that the Perl installation is correct. You can use apt install debsums and then run debsums -s libperl5.36 perl-base, or to verify all packages (can take a while) debsums -s. If that doesn't show anything wrong, I'd run a memory test just to be sure.

Hi Fiona, thanks for the reply. Here are those results. Should I proceed to memtest?

root@ServerB:~# debsums -s libperl5.36 perl-base
debsums: package libperl5.36 is not installed

No output for just debsums -s

root@ServerB:~# debsums -s
root@ServerB:~#

lastly I also checked config files but this all looks correct to me.

root@ServerB:~# debsums -as
debsums: changed file /etc/issue (from base-files package)
debsums: changed file /etc/lvm/lvm.conf (from lvm2 package)
debsums: changed file /etc/nut/upsmon.conf (from nut-client package)
debsums: changed file /etc/nut/nut.conf (from nut-client package)
debsums: changed file /etc/nut/upsd.users (from nut-server package)
debsums: changed file /etc/nut/upsd.conf (from nut-server package)
debsums: changed file /etc/nut/ups.conf (from nut-server package)
debsums: changed file /etc/vzdump.conf (from pve-manager package)
debsums: changed file /etc/apt/sources.list.d/pve-enterprise.list (from pve-manager package)
debsums: changed file /etc/rsyslog.conf (from rsyslog package)
root@TracheNodeB:~#

I'm running PVE 7.4-16 currently