PVE 9: SCTP Corosync fails after restart with libknet1t64 1.35-pve1 (EINVAL / status 8)

[itNGO]

After a Corosync service restart, one node failed to rejoin our cluster. The installed libknet has no usable SCTP transport, despite SCTP being configured. The diagnosis is confirmed; production recovery after a rollback has not been verified.

Affected node:

Component	Version
pve-manager	9.2.18
Running kernel	7.0.2-2-pve
corosync	3.1.10-pve3
libknet1t64	1.35-pve1

The cluster has four nodes and two links per node. Corosync uses KNET with knet_transport: sctp and ip_version: ipv4-6. Every ring*_addr is numeric IPv4; anonymized examples are 192.0.2.11 and 198.51.100.11.

Selected log lines (hostname anonymized; timestamps and PIDs omitted):

node-a corosync: [TOTEM ] WARNING SCTP transport is deprecated and will be removed in a future release.
node-a corosync: [TOTEM ] WARNING SCTP transport is deprecated and will be removed in a future release.
node-a corosync: [TOTEM ] knet_link_set_config failed: Invalid argument (22)
node-a corosync: [MAIN ] Corosync Cluster Engine exiting with status 8 at totemknet.c:864.
node-a pmxcfs: [dcdb] crit: cpg_initialize failed: CS_ERR_LIBRARY (failed to connect to corosync)
node-a pmxcfs: [quorum] crit: quorum_initialize failed: CS_ERR_LIBRARY (failed to connect to corosync)
node-a pmxcfs: [confdb] crit: cmap_initialize failed: CS_ERR_LIBRARY (failed to connect to corosync)
The pmxcfs messages follow Corosync's startup failure. pve-cluster itself remained active.

This read-only Python probe queries the installed library's transport API:

python3 - <<'PY'
import ctypes as c
knet = c.CDLL('/lib/x86_64-linux-gnu/libknet.so.1', use_errno=True)
lookup = knet.knet_get_transport_id_by_name
lookup.argtypes = [c.c_char_p]
lookup.restype = c.c_uint8
for name in (b'UDP', b'SCTP'):
c.set_errno(0)
transport_id = lookup(name)
print(f'{name.decode()} id={transport_id} errno={c.get_errno()}')
PY
Results on the affected node:

UDP id=1 errno=0
SCTP id=255 errno=22
SCTP is unavailable in this installed userspace library. Upstream Kronosnet 1.32 deprecated SCTP and disabled it by default. A Proxmox patch submitted on 7 September 2026 adds --enable-libknet-sctp to preserve existing installations. It has a Tested-by reply; I have not verified availability in a released package.

IPv6 was initially suspected: net.ipv6.conf.{all,default,lo}.disable_ipv6 were all 1, with no ipv6.disable=1 kernel parameter. However, all configured links were IPv4, and the library rejected SCTP independently of any link address. The IPv6-link explanation does not account for this failure. IPv6 settings may still matter in other SCTP cases.

For prevention, I am attaching pve-sctp-hold.sh v1.0.0. It preserves the currently installed, SCTP-capable package combination; it does not prescribe old versions. Run locally as root on each relevant node:

bash pve-sctp-hold.sh --check
bash pve-sctp-hold.sh --apply
--check is the default and is read-only. Before applying holds, the script requires root, PVE 9 on Debian 13, intact installed corosync/libknet1t64 packages, SCTP support, an active Corosync service, and quorum. It then uses APT holds for those two packages at their installed versions. It performs no upgrades, downgrades, service restarts, or configuration changes. --release only releases those two holds. It cannot repair an already affected installation; holds also postpone subsequent fixes.

The script passed 44 isolated tests using simulated package/service state and small compiled ABI fixtures. These were not production Proxmox tests; application on the remaining cluster nodes is still pending. Run package operations one node at a time, with no concurrent package manager during the checks and hold operation.

A possible temporary recovery candidate is corosync=3.1.10-pve2 with libknet1t64=1.31-pve1. First evaluate the package transaction without changing anything:

apt-get -s --no-remove --allow-downgrades install \
corosync=3.1.10-pve2 libknet1t64=1.31-pve1
Package availability and the complete binary dependency set still need verification. The pve3 DSCP change uses the newer libknet API (1.32 or later), so downgrading only libknet is not a verified solution. Returning to pve2 would also remove pve3 CPG fixes for a cluster-wide /etc/pve synchronization hang that could trigger HA watchdog fencing. This is an emergency candidate, not a tested fix, and the script does not execute it.

Could Proxmox confirm which released package restores SCTP support, its repository availability, and the recommended recovery path while retaining the pve3 CPG fixes?

#!/usr/bin/env bash
# pve-sctp-hold.sh 1.0.0 — temporary SCTP update protection for PVE 9.
# Pins the INSTALLED pair with APT holds after read-only checks.
# No package installation, downgrade, service restart or Corosync config edit.
set -Eeuo pipefail
export LC_ALL=C

readonly SCRIPT_VERSION='1.0.0'
readonly -a PACKAGES=(corosync libknet1t64)
mode=${1:---check}

fail() { printf 'FEHLER: %s\n' "$*" >&2; exit 1; }
usage() {
    cat <<'HELP'
Aufruf: bash pve-sctp-hold.sh [--check|--apply|--release|--help]
  --check    Nur pruefen (Standard), setzt keine Holds.
  --apply    SCTP und Clusterzustand pruefen, installierte Versionen festhalten.
  --release  Die Holds fuer corosync und libknet1t64 wieder aufheben.

Fuer Proxmox VE 9 / Debian 13. Als root VOR dem Paketupdate ausfuehren.
Keine anderen Paketoperationen parallel starten. Kein Major-Upgrade-Werkzeug.
--release installiert nichts und prueft keinen kuenftigen Paketkandidaten.
HELP
}

[[ $# -le 1 ]] || { usage >&2; exit 2; }
case "$mode" in
    --help|-h) usage; exit 0 ;;
    --check|--apply|--release) ;;
    *) usage >&2; exit 2 ;;
esac
[[ $EUID -eq 0 ]] || fail 'Bitte als root ausfuehren.'
for pin_cmd in dpkg-query apt-mark python3; do
    command -v "$pin_cmd" >/dev/null || fail "Befehl fehlt: $pin_cmd"
done
printf 'pve-sctp-hold.sh %s — %s\n' "$SCRIPT_VERSION" "$mode"

package_version() {
    local pin_row pin_state pin_error pin_version
    pin_row=$(dpkg-query -W -f='${db:Status-Status}\t${db:Status-Eflag}\t${Version}\n' "$1") ||
        fail "Paket nicht abfragbar: $1"
    read -r pin_state pin_error pin_version <<<"$pin_row"
    [[ $pin_state == installed && $pin_error == ok && -n $pin_version ]] ||
        fail "Paket nicht vollstaendig installiert: $1 ($pin_row)"
    printf '%s\n' "$pin_version"
}

pve_version=$(package_version pve-manager)
[[ ${pve_version#*:} == 9.* ]] || fail "Unterstuetzt wird PVE 9; gefunden: $pve_version"
declare -A before_version held
for pin_pkg in "${PACKAGES[@]}"; do
    before_version[$pin_pkg]=$(package_version "$pin_pkg")
    printf '%s = %s\n' "$pin_pkg" "${before_version[$pin_pkg]}"
done

read_holds() {
    local pin_line pin_output
    pin_output=$(apt-mark showhold) || fail 'APT-Holds konnten nicht gelesen werden.'
    held=()
    while IFS= read -r pin_line; do
        [[ -n $pin_line ]] && held[${pin_line%%:*}]=1
    done <<<"$pin_output"
    return 0
}

if [[ $mode == --release ]]; then
    apt-mark unhold "${PACKAGES[@]}" || fail 'Freigabe fehlgeschlagen; Holds kontrollieren.'
    read_holds
    for pin_pkg in "${PACKAGES[@]}"; do
        [[ ${held[$pin_pkg]:-0} == 0 ]] || fail "Hold weiterhin aktiv: $pin_pkg"
    done
    printf 'FREIGEGEBEN: Beide Holds entfernt. Paketversionen wurden nicht geaendert.\n'
    exit 0
fi

for pin_cmd in ldd systemctl pvecm; do
    command -v "$pin_cmd" >/dev/null || fail "Befehl fehlt: $pin_cmd"
done

check_library() {
    python3 - <<'PY'
import ctypes
import os
import re
import shutil
import subprocess

try:
    with open('/etc/os-release', encoding='utf-8') as stream:
        release = dict(line.rstrip().split('=', 1) for line in stream
                       if '=' in line and not line.startswith('#'))
    if (release.get('ID', '').strip('"') != 'debian' or
            release.get('VERSION_ID', '').strip('"') != '13'):
        raise RuntimeError('Unterstuetzt wird Debian 13 mit Proxmox VE 9.')
    binary = shutil.which('corosync')
    if not binary:
        raise RuntimeError('corosync-Binary fehlt.')
    result = subprocess.run(['ldd', '-r', binary], capture_output=True, text=True)
    output = result.stdout + result.stderr
    if result.returncode or re.search(r'not found|undefined symbol:', output):
        raise RuntimeError('Corosync-Bibliotheken/Symbole nicht aufloesbar:\n' + output)
    paths = re.findall(r'^\s*libknet\.so\S*\s+=>\s+(\S+)', output, re.MULTILINE)
    if len(paths) != 1 or not os.path.isfile(paths[0]):
        raise RuntimeError('libknet-Pfad nicht eindeutig: ' + repr(paths))
    library = ctypes.CDLL(paths[0], use_errno=True)
    lookup = library.knet_get_transport_id_by_name
    lookup.argtypes = [ctypes.c_char_p]
    lookup.restype = ctypes.c_uint8
    ctypes.set_errno(0)
    transport_id = lookup(b'SCTP')
    error = ctypes.get_errno()
    print(f'SCTP: ID={transport_id}, errno={error}; Bibliothek: {paths[0]}')
    if transport_id != 2 or error != 0:
        raise RuntimeError('SCTP fehlt in der installierten libknet. '
                           'Dieser Paketstand wird nicht als updatebereit freigegeben. '
                           'Auch ein noch laufender Corosync kann nach Neustart ausfallen.')
except (OSError, ValueError, AttributeError, RuntimeError) as exc:
    raise SystemExit('FEHLER: ' + str(exc))
PY
}

check_library || fail 'Bibliothekspruefung fehlgeschlagen; keine neuen Holds gesetzt.'
systemctl is-active --quiet corosync || fail 'Corosync laeuft nicht; erst Dienstzustand klaeren.'
quorum_output=$(pvecm status) || fail 'Clusterstatus nicht abfragbar.'
[[ $quorum_output =~ (^|$'\n')[[:space:]]*Quorate:[[:space:]]+Yes([[:space:]]|$) ]] ||
    fail 'Kein bestaetigtes Quorum; kein Update starten.'
read_holds
for pin_pkg in "${PACKAGES[@]}"; do
    printf 'Hold %s: %s\n' "$pin_pkg" "${held[$pin_pkg]:-0}"
done

if [[ $mode == --check ]]; then
    printf 'PRUEFUNG OK: SCTP vorhanden, Corosync aktiv, Quorum vorhanden.\n'
    printf 'Mit --apply diese beiden installierten Paketversionen festhalten.\n'
    exit 0
fi

for pin_pkg in "${PACKAGES[@]}"; do
    [[ $(package_version "$pin_pkg") == "${before_version[$pin_pkg]}" ]] ||
        fail 'Paketversion hat sich waehrend der Pruefung geaendert; nicht parallel updaten.'
done
apt-mark hold "${PACKAGES[@]}" ||
    fail 'Hold fehlgeschlagen oder nur teilweise gesetzt; apt-mark showhold kontrollieren.'
read_holds
for pin_pkg in "${PACKAGES[@]}"; do
    [[ ${held[$pin_pkg]:-0} == 1 ]] || fail "Hold fehlt: $pin_pkg; nicht updaten."
    [[ $(package_version "$pin_pkg") == "${before_version[$pin_pkg]}" ]] ||
        fail 'Paketversion hat sich beim Setzen geaendert. Holds kontrollieren; nicht updaten.'
done
printf 'GESCHUETZT: corosync=%s und libknet1t64=%s stehen auf Hold.\n' \
    "${before_version[corosync]}" "${before_version[libknet1t64]}"
printf 'Der Schutz gilt fuer APT-Updates, die Holds beachten; kein Major-Upgrade.\n'