Proxmox cluster, loss of node causing process hang

FingerlessGloves

Well-Known Member
Oct 22, 2019
43
6
48
I have a 2 node proxmox cluster running version 8.4.14, with a Raspberry Pi acting a as a qdevice for quorum. Kernel 6.8.12-15-pve

I've taken pve-2 out of the datacentre to do some hardware work on it, and pve-1 had all the VMs running on it prior to powering it off. I then noticed my monitoring was having issues monitoring pve-1, but all the VMs on pve-1 were reporting just fine.

After a closer look, I couldn't SSH the PVE host or login via the WebUI. I can login using the console just fine. When I try do anything where I believe dns is involved the process just hang, and I can't even kill them using `kill -s 9`. pvestatd is usually the first process to hang in dmesg.

I've done this before without issues, and I have pve-1 out of action for a few weeks last month, because the motherboard had a fault, which has since been fixed, pve-1 hasn't been reinstalled though.

`ip address`
`ping google.com`
`lspci` hangs when it does to lookup `openat(AT_FDCWD, "/sys/bus/pci/devices/0000:03:00.0/config", O_RDONLY) = 3` which is the enp3s0 which is the migration network NIC which is down, as it's directly connected to pve-2 which isn't there.

The following commands do work as expected
`ping 1.1.1.1`
`dig google.com`

Feels like corosync or ha-manager is doing something really strange, to make part of the network stack hang?

During trying to debug this myself, I've tried restarting pve-1, but after a few minutes it just goes back to this broken state. The HA VMs start and things work, and then suddenly the problem comes back. I can't find anything in dmesg or journalctl --boot which I can see that's causing this result. pve-1 been working great for weeks, until I took pve-2 out, also rebooted pve-1 a few times during that 2 week period too.

`pvecm status`
Code:
Cluster information
-------------------
Name:             pve-coventry
Config Version:   3
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Fri Oct 17 23:16:17 2025
Quorum provider:  corosync_votequorum
Nodes:            1
Node ID:          0x00000001
Ring ID:          1.395
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   2
Highest expected: 2
Total votes:      2
Quorum:           2
Flags:            Quorate Qdevice

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
0x00000001          1    A,V,NMW 172.30.1.10 (local)
0x00000000          1            Qdevice

`ha-manager status`
Code:
quorum OK
master pve-1 (active, Fri Oct 17 23:16:44 2025)
lrm pve-1 (active, Fri Oct 17 23:16:45 2025)
lrm pve-2 (old timestamp - dead?, Fri Oct 17 17:24:41 2025)
service vm:102 (pve-1, started)
service vm:103 (pve-1, started)
service vm:105 (pve-1, started)
service vm:106 (pve-1, started)
service vm:107 (pve-1, started)
service vm:108 (pve-1, started)
service vm:109 (pve-1, started)
service vm:110 (pve-1, started)
service vm:111 (pve-1, started)
service vm:112 (pve-1, started)
service vm:114 (pve-1, started)

Code:
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.

passwd:         files systemd
group:          files systemd
shadow:         files systemd
gshadow:        files systemd

hosts:          files dns
networks:       files

protocols:      db files
services:       db files
ethers:         db files
rpc:            db files

netgroup:       nis

`/etc/resolv.conf`
Code:
nameserver 172.30.0.1


Also then causes webui VMs to go to unknown state, yet the HA stays it's OK
1760740277783.png

Love to get some help on this, as this was working and wondering if an update has broke something here...
 
Last edited:
I also did some strace's to hopefully hope show where the issue happens.

strace of `ping google.com`
Code:
strace ping google.com
execve("/usr/bin/ping", ["ping", "google.com"], 0x7ffdeff82248 /* 17 vars */) = 0
brk(NULL)                               = 0x576275af1000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x78a550e2d000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=30127, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 30127, PROT_READ, MAP_PRIVATE, 3, 0) = 0x78a550e25000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libcap.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260t\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=47288, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 45128, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x78a550e19000
mmap(0x78a550e1c000, 20480, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x78a550e1c000
mmap(0x78a550e21000, 8192, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8000) = 0x78a550e21000
mmap(0x78a550e23000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xa000) = 0x78a550e23000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libidn2.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=198776, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 196760, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x78a550de8000
mmap(0x78a550dea000, 28672, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x78a550dea000
mmap(0x78a550df1000, 155648, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x9000) = 0x78a550df1000
mmap(0x78a550e17000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2f000) = 0x78a550e17000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20t\2\0\0\0\0\0"..., 832) = 832
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=1926232, ...}, AT_EMPTY_PATH) = 0
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
mmap(NULL, 1974096, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x78a550c06000
mmap(0x78a550c2c000, 1400832, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x26000) = 0x78a550c2c000
mmap(0x78a550d82000, 339968, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17c000) = 0x78a550d82000
mmap(0x78a550dd5000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1cf000) = 0x78a550dd5000
mmap(0x78a550ddb000, 53072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x78a550ddb000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libunistring.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=1792040, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 1792264, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x78a550a50000
mmap(0x78a550a62000, 262144, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x12000) = 0x78a550a62000
mmap(0x78a550aa2000, 1437696, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x52000) = 0x78a550aa2000
mmap(0x78a550c01000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b1000) = 0x78a550c01000
close(3)                                = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x78a550a4e000
arch_prctl(ARCH_SET_FS, 0x78a550a4ed00) = 0
set_tid_address(0x78a550a4efd0)         = 17592
set_robust_list(0x78a550a4efe0, 24)     = 0
rseq(0x78a550a4f620, 0x20, 0, 0x53053053) = 0
mprotect(0x78a550dd5000, 16384, PROT_READ) = 0
mprotect(0x78a550c01000, 16384, PROT_READ) = 0
mprotect(0x78a550e17000, 4096, PROT_READ) = 0
mprotect(0x78a550e23000, 4096, PROT_READ) = 0
mprotect(0x576238070000, 4096, PROT_READ) = 0
mprotect(0x78a550e60000, 8192, PROT_READ) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
munmap(0x78a550e25000, 30127)           = 0
prctl(PR_CAPBSET_READ, CAP_MAC_OVERRIDE) = 1
prctl(PR_CAPBSET_READ, 0x30 /* CAP_??? */) = -1 EINVAL (Invalid argument)
prctl(PR_CAPBSET_READ, CAP_CHECKPOINT_RESTORE) = 1
prctl(PR_CAPBSET_READ, 0x2c /* CAP_??? */) = -1 EINVAL (Invalid argument)
prctl(PR_CAPBSET_READ, 0x2a /* CAP_??? */) = -1 EINVAL (Invalid argument)
prctl(PR_CAPBSET_READ, 0x29 /* CAP_??? */) = -1 EINVAL (Invalid argument)
getrandom("\x83\xa2\x28\xa2\xc8\x41\xdd\x34", 8, GRND_NONBLOCK) = 8
brk(NULL)                               = 0x576275af1000
brk(0x576275b12000)                     = 0x576275b12000
capget({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, NULL) = 0
capget({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, {effective=1<<CAP_CHOWN|1<<CAP_DAC_OVERRIDE|1<<CAP_DAC_READ_SEARCH|1<<CAP_FOWNER|1<<CAP_FSETID|1<<CAP_KILL|1<<CAP_SETGID|1<<CAP_SETUID|1<<CAP_SETPCAP|1<<CAP_LINUX_IMMUTABLE|1<<CAP_NET_BIND_SERVICE|1<<CAP_NET_BROADCAST|1<<CAP_NET_ADMIN|1<<CAP_NET_RAW|1<<CAP_IPC_LOCK|1<<CAP_IPC_OWNER|1<<CAP_SYS_MODULE|1<<CAP_SYS_RAWIO|1<<CAP_SYS_CHROOT|1<<CAP_SYS_PTRACE|1<<CAP_SYS_PACCT|1<<CAP_SYS_ADMIN|1<<CAP_SYS_BOOT|1<<CAP_SYS_NICE|1<<CAP_SYS_RESOURCE|1<<CAP_SYS_TIME|1<<CAP_SYS_TTY_CONFIG|1<<CAP_MKNOD|1<<CAP_LEASE|1<<CAP_AUDIT_WRITE|1<<CAP_AUDIT_CONTROL|1<<CAP_SETFCAP|1<<CAP_MAC_OVERRIDE|1<<CAP_MAC_ADMIN|1<<CAP_SYSLOG|1<<CAP_WAKE_ALARM|1<<CAP_BLOCK_SUSPEND|1<<CAP_AUDIT_READ|1<<CAP_PERFMON|1<<CAP_BPF|1<<CAP_CHECKPOINT_RESTORE, permitted=1<<CAP_CHOWN|1<<CAP_DAC_OVERRIDE|1<<CAP_DAC_READ_SEARCH|1<<CAP_FOWNER|1<<CAP_FSETID|1<<CAP_KILL|1<<CAP_SETGID|1<<CAP_SETUID|1<<CAP_SETPCAP|1<<CAP_LINUX_IMMUTABLE|1<<CAP_NET_BIND_SERVICE|1<<CAP_NET_BROADCAST|1<<CAP_NET_ADMIN|1<<CAP_NET_RAW|1<<CAP_IPC_LOCK|1<<CAP_IPC_OWNER|1<<CAP_SYS_MODULE|1<<CAP_SYS_RAWIO|1<<CAP_SYS_CHROOT|1<<CAP_SYS_PTRACE|1<<CAP_SYS_PACCT|1<<CAP_SYS_ADMIN|1<<CAP_SYS_BOOT|1<<CAP_SYS_NICE|1<<CAP_SYS_RESOURCE|1<<CAP_SYS_TIME|1<<CAP_SYS_TTY_CONFIG|1<<CAP_MKNOD|1<<CAP_LEASE|1<<CAP_AUDIT_WRITE|1<<CAP_AUDIT_CONTROL|1<<CAP_SETFCAP|1<<CAP_MAC_OVERRIDE|1<<CAP_MAC_ADMIN|1<<CAP_SYSLOG|1<<CAP_WAKE_ALARM|1<<CAP_BLOCK_SUSPEND|1<<CAP_AUDIT_READ|1<<CAP_PERFMON|1<<CAP_BPF|1<<CAP_CHECKPOINT_RESTORE, inheritable=0}) = 0
capget({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, NULL) = 0
capset({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, {effective=0, permitted=1<<CAP_NET_ADMIN|1<<CAP_NET_RAW, inheritable=0}) = 0
prctl(PR_SET_KEEPCAPS, 1)               = 0
getuid()                                = 0
setuid(0)                               = 0
prctl(PR_SET_KEEPCAPS, 0)               = 0
getuid()                                = 0
openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=3048928, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 3048928, PROT_READ, MAP_PRIVATE, 3, 0) = 0x78a550765000
close(3)                                = 0
capget({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, NULL) = 0
capget({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, {effective=0, permitted=1<<CAP_NET_ADMIN|1<<CAP_NET_RAW, inheritable=0}) = 0
capset({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, {effective=1<<CAP_NET_RAW, permitted=1<<CAP_NET_ADMIN|1<<CAP_NET_RAW, inheritable=0}) = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_ICMP) = -1 EACCES (Permission denied)
socket(AF_INET, SOCK_RAW, IPPROTO_ICMP) = 3
socket(AF_INET6, SOCK_DGRAM, IPPROTO_ICMPV6) = -1 EACCES (Permission denied)
socket(AF_INET6, SOCK_RAW, IPPROTO_ICMPV6) = 4
capget({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, NULL) = 0
capget({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, {effective=1<<CAP_NET_RAW, permitted=1<<CAP_NET_ADMIN|1<<CAP_NET_RAW, inheritable=0}) = 0
capset({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, {effective=0, permitted=1<<CAP_NET_ADMIN|1<<CAP_NET_RAW, inheritable=0}) = 0
openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache", O_RDONLY) = 5
newfstatat(5, "", {st_mode=S_IFREG|0644, st_size=27028, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 27028, PROT_READ, MAP_SHARED, 5, 0) = 0x78a550e26000
close(5)                                = 0
futex(0x78a550ddaa4c, FUTEX_WAKE_PRIVATE, 2147483647) = 0
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 5
connect(5, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(5)                                = 0
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 5
connect(5, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(5)                                = 0
newfstatat(AT_FDCWD, "/etc/nsswitch.conf", {st_mode=S_IFREG|0644, st_size=526, ...}, 0) = 0
newfstatat(AT_FDCWD, "/", {st_mode=S_IFDIR|0755, st_size=22, ...}, 0) = 0
openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 5
newfstatat(5, "", {st_mode=S_IFREG|0644, st_size=526, ...}, AT_EMPTY_PATH) = 0
read(5, "# /etc/nsswitch.conf\n#\n# Example"..., 1024) = 526
read(5, "", 1024)                       = 0
newfstatat(5, "", {st_mode=S_IFREG|0644, st_size=526, ...}, AT_EMPTY_PATH) = 0
close(5)                                = 0
newfstatat(AT_FDCWD, "/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=22, ...}, 0) = 0
openat(AT_FDCWD, "/etc/host.conf", O_RDONLY|O_CLOEXEC) = 5
newfstatat(5, "", {st_mode=S_IFREG|0644, st_size=9, ...}, AT_EMPTY_PATH) = 0
read(5, "multi on\n", 512)              = 9
read(5, "", 512)                        = 0
close(5)                                = 0
futex(0x78a550de242c, FUTEX_WAKE_PRIVATE, 2147483647) = 0
openat(AT_FDCWD, "/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 5
newfstatat(5, "", {st_mode=S_IFREG|0644, st_size=22, ...}, AT_EMPTY_PATH) = 0
read(5, "nameserver 172.30.0.1\n", 512) = 22
read(5, "", 512)                        = 0
uname({sysname="Linux", nodename="pve-1", ...}) = 0
newfstatat(5, "", {st_mode=S_IFREG|0644, st_size=22, ...}, AT_EMPTY_PATH) = 0
close(5)                                = 0
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 5
newfstatat(5, "", {st_mode=S_IFREG|0644, st_size=363, ...}, AT_EMPTY_PATH) = 0
lseek(5, 0, SEEK_SET)                   = 0
read(5, "127.0.0.1 localhost.localdomain "..., 512) = 363
read(5, "", 512)                        = 0
close(5)                                = 0
socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 5
setsockopt(5, SOL_IP, IP_RECVERR, [1], 4) = 0
connect(5, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("172.30.0.1")}, 16) = 0
poll([{fd=5, events=POLLOUT}], 1, 0)    = 1 ([{fd=5, revents=POLLOUT}])
sendmmsg(5, [{msg_hdr={msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="M)\1\0\0\1\0\0\0\0\0\0\6google\3com\0\0\1\0\1", iov_len=28}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, msg_len=28}, {msg_hdr={msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\214(\1\0\0\1\0\0\0\0\0\0\6google\3com\0\0\34\0\1", iov_len=28}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, msg_len=28}], 2, MSG_NOSIGNAL) = 2
poll([{fd=5, events=POLLIN}], 1, 5000)  = 1 ([{fd=5, revents=POLLIN}])
ioctl(5, FIONREAD, [124])               = 0
recvfrom(5, "M)\201\200\0\1\0\6\0\0\0\0\6google\3com\0\0\1\0\1\300\f\0\1"..., 2048, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("172.30.0.1")}, [28 => 16]) = 124
poll([{fd=5, events=POLLIN}], 1, 4994)  = 1 ([{fd=5, revents=POLLIN}])
ioctl(5, FIONREAD, [140])               = 0
recvfrom(5, "\214(\201\200\0\1\0\4\0\0\0\0\6google\3com\0\0\34\0\1\300\f\0\34"..., 65536, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("172.30.0.1")}, [28 => 16]) = 140
close(5)                                = 0
openat(AT_FDCWD, "/etc/gai.conf", O_RDONLY|O_CLOEXEC) = 5
newfstatat(5, "", {st_mode=S_IFREG|0644, st_size=2584, ...}, AT_EMPTY_PATH) = 0
newfstatat(5, "", {st_mode=S_IFREG|0644, st_size=2584, ...}, AT_EMPTY_PATH) = 0
read(5, "# Configuration for getaddrinfo("..., 3072) = 2584
read(5, "", 3072)                       = 0
close(5)                                = 0
futex(0x78a550de12e4, FUTEX_WAKE_PRIVATE, 2147483647) = 0
socket(AF_NETLINK, SOCK_RAW|SOCK_CLOEXEC, NETLINK_ROUTE) = 5
bind(5, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 0
getsockname(5, {sa_family=AF_NETLINK, nl_pid=17592, nl_groups=00000000}, [12]) = 0
sendto(5, [{nlmsg_len=20, nlmsg_type=RTM_GETADDR, nlmsg_flags=NLM_F_REQUEST|NLM_F_DUMP, nlmsg_seq=1760739954, nlmsg_pid=0}, {ifa_family=AF_UNSPEC, ...}], 20, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12
 
strace of `ip address`
Code:
execve("/usr/sbin/ip", ["ip", "address"], 0x7ffdf8653928 /* 17 vars */) = 0
brk(NULL)                               = 0x649ac8d2e000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x76d5b1a22000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=30127, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 30127, PROT_READ, MAP_PRIVATE, 3, 0) = 0x76d5b1a1a000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libbpf.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=358880, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 361120, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x76d5b19c1000
mmap(0x76d5b19cb000, 217088, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xa000) = 0x76d5b19cb000
mmap(0x76d5b1a00000, 90112, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3f000) = 0x76d5b1a00000
mmap(0x76d5b1a16000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x54000) = 0x76d5b1a16000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libelf.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=109088, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 106928, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x76d5b19a6000
mmap(0x76d5b19a9000, 73728, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x76d5b19a9000
mmap(0x76d5b19bb000, 16384, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15000) = 0x76d5b19bb000
mmap(0x76d5b19bf000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19000) = 0x76d5b19bf000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libmnl.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\"\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=26616, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 28688, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x76d5b199e000
mprotect(0x76d5b19a0000, 16384, PROT_NONE) = 0
mmap(0x76d5b19a0000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x76d5b19a0000
mmap(0x76d5b19a2000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x76d5b19a2000
mmap(0x76d5b19a4000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x76d5b19a4000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libbsd.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=84840, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 86224, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x76d5b1988000
mmap(0x76d5b198c000, 49152, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x76d5b198c000
mmap(0x76d5b1998000, 12288, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x10000) = 0x76d5b1998000
mmap(0x76d5b199b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x13000) = 0x76d5b199b000
mmap(0x76d5b199d000, 208, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x76d5b199d000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libcap.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260t\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=47288, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 45128, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x76d5b197c000
mmap(0x76d5b197f000, 20480, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x76d5b197f000
mmap(0x76d5b1984000, 8192, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8000) = 0x76d5b1984000
mmap(0x76d5b1986000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xa000) = 0x76d5b1986000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20t\2\0\0\0\0\0"..., 832) = 832
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=1926232, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x76d5b197a000
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
mmap(NULL, 1974096, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x76d5b1798000
mmap(0x76d5b17be000, 1400832, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x26000) = 0x76d5b17be000
mmap(0x76d5b1914000, 339968, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17c000) = 0x76d5b1914000
mmap(0x76d5b1967000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1cf000) = 0x76d5b1967000
mmap(0x76d5b196d000, 53072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x76d5b196d000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libz.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=121280, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 123280, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x76d5b1779000
mmap(0x76d5b177c000, 77824, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x76d5b177c000
mmap(0x76d5b178f000, 28672, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0x76d5b178f000
mmap(0x76d5b1796000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c000) = 0x76d5b1796000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libmd.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=47312, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 49384, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x76d5b176c000
mmap(0x76d5b176e000, 28672, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x76d5b176e000
mmap(0x76d5b1775000, 8192, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x9000) = 0x76d5b1775000
mmap(0x76d5b1777000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xa000) = 0x76d5b1777000
close(3)                                = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x76d5b176a000
arch_prctl(ARCH_SET_FS, 0x76d5b176b300) = 0
set_tid_address(0x76d5b176b5d0)         = 18001
set_robust_list(0x76d5b176b5e0, 24)     = 0
rseq(0x76d5b176bc20, 0x20, 0, 0x53053053) = 0
mprotect(0x76d5b1967000, 16384, PROT_READ) = 0
mprotect(0x76d5b1777000, 4096, PROT_READ) = 0
mprotect(0x76d5b1796000, 4096, PROT_READ) = 0
mprotect(0x76d5b1986000, 4096, PROT_READ) = 0
mprotect(0x76d5b199b000, 4096, PROT_READ) = 0
mprotect(0x76d5b19a4000, 4096, PROT_READ) = 0
mprotect(0x76d5b19bf000, 4096, PROT_READ) = 0
mprotect(0x76d5b1a16000, 12288, PROT_READ) = 0
mprotect(0x649aaebe3000, 12288, PROT_READ) = 0
mprotect(0x76d5b1a55000, 8192, PROT_READ) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
munmap(0x76d5b1a1a000, 30127)           = 0
prctl(PR_CAPBSET_READ, CAP_MAC_OVERRIDE) = 1
prctl(PR_CAPBSET_READ, 0x30 /* CAP_??? */) = -1 EINVAL (Invalid argument)
prctl(PR_CAPBSET_READ, CAP_CHECKPOINT_RESTORE) = 1
prctl(PR_CAPBSET_READ, 0x2c /* CAP_??? */) = -1 EINVAL (Invalid argument)
prctl(PR_CAPBSET_READ, 0x2a /* CAP_??? */) = -1 EINVAL (Invalid argument)
prctl(PR_CAPBSET_READ, 0x29 /* CAP_??? */) = -1 EINVAL (Invalid argument)
getuid()                                = 0
socket(AF_NETLINK, SOCK_RAW|SOCK_CLOEXEC, NETLINK_ROUTE) = 3
setsockopt(3, SOL_SOCKET, SO_SNDBUF, [32768], 4) = 0
setsockopt(3, SOL_SOCKET, SO_RCVBUF, [1048576], 4) = 0
setsockopt(3, SOL_NETLINK, NETLINK_EXT_ACK, [1], 4) = 0
bind(3, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 0
getsockname(3, {sa_family=AF_NETLINK, nl_pid=18001, nl_groups=00000000}, [12]) = 0
setsockopt(3, SOL_NETLINK, NETLINK_GET_STRICT_CHK, [1], 4) = 0
sendto(3, [{nlmsg_len=40, nlmsg_type=RTM_GETLINK, nlmsg_flags=NLM_F_REQUEST|NLM_F_DUMP, nlmsg_seq=1760740011, nlmsg_pid=0}, {ifi_family=AF_UNSPEC, ifi_type=ARPHRD_NETROM, ifi_index=0, ifi_flags=0, ifi_change=0}, [{nla_len=8, nla_type=IFLA_EXT_MASK}, RTEXT_FILTER_VF]], 40, 0, NULL, 0
 
Example kernel reported hangs

Code:
Oct 17 23:06:07 pve-1 kernel: INFO: task kworker/11:1:201 blocked for more than 122 seconds.
Oct 17 23:06:07 pve-1 kernel:       Tainted: P           O       6.8.12-15-pve #1
Oct 17 23:06:07 pve-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 17 23:06:07 pve-1 kernel: task:kworker/11:1    state:D stack:0     pid:201   tgid:201   ppid:2      flags:0x00004000
Oct 17 23:06:07 pve-1 kernel: Workqueue: events switchdev_deferred_process_work
Oct 17 23:06:07 pve-1 kernel: Call Trace:
Oct 17 23:06:07 pve-1 kernel:  <TASK>
Oct 17 23:06:07 pve-1 kernel:  __schedule+0x42b/0x1500
Oct 17 23:06:07 pve-1 kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Oct 17 23:06:07 pve-1 kernel:  schedule+0x33/0x110
Oct 17 23:06:07 pve-1 kernel:  schedule_preempt_disabled+0x15/0x30
Oct 17 23:06:07 pve-1 kernel:  __mutex_lock.constprop.0+0x3f8/0x7a0
Oct 17 23:06:07 pve-1 kernel:  ? add_timer_on+0xf9/0x150
Oct 17 23:06:07 pve-1 kernel:  __mutex_lock_slowpath+0x13/0x20
Oct 17 23:06:07 pve-1 kernel:  mutex_lock+0x3c/0x50
Oct 17 23:06:07 pve-1 kernel:  rtnl_lock+0x15/0x20
Oct 17 23:06:07 pve-1 kernel:  switchdev_deferred_process_work+0xe/0x20
Oct 17 23:06:07 pve-1 kernel:  process_one_work+0x17f/0x3a0
Oct 17 23:06:07 pve-1 kernel:  worker_thread+0x306/0x440
Oct 17 23:06:07 pve-1 kernel:  ? __pfx_worker_thread+0x10/0x10
Oct 17 23:06:07 pve-1 kernel:  kthread+0xef/0x120
Oct 17 23:06:07 pve-1 kernel:  ? __pfx_kthread+0x10/0x10
Oct 17 23:06:07 pve-1 kernel:  ret_from_fork+0x44/0x70
Oct 17 23:06:07 pve-1 kernel:  ? __pfx_kthread+0x10/0x10
Oct 17 23:06:07 pve-1 kernel:  ret_from_fork_asm+0x1b/0x30
Oct 17 23:06:07 pve-1 kernel:  </TASK>
Oct 17 23:06:07 pve-1 kernel: INFO: task pvestatd:2189 blocked for more than 122 seconds.
Oct 17 23:06:07 pve-1 kernel:       Tainted: P           O       6.8.12-15-pve #1
Oct 17 23:06:07 pve-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 17 23:06:07 pve-1 kernel: task:pvestatd        state:D stack:0     pid:2189  tgid:2189  ppid:1      flags:0x00000002
Oct 17 23:06:07 pve-1 kernel: Call Trace:
Oct 17 23:06:07 pve-1 kernel:  <TASK>
Oct 17 23:06:07 pve-1 kernel:  __schedule+0x42b/0x1500
Oct 17 23:06:07 pve-1 kernel:  ? kmem_cache_alloc+0x133/0x370
Oct 17 23:06:07 pve-1 kernel:  schedule+0x33/0x110
Oct 17 23:06:07 pve-1 kernel:  schedule_preempt_disabled+0x15/0x30
Oct 17 23:06:07 pve-1 kernel:  __mutex_lock.constprop.0+0x3f8/0x7a0
Oct 17 23:06:07 pve-1 kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Oct 17 23:06:07 pve-1 kernel:  __mutex_lock_slowpath+0x13/0x20
Oct 17 23:06:07 pve-1 kernel:  mutex_lock+0x3c/0x50
Oct 17 23:06:07 pve-1 kernel:  netlink_dump+0x34f/0x3c0
Oct 17 23:06:07 pve-1 kernel:  __netlink_dump_start+0x1f4/0x2a0
Oct 17 23:06:07 pve-1 kernel:  ? __pfx_rtnl_dump_all+0x10/0x10
Oct 17 23:06:07 pve-1 kernel:  rtnetlink_rcv_msg+0x298/0x3e0
Oct 17 23:06:07 pve-1 kernel:  ? __pfx_rtnl_dump_all+0x10/0x10
Oct 17 23:06:07 pve-1 kernel:  ? __pfx_rtnetlink_rcv_msg+0x10/0x10
Oct 17 23:06:07 pve-1 kernel:  netlink_rcv_skb+0x5a/0x110
Oct 17 23:06:07 pve-1 kernel:  rtnetlink_rcv+0x15/0x30
Oct 17 23:06:07 pve-1 kernel:  netlink_unicast+0x1b0/0x2a0
Oct 17 23:06:07 pve-1 kernel:  netlink_sendmsg+0x214/0x470
Oct 17 23:06:07 pve-1 kernel:  __sys_sendto+0x21b/0x230
Oct 17 23:06:07 pve-1 kernel:  __x64_sys_sendto+0x24/0x40
Oct 17 23:06:07 pve-1 kernel:  x64_sys_call+0x1f5c/0x2480
Oct 17 23:06:07 pve-1 kernel:  do_syscall_64+0x81/0x170
Oct 17 23:06:07 pve-1 kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Oct 17 23:06:07 pve-1 kernel:  ? arch_exit_to_user_mode_prepare.constprop.0+0x1a/0xe0
Oct 17 23:06:07 pve-1 kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Oct 17 23:06:07 pve-1 kernel:  ? syscall_exit_to_user_mode+0x43/0x1e0
Oct 17 23:06:07 pve-1 kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Oct 17 23:06:07 pve-1 kernel:  ? do_syscall_64+0x8d/0x170
Oct 17 23:06:07 pve-1 kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Oct 17 23:06:07 pve-1 kernel:  ? arch_exit_to_user_mode_prepare.constprop.0+0x1a/0xe0
Oct 17 23:06:07 pve-1 kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Oct 17 23:06:07 pve-1 kernel:  ? syscall_exit_to_user_mode+0x43/0x1e0
Oct 17 23:06:07 pve-1 kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Oct 17 23:06:07 pve-1 kernel:  ? do_syscall_64+0x8d/0x170
Oct 17 23:06:07 pve-1 kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Oct 17 23:06:07 pve-1 kernel:  ? arch_exit_to_user_mode_prepare.constprop.0+0x1a/0xe0
Oct 17 23:06:07 pve-1 kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Oct 17 23:06:07 pve-1 kernel:  ? irqentry_exit_to_user_mode+0x38/0x1e0
Oct 17 23:06:07 pve-1 kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Oct 17 23:06:07 pve-1 kernel:  ? irqentry_exit+0x43/0x50
Oct 17 23:06:07 pve-1 kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Oct 17 23:06:07 pve-1 kernel:  ? exc_page_fault+0x94/0x1b0
Oct 17 23:06:07 pve-1 kernel:  entry_SYSCALL_64_after_hwframe+0x78/0x80
Oct 17 23:06:07 pve-1 kernel: RIP: 0033:0x7fea5b180ca3
Oct 17 23:06:07 pve-1 kernel: RSP: 002b:00007ffd0d138a98 EFLAGS: 00000202 ORIG_RAX: 000000000000002c
Oct 17 23:06:07 pve-1 kernel: RAX: ffffffffffffffda RBX: 00007ffd0d139b80 RCX: 00007fea5b180ca3
Oct 17 23:06:07 pve-1 kernel: RDX: 0000000000000014 RSI: 00007ffd0d139b80 RDI: 0000000000000009
Oct 17 23:06:07 pve-1 kernel: RBP: 00007ffd0d139bd0 R08: 00007ffd0d139b24 R09: 000000000000000c
Oct 17 23:06:07 pve-1 kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000009
Oct 17 23:06:07 pve-1 kernel: R13: 00007ffd0d139c78 R14: 00007ffd0d139c80 R15: 000055f06c2842a0
Oct 17 23:06:07 pve-1 kernel:  </TASK>
 
Many hours of trial and error, I think I found the issue. Powertop was putting the i226 card in to a different power control mode to allow better power saving, but clearly when nothing is connected to the i226, the driver locks up. So I've now stopped it doing that and so far it's been stable as I expect

If you don't hear back from me this will have been the resolution.
 
  • Like
Reactions: MarkusKo
That's for sure the issue, hopefully that'll get fixed in the driver at some point, so whole computer doesn't start to lockup when there's no ethernet plugged in. It may already be fixed in a later kernel, but since it'll be in use 99% of the time, I've just excluded it from getting power managed.