Hi, my standalone PVE node crashed 2 times this week. I noticed the GUI became unresponsive the first time. Second time it migrated to another NIC due to network crashes. I wasn't paying much attention with New Year and all.
My journalctl was flooded with 1300000 lines. I picked up the most important ones.
For as far as I can tell it seems to be caused by a network issue.
I have no idea how to troubleshoot this.
Here's my /etc/network/interfaces:
Can anyone help me find what's going on?
My system is Supermicro X10SRL-F with E5-2697A V4 and 256GB RAM.
My journalctl was flooded with 1300000 lines. I picked up the most important ones.
Code:
Dec 30 00:33:48 pph1 dhclient[480613]: DHCPDISCOVER on vmbr1 to 255.255.255.255 port 67 interval 1
Dec 30 00:33:48 pph1 dhclient[480613]: DHCPDISCOVER on fwpr2116p2 to 255.255.255.255 port 67 interval 1
Dec 30 00:33:48 pph1 dhclient[480613]: DHCPDISCOVER on veth2100i0 to 255.255.255.255 port 67 interval 1
Dec 30 00:33:48 pph1 dhclient[480613]: DHCPDISCOVER on fwpr2110p1 to 255.255.255.255 port 67 interval 1
Dec 30 00:33:48 pph1 dhclient[480613]: DHCPDISCOVER on veth2112i1 to 255.255.255.255 port 67 interval 1
Dec 30 00:33:48 pph1 dhclient[480613]: DHCPDISCOVER on veth2051i1 to 255.255.255.255 port 67 interval 1
Dec 30 00:33:49 pph1 dhclient[480613]: No DHCPOFFERS received.
Dec 30 00:33:49 pph1 dhclient[480613]: No working leases in persistent database - sleeping.
Dec 30 00:33:49 pph1 dhclient[480613]: No DHCPOFFERS received.
Dec 30 00:33:49 pph1 dhclient[480613]: No working leases in persistent database - sleeping.
Dec 30 00:33:49 pph1 dhclient[480613]: No DHCPOFFERS received.
Dec 30 00:33:49 pph1 dhclient[480613]: No working leases in persistent database - sleeping.
Dec 30 00:33:49 pph1 dhclient[480613]: No DHCPOFFERS received.
Dec 30 00:33:49 pph1 dhclient[480613]: No working leases in persistent database - sleeping.
Dec 30 00:33:53 pph1 iptag[3026]: Checking fw net interface...
Dec 30 00:33:58 pph1 kernel: veth2051i2: left allmulticast mode
Dec 30 00:33:58 pph1 kernel: veth2051i2: left promiscuous mode
Dec 30 00:33:58 pph1 kernel: fwbr2051i2: port 2(veth2051i2) entered disabled state
Dec 30 00:35:20 pph1 iptag[3026]: Skipping 2113 cause ip no changes
Dec 30 00:35:23 pph1 iptag[3026]: Skipping 2114 cause ip no changes
Dec 30 00:35:25 pph1 kernel: veth2200i2: left allmulticast mode
Dec 30 00:35:25 pph1 kernel: veth2200i2: left promiscuous mode
Dec 30 00:35:25 pph1 kernel: fwbr2200i2: port 2(veth2200i2) entered disabled state
Dec 30 00:35:25 pph1 dhclient[480613]: receive_packet failed on fwln2200i2: Network is down
Dec 30 00:35:25 pph1 dhclient[480613]: receive_packet failed on fwpr2200p2: Network is down
Dec 30 00:35:25 pph1 kernel: fwbr2200i2: port 1(fwln2200i2) entered disabled state
Dec 30 00:35:25 pph1 kernel: vmbr2: port 12(fwpr2200p2) entered disabled state
Dec 30 00:35:25 pph1 kernel: fwln2200i2 (unregistering): left allmulticast mode
Dec 30 00:35:25 pph1 kernel: fwln2200i2 (unregistering): left promiscuous mode
Dec 30 00:35:25 pph1 kernel: fwbr2200i2: port 1(fwln2200i2) entered disabled state
Dec 30 00:35:25 pph1 kernel: fwpr2200p2 (unregistering): left allmulticast mode
Dec 30 00:35:25 pph1 kernel: fwpr2200p2 (unregistering): left promiscuous mode
Dec 30 00:35:25 pph1 kernel: vmbr2: port 12(fwpr2200p2) entered disabled state
Dec 30 00:35:25 pph1 dhclient[480613]: receive_packet failed on fwbr2200i2: Network is down
Dec 30 00:35:26 pph1 kernel: out: port 12(fwpr2200p2) entered blocking state
Dec 30 00:35:26 pph1 kernel: out: port 12(fwpr2200p2) entered disabled state
Dec 30 00:35:26 pph1 kernel: fwpr2200p2: entered allmulticast mode
Dec 30 00:35:26 pph1 kernel: fwpr2200p2: entered promiscuous mode
Dec 30 00:35:26 pph1 kernel: out: port 12(fwpr2200p2) entered blocking state
Dec 30 00:35:26 pph1 kernel: out: port 12(fwpr2200p2) entered forwarding state
Dec 30 00:35:26 pph1 kernel: fwbr2200i2: port 1(fwln2200i2) entered blocking state
Dec 30 00:35:26 pph1 kernel: fwbr2200i2: port 1(fwln2200i2) entered disabled state
Dec 30 00:35:26 pph1 kernel: fwln2200i2: entered allmulticast mode
Dec 30 00:35:26 pph1 kernel: fwln2200i2: entered promiscuous mode
Dec 30 00:35:26 pph1 kernel: fwbr2200i2: port 1(fwln2200i2) entered blocking state
Dec 30 00:35:26 pph1 kernel: fwbr2200i2: port 1(fwln2200i2) entered forwarding state
Dec 30 00:35:26 pph1 kernel: fwbr2200i2: port 2(veth2200i2) entered blocking state
Dec 30 00:35:26 pph1 kernel: fwbr2200i2: port 2(veth2200i2) entered disabled state
Dec 30 00:35:26 pph1 kernel: veth2200i2: entered allmulticast mode
Dec 30 00:35:26 pph1 kernel: veth2200i2: entered promiscuous mode
Dec 30 00:35:26 pph1 kernel: fwbr2200i2: port 2(veth2200i2) entered blocking state
Dec 30 00:35:26 pph1 kernel: fwbr2200i2: port 2(veth2200i2) entered forwarding state
Dec 30 00:35:26 pph1 iptag[3026]: Skipping 2115 cause ip no changes
Dec 30 00:35:29 pph1 iptag[3026]: Skipping 2116 cause ip no changes
Dec 30 00:35:32 pph1 iptag[3026]: Skipping 2117 cause ip no changes
Dec 30 00:35:35 pph1 iptag[3026]: Skipping 2200 cause ip no changes
Dec 30 00:35:38 pph1 iptag[3026]: Skipping 2252 cause ip no changes
Dec 30 00:36:20 pph1 dhclient[480613]: DHCPDISCOVER on fwpr2111p2 to 255.255.255.255 port 67 interval 5
Dec 30 00:36:20 pph1 dhclient[480613]: send_packet: No such device or address
Dec 30 00:36:20 pph1 dhclient[480613]: dhclient.c:2600: Failed to send 300 byte long packet over fwpr2111p2 interface.
Dec 30 00:36:20 pph1 dhclient[480613]: DHCPDISCOVER on veth2115i2 to 255.255.255.255 port 67 interval 7
Dec 30 00:36:25 pph1 dhclient[480613]: DHCPDISCOVER on fwpr2111p2 to 255.255.255.255 port 67 interval 14
Dec 30 00:36:25 pph1 dhclient[480613]: send_packet: No such device or address
Dec 30 00:36:25 pph1 dhclient[480613]: dhclient.c:2600: Failed to send 300 byte long packet over fwpr2111p2 interface.
Dec 30 00:36:27 pph1 dhclient[480613]: DHCPDISCOVER on veth2115i2 to 255.255.255.255 port 67 interval 8
Dec 30 00:36:29 pph1 dhclient[480613]: DHCPDISCOVER on fwbr2113i2 to 255.255.255.255 port 67 interval 8
Dec 30 00:36:29 pph1 dhclient[480613]: send_packet: No such device or address
Dec 30 00:36:29 pph1 dhclient[480613]: dhclient.c:2600: Failed to send 300 byte long packet over fwbr2113i2 interface.
Dec 30 00:36:29 pph1 dhclient[480613]: DHCPDISCOVER on fwln2100i2 to 255.255.255.255 port 67 interval 3
Dec 30 00:36:29 pph1 dhclient[480613]: send_packet: No such device or address
Dec 30 00:36:29 pph1 dhclient[480613]: dhclient.c:2600: Failed to send 300 byte long packet over fwln2100i2 interface.
Dec 30 00:36:31 pph1 dhclient[480613]: DHCPDISCOVER on fwbr2113i1 to 255.255.255.255 port 67 interval 8
Dec 30 00:36:32 pph1 dhclient[480613]: DHCPDISCOVER on fwln2100i2 to 255.255.255.255 port 67 interval 3
Dec 30 00:36:32 pph1 dhclient[480613]: send_packet: No such device or address
Dec 30 00:36:32 pph1 dhclient[480613]: dhclient.c:2600: Failed to send 300 byte long packet over fwln2100i2 interface.
Dec 30 00:36:35 pph1 dhclient[480613]: DHCPDISCOVER on veth2115i2 to 255.255.255.255 port 67 interval 11
Dec 30 00:36:35 pph1 dhclient[480613]: DHCPDISCOVER on fwln2100i2 to 255.255.255.255 port 67 interval 3
Dec 30 00:36:35 pph1 dhclient[480613]: send_packet: No such device or address
Dec 30 00:36:35 pph1 dhclient[480613]: dhclient.c:2600: Failed to send 300 byte long packet over fwln2100i2 interface.
Dec 30 00:36:37 pph1 dhclient[480613]: DHCPDISCOVER on fwbr2113i2 to 255.255.255.255 port 67 interval 7
Dec 30 00:36:37 pph1 dhclient[480613]: send_packet: No such device or address
Dec 30 00:36:37 pph1 dhclient[480613]: dhclient.c:2600: Failed to send 300 byte long packet over fwbr2113i2 interface.
Dec 30 00:36:38 pph1 dhclient[480613]: DHCPDISCOVER on veth2114i0 to 255.255.255.255 port 67 interval 8
Dec 30 00:36:38 pph1 dhclient[480613]: DHCPDISCOVER on fwln2100i2 to 255.255.255.255 port 67 interval 8
Dec 30 00:36:38 pph1 dhclient[480613]: send_packet: No such device or address
Dec 30 00:36:38 pph1 dhclient[480613]: dhclient.c:2600: Failed to send 300 byte long packet over fwln2100i2 interface.
Dec 30 00:36:38 pph1 iptag[3026]: Checking fw net interface...
Dec 30 00:36:38 pph1 dhclient[480613]: DHCPDISCOVER on fwln2113i2 to 255.255.255.255 port 67 interval 6
Dec 30 00:36:38 pph1 dhclient[480613]: send_packet: No such device or address
Dec 30 00:36:38 pph1 dhclient[480613]: dhclient.c:2600: Failed to send 300 byte long packet over fwln2113i2 interface.
Dec 30 00:36:38 pph1 dhclient[480613]: DHCPDISCOVER on veth2010i0 to 255.255.255.255 port 67 interval 3
Dec 30 00:36:39 pph1 dhclient[480613]: DHCPDISCOVER on fwpr2111p2 to 255.255.255.255 port 67 interval 14
Dec 30 00:36:39 pph1 dhclient[480613]: send_packet: No such device or address
Dec 30 00:36:39 pph1 dhclient[480613]: dhclient.c:2600: Failed to send 300 byte long packet over fwpr2111p2 interface.
Dec 30 00:36:39 pph1 dhclient[480613]: DHCPDISCOVER on fwbr2113i1 to 255.255.255.255 port 67 interval 17
Dec 30 00:36:41 pph1 dhclient[480613]: DHCPDISCOVER on veth2111i0 to 255.255.255.255 port 67 interval 4
Dec 30 00:36:41 pph1 dhclient[480613]: DHCPDISCOVER on veth2010i0 to 255.255.255.255 port 67 interval 5
Dec 30 00:36:44 pph1 dhclient[480613]: DHCPDISCOVER on fwbr2050i1 to 255.255.255.255 port 67 interval 4
Dec 30 00:36:44 pph1 dhclient[480613]: DHCPDISCOVER on fwbr2113i2 to 255.255.255.255 port 67 interval 13
Dec 30 00:36:44 pph1 dhclient[480613]: send_packet: No such device or address
eases in persistent database - sleeping.
Dec 31 00:32:48 pph1 dhclient[480613]: DHCPDISCOVER on fwbr2117i2 to 255.255.255.255 port 67 interval 2
Dec 31 00:32:48 pph1 dhclient[480613]: send_packet: No such device or address
Dec 31 00:32:48 pph1 dhclient[480613]: dhclient.c:2600: Failed to send 300 byte long packet over fwbr2117i2 interface.
Dec 31 00:32:49 pph1 dhclient[480613]: DHCPDISCOVER on fwln2113i2 to 255.255.255.255 port 67 interval 1
Dec 31 00:32:49 pph1 dhclient[480613]: send_packet: No such device or address
Dec 31 00:32:49 pph1 dhclient[480613]: dhclient.c:2600: Failed to send 300 byte long packet over fwln2113i2 interface.
Dec 31 00:32:50 pph1 dhclient[480613]: No DHCPOFFERS received.
Dec 31 00:32:50 pph1 dhclient[480613]: No working leases in persistent database - sleeping.
Dec 31 00:32:50 pph1 dhclient[480613]: No DHCPOFFERS received.
Dec 31 00:32:50 pph1 dhclient[480613]: No working leases in persistent database - sleeping.
Dec 31 00:32:51 pph1 dhclient[480613]: DHCPDISCOVER on fwbr2117i1 to 255.255.255.255 port 67 interval 12
Dec 31 00:32:51 pph1 dhclient[480613]: DHCPDISCOVER on veth2200i1 to 255.255.255.255 port 67 interval 11
Dec 31 00:32:52 pph1 dhclient[480613]: DHCPDISCOVER on veth2117i0 to 255.255.255.255 port 67 interval 3
Dec 31 00:32:52 pph1 dhclient[480613]: DHCPDISCOVER on fwln2051i1 to 255.255.255.255 port 67 interval 19
Dec 31 00:32:52 pph1 dhclient[480613]: send_packet: No such device or address
Dec 31 00:32:52 pph1 dhclient[480613]: dhclient.c:2600: Failed to send 300 byte long packet over fwln2051i1 interface.
Dec 31 00:32:52 pph1 dhclient[480613]: DHCPDISCOVER on fwln2115i1 to 255.255.255.255 port 67 interval 17
Dec 31 00:32:53 pph1 dhclient[480613]: DHCPDISCOVER on veth2010i1 to 255.255.255.255 port 67 interval 8
Dec 31 00:32:54 pph1 dhclient[480613]: DHCPDISCOVER on fwbr2110i2 to 255.255.255.255 port 67 interval 11
Dec 31 00:32:54 pph1 dhclient[480613]: send_packet: No such device or address
Dec 31 00:32:54 pph1 dhclient[480613]: dhclient.c:2600: Failed to send 300 byte long packet over fwbr2110i2 interface.
Dec 31 00:32:54 pph1 dhclient[480613]: DHCPDISCOVER on eno2 to 255.255.255.255 port 67 interval 3
Dec 31 00:32:54 pph1 dhclient[480613]: DHCPOFFER of 172.16.32.2 from 172.16.32.1
Dec 31 00:32:54 pph1 dhclient[480613]: DHCPREQUEST for 172.16.32.2 on eno2 to 255.255.255.255 port 67
Dec 31 00:32:54 pph1 dhclient[480613]: DHCPACK of 172.16.32.2 from 172.16.32.1
Dec 31 00:32:54 pph1-out dhclient[480613]: bound to 172.16.32.2 -- renewal in 34334 seconds.
Dec 31 00:32:54 pph1-out dhclient[480613]: DHCPDISCOVER on fwbr2114i2 to 255.255.255.255 port 67 interval 16
Dec 31 00:32:54 pph1-out dhclient[480613]: send_packet: No such device or address
Dec 31 00:32:54 pph1-out dhclient[480613]: dhclient.c:2600: Failed to send 300 byte long packet over fwbr2114i2 interface.
Dec 31 00:32:54 pph1-out dhclient[480613]: DHCPDISCOVER on veth2051i0 to 255.255.255.255 port 67 interval 16
Dec 31 00:32:54 pph1-out dhclient[480613]: send_packet: No such device or address
Dec 31 00:32:54 pph1-out dhclient[480613]: dhclient.c:2600: Failed to send 300 byte long packet over veth2051i0 interface.
Dec 31 00:32:55 pph1-out dhclient[480613]: DHCPDISCOVER on fwbr2111i2 to 255.255.255.255 port 67 interval 7
Dec 31 00:32:55 pph1-out dhclient[480613]: send_packet: No such device or address
Dec 31 00:32:55 pph1-out dhclient[480613]: dhclient.c:2600: Failed to send 300 byte long packet over fwbr2111i2 interface.
Dec 31 00:32:55 pph1-out dhclient[480613]: DHCPREQUEST for 172.16.32.2 on out to 172.16.32.1 port 67
Dec 31 00:32:55 pph1-out dhclient[480613]: DHCPACK of 172.16.32.2 from 172.16.32.1
Dec 31 00:32:55 pph1-out dhclient[480613]: bound to 172.16.32.2 -- renewal in 34341 seconds.
Dec 31 00:32:55 pph1-out dhclient[480613]: DHCPDISCOVER on fwln2110i1 to 255.255.255.255 port 67 interval 7
Dec 31 00:32:55 pph1-out dhclient[480613]: DHCPDISCOVER on veth2117i0 to 255.255.255.255 port 67 interval 4
Dec 31 00:32:55 pph1-out dhclient[480613]: DHCPDISCOVER on veth2116i0 to 255.255.255.255 port 67 interval 12
Dec 31 00:32:55 pph1-out dhclient[480613]: DHCPDISCOVER on veth2113i1 to 255.255.255.255 port 67 interval 3
Dec 31 00:32:55 pph1-out dhclient[480613]: DHCPDISCOVER on fwpr2117p1 to 255.255.255.255 port 67 interval 13
Dec 31 00:32:56 pph1-out dhclient[480613]: DHCPDISCOVER on veth2113i2 to 255.255.255.255 port 67 interval 21
Dec 31 00:32:57 pph1-out dhclient[480613]: DHCPDISCOVER on veth2114i0 to 255.255.255.255 port 67 interval 5
Dec 31 00:32:58 pph1-out dhclient[480613]: DHCPDISCOVER on veth2115i0 to 255.255.255.255 port 67 interval 16
Dec 31 00:32:58 pph1-out dhclient[480613]: DHCPDISCOVER on veth2113i1 to 255.255.255.255 port 67 interval 5
Dec 31 00:32:59 pph1-out dhclient[480613]: DHCPDISCOVER on fwbr2112i2 to 255.255.255.255 port 67 interval 5
Dec 31 00:32:59 pph1-out dhclient[480613]: send_packet: No such device or address
Dec 31 00:32:59 pph1-out dhclient[480613]: dhclient.c:2600: Failed to send 300 byte long packet over fwbr2112i2 interface.
For as far as I can tell it seems to be caused by a network issue.
I have no idea how to troubleshoot this.
Here's my /etc/network/interfaces:
Code:
auto lo
iface lo inet loopback
iface eno1 inet manual
iface eno2 inet manual
auto vmbr0
iface vmbr0 inet static
address 172.16.20.2/23
gateway 172.16.20.1
bridge-ports eno1
bridge-stp off
bridge-fd 0
auto vmbr1
iface vmbr1 inet static
address 172.16.40.1/23
bridge-ports none
bridge-stp off
bridge-fd 0
#Virtual LXC Bridge
auto vmbr2
iface vmbr2 inet static
address 172.16.42.1/23
bridge-ports none
bridge-stp off
bridge-fd 0
mtu 9000
#Virtual Bridge To SMB
auto out
iface out inet manual
bridge-ports eno2
bridge-stp off
bridge-fd 0
source /etc/network/interfaces.d/*
Can anyone help me find what's going on?
My system is Supermicro X10SRL-F with E5-2697A V4 and 256GB RAM.