I've been fighting increasing load averages over time on one of my systems that leads to an eventual unresponsive situation that requires a hard reboot to recover from. This machine hosts a VM for openmediavault, a LXC that I use for docker, and a PBS LXC. I'm not sure how to diagnose the true underlying issue, the syslog and kern.log show a lot of chatter about networking, but there's no change in behavior around the time of the spike. The load average slowly creeps up into the multiple hundreds before the whole system becomes unresponsive, this usually takes a few days.
Here's a snippit of the syslog around the time of the spike
Here's a snippit of the syslog around the time of the spike
Code:
2024-07-24T06:06:05.982331-05:00 phpve01 kernel: [124748.207500] CIFS: Attempting to mount //vmnas01.home/Data/Cloud
2024-07-24T06:06:06.040342-05:00 phpve01 kernel: [124748.265375] br-eac7ccf10a5a: port 11(veth5ad8805) entered blocking state
2024-07-24T06:06:06.040378-05:00 phpve01 kernel: [124748.265385] br-eac7ccf10a5a: port 11(veth5ad8805) entered disabled state
2024-07-24T06:06:06.040381-05:00 phpve01 kernel: [124748.265441] veth5ad8805: entered allmulticast mode
2024-07-24T06:06:06.040382-05:00 phpve01 kernel: [124748.265556] veth5ad8805: entered promiscuous mode
2024-07-24T06:06:06.049332-05:00 phpve01 kernel: [124748.274116] br-0e02950d286a: port 13(vethfdfbfec) entered blocking state
2024-07-24T06:06:06.049348-05:00 phpve01 kernel: [124748.274122] br-0e02950d286a: port 13(vethfdfbfec) entered disabled state
2024-07-24T06:06:06.050327-05:00 phpve01 kernel: [124748.275065] vethfdfbfec: entered allmulticast mode
2024-07-24T06:06:06.050346-05:00 phpve01 kernel: [124748.275122] vethfdfbfec: entered promiscuous mode
2024-07-24T06:06:06.054441-05:00 phpve01 kernel: [124748.279233] br-c9cb6dc2a213: port 8(veth8b04f00) entered blocking state
2024-07-24T06:06:06.054467-05:00 phpve01 kernel: [124748.279242] br-c9cb6dc2a213: port 8(veth8b04f00) entered disabled state
2024-07-24T06:06:06.054469-05:00 phpve01 kernel: [124748.279562] veth8b04f00: entered allmulticast mode
2024-07-24T06:06:06.055319-05:00 phpve01 kernel: [124748.280413] veth8b04f00: entered promiscuous mode
2024-07-24T06:06:06.099364-05:00 phpve01 kernel: [124748.323575] eth0: renamed from veth264323b
2024-07-24T06:06:06.114366-05:00 phpve01 kernel: [124748.338913] br-0e02950d286a: port 4(vethca0048c) entered blocking state
2024-07-24T06:06:06.114396-05:00 phpve01 kernel: [124748.338932] br-0e02950d286a: port 4(vethca0048c) entered forwarding state
2024-07-24T06:06:06.143343-05:00 phpve01 kernel: [124748.368512] br-eac7ccf10a5a: port 12(vethb3b827f) entered blocking state
2024-07-24T06:06:06.143379-05:00 phpve01 kernel: [124748.368524] br-eac7ccf10a5a: port 12(vethb3b827f) entered disabled state
2024-07-24T06:06:06.143386-05:00 phpve01 kernel: [124748.368615] vethb3b827f: entered allmulticast mode
2024-07-24T06:06:06.144338-05:00 phpve01 kernel: [124748.369064] vethb3b827f: entered promiscuous mode
2024-07-24T06:06:06.147340-05:00 phpve01 kernel: [124748.371631] br-0e02950d286a: port 14(vethe2943bc) entered blocking state
2024-07-24T06:06:06.147369-05:00 phpve01 kernel: [124748.371639] br-0e02950d286a: port 14(vethe2943bc) entered disabled state
2024-07-24T06:06:06.147371-05:00 phpve01 kernel: [124748.371673] vethe2943bc: entered allmulticast mode
2024-07-24T06:06:06.147372-05:00 phpve01 kernel: [124748.371732] vethe2943bc: entered promiscuous mode
2024-07-24T06:06:06.162341-05:00 phpve01 kernel: [124748.387569] eth0: renamed from veth8f120cc
2024-07-24T06:06:06.171352-05:00 phpve01 kernel: [124748.395976] br-eac7ccf10a5a: port 5(veth84d92d8) entered blocking state
2024-07-24T06:06:06.171384-05:00 phpve01 kernel: [124748.395986] br-eac7ccf10a5a: port 5(veth84d92d8) entered forwarding state
2024-07-24T06:06:06.245339-05:00 phpve01 kernel: [124748.469632] br-0e02950d286a: port 15(vethc4ae6bf) entered blocking state
2024-07-24T06:06:06.245366-05:00 phpve01 kernel: [124748.469639] br-0e02950d286a: port 15(vethc4ae6bf) entered disabled state
2024-07-24T06:06:06.245368-05:00 phpve01 kernel: [124748.469668] vethc4ae6bf: entered allmulticast mode
2024-07-24T06:06:06.245370-05:00 phpve01 kernel: [124748.469719] vethc4ae6bf: entered promiscuous mode
2024-07-24T06:06:06.261326-05:00 phpve01 kernel: [124748.485875] eth0: renamed from veth75ff935
2024-07-24T06:06:06.293328-05:00 phpve01 kernel: [124748.518554] br-c9cb6dc2a213: port 6(veth0195ea3) entered blocking state
2024-07-24T06:06:06.293348-05:00 phpve01 kernel: [124748.518561] br-c9cb6dc2a213: port 6(veth0195ea3) entered forwarding state
2024-07-24T06:06:06.294365-05:00 phpve01 kernel: [124748.519195] br-eac7ccf10a5a: port 13(veth4996821) entered blocking state
2024-07-24T06:06:06.294413-05:00 phpve01 kernel: [124748.519202] br-eac7ccf10a5a: port 13(veth4996821) entered disabled state
2024-07-24T06:06:06.294415-05:00 phpve01 kernel: [124748.519221] veth4996821: entered allmulticast mode
2024-07-24T06:06:06.295315-05:00 phpve01 kernel: [124748.520407] veth4996821: entered promiscuous mode
2024-07-24T06:06:06.298338-05:00 phpve01 kernel: [124748.523225] br-0e02950d286a: port 16(vethffd3083) entered blocking state
2024-07-24T06:06:06.298366-05:00 phpve01 kernel: [124748.523230] br-0e02950d286a: port 16(vethffd3083) entered disabled state
2024-07-24T06:06:06.302532-05:00 phpve01 kernel: [124748.527560] vethffd3083: entered allmulticast mode
2024-07-24T06:06:06.302558-05:00 phpve01 kernel: [124748.527716] vethffd3083: entered promiscuous mode
2024-07-24T06:06:06.396334-05:00 phpve01 kernel: [124748.620796] eth0: renamed from vethd29ecdf
2024-07-24T06:06:06.429393-05:00 phpve01 kernel: [124748.653784] eth0: renamed from vethf7c49ec
2024-07-24T06:06:06.446331-05:00 phpve01 kernel: [124748.670754] br-c9cb6dc2a213: port 5(veth9bd5b31) entered blocking state
2024-07-24T06:06:06.446353-05:00 phpve01 kernel: [124748.670765] br-c9cb6dc2a213: port 5(veth9bd5b31) entered forwarding state
2024-07-24T06:06:06.447347-05:00 phpve01 kernel: [124748.671980] eth0: renamed from vetha578881
2024-07-24T06:06:06.455328-05:00 phpve01 kernel: [124748.680060] eth0: renamed from veth4341b2f
2024-07-24T06:06:06.480323-05:00 phpve01 kernel: [124748.705382] eth0: renamed from veth11b57d9
2024-07-24T06:06:06.512365-05:00 phpve01 kernel: [124748.736635] br-eac7ccf10a5a: port 6(veth4de78b2) entered blocking state
2024-07-24T06:06:06.512397-05:00 phpve01 kernel: [124748.736652] br-eac7ccf10a5a: port 6(veth4de78b2) entered forwarding state
2024-07-24T06:06:06.512399-05:00 phpve01 kernel: [124748.736717] br-c9cb6dc2a213: port 7(veth5213ef2) entered blocking state
2024-07-24T06:06:06.512400-05:00 phpve01 kernel: [124748.736720] br-c9cb6dc2a213: port 7(veth5213ef2) entered forwarding state
2024-07-24T06:06:06.515347-05:00 phpve01 kernel: [124748.739929] br-0e02950d286a: port 9(veth94d2cd5) entered blocking state
2024-07-24T06:06:06.515375-05:00 phpve01 kernel: [124748.739938] br-0e02950d286a: port 9(veth94d2cd5) entered forwarding state
2024-07-24T06:06:06.516573-05:00 phpve01 kernel: [124748.741281] br-eac7ccf10a5a: port 14(vethb49ed28) entered blocking state
2024-07-24T06:06:06.516601-05:00 phpve01 kernel: [124748.741286] br-eac7ccf10a5a: port 14(vethb49ed28) entered disabled state
2024-07-24T06:06:06.516602-05:00 phpve01 kernel: [124748.741529] vethb49ed28: entered allmulticast mode
2024-07-24T06:06:06.517318-05:00 phpve01 kernel: [124748.742116] vethb49ed28: entered promiscuous mode
2024-07-24T06:06:06.517331-05:00 phpve01 kernel: [124748.742282] eth0: renamed from vethec22cc8
2024-07-24T06:06:06.524349-05:00 phpve01 kernel: [124748.749011] br-0e02950d286a: port 8(veth2e29f6b) entered blocking state
2024-07-24T06:06:06.524368-05:00 phpve01 kernel: [124748.749022] br-0e02950d286a: port 8(veth2e29f6b) entered forwarding state
2024-07-24T06:06:06.533574-05:00 phpve01 kernel: [124748.758509] br-eac7ccf10a5a: port 7(veth9f1e31e) entered blocking state
2024-07-24T06:06:06.533601-05:00 phpve01 kernel: [124748.758527] br-eac7ccf10a5a: port 7(veth9f1e31e) entered forwarding state
2024-07-24T06:06:06.533603-05:00 phpve01 kernel: [124748.758559] br-0e02950d286a: port 17(vethdaa26ce) entered blocking state
2024-07-24T06:06:06.533604-05:00 phpve01 kernel: [124748.758564] br-0e02950d286a: port 17(vethdaa26ce) entered disabled state
2024-07-24T06:06:06.533714-05:00 phpve01 kernel: [124748.758577] vethdaa26ce: entered allmulticast mode
2024-07-24T06:06:06.533716-05:00 phpve01 kernel: [124748.758684] vethdaa26ce: entered promiscuous mode
2024-07-24T06:06:06.743327-05:00 phpve01 kernel: [124748.968233] br-0e02950d286a: port 18(veth4accaf8) entered blocking state
2024-07-24T06:06:06.743348-05:00 phpve01 kernel: [124748.968243] br-0e02950d286a: port 18(veth4accaf8) entered disabled state
2024-07-24T06:06:06.743350-05:00 phpve01 kernel: [124748.968273] veth4accaf8: entered allmulticast mode
2024-07-24T06:06:06.743351-05:00 phpve01 kernel: [124748.968327] veth4accaf8: entered promiscuous mode
2024-07-24T06:06:06.852342-05:00 phpve01 kernel: [124749.077562] br-eac7ccf10a5a: port 15(veth3029607) entered blocking state
2024-07-24T06:06:06.852378-05:00 phpve01 kernel: [124749.077571] br-eac7ccf10a5a: port 15(veth3029607) entered disabled state
2024-07-24T06:06:06.852387-05:00 phpve01 kernel: [124749.077630] veth3029607: entered allmulticast mode
2024-07-24T06:06:06.853322-05:00 phpve01 kernel: [124749.077773] veth3029607: entered promiscuous mode
2024-07-24T06:06:06.875333-05:00 phpve01 kernel: [124749.100299] eth0: renamed from vethe2786e3
2024-07-24T06:06:06.887338-05:00 phpve01 kernel: [124749.111652] br-eac7ccf10a5a: port 11(veth5ad8805) entered blocking state
2024-07-24T06:06:06.887363-05:00 phpve01 kernel: [124749.111667] br-eac7ccf10a5a: port 11(veth5ad8805) entered forwarding state
2024-07-24T06:06:06.891558-05:00 phpve01 kernel: [124749.116233] br-0e02950d286a: port 19(veth4aab65c) entered blocking state
2024-07-24T06:06:06.891577-05:00 phpve01 kernel: [124749.116248] br-0e02950d286a: port 19(veth4aab65c) entered disabled state
2024-07-24T06:06:06.891578-05:00 phpve01 kernel: [124749.116267] veth4aab65c: entered allmulticast mode
2024-07-24T06:06:06.893345-05:00 phpve01 kernel: [124749.117625] veth4aab65c: entered promiscuous mode
2024-07-24T06:06:06.958336-05:00 phpve01 kernel: [124749.182873] eth0: renamed from veth3f14cfc
2024-07-24T06:06:06.973349-05:00 phpve01 kernel: [124749.198102] br-eac7ccf10a5a: port 10(veth21eee53) entered blocking state
2024-07-24T06:06:06.973379-05:00 phpve01 kernel: [124749.198114] br-eac7ccf10a5a: port 10(veth21eee53) entered forwarding state
2024-07-24T06:06:07.058334-05:00 phpve01 kernel: [124749.282769] eth0: renamed from veth003516b
2024-07-24T06:06:07.075349-05:00 phpve01 kernel: [124749.299924] br-0e02950d286a: port 14(vethe2943bc) entered blocking state
2024-07-24T06:06:07.075372-05:00 phpve01 kernel: [124749.299930] br-0e02950d286a: port 14(vethe2943bc) entered forwarding state
2024-07-24T06:06:07.143390-05:00 phpve01 kernel: [124749.368238] br-0e02950d286a: port 20(veth1adcfe0) entered blocking state
2024-07-24T06:06:07.143442-05:00 phpve01 kernel: [124749.368249] br-0e02950d286a: port 20(veth1adcfe0) entered disabled state
2024-07-24T06:06:07.143448-05:00 phpve01 kernel: [124749.368329] veth1adcfe0: entered allmulticast mode
2024-07-24T06:06:07.143450-05:00 phpve01 kernel: [124749.368630] veth1adcfe0: entered promiscuous mode
2024-07-24T06:06:07.163344-05:00 phpve01 kernel: [124749.388181] br-eac7ccf10a5a: port 16(veth47e6426) entered blocking state
2024-07-24T06:06:07.163370-05:00 phpve01 kernel: [124749.388187] br-eac7ccf10a5a: port 16(veth47e6426) entered disabled state
2024-07-24T06:06:07.163371-05:00 phpve01 kernel: [124749.388203] veth47e6426: entered allmulticast mode
2024-07-24T06:06:07.163372-05:00 phpve01 kernel: [124749.388258] veth47e6426: entered promiscuous mode
2024-07-24T06:06:07.163373-05:00 phpve01 kernel: [124749.388331] eth0: renamed from veth8fecab1
2024-07-24T06:06:07.176337-05:00 phpve01 kernel: [124749.400788] br-c9cb6dc2a213: port 9(vethcad35a4) entered blocking state
2024-07-24T06:06:07.176377-05:00 phpve01 kernel: [124749.400795] br-c9cb6dc2a213: port 9(vethcad35a4) entered disabled state
2024-07-24T06:06:07.176379-05:00 phpve01 kernel: [124749.400810] vethcad35a4: entered allmulticast mode
2024-07-24T06:06:07.176380-05:00 phpve01 kernel: [124749.400856] vethcad35a4: entered promiscuous mode
2024-07-24T06:06:07.182336-05:00 phpve01 kernel: [124749.407031] br-eac7ccf10a5a: port 12(vethb3b827f) entered blocking state
2024-07-24T06:06:07.182358-05:00 phpve01 kernel: [124749.407042] br-eac7ccf10a5a: port 12(vethb3b827f) entered forwarding state
2024-07-24T06:06:07.293332-05:00 phpve01 kernel: [124749.517814] eth0: renamed from vetha1d4392
2024-07-24T06:06:07.305319-05:00 phpve01 kernel: [124749.529925] br-0e02950d286a: port 21(veth265be6f) entered blocking state
2024-07-24T06:06:07.305334-05:00 phpve01 kernel: [124749.529933] br-0e02950d286a: port 21(veth265be6f) entered disabled state
2024-07-24T06:06:07.305335-05:00 phpve01 kernel: [124749.529946] veth265be6f: entered allmulticast mode
2024-07-24T06:06:07.305336-05:00 phpve01 kernel: [124749.529994] veth265be6f: entered promiscuous mode
2024-07-24T06:06:07.305336-05:00 phpve01 kernel: [124749.530282] br-eac7ccf10a5a: port 13(veth4996821) entered blocking state
2024-07-24T06:06:07.305337-05:00 phpve01 kernel: [124749.530286] br-eac7ccf10a5a: port 13(veth4996821) entered forwarding state
2024-07-24T06:06:07.416345-05:00 phpve01 kernel: [124749.640858] br-eac7ccf10a5a: port 17(veth7d4d45e) entered blocking state
2024-07-24T06:06:07.416376-05:00 phpve01 kernel: [124749.640867] br-eac7ccf10a5a: port 17(veth7d4d45e) entered disabled state
2024-07-24T06:06:07.416377-05:00 phpve01 kernel: [124749.640899] veth7d4d45e: entered allmulticast mode
2024-07-24T06:06:07.416378-05:00 phpve01 kernel: [124749.640977] veth7d4d45e: entered promiscuous mode
2024-07-24T06:06:07.474345-05:00 phpve01 kernel: [124749.698893] br-0e02950d286a: port 22(vethe996a66) entered blocking state
2024-07-24T06:06:07.474382-05:00 phpve01 kernel: [124749.698903] br-0e02950d286a: port 22(vethe996a66) entered disabled state