[SOLVED] WebUI gradually failing

[Oscar_Hill]

I have a strange problem.

Versions: PVE 5.4-13 on Debian GNU/Linux 9 (stretch)

The WebUI is loosing connection gradually to first storage and then VM's/Containers and after a while(not sure how long) to everything(can't log in).

None of the grafs work no matter what I have done, so far.


You can see the different stages of failure below.

Stage 1: Communication failure on storage.



Stage 2 the same on all VM's/LXC's and Host


Stage 3 Can't login.


I get back to "Stage 1" if I restart PVEstatd sudo systemctl restart pvestatd.service
I get back to "Stage 2" if I restart PVEdaemon sudo systemctl restart pvedaemon.service


Can anyone help me with this?


I have seen references to these log entries:

Nov 22 09:17:15 pve kernel: ACPI Error: SMBus/IPMI/GenericSerialBus write requires Buffer of length 66, found length 32 (20170831/exfield-427)
Nov 22 09:17:15 pve kernel: No Local Variables are initialized for Method [_PMM]
Nov 22 09:17:15 pve kernel: No Arguments are initialized for method [_PMM]
Nov 22 09:17:15 pve kernel: ACPI Error: Method parse/execution failed \_SB.PMI0._PMM, AE_AML_BUFFER_LIMIT (20170831/psparse-550)
Nov 22 09:17:15 pve kernel: ACPI Exception: AE_AML_BUFFER_LIMIT, Evaluating _PMM (20170831/power_meter-338)
Nov 22 09:17:30 pve kernel: ACPI Error: SMBus/IPMI/GenericSerialBus write requires Buffer of length 66, found length 32 (20170831/exfield-427)
Nov 22 09:17:30 pve kernel: No Local Variables are initialized for Method [_PMM]
Nov 22 09:17:30 pve kernel: No Arguments are initialized for method [_PMM]
Nov 22 09:17:30 pve kernel: ACPI Error: Method parse/execution failed \_SB.PMI0._PMM, AE_AML_BUFFER_LIMIT (20170831/psparse-550)
Nov 22 09:17:30 pve kernel: ACPI Exception: AE_AML_BUFFER_LIMIT, Evaluating _PMM (20170831/power_meter-338)
Nov 22 09:17:45 pve kernel: ACPI Error: SMBus/IPMI/GenericSerialBus write requires Buffer of length 66, found length 32 (20170831/exfield-427)
Nov 22 09:17:45 pve kernel: No Local Variables are initialized for Method [_PMM]
Nov 22 09:17:45 pve kernel: No Arguments are initialized for method [_PMM]
Nov 22 09:17:45 pve kernel: ACPI Error: Method parse/execution failed \_SB.PMI0._PMM, AE_AML_BUFFER_LIMIT (20170831/psparse-550)
Nov 22 09:17:45 pve kernel: ACPI Exception: AE_AML_BUFFER_LIMIT, Evaluating _PMM (20170831/power_meter-338)
Nov 22 09:18:00 pve kernel: ACPI Error: SMBus/IPMI/GenericSerialBus write requires Buffer of length 66, found length 32 (20170831/exfield-427)
Nov 22 09:18:00 pve kernel: No Local Variables are initialized for Method [_PMM]
Nov 22 09:18:00 pve kernel: No Arguments are initialized for method [_PMM]
Nov 22 09:18:00 pve kernel: ACPI Error: Method parse/execution failed \_SB.PMI0._PMM, AE_AML_BUFFER_LIMIT (20170831/psparse-550)
Nov 22 09:18:00 pve kernel: ACPI Exception: AE_AML_BUFFER_LIMIT, Evaluating _PMM (20170831/power_meter-338)

 

[t.lamprecht]

I have seen references to these log entries:

those kernel logs seem unrelated.

Can you check if there's some errors in journalctl --since -4hour

The WebUI is loosing connection gradually to first storage and then VM's/Containers and after a while(not sure how long) to everything(can't log in).

When did this start, or did it never worked? If things start to go bad some change (configuration or environment) is often the cause.

 

[Oscar_Hill]

I can't find anything obvious in the journal. I attached it here (after some pruning of personal data).

Not sure about the time, but I first notised it yesterday.
I have been using backups and restore to fix a GitLab server witch had failed. The last week or so.

This is the update history:

Start-Date: 2019-10-09  10:41:27
Commandline: apt upgrade
Requested-By: oscar (1000)
Install: pve-kernel-4.15.18-21-pve:amd64 (4.15.18-48, automatic)
Upgrade: libcomerr2:amd64 (1.43.4-2, 1.43.4-2+deb9u1), linux-libc-dev:amd64 (4.9.189-3, 4.9.189-3+deb9u1), openssl:amd64 (1.1.0k-1~deb9u1, 1.1.0l-1~deb9u1), e2fsprogs:amd64 (1.43.4-2, 1.43.4-2+deb9u1), e2fslibs:amd64 (1.43.4-2, 1.43.4-2+deb9u1), libexpat1:amd64 (2.2.0-2+deb9u2, 2.2.0-2+deb9u3), libss2:amd64 (1.43.4-2, 1.43.4-2+deb9u1), libpve-common-perl:amd64 (5.0-54, 5.0-55), lxc-pve:amd64 (3.1.0-6, 3.1.0-7), pve-kernel-4.15:amd64 (5.4-8, 5.4-9), libssl1.1:amd64 (1.1.0k-1~deb9u1, 1.1.0l-1~deb9u1), libssl1.0.2:amd64 (1.0.2s-1~deb9u1, 1.0.2t-1~deb9u1)
End-Date: 2019-10-09  10:43:18

Start-Date: 2019-10-15  15:22:39
Commandline: apt upgrade
Requested-By: oscar (1000)
Upgrade: sudo:amd64 (1.8.19p1-2.1, 1.8.19p1-2.1+deb9u1)
End-Date: 2019-10-15  15:22:45


Start-Date: 2019-11-11  09:13:22
Commandline: apt upgrade
Requested-By: oscar (1000)
Upgrade: tcpdump:amd64 (4.9.2-1~deb9u1, 4.9.3-1~deb9u1), libarchive13:amd64 (3.2.2-2+deb9u1, 3.2.2-2+deb9u2), libmagic1:amd64 (1:5.30-1+deb9u2, 1:5.30-1+deb9u3), libmagic-mgc:amd64 (1:5.30-1+deb9u2, 1:5.30-1+deb9u3), file:amd64 (1:5.30-1+deb9u2, 1:5.30-1+deb9u3)
End-Date: 2019-11-11  09:13:25

Start-Date: 2019-11-21  10:03:37
Commandline: apt upgrade
Requested-By: oscar (1000)
Upgrade: linux-libc-dev:amd64 (4.9.189-3+deb9u1, 4.9.189-3+deb9u2)
End-Date: 2019-11-21  10:03:40

 

[Oscar_Hill]

I have found the problem.

I had a NFS storage witch failed some how. Don't know what happened.
So I had backup-jobs that was, I guess waiting for NFS connection indefinitely. And the WebUI was timing out everywhere.

When I removed and reattached it everything started working.