[SOLVED] Server freezes after 1 hour

[Palerius]

Good morning everyone.

I am having a new and rather unique issue with my proxmox setup.
The server was running perfectly fine for the last months and now all of a sudden this issue appears:

The server will completely freeze. No VM will be available neither via FTP, SSH nor on the web.
I connected a monitor to see if any messages pop up but noticed that the monitor just turns off when the issue happens.
The server stays running for something in between one hour, that's it.

I have no idea what is happening and it is pretty frustrating.

I did look in the /var/log/syslog but didn't really notice anything. (But am also not sure what I am looking for)

Any ideas?

Running Proxmox 6.4-13

 

[VictorSTS]

Could be a hardware issue. When the freeze happens:
- does the Proxmox web UI work?
- can you connect vi SSH to Proxmox host?
- does the host answers to ping requests?
- If there's a keyboard connected to the server, does switching the Caps Lock key actually lit the keyboard "caps lock" led?

 

[SOLTECSIS - Carles Munyoz]

It seems a hardware issue. I understand that when this happens the only solution is power off/on the server, is it right ?

Do you have an IPMI interface or similar?
If the answer is yes, then you can access it and see analyze the logs, maybe you can find the explanation to the problem here.

 

[Palerius]

Thanks for your responses.

When the freeze happens:
- The web UI does not work anymore.
- I can not ssh into the host anymore
- No ping return. Everything is basically dead
- no reaction from cap/numpad light once it freezes

It does also not always run an hour. this time it froze after 15 mins.

I understand that when this happens the only solution is power off/on the server, is it right ?

Correct. The machine is stuck in this state until reboot.

I do not have any IPMI as this is not deployed commercially but "only" a private server hosting personal things.

Also: Just throwing this out there as I did have issues with this in the past:
Could this be related in any way to an attack on the server? The server IP is public-facing as it is hosting a webserver as well.

EDIT:
The machine does still do "something" as the fans still ramp up every now and then even when in a frozen state.
Router also shows the host as well as all the VMs as online in the network (but this might be due to slow update on the side of my router)

 

[VictorSTS]

But... the web server is running in the same Poxmox host or is it running inside a VM? Are SSH, webUI and other Proxmox port exposed on the Internet? Whether the freezes are related or not, you must protect your host as much as possible.

You can try to use netdata (apt install netdata) to fetch some metrics in realtime and review them after a freeze. Try to find spikes in CPU, memory or swap usage.

Anyway, I still think this is hardware related by the simptoms you describe.

 

[Palerius]

The only thing exposed outside the local network is my router's IP. Everything from there gets forwarded.
Anything like a web server, game server, or anything that gets passed through the router to the server will only go to a dedicated VM. Nothing is run or hosted on the actual host except some monitoring software that is only available within the local network.

Mentioned monitoring does work until the freeze and Grafana shows no spikes in CPU load, RAM load, or specific high network traffic. Everything seems to run "as normal" till the moment of freeze.

I am so beyond understanding how this can be.

 

[Palerius]

Update:

In my desperation, I started to remove RAM sticks one at a time and this seems to have fixed the issue. After many freezes the server is now running for over 48h again so I assume there was an issue with either the RAM I removed or the slot on the board.

I appreciate y'alls input!

 

[SOLTECSIS - Carles Munyoz]

You can run a Memtest for check your RAM:
https://linuxhint.com/run_memtest_ubuntu/