No GUI and not showing up in network.

[Nercules]

Hello

Yesterday my Proxmox home server stopped working. I can no longer access my GUI or SSH in the server. I have a keyboard and screen connected to the physical server to troubleshoot, but I am very new to virtualization and have no idea what I am doing.

I have 1 SSD where proxmox is running, then I have 2 HDDs in mirror for fileserver and one ssd where I store all my VMs (all mounted as ZFS partitions). The goal is to keep all this data but fix the proxmox on the ssd.

Is it as simple as reinstalling a new proxmox iso on the ssd, and can proxmox find the VMs and ZFSs? I read something online where I need to backup some /etc/pve files, but this folder is empty for me.

I could also try to fix the problem instead of reinstalling proxmox, but if I want to do that, I need to fix the SSH problem first so I can show what the terminal prints. The homeserver is also not showing up in my network.

I looked at some other threads on this forum, and I saw that my pve-cluster is down. But I have no idea how to proceed now.

Please note that I am a dummy in this if you want to help me and explain things.

 

[cheiss]

Hi,

first of: Can you ping the server? Did you have a power outage or similar which might have caused an abrupt shutdown?

And secondly, since you do have physical access to the server: Can you see what ip a (to see if the network configuration is sane) and systemctl status sshd pveproxy outputs?

You can also attach actual pictures of the screen with the output readable - that's a pretty common solution for such situations where you have no ssh access.

Reinstalling Proxmox as a whole is probably way overkill for this ..

I read something online where I need to backup some /etc/pve files, but this folder is empty for me.

.. although this sounds a bit concerning. This folder holds all the configuration for PVE, esp. VM/CT configs.

 

[Nercules]

I forced the machine to turn off by holding the power button after a vm wouldn't stop running. The server is not pingable "Destination Host Unreachable".

The command outputs: https://www.linkpicture.com/q/331944264_2500319130121031_7243623729352092341_n.jpg

 

[cheiss]

Is the network cable properly connected/intact? The IP is at least correctly set (assuming 172.20.70.100 is the correct one), but the actual network interface (enp4s0 I guess) is not UP, thus the vmbr0 also has NO-CARRIER.
That explains why you cannot ping or reach it via ssh, since it is not connected to the network.

Regarding the other problem with /etc/pve (which is also why you cannot reach the web interface):
What does systemctl status pve-cluster.service say? Also, are there any other warnings/errors in the syslog journalctl -b that might be relevant for this?

 

[Nercules]

Here is a video showing the errors displayed at boot: https://youtu.be/cIW8j-MMIZA

Here is the output of systemctl status pve-cluster.service: https://www.linkpicture.com/q/325907488_2985432465093424_6826011091129851385_n.jpg

And here are pictures of the journalctl -b: https://we.tl/t-yrOlISujON

Also, the ethernet cable and port are working properly. I was able to connect to an other device when swapping.

 

[cheiss]

Well, the syslog mentions several times "No space left on device", meaning that the root partition seems to be full.
That also explains why everything else is just failing horribly ..
The current "situation" can be checked using du -h.

So you just need to clear some space on the root partition, after that everything should work normally again.
There are several things for example you can try:
- Delete apt package cache: apt clean
- Clear old syslogs: journalctl --vacuum-time=1h
- Clear old PVE logs: rm -v /var/log/pveproxy/access.log.*
- Clearing other old logfiles, you can identify these folders using du -hd1 /var/log | sort -h
- Deleting (old) ISOs in /var/lib/vz/template/iso

After deleting some things (try do have at least 1-2GiB free space on / again), try rebooting and all these things should hopefully start working again (if nothing got e.g. corrupted due to running out of space).

 

[Nercules]

I removed some of the logs and iso's, but the system still wont work. As you can see, I should have plenty of space left on the boot disk.

What has changed thow, is the

status pve-cluster.service

, that is now active (running).

ip a

gives a different output now: https://www.linkpicture.com/q/331546132_1256622445065371_6953691490828605799_n.jpg

outputs from

ournalctl -b

https://we.tl/t-9qwzuBmY1M

 

[Nercules]

I don't understand why the disk is so full. I recently moved a few vm's from the bootdisk to an other disk, I checked the remove old partition toggle, but maybe it didn't delete them? Anyway, the goal is to fully reserve the boot disk for proxmox.

How can I see what is taking up all of this space, and clean up?

 

[cheiss]

Using df -h you can get a good overview of the current disk usage.

To inspect the root partition a bit more du -hd1 /<directory> is a good starting point.
Some usual "suspects" are: /var, /var/lib/vz, /var/cache, /var/backups, /var/log, /root.

If you want a more "graphical tool, there's ncdu / (you need to apt install ncdu before).