[SOLVED] Proxmox like dead after unexpected shutdown

prox_noob

New Member
Jun 22, 2022
6
1
3
I need help pretty quick.
After power went out and my server restarted none of the VMs came back to life neither did proxmox. Can´t access webinterface only over putty and the IP. Can´t seem to find the problem since I dunno that much ´bout proxmox.
 
hi,

Can´t access webinterface only over putty and the IP. Can´t seem to find the problem since I dunno that much ´bout proxmox.
if you can SSH to your server then you can check the journals for any apparent error messages.

please run journalctl -b0 > journal.txt and attach your journal here if you need help, along with the output from pveversion -v on your server.
 
hi,


if you can SSH to your server then you can check the journals for any apparent error messages.

please run journalctl -b0 > journal.txt and attach your journal here if you need help, along with the output from pveversion -v on your server.
alright. ty for the fast answer.
its really much tho..
pveversion is at the buttom
 

Attachments

seems like pve-cluster service is not running correctly, that's why the other services aren't coming up either.

excerpt from your log:

Code:
1258 Jun 22 10:27:24 pve systemd[1]: Starting The Proxmox VE cluster filesystem...                                                                                                                                                                                                          
1259 Jun 22 10:27:24 pve pmxcfs[1087]: fuse: mountpoint is not empty                                                                                                                                                                                                                        
1260 Jun 22 10:27:24 pve pmxcfs[1087]: fuse: if you are sure this is safe, use the 'nonempty' mount option                                                                                                                                                                                  
1261 Jun 22 10:27:24 pve pmxcfs[1087]: [main] crit: fuse_mount error: File exists                                                                                                                                                                                                           
1262 Jun 22 10:27:24 pve pmxcfs[1087]: [main] notice: exit proxmox configuration filesystem (-1)                                                                                                                                                                                            
1263 Jun 22 10:27:24 pve pmxcfs[1087]: [main] crit: fuse_mount error: File exists                                                                                                                                                                                                           
1264 Jun 22 10:27:24 pve pmxcfs[1087]: [main] notice: exit proxmox configuration filesystem (-1)                                                                                                                                                                                            
1265 Jun 22 10:27:24 pve systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION                                                                                                                                                                         
1266 Jun 22 10:27:24 pve systemd[1]: pve-cluster.service: Failed with result 'exit-code'.                                                                                                                                                                                                   
1267 Jun 22 10:27:24 pve systemd[1]: Failed to start The Proxmox VE cluster filesystem.

the error mentions that the mountpoint is not empty, it's talking about the cluster filesystem that's normally mounted in /etc/pve/ directory [0].
so it seems like you might have some files on there.

you could do the following to take a backup of the directory and restart the service to see if it helps:
Code:
tar czvf /root/etc-pve.tar.gz /etc/pve # create a backup archive of the directory contents
systemctl stop pve-cluster # stop the service just in case
rm -rf /etc/pve # clean it up
systemctl start pve-cluster

if that works you should be able to restart the rest of the services

hope this helps

[0]: https://pve.proxmox.com/pve-docs/chapter-pmxcfs.html
 
  • Like
Reactions: prox_noob
seems like pve-cluster service is not running correctly, that's why the other services aren't coming up either.

excerpt from your log:

Code:
1258 Jun 22 10:27:24 pve systemd[1]: Starting The Proxmox VE cluster filesystem...                                                                                                                                                                                                         
1259 Jun 22 10:27:24 pve pmxcfs[1087]: fuse: mountpoint is not empty                                                                                                                                                                                                                       
1260 Jun 22 10:27:24 pve pmxcfs[1087]: fuse: if you are sure this is safe, use the 'nonempty' mount option                                                                                                                                                                                 
1261 Jun 22 10:27:24 pve pmxcfs[1087]: [main] crit: fuse_mount error: File exists                                                                                                                                                                                                          
1262 Jun 22 10:27:24 pve pmxcfs[1087]: [main] notice: exit proxmox configuration filesystem (-1)                                                                                                                                                                                           
1263 Jun 22 10:27:24 pve pmxcfs[1087]: [main] crit: fuse_mount error: File exists                                                                                                                                                                                                          
1264 Jun 22 10:27:24 pve pmxcfs[1087]: [main] notice: exit proxmox configuration filesystem (-1)                                                                                                                                                                                           
1265 Jun 22 10:27:24 pve systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION                                                                                                                                                                        
1266 Jun 22 10:27:24 pve systemd[1]: pve-cluster.service: Failed with result 'exit-code'.                                                                                                                                                                                                  
1267 Jun 22 10:27:24 pve systemd[1]: Failed to start The Proxmox VE cluster filesystem.

the error mentions that the mountpoint is not empty, it's talking about the cluster filesystem that's normally mounted in /etc/pve/ directory [0].
so it seems like you might have some files on there.

you could do the following to take a backup of the directory and restart the service to see if it helps:
Code:
tar czvf /root/etc-pve.tar.gz /etc/pve # create a backup archive of the directory contents
systemctl stop pve-cluster # stop the service just in case
rm -rf /etc/pve # clean it up
systemctl start pve-cluster

if that works you should be able to restart the rest of the services

hope this helps

[0]: https://pve.proxmox.com/pve-docs/chapter-pmxcfs.html
omg it worked. I love you
thanks
 
  • Like
Reactions: oguz
For us to avoid such future issues, could you please post what files were wrongly in the mountpoint before pmxcfs being started:
tar tf /root/etc-pve.tar.gz
when i paste that code and enter I only get this:

root@pve:~# tar tf /root/etc-pve.tar.gz
etc/pve/
etc/pve/pve-root-ca.pem

Idk if thats what you where looking for..
 
A hard reset of a server is never good and you will loose more or less data, especially if you got consumer SSDs and so one without a powerloss protection. So its always a good idea to put your server behind a UPS (useful ones start at 50$) that gracefully shutdowns your server on an power outage so such problems won't happen in the first place.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!