no web gui, qm list: failed connection refused

[kblknight]

Hi everyone

just right now intalled the latest Updates 7.1-10 and rebooted my server.

no GUI, i cant see my qm lists.

#systemctl status pve-cluster pveprox pvedaemon

pve-cluster.service - the proxmox ve cluster filesystem
loaded : loaded (/lib/systemd/system/pve-cluster.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2022-01-19 10:43:54 CET; 6 mint ago
process: 999 ExecStart=/usr/bin/pmxcfs (code=exited, status=255/EXCEPTION)

THE other two are Active and running.

i would be happy to hear from you guys.

Regards
KBLKnight

 

[oguz]

hi,

no GUI, i cant see my qm lists.

yes sounds like it's not alive?

can you please post the output from journalctl -u pve* -b0? and your pveversion -v please

 

[kblknight]

hi,


yes sounds like it's not alive?

can you please post the output from journalctl -u pve* -b0? and your pveversion -v please

hey hi,

the journalctl -u pve* -b0 output is huge and on the other server i cant write it all

but the pveversion -v is as following:

proxmox-ve: 7.1-1 (running kernel: 5.13.19-3-pve)
pve-manager : 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-helper: 7.1-8
pve-kernel: 5.13: 7.1-6
and .......


i would be happy if you can tell me which line do you need so i can write it

Regards

 

[oguz]

the journalctl -u pve* -b0 output is huge and on the other server i cant write it all

you could try to attach it as a text file here, or if it's too big you can use pastebin etc. or a file upload service.
journalctl -u pve* -b0 > journal-pve1.txt and then you should have a file with the contents of the journal

but the pveversion -v is as following:

could you post the whole thing? (tip: use [code][/code] tags)
also please make sure you've upgraded all the hosts to the same versions and rebooted all of them. (pveversion outputs should be matching)

would be good to see pvecm status as well, since it seems like pmxcfs isn't running.. did you upgrade all the servers or just this one?

 

[Patricktslee]

Hi All,

I got the same issue. After reboot the PVE server, the hostname is automatically changed with the one of the VM hostname. I can’t change the hostsname back to original one.

Can someone help to check?

Attached please find the journal-pve1.txt and PVE version file

Thanks with the best regards

Patrick Lee

 

[oguz]

Hi All,

I got the same issue. After reboot the PVE server, the hostname is automatically changed with the one of the VM hostname. I can’t change the hostsname back to original one.

Can someone help to check?

Attached please find the journal-pve1.txt and PVE version file

Thanks with the best regards

Patrick Lee

check the contents of cat /etc/hosts file, make sure that the IP address for your node matches the correct one.
hostname -i should return the IP address that you normally use for accessing the GUI

 

[Patricktslee]

check the contents of cat /etc/hosts file, make sure that the IP address for your node matches the correct one.
hostname -i should return the IP address that you normally use for accessing the GUI

I set “preserve_hostname” to true on the /etc/cloud/cloud.cfg to ensure the hosts file not updated by system.

I forced to correct the hostname by running the below command and ensure the hostname is set to correct one.

hostnamectl set-hostname pve-morefine

I manually edit the /etc/hosts on below and reboot the system.

# Your system has configured 'manage_etc_hosts' as True.
# As a result, if you wish for changes to this file to persist
# then you will need to either
# a.) make changes to the master file in /etc/cloud/templates/hosts.debian.tmpl
# b.) change or remove the value of 'manage_etc_hosts' in
# /etc/cloud/cloud.cfg or cloud-config from user-data
#
#127.0.0.1 pve-morefine appwrite
#127.0.0.1 PVE-morefine
#127.0.0.1 localhost
127.0.0.1 pve-morefine pve-morefine

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

Then run the command

hostname -i
127.0.0.1

But my GUI IP address is 10.0.0.239. And it is failed to access. The journalctl also show unable to get local IP Address.

What is the next steps to troubleshoot it?

Thanks

Patrick Lee

 

[oguz]

But my GUI IP address is 10.0.0.239

in that case that address should be in your hosts file (instead of 127.0.0.1)

try something like this:

127.0.0.1 localhost
10.0.0.239 pve-morefine

 

[Patricktslee]

in that case that address should be in your hosts file (instead of 127.0.0.1)

try something like this:

127.0.0.1 localhost
10.0.0.239 pve-morefine

I updated the hosts and hostname -i command shown 10.0.0.239. However it is failed and I checked journalctl -b -u pve-cluster. It has another error. Attached please find the file for pve-cluster.txt.

And I checked the /etc/pve files and it is strange thing that it is only has 1 files.

ls /etc/pve
'ystemctl status pve-cluster'

Thanks

 

[oguz]

the relevant error is:

pmxcfs[3953]: fuse: mountpoint is not empty

that's because there are some files on /etc/pve/ (but it should be empty while the pve-cluster service isn't working)

is your node part of a cluster?

you can try the following:

tar czvf etc-pve-backup.tar.gz /etc/pve
systemctl stop pve-cluster
rm -rf /etc/pve/*
systemctl start pve-cluster

 

[Patricktslee]

the relevant error is:

pmxcfs[3953]: fuse: mountpoint is not empty

that's because there are some files on /etc/pve/ (but it should be empty while the pve-cluster service isn't working)

is your node part of a cluster?

you can try the following:

tar czvf etc-pve-backup.tar.gz /etc/pve
systemctlsyst stop pve-cluster
rm -rf /etc/pve/*
systemctl start pve-cluster

This is standalone single server.

I run it and it would login GUI but the VM failed to start

please see the attached screen dump.

 

[oguz]

This is standalone single server.

I run it and it would login GUI but the VM failed to start

please see the attached screen dump.

okay, good that the GUI starts correctly now.
check your journal again to see if there are any errors

 

[Patricktslee]

okay, good that the GUI starts correctly now.
check your journal again to see if there are any errors

Check the journalctl -b -u pve-cluster and attached the log for detail.

-- Journal begins at Sat 2022-05-28 07:45:38 HKT, ends at Thu 2022-07-14 20:18:16 HKT. --
Jul 14 19:47:25 pve-morefine systemd[1]: Starting The Proxmox VE cluster filesystem...
Jul 14 19:47:25 pve-morefine pmxcfs[3054]: fuse: mountpoint is not empty
Jul 14 19:47:25 pve-morefine pmxcfs[3054]: fuse: if you are sure this is safe, use the 'nonempty' mount option
Jul 14 19:47:25 pve-morefine pmxcfs[3054]: [main] crit: fuse_mount error: File exists
Jul 14 19:47:25 pve-morefine pmxcfs[3054]: [main] notice: exit proxmox configuration filesystem (-1)
Jul 14 19:47:25 pve-morefine pmxcfs[3054]: [main] crit: fuse_mount error: File exists
Jul 14 19:47:25 pve-morefine pmxcfs[3054]: [main] notice: exit proxmox configuration filesystem (-1)
Jul 14 19:47:25 pve-morefine systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Jul 14 19:47:25 pve-morefine systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Jul 14 19:47:25 pve-morefine systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Jul 14 19:47:25 pve-morefine systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 1.
Jul 14 19:47:25 pve-morefine systemd[1]: Stopped The Proxmox VE cluster filesystem.
Jul 14 19:47:25 pve-morefine systemd[1]: Starting The Proxmox VE cluster filesystem...
Jul 14 19:47:25 pve-morefine pmxcfs[3788]: fuse: mountpoint is not empty
Jul 14 19:47:25 pve-morefine pmxcfs[3788]: fuse: if you are sure this is safe, use the 'nonempty' mount option
Jul 14 19:47:25 pve-morefine pmxcfs[3788]: [main] crit: fuse_mount error: File exists
Jul 14 19:47:25 pve-morefine pmxcfs[3788]: [main] notice: exit proxmox configuration filesystem (-1)
Jul 14 19:47:25 pve-morefine pmxcfs[3788]: [main] crit: fuse_mount error: File exists

And try to list the /etc/pve and show the below files.

ls /etc/pve
authkey.pub datacenter.cfg jobs.cfg lxc openvz pve-root-ca.pem qemu-server storage.cfg virtual-guest
authkey.pub.old ha local nodes priv pve-www.key sdn user.cfg vzdump.cron

 

[oguz]

And try to list the /etc/pve and show the below files.

seems good, so pve-cluster is working.

the journal you've posted now is the same one from before (the entries have the same timestamps) so those are the old errors.
please check the journal of the other PVE services (not just pve-cluster).
you can do that with journalctl -b0 -u "pve*"

 

[Patricktslee]

seems good, so pve-cluster is working.

the journal you've posted now is the same one from before (the entries have the same timestamps) so those are the old errors.
please check the journal of the other PVE services (not just pve-cluster).
you can do that with journalctl -b0 -u "pve*"

Here is attached file for journalctl -b0 -u "pve*".

 

[oguz]

Here is attached file for journalctl -b0 -u "pve*".

try:

pvecm updatecerts --force

also check if /etc/hostname contains the right name, then restart the PVE services systemctl restart pvedaemon pveproxy pvestatd

 

[Patricktslee]

try:

pvecm updatecerts --force

also check if /etc/hostname contains the right name, then restart the PVE services systemctl restart pvedaemon pveproxy pvestatd

The hostname is the same pve-morefine

And it is failed to launch GUI right now.

Here is the journalctl log.

 

[oguz]

The hostname is the same pve-morefine

And it is failed to launch GUI right now.

Here is the journalctl log.

i don't see any new errors in your journal.
how does the GUI fail exactly?
you can check if the service is running, and also if the port is open: ss -anltp | grep pve should show you if port 8006 is open

 

[Patricktslee]

i don't see any new errors in your journal.
how does the GUI fail exactly?
you can check if the service is running, and also if the port is open: ss -anltp | grep pve should show you if port 8006 is open

I believe it works. After I updateCerts and load the PVE GUI on safari, it is looped. I believe the new cert not taken effect. So I use Chrome to launch the GUI, it can show the GUI.

I am going to backup all my VM into the NFS storage in case the PVE is going wrong.

BTW, do you know why my PVE server experienced this issue? Actually, I didn’t change anything and the machine is rebooted and cause the hostname change.

Thanks your help again.

 

[oguz]

So I use Chrome to launch the GUI, it can show the GUI.

great, safari might have the previous certificate cached.

I am going to backup all my VM into the NFS storage in case the PVE is going wrong.

always a good idea

BTW, do you know why my PVE server experienced this issue? Actually, I didn’t change anything and the machine is rebooted and cause the hostname change.

not sure, but the first /etc/hosts file you've sent had "appwrite" entry in it which could have caused the hostname change...
you also mentioned cloud-init on the host so i would take a look there as well.
otherwise i don't have other ideas for you