Interface comes up with all question marks

[ERD]

Hello,

Every 2-3 days my proxmox server comes up with all questions marks. See picture.

If I run the following commands, or reboot, it comes back:

systemctl restart pve-cluster
systemctl restart pvedaemon
systemctl restart pveproxy
systemctl restart pvestatd.service

I am using the enterprise repository. This is my version:

root@proxmox:~# pveversion -v
proxmox-ve: 6.3-1 (running kernel: 5.4.78-2-pve)
pve-manager: 6.3-3 (running version: 6.3-3/eee5f901)
pve-kernel-5.4: 6.3-3
pve-kernel-helper: 6.3-3
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.73-1-pve: 5.4.73-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.7
libproxmox-backup-qemu0: 1.0.2-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-3
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.6-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-3
pve-cluster: 6.2-1
pve-container: 3.3-2
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-7
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-2
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.5-pve1

Any assistance would be greatly appreciated. Thanks!

 

[ERD]

I found that if I issue this single command it will fix the interface.

systemctl restart pvestatd.service

 

[fiona]

Hi,
are there any errors in /var/log/syslog from such a failed startup?

 

[ERD]

I do not see any errors, other than my NFS backup which I leave off except when I'm actively running a backup

Feb  2 12:07:37 proxmox pvestatd[15672]: storage 'backup' is not online

Here is the syslog while I issue the command systemctl restart pvestatd.service which fixes the problem.

Feb  2 12:05:41 proxmox systemd[1]: Started Session 24106 of user root.
Feb  2 12:05:47 proxmox systemd[1]: Stopping PVE Status Daemon...
Feb  2 12:05:52 proxmox pvestatd[15641]: server still running - send KILL
Feb  2 12:05:52 proxmox systemd[1]: pvestatd.service: Main process exited, code=killed, status=9/KILL
Feb  2 12:05:53 proxmox systemd[1]: pvestatd.service: Failed with result 'signal'.
Feb  2 12:05:53 proxmox systemd[1]: Stopped PVE Status Daemon.
Feb  2 12:05:53 proxmox systemd[1]: Starting PVE Status Daemon...
Feb  2 12:05:53 proxmox pvestatd[15672]: starting server
Feb  2 12:05:53 proxmox systemd[1]: Started PVE Status Daemon.
Feb  2 12:06:00 proxmox systemd[1]: Starting Proxmox VE replication runner...
Feb  2 12:06:00 proxmox systemd[1]: pvesr.service: Succeeded.
Feb  2 12:06:00 proxmox systemd[1]: Started Proxmox VE replication runner.

 

[fiona]

I'm guessing it's hanging when checking the connection to the NFS server, but the good news is that this should only affect the status daemon. The easiest solution is to mark the storage as disabled while you are not using it. Otherwise the status daemon will continually try to check if it's there and hang (it should time out after 10 seconds, but maybe it's stuck in an uninterruptible state somehow?). We are planning to make it more modular to avoid such problems in the future.

That said, how long did you wait before restarting it? When trying to reproduce the problem here, the question marks go away after a while.

Are there any messages from pvestatd before you restarted it? Is the NFS v4 only?

 

[ERD]

The first time I logged in and saw this problem, I have left it about an hour while searching the forums on suggestions. They stayed question marks over that hour.

I marked the storage as disabled. I will let it run for a few days, see if that fixes the issue, and report back.

Yes, this is NFS v4. I didn't remember seeing any messages when restarting pvestatd. If it happens again, I will take note of any messages when restarting the service.

Thanks for helping be debug this issue!

 

[fiona]

I was asking about v4, because there is a recent change how we check the connection in that case. It's currently not packaged yet, but it might also improve the situation.

 

[ERD]

After I changed the NFS storage as disabled when its powered off, this issue has not come up again.

I've run this setup where I leave my NFS backup server off when not in use for a few years. So it must a recent-ish change in the past few months that is causing this to happen now. I would be interested to try to replicate this again when that recent v4 change gets packaged in.

Thanks!

 

[fiona]

Hi,
a package with the change is now available in our testing repository. You can

1. Add the repository
2. Use apt update && apt install libpve-storage-perl
3. Remove/disable the repository again

to only upgrade the relevant package and its dependencies. The version should be 6.3-6.

 

[ERD]

I updated my server.

Even with the the storage enabled, the question marks are gone!

Thanks for your help!

 

[Ramalama]

@Fabian_E
I have the same issue, if my dns server is down.
(And yes, i connect to proxmox through the ip, but without dns, it looks similar to erd) xD

But that's totally weird, why do i need the dns server to see the names of the containers/vms?

 

[fiona]

Hi,

@Fabian_E
I have the same issue, if my dns server is down.
(And yes, i connect to proxmox through the ip, but without dns, it looks similar to erd) xD

But that's totally weird, why do i need the dns server to see the names of the containers/vms?

do you have any storage that's configured with a hostname (and requires DNS resolution) instead of an IP?

The pvestatd collects information about storages and guests. If it hangs on a storage, you won't see the information about the guests either.

 

[Ramalama]

Hi,

do you have any storage that's configured with a hostname (and requires DNS resolution) instead of an IP?

The pvestatd collects information about storages and guests. If it hangs on a storage, you won't see the information about the guests either.

No, it's all local storage.

The only thing that i map with dns is monitoring. The influx db hostname for the metrics.

Edit:
To be clear, it's just an Webinterface bug.
Cli works as expected.
pct list
qm list
And everything else in cli works xD

 

[fiona]

No, it's all local storage.

The only thing that i map with dns is monitoring. The influx db hostname for the metrics.

That's most likely the problem as the PVE status daemon won't be able to communicate with the metric server.

Edit:
To be clear, it's just an Webinterface bug.
Cli works as expected.
pct list
qm list
And everything else in cli works xD

Do you see any errors from pvestatd in the /var/log/syslog?

 

[dengolius]

That's most likely the problem as the PVE status daemon won't be able to communicate with the metric server.


Do you see any errors from pvestatd in the /var/log/syslog?

Here is my syslog

Apr  2 10:31:01 px4 systemd[1]: Started Proxmox VE replication runner.
Apr  2 10:31:28 px4 pvedaemon[20925]: removing influxdb credentials file '/etc/pve/priv/metricserver/p4.pw' failed: No such file or directory
Apr  2 10:31:29 px4 pvestatd[1130]: node status update error: can't open '/etc/pve/priv/metricserver/p4.pw' - No such file or directory
Apr  2 10:31:29 px4 pvestatd[1130]: qemu status update error: can't open '/etc/pve/priv/metricserver/p4.pw' - No such file or directory
Apr  2 10:31:29 px4 pvestatd[1130]: lxc status update error: can't open '/etc/pve/priv/metricserver/p4.pw' - No such file or directory
Apr  2 10:31:30 px4 pvestatd[1130]: storage status update error: can't open '/etc/pve/priv/metricserver/p4.pw' - No such file or directory
Apr  2 10:31:39 px4 pvestatd[1130]: node status update error: can't open '/etc/pve/priv/metricserver/p4.pw' - No such file or directory
Apr  2 10:31:39 px4 pvestatd[1130]: qemu status update error: can't open '/etc/pve/priv/metricserver/p4.pw' - No such file or directory
Apr  2 10:31:39 px4 pvestatd[1130]: lxc status update error: can't open '/etc/pve/priv/metricserver/p4.pw' - No such file or directory
Apr  2 10:31:39 px4 pvestatd[1130]: storage status update error: can't open '/etc/pve/priv/metricserver/p4.pw' - No such file or directory
Apr  2 10:31:49 px4 pvestatd[1130]: node status update error: can't open '/etc/pve/priv/metricserver/p4.pw' - No such file or directory
Apr  2 10:31:49 px4 pvestatd[1130]: qemu status update error: can't open '/etc/pve/priv/metricserver/p4.pw' - No such file or directory
Apr  2 10:31:49 px4 pvestatd[1130]: lxc status update error: can't open '/etc/pve/priv/metricserver/p4.pw' - No such file or directory
Apr  2 10:31:49 px4 pvestatd[1130]: storage status update error: can't open '/etc/pve/priv/metricserver/p4.pw' - No such file or directory
Apr  2 10:31:59 px4 pvestatd[1130]: node status update error: can't open '/etc/pve/priv/metricserver/p4.pw' - No such file or directory
Apr  2 10:31:59 px4 pvestatd[1130]: qemu status update error: can't open '/etc/pve/priv/metricserver/p4.pw' - No such file or directory
Apr  2 10:31:59 px4 pvestatd[1130]: lxc status update error: can't open '/etc/pve/priv/metricserver/p4.pw' - No such file or directory
Apr  2 10:31:59 px4 pvestatd[1130]: storage status update error: can't open '/etc/pve/priv/metricserver/p4.pw' - No such file or directory
Apr  2 10:32:00 px4 systemd[1]: Starting Proxmox VE replication runner...
Apr  2 10:32:01 px4 systemd[1]: pvesr.service: Succeeded.
Apr  2 10:32:01 px4 systemd[1]: Started Proxmox VE replication runner.
Apr  2 10:32:09 px4 pvestatd[1130]: node status update error: can't open '/etc/pve/priv/metricserver/p4.pw' - No such file or directory
Apr  2 10:32:09 px4 pvestatd[1130]: qemu status update error: can't open '/etc/pve/priv/metricserver/p4.pw' - No such file or directory
Apr  2 10:32:09 px4 pvestatd[1130]: lxc status update error: can't open '/etc/pve/priv/metricserver/p4.pw' - No such file or directory
Apr  2 10:32:09 px4 pvestatd[1130]: storage status update error: can't open '/etc/pve/priv/metricserver/p4.pw' - No such file or directory
Apr  2 10:32:19 px4 pvestatd[1130]: node status update error: can't open '/etc/pve/priv/metricserver/p4.pw' - No such file or directory
Apr  2 10:32:19 px4 pvestatd[1130]: qemu status update error: can't open '/etc/pve/priv/metricserver/p4.pw' - No such file or directory
Apr  2 10:32:19 px4 pvestatd[1130]: lxc status update error: can't open '/etc/pve/priv/metricserver/p4.pw' - No such file or directory
Apr  2 10:32:19 px4 pvestatd[1130]: storage status update error: can't open '/etc/pve/priv/metricserver/p4.pw' - No such file or directory

Also here is what I have in /etc/pve/status.cfg

influxdb: p4
        port 848
        server 192.168.8.11
        influxdbproto http
        max-body-size 1000
        organization px

I use VictoriaMetrics as InfluxDB (it could serve as an alternative to InfluxDB) https://github.com/VictoriaMetrics/...m-influxdb-compatible-agents-such-as-telegraf

I run VictoriaMetrics with docker-compose: https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker

 

[dengolius]

Also I have tried to add dns-name into /etc/hosts 192.168.8.11 victoria


cat /etc/pve/status.cfg

influxdb: px4
    port 8428
    server victoria
    influxdbproto http

But result is the same:

tail -f /var/log/syslog

Apr  2 11:16:49 px4 pvestatd[1130]: lxc status update error: can't open '/etc/pve/priv/metricserver/px4.pw' - No such file or directory
Apr  2 11:16:49 px4 pvestatd[1130]: storage status update error: can't open '/etc/pve/priv/metricserver/px4.pw' - No such file or directory
Apr  2 11:16:59 px4 pvestatd[1130]: node status update error: can't open '/etc/pve/priv/metricserver/px4.pw' - No such file or directory
Apr  2 11:16:59 px4 pvestatd[1130]: qemu status update error: can't open '/etc/pve/priv/metricserver/px4.pw' - No such file or directory
Apr  2 11:16:59 px4 pvestatd[1130]: lxc status update error: can't open '/etc/pve/priv/metricserver/px4.pw' - No such file or directory
Apr  2 11:16:59 px4 pvestatd[1130]: storage status update error: can't open '/etc/pve/priv/metricserver/px4.pw' - No such file or directory

 

[dengolius]

@Fabian_E do you have any idea how to solve this issue?

 

[fiona]

Hi,
so you also have all question marks in the interface? Do they go away when you disable the metric server?

tail -f /var/log/syslog

Apr  2 11:16:49 px4 pvestatd[1130]: lxc status update error: can't open '/etc/pve/priv/metricserver/px4.pw' - No such file or directory
Apr  2 11:16:49 px4 pvestatd[1130]: storage status update error: can't open '/etc/pve/priv/metricserver/px4.pw' - No such file or directory
Apr  2 11:16:59 px4 pvestatd[1130]: node status update error: can't open '/etc/pve/priv/metricserver/px4.pw' - No such file or directory
Apr  2 11:16:59 px4 pvestatd[1130]: qemu status update error: can't open '/etc/pve/priv/metricserver/px4.pw' - No such file or directory
Apr  2 11:16:59 px4 pvestatd[1130]: lxc status update error: can't open '/etc/pve/priv/metricserver/px4.pw' - No such file or directory
Apr  2 11:16:59 px4 pvestatd[1130]: storage status update error: can't open '/etc/pve/priv/metricserver/px4.pw' - No such file or directory

AFAICT these error messages should only be a cosmetic issue (except if you require credentials, you should of course put them in that file).

EDIT: the errors are only cosmetic in git master, not the currently released version.

Also I have tried to add dns-name into /etc/hosts 192.168.8.11 victoria


cat /etc/pve/status.cfg

influxdb: px4
    port 8428
    server victoria
    influxdbproto http

Using the IP is preferable, as you don't need to resolve the name every time. Can you ping the server normally? Does it receive the statistics?

 

[dengolius]

1. so you also have all question marks in the interface? Do they go away when you disable the metric server?
Nope, I have all question marks when I enabled this metrics server. See attached screenshot.

But Virtual Machines works fine, maybe because they works on Linux and not on Windows.




2. AFAICT these error messages should only be a cosmetic issue (except if you require credentials, you should of course put them in that file).

Could you provide example of config that I need to put in this file /etc/pve/priv/metricserver/px4.pw ?

 

[dengolius]

This information can be helpful

root@px4 /home/dengolius #pveversion --verbose
proxmox-ve: 6.3-1 (running kernel: 5.4.103-1-pve)
pve-manager: 6.3-6 (running version: 6.3-6/2184247e)
pve-kernel-5.4: 6.3-7
pve-kernel-helper: 6.3-7
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.103-1-pve: 5.4.103-1
pve-kernel-5.4.44-2-pve: 5.4.44-2
pve-kernel-4.15: 5.4-16
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-4.15.18-27-pve: 4.15.18-55
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.0-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.7
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-5
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-7
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.9-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-6
pve-cluster: 6.2-1
pve-container: 3.3-4
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-2
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.2.0-3
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-7
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.3-pve2