[Solved] Question mark

[HomeLabNerd]

I have a question mark on my storage in the Promox gui. The storage is available and accessible but the question won't disappear. How can I solve this without rebooting?

 

[dcsapak]

the storage gets a question mark if the pvestatd daemon is not able to collect stats for > 2 minutes. so either check if pvestat is not hanging crashing on the relevant node, or if stats collection works for it (e.g. look in the journal/syslog if there is any issue with the storage)

 

[HomeLabNerd]

the storage gets a question mark if the pvestatd daemon is not able to collect stats for > 2 minutes. so either check if pvestat is not hanging crashing on the relevant node, or if stats collection works for it (e.g. look in the journal/syslog if there is any issue with the storage)

Thanks for the reply!

I've restarted pvestatd on my 3 node cluster but it didn't solve the issue. It was complaining that the storage wasn't online but it is. I can access the shares via an LXC container. After the restart pvestatd status is okay. I've checked the log but can't find anything strange..

Update: Also when I check with df -h on the node I see that the shares are mounted.

 

[dcsapak]

ok what kind of storage is it? can you post the config? also can you maybe post an excerpt of the journal when the issue occurs?

 

[HomeLabNerd]

ok what kind of storage is it? can you post the config? also can you maybe post an excerpt of the journal when the issue occurs?

For storage I use a Unifi UNAS Pro. The share is accessed over NFS. I found something in the syslog. This is before I restarted pvestatd. After that I didn't receive these messages anymore. Hope it helps a bit?

Nov 11 12:47:23 pve-3 sshd[558896]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Nov 11 12:47:23 pve-3 systemd-logind[897]: New session 274 of user root.
Nov 11 12:47:23 pve-3 systemd[1]: Started session-274.scope - Session 274 of User root.
Nov 11 12:47:23 pve-3 sshd[558896]: pam_env(sshd:session): deprecated reading of user environment enabled
Nov 11 12:47:28 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:47:28 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:47:38 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:47:38 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:47:48 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:47:48 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:47:59 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:47:59 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:48:08 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:48:08 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:48:18 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:48:18 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:48:28 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:48:28 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:48:38 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:48:38 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:48:48 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:48:48 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:48:58 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:48:58 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:49:08 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:49:09 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:49:18 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:49:18 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:49:28 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:49:28 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:49:38 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:49:38 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:49:48 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:49:48 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:49:58 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:49:58 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:50:08 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:50:08 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:50:19 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:50:19 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:50:28 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:50:28 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:50:29 pve-3 pvedaemon[184846]: <root@pam> starting task UPID:pve-3:00088EBB:0076C824:6731EF85:vzshutdown:310:root@pam:
Nov 11 12:50:29 pve-3 pvedaemon[560827]: shutdown CT 310: UPID:pve-3:00088EBB:0076C824:6731EF85:vzshutdown:310:root@pam:
Nov 11 12:50:38 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:50:38 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:50:38 pve-3 sshd[558896]: Received disconnect from 192.168.1.21 port 43314:11: disconnected by user
Nov 11 12:50:38 pve-3 sshd[558896]: Disconnected from user root 192.168.1.21 port 43314
Nov 11 12:50:38 pve-3 sshd[558896]: pam_unix(sshd:session): session closed for user root
Nov 11 12:50:38 pve-3 systemd-logind[897]: Session 274 logged out. Waiting for processes to exit.
Nov 11 12:50:38 pve-3 pmxcfs[1110]: [status] notice: received log
Nov 11 12:50:41 pve-3 pmxcfs[1110]: [status] notice: received log
Nov 11 12:50:41 pve-3 pmxcfs[1110]: [status] notice: received log
Nov 11 12:50:41 pve-3 sshd[560933]: Accepted publickey for root from 192.168.1.21 port 50588 ssh2: RSA SHA256:HEukI8Wh2VMl7KEzPhrNl5L0M3TkYokA2Xa2YsynmpM
Nov 11 12:50:41 pve-3 sshd[560933]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Nov 11 12:50:41 pve-3 systemd-logind[897]: New session 277 of user root.
Nov 11 12:50:41 pve-3 systemd[1]: Started session-277.scope - Session 277 of User root.
Nov 11 12:50:41 pve-3 sshd[560933]: pam_env(sshd:session): deprecated reading of user environment enabled
Nov 11 12:50:41 pve-3 login[560939]: pam_unix(login:session): session opened for user root(uid=0) by root(uid=0)
Nov 11 12:50:41 pve-3 login[560944]: ROOT LOGIN on '/dev/pts/0' from '192.168.1.21'
Nov 11 12:50:48 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:50:49 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:50:49 pve-3 kernel: vmbr0v3: port 2(veth310i0) entered disabled state
Nov 11 12:50:49 pve-3 kernel: veth310i0 (unregistering): left allmulticast mode
Nov 11 12:50:49 pve-3 kernel: veth310i0 (unregistering): left promiscuous mode
Nov 11 12:50:49 pve-3 kernel: vmbr0v3: port 2(veth310i0) entered disabled state
Nov 11 12:50:49 pve-3 audit[561064]: AVC apparmor="STATUS" operation="profile_remove" profile="/usr/bin/lxc-start" name="lxc-310_</var/lib/lxc>" pid=561064 comm="apparmor_parser"
Nov 11 12:50:49 pve-3 kernel: kauditd_printk_skb: 12 callbacks suppressed
Nov 11 12:50:49 pve-3 kernel: audit: type=1400 audit(1731325849.480:343): apparmor="STATUS" operation="profile_remove" profile="/usr/bin/lxc-start" name="lxc-310_</var/lib/lxc>" pid=561064 comm="apparmor_parser"
Nov 11 12:50:49 pve-3 kernel: EXT4-fs (dm-18): unmounting filesystem 8290bd07-e720-43b0-96cd-e25944d3cc0e.
Nov 11 12:50:50 pve-3 systemd[1]: session-274.scope: Deactivated successfully.
Nov 11 12:50:50 pve-3 pvedaemon[184846]: <root@pam> end task UPID:pve-3:00088EBB:0076C824:6731EF85:vzshutdown:310:root@pam: OK
Nov 11 12:50:50 pve-3 systemd-logind[897]: Removed session 274.
Nov 11 12:50:50 pve-3 systemd[1]: [EMAIL]pve-container@310.service[/EMAIL]: Deactivated successfully.
Nov 11 12:50:58 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:50:58 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:51:08 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:51:08 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:51:18 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:51:18 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:51:28 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:51:28 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:51:38 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:51:38 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:51:48 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:51:48 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:51:59 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:51:59 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:52:08 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:52:08 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:52:18 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:52:18 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:52:28 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:52:28 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:52:38 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:52:38 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:52:42 pve-3 pvedaemon[183197]: <root@pam> starting task UPID:pve-3:00089460:0076FC08:6731F00A:vzdestroy:310:root@pam:
Nov 11 12:52:42 pve-3 pvedaemon[562272]: storage 'proxmox' is not online
Nov 11 12:52:42 pve-3 pvedaemon[183197]: <root@pam> end task UPID:pve-3:00089460:0076FC08:6731F00A:vzdestroy:310:root@pam: storage 'proxmox' is not online
Nov 11 12:52:48 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:52:48 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:52:58 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:52:58 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:53:00 pve-3 sshd[560933]: Received disconnect from 192.168.1.21 port 50588:11: disconnected by user
Nov 11 12:53:00 pve-3 sshd[560933]: Disconnected from user root 192.168.1.21 port 50588
Nov 11 12:53:00 pve-3 sshd[560933]: pam_unix(sshd:session): session closed for user root
Nov 11 12:53:00 pve-3 systemd-logind[897]: Session 277 logged out. Waiting for processes to exit.
Nov 11 12:53:00 pve-3 pmxcfs[1110]: [status] notice: received log
Nov 11 12:53:00 pve-3 systemd[1]: session-277.scope: Deactivated successfully.
Nov 11 12:53:00 pve-3 systemd[1]: session-277.scope: Consumed 3.856s CPU time.
Nov 11 12:53:00 pve-3 systemd-logind[897]: Removed session 277.
Nov 11 12:53:08 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:53:08 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:53:18 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:53:18 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:53:28 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:53:28 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:53:38 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:53:39 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:53:48 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:53:48 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:53:58 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:53:58 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:54:08 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:54:08 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:54:19 pve-3 pvestatd[1260]: storage 'data_nfs' is not online
Nov 11 12:54:19 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:54:28 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:54:38 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:54:48 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:54:58 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:55:08 pve-3 pvestatd[1260]: storage 'proxmox' is not online
Nov 11 12:56:06 pve-3 pvedaemon[184846]: <root@pam> starting task UPID:pve-3:00089B4C:00774BD4:6731F0D6:vzdestroy:310:root@pam:
Nov 11 12:56:07 pve-3 pvedaemon[184846]: <root@pam> end task UPID:pve-3:00089B4C:00774BD4:6731F0D6:vzdestroy:310:root@pam: OK
Nov 11 12:56:35 pve-3 pmxcfs[1110]: [status] notice: received log
Nov 11 12:56:35 pve-3 pmxcfs[1110]: [status] notice: received log
Nov 11 12:56:35 pve-3 sshd[564287]: Accepted publickey for root from 192.168.1.21 port 46382 ssh2: RSA SHA256:HEukI8Wh2VMl7KEzPhrNl5L0M3TkYokA2Xa2YsynmpM
Nov 11 12:56:35 pve-3 sshd[564287]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Nov 11 12:56:35 pve-3 systemd-logind[897]: New session 279 of user root.
Nov 11 12:56:35 pve-3 systemd[1]: Started session-279.scope - Session 279 of User root.
Nov 11 12:56:35 pve-3 sshd[564287]: pam_env(sshd:session): deprecated reading of user environment enabled
Nov 11 12:56:35 pve-3 login[564293]: pam_unix(login:session): session opened for user root(uid=0) by root(uid=0)
Nov 11 12:56:35 pve-3 login[564298]: ROOT LOGIN on '/dev/pts/0' from '192.168.1.21'
Nov 11 12:57:01 pve-3 cron[1227]: (*system*vzdump) RELOAD (/etc/cron.d/vzdump)
Nov 11 12:57:50 pve-3 pct[565206]: <root@pam> starting task UPID:pve-3:00089FDF:0077743D:6731F13E:vzcreate:310:root@pam:
Nov 11 12:57:50 pve-3 kernel: EXT4-fs (dm-18): mounted filesystem ce2cb456-dd88-4db9-9406-5aa77db34e22 r/w with ordered data mode. Quota mode: none.
Nov 11 12:57:51 pve-3 systemd[1]: var-lib-lxc-310-rootfs.mount: Deactivated successfully.

 

[dcsapak]

For storage I use a Unifi UNAS Pro. The share is accessed over NFS. I found something in the syslog. This is before I restarted pvestatd. After that I didn't receive these messages anymore. Hope it helps a bit?

do you also have the logs from before the issue starts? e.g. from when it's working good to when it's showing the question marks?
did it work again after a restart?

also the storage config would be good (/etc/pve/storage.cfg)

generally if the nfs server or the connection is flaky, pvestatd sadly tends to hang (this is generally an issue with nfs in the kernel)

 

[HomeLabNerd]

I restarted all nodes in the cluster because of all updates available. The issue is solved by restarting the nodes.

Dominik thanks for the help and explanation. I think it is because what you suggest in your last comment.