grey question mark

H

hmmatos

Renowned Member
Sep 8, 2015
11
1
68
Hello,

I have been having this recurring problem, grey question mark

Screenshot 2023-03-08 at 15.18.18.png

I tried to restart the services but the problem remains

Code: 
systemctl stop pve-cluster && systemctl stop corosync && systemctl stop pvedaemon && systemctl stop pveproxy && systemctl stop pvestatd
systemctl start pve-cluster && systemctl start corosync && systemctl start pvedaemon && systemctl start pveproxy && systemctl start pvestatd

The only way to fix it in restart the node.

--------------------------

cat /var/log/syslog

Code: 
Mar  8 15:23:43 cofgr-pve-001 pveproxy[1375231]: proxy detected vanished client connection
Mar  8 15:23:43 cofgr-pve-001 pveproxy[1375230]: proxy detected vanished client connection
Mar  8 15:23:46 cofgr-pve-001 pveproxy[1375230]: proxy detected vanished client connection
Mar  8 15:23:46 cofgr-pve-001 pveproxy[1375230]: proxy detected vanished client connection
Mar  8 15:23:47 cofgr-pve-001 pveproxy[1375230]: proxy detected vanished client connection

journalctl -f

Code: 
Mar 08 15:32:29 cofgr-pve-001 pveproxy[1375230]: proxy detected vanished client connection
Mar 08 15:32:30 cofgr-pve-001 pveproxy[1375232]: proxy detected vanished client connection
Mar 08 15:32:33 cofgr-pve-001 pveproxy[1375232]: proxy detected vanished client connection
Mar 08 15:32:43 cofgr-pve-001 pveproxy[1375232]: proxy detected vanished client connection
Mar 08 15:32:53 cofgr-pve-001 pveproxy[1375232]: proxy detected vanished client connection

pveversion -v

Code: 
proxmox-ve: 7.3-1 (running kernel: 5.15.85-1-pve)
pve-manager: 7.3-6 (running version: 7.3-6/723bb6ec)
pve-kernel-helper: 7.3-5
pve-kernel-5.15: 7.3-2
pve-kernel-5.13: 7.1-9
pve-kernel-5.11: 7.0-10
pve-kernel-5.15.85-1-pve: 5.15.85-1
pve-kernel-5.15.83-1-pve: 5.15.83-1
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-4-pve: 5.11.22-9
ceph-fuse: 16.2.11-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.3-1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-2
libpve-guest-common-perl: 4.2-3
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.3-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.3.3-1
proxmox-backup-file-restore: 2.3.3-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.1-1
proxmox-widget-toolkit: 3.5.5
pve-cluster: 7.3-2
pve-container: 4.4-2
pve-docs: 7.3-1
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-7
pve-firmware: 3.6-3
pve-ha-manager: 3.5.1
pve-i18n: 2.8-3
pve-qemu-kvm: 7.2.0-5
pve-xtermjs: 4.16.0-1
qemu-server: 7.3-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.9-pve1

uptime

Code: 
15:48:43 up 28 days, 22:21,  1 user,  load average: 1.75, 1.96, 2.08

any ideas on how to fix this?
 
Hello,

yes, Firefox and Safari is the same.

another thing that happens is in the login to be able to login I first have to restart the services.
 
Can you provide us more verbose from the syslog/journalctl?

From the image you posted, I see that there is no cluster, right? if it's no a cluster, you not need to restart the corosync and pve-cluster services.


Another thing I would check is the (Storage) is all storage are online pvesm status?
 
right there is no cluster.

Code: 
root@cofgr-pve-001:~# pvesm status
Name                      Type     Status           Total            Used       Available        %
Veeam-BackupServer        cifs     active     62507778048     55362366528      7145411520   88.57%
loca-zfs               zfspool     active      2901934080        73251116      2828682964    2.52%
local                      dir     active        57225328        15028124        39257916   26.26%

Code: 
root@cofgr-pve-001:~# journalctl -f
-- Journal begins at Tue 2022-12-20 01:20:42 WET. --
Mar 09 09:20:17 cofgr-pve-001 pveproxy[3429953]: proxy detected vanished client connection
Mar 09 09:20:23 cofgr-pve-001 pveproxy[3429953]: proxy detected vanished client connection
Mar 09 09:20:29 cofgr-pve-001 pveproxy[3429954]: proxy detected vanished client connection
Mar 09 09:20:33 cofgr-pve-001 pveproxy[3429952]: proxy detected vanished client connection
Mar 09 09:20:39 cofgr-pve-001 pveproxy[3429953]: proxy detected vanished client connection
Mar 09 09:20:43 cofgr-pve-001 pveproxy[3429952]: proxy detected vanished client connection
Mar 09 09:20:43 cofgr-pve-001 pveproxy[3429954]: proxy detected vanished client connection
Mar 09 09:20:52 cofgr-pve-001 pveproxy[3429952]: proxy detected vanished client connection
Mar 09 09:20:53 cofgr-pve-001 pveproxy[3429952]: proxy detected vanished client connection
Mar 09 09:21:03 cofgr-pve-001 pveproxy[3429953]: proxy detected vanished client connection
Mar 09 09:21:13 cofgr-pve-001 pveproxy[3429952]: proxy detected vanished client connection
Mar 09 09:21:16 cofgr-pve-001 pveproxy[3429952]: proxy detected vanished client connection
Mar 09 09:21:22 cofgr-pve-001 pveproxy[3429952]: proxy detected vanished client connection
Mar 09 09:21:23 cofgr-pve-001 pveproxy[3429952]: proxy detected vanished client connection

Code: 
root@cofgr-pve-001:~# tail -f /var/log/syslog
Mar  9 09:21:29 cofgr-pve-001 pveproxy[3429953]: proxy detected vanished client connection
Mar  9 09:21:30 cofgr-pve-001 pveproxy[3429953]: proxy detected vanished client connection
Mar  9 09:21:33 cofgr-pve-001 pveproxy[3429953]: proxy detected vanished client connection
Mar  9 09:21:43 cofgr-pve-001 pveproxy[3429954]: proxy detected vanished client connection
Mar  9 09:21:43 cofgr-pve-001 pveproxy[3429954]: proxy detected vanished client connection
Mar  9 09:21:43 cofgr-pve-001 pveproxy[3429952]: proxy detected vanished client connection
Mar  9 09:21:46 cofgr-pve-001 pveproxy[3429952]: proxy detected vanished client connection
Mar  9 09:21:52 cofgr-pve-001 pveproxy[3429954]: proxy detected vanished client connection
Mar  9 09:21:53 cofgr-pve-001 pveproxy[3429952]: proxy detected vanished client connection
Mar  9 09:22:03 cofgr-pve-001 pveproxy[3429954]: proxy detected vanished client connection
Mar  9 09:22:13 cofgr-pve-001 pveproxy[3429954]: proxy detected vanished client connection
Mar  9 09:22:16 cofgr-pve-001 pveproxy[3429954]: proxy detected vanished client connection
Mar  9 09:22:16 cofgr-pve-001 pveproxy[3429952]: proxy detected vanished client connection
Mar  9 09:22:22 cofgr-pve-001 pveproxy[3429954]: proxy detected vanished client connection
Mar  9 09:22:23 cofgr-pve-001 pveproxy[3429952]: proxy detected vanished client connection


Screenshot 2023-03-09 at 09.27.02.png
 
Can you restart only the pveproxy and pvedaemon services and then post the status output here:

Bash: 
systemctl restart pveproxy.service
systemctl status pveproxy.service

systemctl restart pvedaemon.service
systemctl status pvedaemon.service
 
Code: 
root@cofgr-pve-001:~# systemctl restart pveproxy.service
root@cofgr-pve-001:~# systemctl status pveproxy.service
● pveproxy.service - PVE API Proxy Server
     Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-03-09 09:45:33 WET; 8s ago
    Process: 3551694 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)
    Process: 3551697 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
   Main PID: 3551699 (pveproxy)
      Tasks: 4 (limit: 154305)
     Memory: 206.7M
        CPU: 1.934s
     CGroup: /system.slice/pveproxy.service
             ├─3551699 pveproxy
             ├─3551700 pveproxy worker
             ├─3551701 pveproxy worker
             └─3551702 pveproxy worker

Mar 09 09:45:31 cofgr-pve-001 systemd[1]: Starting PVE API Proxy Server...
Mar 09 09:45:33 cofgr-pve-001 pveproxy[3551699]: starting server
Mar 09 09:45:33 cofgr-pve-001 pveproxy[3551699]: starting 3 worker(s)
Mar 09 09:45:33 cofgr-pve-001 pveproxy[3551699]: worker 3551700 started
Mar 09 09:45:33 cofgr-pve-001 pveproxy[3551699]: worker 3551701 started
Mar 09 09:45:33 cofgr-pve-001 pveproxy[3551699]: worker 3551702 started
Mar 09 09:45:33 cofgr-pve-001 systemd[1]: Started PVE API Proxy Server.

Code: 
root@cofgr-pve-001:~# systemctl restart pvedaemon.service
root@cofgr-pve-001:~# systemctl status pvedaemon.service
● pvedaemon.service - PVE API Daemon
     Loaded: loaded (/lib/systemd/system/pvedaemon.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-03-09 09:46:42 WET; 3s ago
    Process: 3553733 ExecStart=/usr/bin/pvedaemon start (code=exited, status=0/SUCCESS)
   Main PID: 3553735 (pvedaemon)
      Tasks: 5 (limit: 154305)
     Memory: 137.4M
        CPU: 1.350s
     CGroup: /system.slice/pvedaemon.service
             ├─3553735 pvedaemon
             ├─3553736 pvedaemon worker
             ├─3553737 pvedaemon worker
             ├─3553738 pvedaemon worker
             └─3553856 lxc-info -n 103 -p

Mar 09 09:46:40 cofgr-pve-001 systemd[1]: Starting PVE API Daemon...
Mar 09 09:46:41 cofgr-pve-001 pvedaemon[3553735]: starting server
Mar 09 09:46:41 cofgr-pve-001 pvedaemon[3553735]: starting 3 worker(s)
Mar 09 09:46:41 cofgr-pve-001 pvedaemon[3553735]: worker 3553736 started
Mar 09 09:46:41 cofgr-pve-001 pvedaemon[3553735]: worker 3553737 started
Mar 09 09:46:41 cofgr-pve-001 pvedaemon[3553735]: worker 3553738 started
Mar 09 09:46:42 cofgr-pve-001 systemd[1]: Started PVE API Daemon.

I have a connection issue on postfix but I thinks is not related

Code: 
Mar 09 09:41:36 cofgr-pve-001 postfix/smtp[3535933]: connect to XXX.com[192.168.XXX.XXX]:25: Connection timed out
Mar 09 09:41:36 cofgr-pve-001 postfix/smtp[3535933]: DF3662E104E: to=<XXX@XXX.com>, relay=none, delay=380495, delays=380345/0.02/150/0, dsn=4.4.1, status=deferred (connect to XXX.com[192.168.XXX.XXX]:25: Connection timed out)
Mar 09 09:41:36 cofgr-pve-001 postfix/smtp[3535934]: connect to XXX.com[192.168.XXX.XXX]:25: Connection timed out
Mar 09 09:41:36 cofgr-pve-001 postfix/smtp[3535934]: 66E1A2E1069: to=<XXX@XXX.com>, relay=none, delay=405692, delays=405542/0.02/150/0, dsn=4.4.1, status=deferred (connect to XXX.com[192.168.XXX.XXX]:25: Connection timed out)
 
2 or 3 minutes after restart proxy service starts showing "proxy detected vanished client connection"

Code: 
root@cofgr-pve-001:~# systemctl status pveproxy.service
● pveproxy.service - PVE API Proxy Server
     Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-03-09 09:45:33 WET; 15min ago
    Process: 3551694 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)
    Process: 3551697 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
   Main PID: 3551699 (pveproxy)
      Tasks: 4 (limit: 154305)
     Memory: 222.6M
        CPU: 4.832s
     CGroup: /system.slice/pveproxy.service
             ├─3551699 pveproxy
             ├─3551700 pveproxy worker
             ├─3551701 pveproxy worker
             └─3551702 pveproxy worker

Mar 09 10:00:52 cofgr-pve-001 pveproxy[3551701]: proxy detected vanished client connection
Mar 09 10:00:53 cofgr-pve-001 pveproxy[3551702]: proxy detected vanished client connection
Mar 09 10:00:59 cofgr-pve-001 pveproxy[3551701]: proxy detected vanished client connection
Mar 09 10:01:03 cofgr-pve-001 pveproxy[3551700]: proxy detected vanished client connection
Mar 09 10:01:03 cofgr-pve-001 pveproxy[3551701]: proxy detected vanished client connection
Mar 09 10:01:13 cofgr-pve-001 pveproxy[3551702]: proxy detected vanished client connection
Mar 09 10:01:13 cofgr-pve-001 pveproxy[3551701]: proxy detected vanished client connection
Mar 09 10:01:16 cofgr-pve-001 pveproxy[3551700]: proxy detected vanished client connection
Mar 09 10:01:16 cofgr-pve-001 pveproxy[3551700]: proxy detected vanished client connection
Mar 09 10:01:17 cofgr-pve-001 pveproxy[3551700]: proxy detected vanished client connection
 
Can you post the full syslog after you restart the proxy service?

I would also check the network configuration of the server if everything is ok.
 
Code: 
Mar  9 15:23:49 cofgr-pve-001 pvedaemon[24551]: <root@pam> successful auth for user 'root@pam'
Mar  9 15:24:00 cofgr-pve-001 pvedaemon[24552]: <root@pam> successful auth for user 'root@pam'
Mar  9 15:24:13 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:25:13 cofgr-pve-001 pveproxy[24694]: proxy detected vanished client connection
Mar  9 15:26:13 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:26:16 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:26:16 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:26:17 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:26:23 cofgr-pve-001 pveproxy[24694]: proxy detected vanished client connection
Mar  9 15:26:43 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:26:46 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:26:46 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:26:47 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:26:52 cofgr-pve-001 pveproxy[24694]: proxy detected vanished client connection
Mar  9 15:26:53 cofgr-pve-001 pveproxy[24694]: proxy detected vanished client connection
Mar  9 15:27:13 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:27:16 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:27:16 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:27:17 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:27:23 cofgr-pve-001 pveproxy[24694]: proxy detected vanished client connection
Mar  9 15:27:29 cofgr-pve-001 pveproxy[24694]: proxy detected vanished client connection
Mar  9 15:27:33 cofgr-pve-001 pveproxy[24692]: proxy detected vanished client connection
Mar  9 15:27:39 cofgr-pve-001 pveproxy[24694]: proxy detected vanished client connection
Mar  9 15:27:43 cofgr-pve-001 pveproxy[24694]: proxy detected vanished client connection
Mar  9 15:27:43 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:27:46 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:27:46 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:27:52 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:27:53 cofgr-pve-001 pveproxy[24694]: proxy detected vanished client connection
Mar  9 15:28:13 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:28:16 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:28:16 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:28:22 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:28:33 cofgr-pve-001 pveproxy[24692]: proxy detected vanished client connection
Mar  9 15:28:39 cofgr-pve-001 pveproxy[24692]: proxy detected vanished client connection
Mar  9 15:28:43 cofgr-pve-001 pveproxy[24694]: proxy detected vanished client connection
Mar  9 15:28:43 cofgr-pve-001 pveproxy[24694]: proxy detected vanished client connection
Mar  9 15:28:43 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:28:46 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:28:46 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:28:52 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:28:53 cofgr-pve-001 pveproxy[24692]: proxy detected vanished client connection
Mar  9 15:29:08 cofgr-pve-001 postfix/qmgr[2910497]: DF3662E104E: from=<pve.cofgr@XXX.com>, size=1751, nrcpt=1 (queue active)
Mar  9 15:29:08 cofgr-pve-001 postfix/qmgr[2910497]: 66E1A2E1069: from=<pve.cofgr@XXX.com>, size=1793, nrcpt=1 (queue active)
Mar  9 15:29:13 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:29:13 cofgr-pve-001 pveproxy[24694]: proxy detected vanished client connection
Mar  9 15:29:16 cofgr-pve-001 pveproxy[24692]: proxy detected vanished client connection
Mar  9 15:29:16 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:29:16 cofgr-pve-001 pveproxy[24692]: proxy detected vanished client connection
Mar  9 15:29:17 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:29:23 cofgr-pve-001 pveproxy[24692]: proxy detected vanished client connection
Mar  9 15:29:29 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:29:33 cofgr-pve-001 pveproxy[24692]: proxy detected vanished client connection
Mar  9 15:29:38 cofgr-pve-001 postfix/smtp[35702]: connect to XXX.com[192.168.XXX.XXX]:25: Connection timed out
Mar  9 15:29:38 cofgr-pve-001 postfix/smtp[35703]: connect to XXX.com[192.168.XXX.XXX]:25: Connection timed out
Mar  9 15:29:38 cofgr-pve-001 postfix/smtp[35703]: connect to XXX.com[192.168.XXX.XXX]:25: Connection refused
Mar  9 15:29:43 cofgr-pve-001 pveproxy[24693]: proxy detected vanished client connection
Mar  9 15:29:46 cofgr-pve-001 pveproxy[24692]: proxy detected vanished client connection
Mar  9 15:29:46 cofgr-pve-001 pveproxy[24692]: proxy detected vanished client connection
Mar  9 15:29:50 cofgr-pve-001 systemd[1]: Stopping PVE API Proxy Server...
Mar  9 15:29:51 cofgr-pve-001 pveproxy[24690]: received signal TERM
Mar  9 15:29:51 cofgr-pve-001 pveproxy[24690]: server closing
Mar  9 15:29:51 cofgr-pve-001 pveproxy[24694]: worker exit
Mar  9 15:29:51 cofgr-pve-001 pveproxy[24692]: worker exit
Mar  9 15:29:51 cofgr-pve-001 pveproxy[24690]: worker 24694 finished
Mar  9 15:29:51 cofgr-pve-001 pveproxy[24690]: worker 24692 finished
Mar  9 15:29:51 cofgr-pve-001 pveproxy[24690]: worker 24693 finished
Mar  9 15:29:51 cofgr-pve-001 pveproxy[24690]: server stopped
Mar  9 15:29:52 cofgr-pve-001 systemd[1]: pveproxy.service: Succeeded.
Mar  9 15:29:52 cofgr-pve-001 systemd[1]: Stopped PVE API Proxy Server.
Mar  9 15:29:52 cofgr-pve-001 systemd[1]: pveproxy.service: Consumed 4.957s CPU time.
Mar  9 15:29:52 cofgr-pve-001 systemd[1]: Starting PVE API Proxy Server...
Mar  9 15:29:54 cofgr-pve-001 pveproxy[36978]: starting server
Mar  9 15:29:54 cofgr-pve-001 pveproxy[36978]: starting 3 worker(s)
Mar  9 15:29:54 cofgr-pve-001 pveproxy[36978]: worker 36979 started
Mar  9 15:29:54 cofgr-pve-001 pveproxy[36978]: worker 36980 started
Mar  9 15:29:54 cofgr-pve-001 pveproxy[36978]: worker 36981 started
Mar  9 15:29:54 cofgr-pve-001 systemd[1]: Started PVE API Proxy Server.

I don't change my network configuration in a long time and everything was working
My container is working, I can reach them and ssh

all except one - 103
 
You must log in or register to reply here.
Top Bottom