I have this single node Proxmox and recently it froze out of the blue.
I could not access its web interface or SSH into it. I ended up having to hard reset it.
After the reboot, I went to the logs to try to make some sense of it, and found a bunch of "pveproxy" messages.
Prior to the problem, I actually upgraded the system, but I could not find anything out of order.
I also noticed a gradual increase in CPU load, but could not find the root cause of it.
The following is a snippet of the log messages the day before up to the point where "pveproxy" spits out these records.
pveproxy continued to print messages until I rebooted the system.
Nothing else showed up in the logs.
There is no "corosync.conf" file.
There are no PVE members other than the node itself.
The cluster log:
I'm confused as to what might have triggered this issue since this is not a cluster set up.
What does the
Any thoughts on this issue?
I could not access its web interface or SSH into it. I ended up having to hard reset it.
After the reboot, I went to the logs to try to make some sense of it, and found a bunch of "pveproxy" messages.
Code:
Jan 22 20:21:05 [single_node] pveproxy[319338]: proxy detected vanished client connectionPrior to the problem, I actually upgraded the system, but I could not find anything out of order.
I also noticed a gradual increase in CPU load, but could not find the root cause of it.
The following is a snippet of the log messages the day before up to the point where "pveproxy" spits out these records.
Code:
Jan 22 18:24:10 [single_node] sshd[232877]: Accepted password for root from 192.168.7.3 port 45512 ssh2
Jan 22 18:24:10 [single_node] sshd[232877]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Jan 22 18:24:10 [single_node] systemd-logind[1090]: New session 2609 of user root.
Jan 22 18:24:10 [single_node] systemd[1]: Created slice user-0.slice - User Slice of UID 0.
Jan 22 18:24:10 [single_node] systemd[1]: Starting user-runtime-dir@0.service - User Runtime Directory /run/user/0...
Jan 22 18:24:10 [single_node] systemd[1]: Finished user-runtime-dir@0.service - User Runtime Directory /run/user/0.
Jan 22 18:24:10 [single_node] systemd[1]: Starting user@0.service - User Manager for UID 0...
Jan 22 18:24:10 [single_node] (systemd)[232880]: pam_unix(systemd-user:session): session opened for user root(uid=0) by (uid=0)
Jan 22 18:24:10 [single_node] systemd[232880]: Queued start job for default target default.target.
Jan 22 18:24:10 [single_node] systemd[232880]: Created slice app.slice - User Application Slice.
Jan 22 18:24:10 [single_node] systemd[232880]: Reached target paths.target - Paths.
Jan 22 18:24:10 [single_node] systemd[232880]: Reached target timers.target - Timers.
Jan 22 18:24:10 [single_node] systemd[232880]: Listening on dirmngr.socket - GnuPG network certificate management daemon.
Jan 22 18:24:10 [single_node] systemd[232880]: Listening on gpg-agent-browser.socket - GnuPG cryptographic agent and passphrase cache (access for web browsers).
Jan 22 18:24:10 [single_node] systemd[232880]: Listening on gpg-agent-extra.socket - GnuPG cryptographic agent and passphrase cache (restricted).
Jan 22 18:24:10 [single_node] systemd[232880]: Listening on gpg-agent-ssh.socket - GnuPG cryptographic agent (ssh-agent emulation).
Jan 22 18:24:10 [single_node] systemd[232880]: Listening on gpg-agent.socket - GnuPG cryptographic agent and passphrase cache.
Jan 22 18:24:10 [single_node] systemd[232880]: Reached target sockets.target - Sockets.
Jan 22 18:24:10 [single_node] systemd[232880]: Reached target basic.target - Basic System.
Jan 22 18:24:10 [single_node] systemd[232880]: Reached target default.target - Main User Target.
Jan 22 18:24:10 [single_node] systemd[232880]: Startup finished in 109ms.
Jan 22 18:24:10 [single_node] systemd[1]: Started user@0.service - User Manager for UID 0.
Jan 22 18:24:10 [single_node] systemd[1]: Started session-2609.scope - Session 2609 of User root.
Jan 22 18:24:10 [single_node] sshd[232877]: pam_env(sshd:session): deprecated reading of user environment enabled
Jan 22 18:24:17 [single_node] sudo[232964]: root : TTY=pts/0 ; PWD=/root ; USER=root ; COMMAND=/usr/bin/apt -y update
Jan 22 18:24:17 [single_node] sudo[232964]: pam_unix(sudo:session): session opened for user root(uid=0) by root(uid=0)
Jan 22 18:24:18 [single_node] sudo[232964]: pam_unix(sudo:session): session closed for user root
Jan 22 18:24:18 [single_node] sudo[233733]: root : TTY=pts/0 ; PWD=/root ; USER=root ; COMMAND=/usr/bin/apt -y full-upgrade
Jan 22 18:24:18 [single_node] sudo[233733]: pam_unix(sudo:session): session opened for user root(uid=0) by root(uid=0)
Jan 22 18:24:18 [single_node] sudo[233733]: pam_unix(sudo:session): session closed for user root
Jan 22 18:24:18 [single_node] sudo[233738]: root : TTY=pts/0 ; PWD=/root ; USER=root ; COMMAND=/usr/bin/apt -y autoremove
Jan 22 18:24:18 [single_node] sudo[233738]: pam_unix(sudo:session): session opened for user root(uid=0) by root(uid=0)
Jan 22 18:24:19 [single_node] sudo[233738]: pam_unix(sudo:session): session closed for user root
Jan 22 18:24:19 [single_node] sudo[233743]: root : TTY=pts/0 ; PWD=/root ; USER=root ; COMMAND=/usr/bin/apt -y clean
Jan 22 18:24:19 [single_node] sudo[233743]: pam_unix(sudo:session): session opened for user root(uid=0) by root(uid=0)
Jan 22 18:24:19 [single_node] sudo[233743]: pam_unix(sudo:session): session closed for user root
Jan 22 18:24:19 [single_node] sudo[233746]: root : TTY=pts/0 ; PWD=/root ; USER=root ; COMMAND=/usr/bin/apt -y -f install
Jan 22 18:24:19 [single_node] sudo[233746]: pam_unix(sudo:session): session opened for user root(uid=0) by root(uid=0)
Jan 22 18:24:19 [single_node] sudo[233746]: pam_unix(sudo:session): session closed for user root
Jan 22 18:24:27 [single_node] sshd[232877]: pam_unix(sshd:session): session closed for user root
Jan 22 18:24:27 [single_node] systemd[1]: session-2609.scope: Deactivated successfully.
Jan 22 18:24:27 [single_node] systemd[1]: session-2609.scope: Consumed 1.892s CPU time.
Jan 22 18:24:27 [single_node] systemd-logind[1090]: Session 2609 logged out. Waiting for processes to exit.
Jan 22 18:24:27 [single_node] systemd-logind[1090]: Removed session 2609.
Jan 22 18:24:37 [single_node] systemd[1]: Stopping user@0.service - User Manager for UID 0...
Jan 22 18:24:37 [single_node] systemd[232880]: Activating special unit exit.target...
Jan 22 18:24:37 [single_node] systemd[232880]: Stopped target default.target - Main User Target.
Jan 22 18:24:37 [single_node] systemd[232880]: Stopped target basic.target - Basic System.
Jan 22 18:24:37 [single_node] systemd[232880]: Stopped target paths.target - Paths.
Jan 22 18:24:37 [single_node] systemd[232880]: Stopped target sockets.target - Sockets.
Jan 22 18:24:37 [single_node] systemd[232880]: Stopped target timers.target - Timers.
Jan 22 18:24:37 [single_node] systemd[232880]: Closed dirmngr.socket - GnuPG network certificate management daemon.
Jan 22 18:24:37 [single_node] systemd[232880]: Closed gpg-agent-browser.socket - GnuPG cryptographic agent and passphrase cache (access for web browsers).
Jan 22 18:24:37 [single_node] systemd[232880]: Closed gpg-agent-extra.socket - GnuPG cryptographic agent and passphrase cache (restricted).
Jan 22 18:24:37 [single_node] systemd[232880]: Closed gpg-agent-ssh.socket - GnuPG cryptographic agent (ssh-agent emulation).
Jan 22 18:24:37 [single_node] systemd[232880]: Closed gpg-agent.socket - GnuPG cryptographic agent and passphrase cache.
Jan 22 18:24:37 [single_node] systemd[232880]: Removed slice app.slice - User Application Slice.
Jan 22 18:24:37 [single_node] systemd[232880]: Reached target shutdown.target - Shutdown.
Jan 22 18:24:37 [single_node] systemd[232880]: Finished systemd-exit.service - Exit the Session.
Jan 22 18:24:37 [single_node] systemd[232880]: Reached target exit.target - Exit the Session.
Jan 22 18:24:37 [single_node] systemd[1]: user@0.service: Deactivated successfully.
Jan 22 18:24:37 [single_node] systemd[1]: Stopped user@0.service - User Manager for UID 0.
Jan 22 18:24:37 [single_node] systemd[1]: Stopping user-runtime-dir@0.service - User Runtime Directory /run/user/0...
Jan 22 18:24:38 [single_node] systemd[1]: run-user-0.mount: Deactivated successfully.
Jan 22 18:24:38 [single_node] systemd[1]: user-runtime-dir@0.service: Deactivated successfully.
Jan 22 18:24:38 [single_node] systemd[1]: Stopped user-runtime-dir@0.service - User Runtime Directory /run/user/0.
Jan 22 18:24:38 [single_node] systemd[1]: Removed slice user-0.slice - User Slice of UID 0.
Jan 22 18:24:38 [single_node] systemd[1]: user-0.slice: Consumed 2.004s CPU time.
Jan 22 18:32:05 [single_node] postfix/qmgr[3544956]: 35465180B0D: from=<root@[single_node].[domain]>, size=3873, nrcpt=1 (queue active)
Jan 22 18:32:35 [single_node] postfix/smtp[240149]: connect to [domain][[domain_ip]]:25: Connection timed out
Jan 22 18:33:05 [single_node] postfix/smtp[240149]: connect to [domain][[domain_ip]]:25: Connection timed out
Jan 22 18:33:35 [single_node] postfix/smtp[240149]: connect to [domain][[domain_ip]]:25: Connection timed out
Jan 22 18:34:05 [single_node] postfix/smtp[240149]: connect to [domain][[domain_ip]]:25: Connection timed out
Jan 22 18:34:05 [single_node] postfix/smtp[240149]: 35465180B0D: to=<[user]@[domain]>, relay=none, delay=414350, delays=414230/0.01/120/0, dsn=4.4.1, status=deferred (connect to [domain][[domain_ip]]:25: Connection timed out)
Jan 22 18:41:34 [single_node] pvedaemon[156628]: worker exit
Jan 22 18:41:34 [single_node] pvedaemon[1482]: worker 156628 finished
Jan 22 18:41:34 [single_node] pvedaemon[1482]: starting 1 worker(s)
Jan 22 18:41:34 [single_node] pvedaemon[1482]: worker 247906 started
Jan 22 19:04:56 [single_node] smartd[1083]: Device: /dev/sda [SAT], is in STANDBY mode, suspending checks
Jan 22 19:17:01 [single_node] CRON[277056]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 22 19:17:01 [single_node] CRON[277057]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jan 22 19:17:01 [single_node] CRON[277056]: pam_unix(cron:session): session closed for user root
Jan 22 19:20:36 [single_node] syncthing[3497355]: [SSF3N] INFO: Lost primary connection to KB4ZVTN at 192.168.8.9:22000-192.168.7.3:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256/WAN-P30-60T34M4DNF5B2A0ENE7ND1R7QK: reading length: read tcp 192.168.8.9:22000->192.168.7.3:22000: read: connection reset by peer (0 remain)
Jan 22 19:20:36 [single_node] syncthing[3497355]: [SSF3N] INFO: Connection to KB4ZVTN at 192.168.8.9:22000-192.168.7.3:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256/WAN-P30-60T34M4DNF5B2A0ENE7ND1R7QK closed: reading length: read tcp 192.168.8.9:22000->192.168.7.3:22000: read: connection reset by peer
Jan 22 19:42:06 [single_node] postfix/qmgr[3544956]: 35465180B0D: from=<root@[single_node].[domain]>, size=3873, nrcpt=1 (queue active)
Jan 22 19:42:36 [single_node] postfix/smtp[297603]: connect to [domain][[domain_ip]]:25: Connection timed out
Jan 22 19:43:06 [single_node] postfix/smtp[297603]: connect to [domain][[domain_ip]]:25: Connection timed out
Jan 22 19:43:36 [single_node] postfix/smtp[297603]: connect to [domain][[domain_ip]]:25: Connection timed out
Jan 22 19:44:06 [single_node] postfix/smtp[297603]: connect to [domain][[domain_ip]]:25: Connection timed out
Jan 22 19:44:06 [single_node] postfix/smtp[297603]: 35465180B0D: to=<[user]@[domain]>, relay=none, delay=418551, delays=418431/0.01/120/0, dsn=4.4.1, status=deferred (connect to [domain][[domain_ip]]:25: Connection timed out)
Jan 22 19:50:07 [single_node] pveproxy[216029]: worker exit
Jan 22 19:50:07 [single_node] pveproxy[1491]: worker 216029 finished
Jan 22 19:50:07 [single_node] pveproxy[1491]: starting 1 worker(s)
Jan 22 19:50:07 [single_node] pveproxy[1491]: worker 304133 started
Jan 22 20:06:57 [single_node] pveproxy[226590]: worker exit
Jan 22 20:06:57 [single_node] pveproxy[1491]: worker 226590 finished
Jan 22 20:06:57 [single_node] pveproxy[1491]: starting 1 worker(s)
Jan 22 20:06:57 [single_node] pveproxy[1491]: worker 317929 started
Jan 22 20:08:36 [single_node] pveproxy[229097]: worker exit
Jan 22 20:08:36 [single_node] pveproxy[1491]: worker 229097 finished
Jan 22 20:08:36 [single_node] pveproxy[1491]: starting 1 worker(s)
Jan 22 20:08:36 [single_node] pveproxy[1491]: worker 319338 started
Jan 22 20:10:26 [single_node] pvedaemon[228242]: worker exit
Jan 22 20:10:26 [single_node] pvedaemon[1482]: worker 228242 finished
Jan 22 20:10:26 [single_node] pvedaemon[1482]: starting 1 worker(s)
Jan 22 20:10:26 [single_node] pvedaemon[1482]: worker 320824 started
Jan 22 20:17:01 [single_node] CRON[326199]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 22 20:17:01 [single_node] CRON[326200]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jan 22 20:17:01 [single_node] CRON[326199]: pam_unix(cron:session): session closed for user root
Jan 22 20:21:05 [single_node] pveproxy[319338]: proxy detected vanished client connection
Jan 22 20:21:16 [single_node] pveproxy[319338]: proxy detected vanished client connection
Jan 22 20:21:36 [single_node] pveproxy[317929]: proxy detected vanished client connection
(...)pveproxy continued to print messages until I rebooted the system.
Nothing else showed up in the logs.
There is no "corosync.conf" file.
There are no PVE members other than the node itself.
Code:
# cat /etc/pve/.members
{
"nodename": "[single_node]",
"version": 0
}The cluster log:
Code:
# cat /etc/pve/.clusterlog
{
"data": [
{"uid": 14, "time": 1737673771, "pri": 6, "tag": "pvedaemon", "pid": 1506, "node": "[single_node]", "user": "root@pam", "msg": "successful auth for user '[user]@pve'"},
{"uid": 13, "time": 1737672871, "pri": 6, "tag": "pvedaemon", "pid": 1506, "node": "[single_node]", "user": "root@pam", "msg": "successful auth for user '[user]@pve'"},
{"uid": 12, "time": 1737671970, "pri": 6, "tag": "pvedaemon", "pid": 1504, "node": "[single_node]", "user": "root@pam", "msg": "successful auth for user '[user]@pve'"},
{"uid": 11, "time": 1737671069, "pri": 6, "tag": "pvedaemon", "pid": 1504, "node": "[single_node]", "user": "root@pam", "msg": "successful auth for user '[user]@pve'"},
{"uid": 10, "time": 1737670169, "pri": 6, "tag": "pvedaemon", "pid": 1504, "node": "[single_node]", "user": "root@pam", "msg": "successful auth for user '[user]@pve'"},
{"uid": 9, "time": 1737669401, "pri": 6, "tag": "pve-guests", "pid": 1529, "node": "[single_node]", "user": "root@pam", "msg": "end task UPID:[single_node]:000005FB:00000409:6792BA65:startall::root@pam: OK"},
{"uid": 8, "time": 1737669356, "pri": 6, "tag": "pve-guests", "pid": 1531, "node": "[single_node]", "user": "root@pam", "msg": "starting task UPID:[single_node]:00000F82:000038CF:6792BAEC:vzstart:910:root@pam:"},
{"uid": 7, "time": 1737669311, "pri": 6, "tag": "pve-guests", "pid": 1531, "node": "[single_node]", "user": "root@pam", "msg": "starting task UPID:[single_node]:00000CEC:00002739:6792BABF:vzstart:908:root@pam:"},
{"uid": 6, "time": 1737669269, "pri": 6, "tag": "pvedaemon", "pid": 1504, "node": "[single_node]", "user": "root@pam", "msg": "successful auth for user '[user]@pve'"},
{"uid": 5, "time": 1737669266, "pri": 6, "tag": "pvedaemon", "pid": 1506, "node": "[single_node]", "user": "root@pam", "msg": "successful auth for user '[user]@pve'"},
{"uid": 4, "time": 1737669266, "pri": 6, "tag": "pve-guests", "pid": 1531, "node": "[single_node]", "user": "root@pam", "msg": "starting task UPID:[single_node]:000008F5:000015A2:6792BA92:vzstart:906:root@pam:"},
{"uid": 3, "time": 1737669221, "pri": 6, "tag": "pve-guests", "pid": 1531, "node": "[single_node]", "user": "root@pam", "msg": "starting task UPID:[single_node]:000005FC:0000040B:6792BA65:vzstart:904:root@pam:"},
{"uid": 2, "time": 1737669221, "pri": 6, "tag": "pve-guests", "pid": 1529, "node": "[single_node]", "user": "root@pam", "msg": "starting task UPID:[single_node]:000005FB:00000409:6792BA65:startall::root@pam:"},
{"uid": 1, "time": 1737669217, "pri": 6, "tag": "cluster", "pid": 1347, "node": "[single_node]", "user": "root", "msg": "starting cluster log"}
]
}I'm confused as to what might have triggered this issue since this is not a cluster set up.
What does the
proxy detected vanished client connection message means?Any thoughts on this issue?