[SOLVED] Unable to authenticate, SSH works fine

M

maddy_in65

New Member
Mar 27, 2024
8
1
3
I am running proxmox cluster having 3 nodes and unable to login to GUI since yesterday, however SSH is working fine on all nodes and few VMs.

Code: 
root@PvE01Ser05:~# pvecm nodes

Membership information
----------------------
    Nodeid      Votes Name
         1          1 PvE01Ser05 (local)
         2          1 PvE02Ser06
         3          1 PvE03Ser13

-------------------
Name:             MNeTPvECL01
Config Version:   3
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Mon Mar 25 20:14:41 2024
Quorum provider:  corosync_votequorum
Nodes:            3
Node ID:          0x00000001
Ring ID:          1.1075
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2  
Flags:            Quorate 

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 192.168.5.15 (local)
0x00000002          1 192.168.5.25
0x00000003          1 192.168.5.35
root@PvE01Ser05:~#
1711562474773.png

I did troubleshooting by searching over internet but still unable to access it. Now some of the VMs were inaccessible which are on main node. Please can someone guide to resolve this issue.
Code: 
root@PvE01Ser05:~# wget --no-check-certificate https://localhost:8006
--2024-03-25 16:07:53--  https://localhost:8006/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:8006... connected.
WARNING: The certificate of ‘localhost’ is not trusted.
WARNING: The certificate of ‘localhost’ doesn't have a known issuer.
HTTP request sent, awaiting response... 200 OK
Length: 2494 (2.4K) [text/html]
Saving to: ‘index.html.1’

index.html.1                                                 100%[==============================================================================================================================================>]   2.44K  --.-KB/s    in 0s     

2024-03-25 16:07:53 (121 MB/s) - ‘index.html.1’ saved [2494/2494]

Tried restarting pvedaemon and pveproxy

Code: 
 systemctl restart pveproxy.service pvedaemon.service
Failed to restart pveproxy.service: Transaction for pveproxy.service/restart is destructive (dev-disk-by\x2did-dm\x2dname\x2dpve\x2dswap.swap has 'stop' job queued, but 'start' is included in transaction).
See system logs and 'systemctl status pveproxy.service' for details.
Failed to restart pvedaemon.service: Transaction for pvedaemon.service/restart is destructive (systemd-binfmt.service has 'stop' job queued, but 'start' is included in transaction).
See system logs and 'systemctl status pvedaemon.service' for details.

other troubleshooting steps:
Code: 
root@PvE01Ser05:~# pvedaemon.service


e - PVE API Daemon
     Loaded: loaded (/lib/systemd/system/pvedaemon.service; enabled; preset: enabled)
     Active: active (running) since Mon 2024-03-25 11:42:04 IST; 48min ago
    Process: 834 ExecStart=/usr/bin/pvedaemon start (code=exited, status=0/SUCCESS)
   Main PID: 863 (pvedaemon)
      Tasks: 4 (limit: 18969)
     Memory: 152.3M
        CPU: 1.260s
     CGroup: /system.slice/pvedaemon.service
             ├─863 pvedaemon
             ├─864 "pvedaemon worker"
             ├─865 "pvedaemon worker"
             └─866 "pvedaemon worker"

Mar 25 11:42:03 PvE01Ser05 systemd[1]: Starting pvedaemon.service - PVE API Daemon...
Mar 25 11:42:04 PvE01Ser05 pvedaemon[863]: starting server
Mar 25 11:42:04 PvE01Ser05 pvedaemon[863]: starting 3 worker(s)
Mar 25 11:42:04 PvE01Ser05 pvedaemon[863]: worker 864 started
Mar 25 11:42:04 PvE01Ser05 pvedaemon[863]: worker 865 started
Mar 25 11:42:04 PvE01Ser05 pvedaemon[863]: worker 866 started
Mar 25 11:42:04 PvE01Ser05 systemd[1]: Started pvedaemon.service - PVE API Daemon.


systemctl status pvestatd.service
○ pvestatd.service - PVE Status Daemon
     Loaded: loaded (/lib/systemd/system/pvestatd.service; enabled; preset: enabled)
     Active: inactive (dead) since Mon 2024-03-25 12:13:08 IST; 20min ago
   Duration: 31min 2.738s
    Process: 820 ExecStart=/usr/bin/pvestatd start (code=exited, status=0/SUCCESS)
    Process: 4323 ExecStop=/usr/bin/pvestatd stop (code=exited, status=0/SUCCESS)
   Main PID: 835 (code=exited, status=0/SUCCESS)
        CPU: 1.571s

Mar 25 11:42:02 PvE01Ser05 systemd[1]: Starting pvestatd.service - PVE Status Daemon...
Mar 25 11:42:03 PvE01Ser05 pvestatd[835]: starting server
Mar 25 11:42:03 PvE01Ser05 systemd[1]: Started pvestatd.service - PVE Status Daemon.
Mar 25 12:13:06 PvE01Ser05 systemd[1]: Stopping pvestatd.service - PVE Status Daemon...
Mar 25 12:13:07 PvE01Ser05 pvestatd[835]: received signal TERM
Mar 25 12:13:07 PvE01Ser05 pvestatd[835]: server closing
Mar 25 12:13:07 PvE01Ser05 pvestatd[835]: server stopped
Mar 25 12:13:08 PvE01Ser05 systemd[1]: pvestatd.service: Deactivated successfully.
Mar 25 12:13:08 PvE01Ser05 systemd[1]: Stopped pvestatd.service - PVE Status Daemon.
Mar 25 12:13:08 PvE01Ser05 systemd[1]: pvestatd.service: Consumed 1.571s CPU time.


 systemctl status pveproxy
× pveproxy.service - PVE API Proxy Server
     Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; preset: enabled)
     Active: failed (Result: timeout) since Mon 2024-03-25 12:12:10 IST; 24min ago
    Process: 3523 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=killed, signal=KILL)
        CPU: 318ms

Mar 25 12:10:40 PvE01Ser05 systemd[1]: pveproxy.service: Killing process 2877 (pvecm) with signal SIGKILL.
Mar 25 12:10:40 PvE01Ser05 systemd[1]: pveproxy.service: Killing process 3524 (pvecm) with signal SIGKILL.
Mar 25 12:12:10 PvE01Ser05 systemd[1]: pveproxy.service: Processes still around after final SIGKILL. Entering failed mode.
Mar 25 12:12:10 PvE01Ser05 systemd[1]: pveproxy.service: Failed with result 'timeout'.
Mar 25 12:12:10 PvE01Ser05 systemd[1]: pveproxy.service: Unit process 869 (pvecm) remains running after unit stopped.
Mar 25 12:12:10 PvE01Ser05 systemd[1]: pveproxy.service: Unit process 1576 (pvecm) remains running after unit stopped.
Mar 25 12:12:10 PvE01Ser05 systemd[1]: pveproxy.service: Unit process 2225 (pvecm) remains running after unit stopped.
Mar 25 12:12:10 PvE01Ser05 systemd[1]: pveproxy.service: Unit process 2877 (pvecm) remains running after unit stopped.
Mar 25 12:12:10 PvE01Ser05 systemd[1]: pveproxy.service: Unit process 3524 (pvecm) remains running after unit stopped.
Mar 25 12:12:10 PvE01Ser05 systemd[1]: Stopped pveproxy.service - PVE API Proxy Server.
 
Don't know exactly what's up, or how you've setup the various users/passwords on the nodes. But maybe try getting in with PVE Authentication Server which is on the GUI login screen (third line down dropdown).
Something else to consider; maybe you've got some keyboard/setting/foreign language that's actually not entering your correct password in the GUI.
 
  • Like
Reactions: maddy_in65
gfngfn256 said:
Don't know exactly what's up, or how you've setup the various users/passwords on the nodes. But maybe try getting in with PVE Authentication Server which is on the GUI login screen (third line down dropdown).
Something else to consider; maybe you've got some keyboard/setting/foreign language that's actually not entering your correct password in the GUI.
Click to expand...
I have not setup auth server, its Linux PAM auth was always selected. the password was saved in browser and was not maunally entered.
I also tested with different browser.
 
I have just tried to login and now got new error message, I am still able to login via SSH. I have also to login via other node but it was not working as well.

1711564158713.png
 
Assuming the data/logs you have included in your post are recent (your problem started yesterday) - why are you showing a date of Mar 25. This indicates that your time/date on your node/s are not correct. This could be the cause of your problem/s.
 
gfngfn256 said:
Assuming the data/logs you have included in your post are recent (your problem started yesterday) - why are you showing a date of Mar 25. This indicates that your time/date on your node/s are not correct. This could be the cause of your problem/s.
Click to expand...
This issue started few days back and some logs were taken at that time. Checked date and time and it is matching as per my region.
Current date and time
Code: 
root@PvE01Ser05:~# date
Thu Mar 28 09:27:04 AM IST 2024
root@PvE01Ser05:~#
 
i think I am close to solve the issue, these services seems working fine but pevedaemon shows auth failure but the password is correct, I have tested multiple browsers. Now When I click on Login on the browser, nothing happens.

Code: 
systemctl status pvedaemon pve-cluster pveproxy
● pvedaemon.service - PVE API Daemon
     Loaded: loaded (/lib/systemd/system/pvedaemon.service; enabled; preset: enabled)
     Active: active (running) since Thu 2024-03-28 09:29:10 IST; 17min ago
    Process: 837 ExecStart=/usr/bin/pvedaemon start (code=exited, status=0/SUCCESS)
   Main PID: 866 (pvedaemon)
      Tasks: 4 (limit: 18969)
     Memory: 169.3M
        CPU: 1.297s
     CGroup: /system.slice/pvedaemon.service
             ├─866 pvedaemon
             ├─867 "pvedaemon worker"
             ├─868 "pvedaemon worker"
             └─869 "pvedaemon worker"

Mar 28 09:29:10 PvE01Ser05 pvedaemon[866]: starting 3 worker(s)
Mar 28 09:29:10 PvE01Ser05 pvedaemon[866]: worker 867 started
Mar 28 09:29:10 PvE01Ser05 pvedaemon[866]: worker 868 started
Mar 28 09:29:10 PvE01Ser05 pvedaemon[866]: worker 869 started
Mar 28 09:29:10 PvE01Ser05 systemd[1]: Started pvedaemon.service - PVE API Daemon.
Mar 28 09:31:26 PvE01Ser05 pvedaemon[867]: <root@pam> successful auth for user 'root@pam'
Mar 28 09:32:03 PvE01Ser05 pvedaemon[869]: <root@pam> successful auth for user 'root@pam'
Mar 28 09:44:49 PvE01Ser05 pvedaemon[867]: <root@pam> successful auth for user 'root@pam'
Mar 28 09:44:57 PvE01Ser05 pvedaemon[869]: authentication failure; rhost=::ffff:192.168.6.6 user=root@pve msg=no such user ('root@pve')
Mar 28 09:45:04 PvE01Ser05 pvedaemon[868]: <root@pam> successful auth for user 'root@pam'

● pve-cluster.service - The Proxmox VE cluster filesystem
     Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; preset: enabled)
     Active: active (running) since Thu 2024-03-28 09:29:09 IST; 17min ago
    Process: 695 ExecStart=/usr/bin/pmxcfs (code=exited, status=0/SUCCESS)
   Main PID: 716 (pmxcfs)
      Tasks: 6 (limit: 18969)
     Memory: 62.7M
        CPU: 935ms
     CGroup: /system.slice/pve-cluster.service
             └─716 /usr/bin/pmxcfs

Mar 28 09:31:19 PvE01Ser05 pmxcfs[716]: [dcdb] notice: received all states
Mar 28 09:31:19 PvE01Ser05 pmxcfs[716]: [dcdb] notice: leader is 1/716
Mar 28 09:31:19 PvE01Ser05 pmxcfs[716]: [dcdb] notice: synced members: 1/716
Mar 28 09:31:19 PvE01Ser05 pmxcfs[716]: [dcdb] notice: start sending inode updates
Mar 28 09:31:19 PvE01Ser05 pmxcfs[716]: [dcdb] notice: sent all (6) updates
Mar 28 09:31:19 PvE01Ser05 pmxcfs[716]: [dcdb] notice: all data is up to date
Mar 28 09:31:19 PvE01Ser05 pmxcfs[716]: [status] notice: received all states
Mar 28 09:31:19 PvE01Ser05 pmxcfs[716]: [status] notice: all data is up to date
Mar 28 09:31:35 PvE01Ser05 pmxcfs[716]: [status] notice: received log
Mar 28 09:31:35 PvE01Ser05 pmxcfs[716]: [status] notice: received log

● pveproxy.service - PVE API Proxy Server
     Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; preset: enabled)
     Active: active (running) since Thu 2024-03-28 09:29:41 IST; 16min ago
    Process: 871 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)
    Process: 951 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
   Main PID: 952 (pveproxy)
      Tasks: 4 (limit: 18969)
     Memory: 194.7M
        CPU: 2.094s
     CGroup: /system.slice/pveproxy.service
             ├─952 pveproxy
             ├─953 "pveproxy worker"
             ├─954 "pveproxy worker"
             └─955 "pveproxy worker"

Mar 28 09:29:10 PvE01Ser05 systemd[1]: Starting pveproxy.service - PVE API Proxy Server...
Mar 28 09:29:40 PvE01Ser05 pvecm[871]: got timeout when trying to ensure cluster certificates and base file hierarchy is set up - no quorum (yet) or hung pmxcfs?
Mar 28 09:29:41 PvE01Ser05 pveproxy[952]: starting server
Mar 28 09:29:41 PvE01Ser05 pveproxy[952]: starting 3 worker(s)
Mar 28 09:29:41 PvE01Ser05 pveproxy[952]: worker 953 started
Mar 28 09:29:41 PvE01Ser05 pveproxy[952]: worker 954 started
Mar 28 09:29:41 PvE01Ser05 pveproxy[952]: worker 955 started
Mar 28 09:29:41 PvE01Ser05 systemd[1]: Started pveproxy.service - PVE API Proxy Server.

[1]+  Stopped                 systemctl status pvedaemon pve-cluster pveproxy
Code: 
root@PvE01Ser05:/var/log# tail syslog
2024-03-28T09:31:35.170414+05:30 PvE01Ser05 pmxcfs[716]: [status] notice: received log
2024-03-28T09:31:35.193153+05:30 PvE01Ser05 pmxcfs[716]: [status] notice: received log
2024-03-28T09:32:03.043925+05:30 PvE01Ser05 pvedaemon[869]: <root@pam> successful auth for user 'root@pam'
2024-03-28T09:44:29.473865+05:30 PvE01Ser05 systemd[1]: Starting systemd-tmpfiles-clean.service - Cleanup of Temporary Directories...
2024-03-28T09:44:29.496534+05:30 PvE01Ser05 systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
2024-03-28T09:44:29.496758+05:30 PvE01Ser05 systemd[1]: Finished systemd-tmpfiles-clean.service - Cleanup of Temporary Directories.
2024-03-28T09:44:29.497851+05:30 PvE01Ser05 systemd[1]: run-credentials-systemd\x2dtmpfiles\x2dclean.service.mount: Deactivated successfully.
2024-03-28T09:44:49.247321+05:30 PvE01Ser05 pvedaemon[867]: <root@pam> successful auth for user 'root@pam'
2024-03-28T09:44:57.760231+05:30 PvE01Ser05 pvedaemon[869]: authentication failure; rhost=::ffff:192.168.6.6 user=root@pve msg=no such user ('root@pve')
2024-03-28T09:45:04.700394+05:30 PvE01Ser05 pvedaemon[868]: <root@pam> successful auth for user 'root@pam'
Code: 
root@PvE01Ser05:/var/log# tail -20 auth.log
2024-03-28T09:26:43.349888+05:30 PvE01Ser05 (systemd): pam_unix(systemd-user:session): session opened for user root(uid=0) by (uid=0)
2024-03-28T09:26:43.474261+05:30 PvE01Ser05 sshd[457424]: pam_env(sshd:session): deprecated reading of user environment enabled
2024-03-28T09:28:21.515472+05:30 PvE01Ser05 systemd-logind[489]: The system will reboot now!
2024-03-28T09:28:21.524918+05:30 PvE01Ser05 systemd-logind[489]: System is rebooting.
2024-03-28T09:28:21.533718+05:30 PvE01Ser05 sshd[457424]: Exiting on signal 15
2024-03-28T09:28:21.533810+05:30 PvE01Ser05 sshd[457424]: pam_unix(sshd:session): session closed for user root
2024-03-28T09:28:21.581029+05:30 PvE01Ser05 systemd-logind[489]: Session 89 logged out. Waiting for processes to exit.
2024-03-28T09:28:21.586892+05:30 PvE01Ser05 systemd-logind[489]: Removed session 89.
2024-03-28T09:28:21.604638+05:30 PvE01Ser05 (sd-pam): pam_unix(systemd-user:session): session closed for user root
2024-03-28T09:29:06.544172+05:30 PvE01Ser05 systemd-logind[491]: New seat seat0.
2024-03-28T09:29:06.549707+05:30 PvE01Ser05 systemd-logind[491]: Watching system buttons on /dev/input/event2 (Power Button)
2024-03-28T09:29:06.549790+05:30 PvE01Ser05 systemd-logind[491]: Watching system buttons on /dev/input/event1 (Power Button)
2024-03-28T09:29:06.549826+05:30 PvE01Ser05 systemd-logind[491]: Watching system buttons on /dev/input/event0 (Sleep Button)
2024-03-28T09:29:08.007418+05:30 PvE01Ser05 sshd[626]: Server listening on 0.0.0.0 port .
2024-03-28T09:29:08.007497+05:30 PvE01Ser05 sshd[626]: Server listening on :: port .
2024-03-28T09:31:10.517215+05:30 PvE01Ser05 sshd[1145]: Accepted password for root from 192.168.12.6 port 56684 ssh2
2024-03-28T09:31:10.518290+05:30 PvE01Ser05 sshd[1145]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
2024-03-28T09:31:10.559927+05:30 PvE01Ser05 systemd-logind[491]: New session 1 of user root.
2024-03-28T09:31:10.568561+05:30 PvE01Ser05 (systemd): pam_unix(systemd-user:session): session opened for user root(uid=0) by (uid=0)
2024-03-28T09:31:10.688884+05:30 PvE01Ser05 sshd[1145]: pam_env(sshd:session): deprecated reading of user environment enabled
 
Syslog messages, its seems aliases db is not available. is this the root cause of this issue, please advise on how to solve this issue.
Code: 
tail syslog
2024-03-28T11:12:16.441882+05:30 PvE01Ser05 pvecm[9345]: got inotify poll request in wrong process - disabling inotify
2024-03-28T11:14:29.756139+05:30 PvE01Ser05 pvecm[9624]: got inotify poll request in wrong process - disabling inotify
2024-03-28T11:15:40.190641+05:30 PvE01Ser05 postfix/qmgr[811]: B025D1A0312: from=<>, size=2900, nrcpt=1 (queue active)
2024-03-28T11:15:40.199357+05:30 PvE01Ser05 postfix/local[9778]: error: open database /etc/aliases.db: No such file or directory
2024-03-28T11:15:40.199483+05:30 PvE01Ser05 postfix/local[9778]: warning: hash:/etc/aliases is unavailable. open database /etc/aliases.db: No such file or directory
2024-03-28T11:15:40.199519+05:30 PvE01Ser05 postfix/local[9778]: warning: hash:/etc/aliases: lookup of 'root' failed
2024-03-28T11:15:40.213500+05:30 PvE01Ser05 postfix/local[9778]: B025D1A0312: to=<root@PvE01Ser05.mnet>, relay=local, delay=384056, delays=384056/0.01/0/0.01, dsn=4.3.0, status=deferred (alias database unavailable)
2024-03-28T11:15:47.514146+05:30 PvE01Ser05 pvedaemon[867]: <root@pam> successful auth for user 'root@pam'
2024-03-28T11:15:52.391774+05:30 PvE01Ser05 pvedaemon[868]: <root@pam> successful auth for user 'root@pam'
2024-03-28T11:17:01.932825+05:30 PvE01Ser05 CRON[9963]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
 
maddy_in65 said:
Syslog messages, its seems aliases db is not available. is this the root cause of this issue, please advise on how to solve this issue.
Click to expand...
The aliases DB looks like a postfix problem.

What does this show:
Code: 
cat /etc/aliases
If it exists & contains content, you probably need to rebuild the /etc/aliases.db by entering:
Code: 
newaliases

Would be surprising if this has anything todo with your GUI login.

What does this show:
Code: 
cat /etc/pve/user.cfg
 
Last edited:
  • Like
Reactions: maddy_in65
gfngfn256 said:
The aliases DB looks like a postfix problem.

What does this show:
Code: 
cat /etc/aliases
If it exists & contains content, you probably need to rebuild the /etc/aliases.db by entering:
Code: 
newaliases

Would be surprising if this has anything todo with your GUI login.

What does this show:
Code: 
cat /etc/pve/user.cfg
Click to expand...

cat /etc/aliases
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back