Login failed. Please try again

when I first did this:

systemctl stop pve-cluster; pmxcfs -d -f

It never stopped scrolling debug messages. When I went back and re-read your instructions, you seemed to imply it would reach a steady state. Then try the 'find' command to get new messages. Since my earlier attempt never stopped scrolling log messages, I thought I'd go back, try it again and maybe wait longer. When I stopped pve-cluster however, I got an error when trying to restart pmxcfs and doing a simple 'ls' got me this:

ls /etc/pve ls: cannot access '/etc/pve': Transport endpoint is not connected
 
RollMops said:
I had this running for like 2 hrs.

Now I CAN touch /etc/pve/something.

GUI on one browser gives me an empty page, the other lets me log in but gives:

Connection error 500: RPCEnvironment init request failed: Unable to load access control list: Connection refused
Click to expand...

this does sound suspicious..

RollMops said:
I'll have to reboot.
What shall I do w. the 170k lines of logs?
Click to expand...

you can upload the compressed somewhere or send them to me (f.gruenbichler@proxmox.com)
 
  • Like
Reactions: Riesling.Dry
robert.kuropkat said:
when I first did this:

systemctl stop pve-cluster; pmxcfs -d -f

It never stopped scrolling debug messages. When I went back and re-read your instructions, you seemed to imply it would reach a steady state. Then try the 'find' command to get new messages. Since my earlier attempt never stopped scrolling log messages, I thought I'd go back, try it again and maybe wait longer. When I stopped pve-cluster however, I got an error when trying to restart pmxcfs and doing a simple 'ls' got me this:

ls /etc/pve ls: cannot access '/etc/pve': Transport endpoint is not connected
Click to expand...
could you please open a new thread? other than one symptom being similar you definitely have a different issue than the OP here ;)
you likely stopped all instances of pmxcfs on that cluster - restarting the pve-cluster service should help.
 
  • Like
Reactions: Riesling.Dry
thanks! I probably should have made this clearer - when you run pmxcfs in foreground/debug mode, you have to keep it running while trying to reproduce the hang ;) if I read the log correctly, you stopped the debug instance before running find, which means find just saw the empty mountpoint but didn't actually access the filesystem at all.

so the following would be what I am looking for:
Code: 
# in terminal 1

$ systemctl stop pve-cluster; pmxcfs -d -f

# in terminal 2

$ touch /etc/pve/something
$ find /etc/pve -type f -print -exec md5sum {} \;
$ touch /etc/pve/something
$ killall pmxcfs; systemctl start pve-cluster

that should give you less output in terminal 1 since the debug instance gets killed after the test commands in terminal 2 are done and the regular instance is started again.. I only need the pmxcfs output and any errors/warnings printed by the find command, the lines with the actual checksums are not important (and might allow people to bruteforce the files, so I'd not post them here ;))
 
  • Like
Reactions: Riesling.Dry
Sorry Fabian, my bad.

I again have the error (Login failed, authentication failure; rhost=::ffff:10.0.0.123 user=root@pam msg=cfs-lock 'file-priv_tfa_cfg' error: got lock request timeout in ), but it It let me touch /etc/pve/something...

Code: 
root@skynet:~# touch /etc/pve/something
root@skynet:~#  find /etc/pve -type f -print -exec md5sum {} \;
root@skynet:~#  find /etc/pve -type f -print -exec md5sum {} \;
/etc/pve/.debug
b026324c6904b2a9cb4b88d6d61c81d1  /etc/pve/.debug
/etc/pve/.vmlist
8f944b2a5811e73a2a9b3904e4212414  /etc/pve/.vmlist
/etc/pve/.members
9f8abdd26c1024fc58f44f650b342ed2  /etc/pve/.members
/etc/pve/.rrd
4cfc0dee198eb4bf7571f2c394067497  /etc/pve/.rrd
/etc/pve/.version
64ad8e6c95099c39b2e9318a95872403  /etc/pve/.version
/etc/pve/.clusterlog
ec1d95e8040ec031a176abd401dcfa42  /etc/pve/.clusterlog
/etc/pve/storage.cfg
30a93272c2cbf31b6a19159aaa1c62dc  /etc/pve/storage.cfg
/etc/pve/user.cfg
4743c56d9c4eb89e0c8f8939989f21f3  /etc/pve/user.cfg
/etc/pve/something
d41d8cd98f00b204e9800998ecf8427e  /etc/pve/something
/etc/pve/pve-root-ca.pem
02ed1d9391a14473b2ce7fffd0d8e8a8  /etc/pve/pve-root-ca.pem
/etc/pve/priv/known_hosts
b263876244016b9fcd8d958cd478c7f2  /etc/pve/priv/known_hosts
/etc/pve/priv/authorized_keys
b270fabc1549317462ed559b23c3e6bd  /etc/pve/priv/authorized_keys
/etc/pve/priv/pve-root-ca.srl
0ade138937c4b9cb36a28e2edb6485fc  /etc/pve/priv/pve-root-ca.srl
/etc/pve/priv/authkey.key
ec26c4c0511d85631f0da526a629c38f  /etc/pve/priv/authkey.key
/etc/pve/priv/pve-root-ca.key
53b177134d8b8b2d6dbef12656e279f2  /etc/pve/priv/pve-root-ca.key
/etc/pve/replication.cfg
d41d8cd98f00b204e9800998ecf8427e  /etc/pve/replication.cfg
/etc/pve/vzdump.cron
72079a453981656bd89a03b20d347789  /etc/pve/vzdump.cron
/etc/pve/datacenter.cfg
86a73fe56f2219cf6856c43395078bc7  /etc/pve/datacenter.cfg
/etc/pve/authkey.pub
bf6c62a1822a013afb637a57622b847e  /etc/pve/authkey.pub
/etc/pve/authkey.pub.old
4519bf617b9c42a3c8344ce69d035759  /etc/pve/authkey.pub.old
/etc/pve/pve-www.key
0ad7ede237d01fa473f82d4cd7b8c8f9  /etc/pve/pve-www.key
/etc/pve/nodes/skynet/lxc/501.conf
93bfd29ff12f1e7bbcab0ade8817f3fc  /etc/pve/nodes/skynet/lxc/501.conf
/etc/pve/nodes/skynet/lxc/311.conf
290475c11cf07db67c0c46c300ae1cbc  /etc/pve/nodes/skynet/lxc/311.conf
/etc/pve/nodes/skynet/lxc/222.conf~
6b3e134b86bd3ef1ee1899083cefa576  /etc/pve/nodes/skynet/lxc/222.conf~
/etc/pve/nodes/skynet/lxc/600.conf
de4f39d89bc296ba07de3553ef5d4f04  /etc/pve/nodes/skynet/lxc/600.conf
/etc/pve/nodes/skynet/lxc/111.conf
8632d98c08a4fdfec8ce8987dd743f8c  /etc/pve/nodes/skynet/lxc/111.conf
/etc/pve/nodes/skynet/lxc/555.conf
3a40ef8a9f92e71a6a4b329d191ed6b6  /etc/pve/nodes/skynet/lxc/555.conf
/etc/pve/nodes/skynet/lxc/777.conf
2fdba982bd997992198400ac0f79bb78  /etc/pve/nodes/skynet/lxc/777.conf
/etc/pve/nodes/skynet/lxc/888.conf
cdd3b720d0dd960cc74800a361c1fa9e  /etc/pve/nodes/skynet/lxc/888.conf
/etc/pve/nodes/skynet/lxc/1111.conf
462fc37a08d7c3ad5bb5e7dda6868e93  /etc/pve/nodes/skynet/lxc/1111.conf
/etc/pve/nodes/skynet/lxc/1010.conf
4dca82d78960b09c4c2fe32ebcc8c424  /etc/pve/nodes/skynet/lxc/1010.conf
/etc/pve/nodes/skynet/lxc/400.conf
1b1c7127aede756f4f61d0dbb2872000  /etc/pve/nodes/skynet/lxc/400.conf
/etc/pve/nodes/skynet/lxc/444.conf
ca7f85680e63858df929aa540c84739a  /etc/pve/nodes/skynet/lxc/444.conf
/etc/pve/nodes/skynet/lxc/333.conf.tmp.2521576
e45ea39b07f08d9e29222c4ca39a0ddd  /etc/pve/nodes/skynet/lxc/333.conf.tmp.2521576
/etc/pve/nodes/skynet/lxc/222.conf
7c50170c43ff08856f66b44131f259a7  /etc/pve/nodes/skynet/lxc/222.conf
/etc/pve/nodes/skynet/lxc/333.conf
e45ea39b07f08d9e29222c4ca39a0ddd  /etc/pve/nodes/skynet/lxc/333.conf
/etc/pve/nodes/skynet/lrm_status.tmp.2294
addd06215b69cea3ad1adda010d2c303  /etc/pve/nodes/skynet/lrm_status.tmp.2294
/etc/pve/nodes/skynet/pve-ssl.key
c36ef0719348412e2a81e94e1afa4293  /etc/pve/nodes/skynet/pve-ssl.key
/etc/pve/nodes/skynet/lrm_status
f72b9e108b1efd2c8840ac4baa2b8921  /etc/pve/nodes/skynet/lrm_status
/etc/pve/nodes/skynet/pve-ssl.pem
14d5b4af87fea004fba66547aef5a008  /etc/pve/nodes/skynet/pve-ssl.pem
/etc/pve/nodes/skynet/lrm_status.tmp.2570099
1fafb33b243dfcde87d7d0015ba782f7  /etc/pve/nodes/skynet/lrm_status.tmp.2570099
/etc/pve/nodes/skynet/lrm_status.tmp.1609
bd1698422ba43343d56237084ec0112f  /etc/pve/nodes/skynet/lrm_status.tmp.1609
/etc/pve/jobs.cfg
cc5afe4006cdc5e5e1f24ad43985538c  /etc/pve/jobs.cfg
root@skynet:~# touch /etc/pve/something
root@skynet:~#  find /etc/pve -type f -print -exec md5sum {} \;
/etc/pve/.debug
b026324c6904b2a9cb4b88d6d61c81d1  /etc/pve/.debug
/etc/pve/.vmlist
8f944b2a5811e73a2a9b3904e4212414  /etc/pve/.vmlist
/etc/pve/.members
9f8abdd26c1024fc58f44f650b342ed2  /etc/pve/.members
/etc/pve/.rrd
f1e12b40fa3ef77d1d02b8a30f96b938  /etc/pve/.rrd
/etc/pve/.version
3a7695dd054ce16e6ca28c3251144ce3  /etc/pve/.version
/etc/pve/.clusterlog
ec1d95e8040ec031a176abd401dcfa42  /etc/pve/.clusterlog
/etc/pve/storage.cfg
30a93272c2cbf31b6a19159aaa1c62dc  /etc/pve/storage.cfg
/etc/pve/user.cfg
4743c56d9c4eb89e0c8f8939989f21f3  /etc/pve/user.cfg
/etc/pve/something
d41d8cd98f00b204e9800998ecf8427e  /etc/pve/something
/etc/pve/pve-root-ca.pem
02ed1d9391a14473b2ce7fffd0d8e8a8  /etc/pve/pve-root-ca.pem
/etc/pve/priv/known_hosts
b263876244016b9fcd8d958cd478c7f2  /etc/pve/priv/known_hosts
/etc/pve/priv/authorized_keys
b270fabc1549317462ed559b23c3e6bd  /etc/pve/priv/authorized_keys
/etc/pve/priv/pve-root-ca.srl
0ade138937c4b9cb36a28e2edb6485fc  /etc/pve/priv/pve-root-ca.srl
/etc/pve/priv/authkey.key
ec26c4c0511d85631f0da526a629c38f  /etc/pve/priv/authkey.key
/etc/pve/priv/pve-root-ca.key
53b177134d8b8b2d6dbef12656e279f2  /etc/pve/priv/pve-root-ca.key
/etc/pve/replication.cfg
d41d8cd98f00b204e9800998ecf8427e  /etc/pve/replication.cfg
/etc/pve/vzdump.cron
72079a453981656bd89a03b20d347789  /etc/pve/vzdump.cron
/etc/pve/datacenter.cfg
86a73fe56f2219cf6856c43395078bc7  /etc/pve/datacenter.cfg
/etc/pve/authkey.pub
bf6c62a1822a013afb637a57622b847e  /etc/pve/authkey.pub
/etc/pve/authkey.pub.old
4519bf617b9c42a3c8344ce69d035759  /etc/pve/authkey.pub.old
/etc/pve/pve-www.key
0ad7ede237d01fa473f82d4cd7b8c8f9  /etc/pve/pve-www.key
/etc/pve/nodes/skynet/lxc/501.conf
93bfd29ff12f1e7bbcab0ade8817f3fc  /etc/pve/nodes/skynet/lxc/501.conf
/etc/pve/nodes/skynet/lxc/311.conf
290475c11cf07db67c0c46c300ae1cbc  /etc/pve/nodes/skynet/lxc/311.conf
/etc/pve/nodes/skynet/lxc/222.conf~
6b3e134b86bd3ef1ee1899083cefa576  /etc/pve/nodes/skynet/lxc/222.conf~
/etc/pve/nodes/skynet/lxc/600.conf
de4f39d89bc296ba07de3553ef5d4f04  /etc/pve/nodes/skynet/lxc/600.conf
/etc/pve/nodes/skynet/lxc/111.conf
8632d98c08a4fdfec8ce8987dd743f8c  /etc/pve/nodes/skynet/lxc/111.conf
/etc/pve/nodes/skynet/lxc/555.conf
3a40ef8a9f92e71a6a4b329d191ed6b6  /etc/pve/nodes/skynet/lxc/555.conf
/etc/pve/nodes/skynet/lxc/777.conf
2fdba982bd997992198400ac0f79bb78  /etc/pve/nodes/skynet/lxc/777.conf
/etc/pve/nodes/skynet/lxc/888.conf
cdd3b720d0dd960cc74800a361c1fa9e  /etc/pve/nodes/skynet/lxc/888.conf
/etc/pve/nodes/skynet/lxc/1111.conf
462fc37a08d7c3ad5bb5e7dda6868e93  /etc/pve/nodes/skynet/lxc/1111.conf
/etc/pve/nodes/skynet/lxc/1010.conf
4dca82d78960b09c4c2fe32ebcc8c424  /etc/pve/nodes/skynet/lxc/1010.conf
/etc/pve/nodes/skynet/lxc/400.conf
1b1c7127aede756f4f61d0dbb2872000  /etc/pve/nodes/skynet/lxc/400.conf
/etc/pve/nodes/skynet/lxc/444.conf
ca7f85680e63858df929aa540c84739a  /etc/pve/nodes/skynet/lxc/444.conf
/etc/pve/nodes/skynet/lxc/333.conf.tmp.2521576
e45ea39b07f08d9e29222c4ca39a0ddd  /etc/pve/nodes/skynet/lxc/333.conf.tmp.2521576
/etc/pve/nodes/skynet/lxc/222.conf
7c50170c43ff08856f66b44131f259a7  /etc/pve/nodes/skynet/lxc/222.conf
/etc/pve/nodes/skynet/lxc/333.conf
e45ea39b07f08d9e29222c4ca39a0ddd  /etc/pve/nodes/skynet/lxc/333.conf
/etc/pve/nodes/skynet/lrm_status.tmp.2294
addd06215b69cea3ad1adda010d2c303  /etc/pve/nodes/skynet/lrm_status.tmp.2294
/etc/pve/nodes/skynet/pve-ssl.key
c36ef0719348412e2a81e94e1afa4293  /etc/pve/nodes/skynet/pve-ssl.key
/etc/pve/nodes/skynet/lrm_status
34f6ac631527a8e5107212b7e281fd20  /etc/pve/nodes/skynet/lrm_status
/etc/pve/nodes/skynet/pve-ssl.pem
14d5b4af87fea004fba66547aef5a008  /etc/pve/nodes/skynet/pve-ssl.pem
/etc/pve/nodes/skynet/lrm_status.tmp.2570099
1fafb33b243dfcde87d7d0015ba782f7  /etc/pve/nodes/skynet/lrm_status.tmp.2570099
/etc/pve/nodes/skynet/lrm_status.tmp.1609
bd1698422ba43343d56237084ec0112f  /etc/pve/nodes/skynet/lrm_status.tmp.1609
/etc/pve/jobs.cfg
cc5afe4006cdc5e5e1f24ad43985538c  /etc/pve/jobs.cfg
root@skynet:~# killall pmxcfs; systemctl start pve-cluster
root@skynet:~#

After that I could log in via WebGUI, but I reckon this won't last too long...
Cheers,
~R.
 
Last edited:
fabian said:
could you please open a new thread? other than one symptom being similar you definitely have a different issue than the OP here ;)
you likely stopped all instances of pmxcfs on that cluster - restarting the pve-cluster service should help.
Click to expand...

I wondered if I should. New thread started here: New Cluster not working (seems to not be communicating)

Thanks for the help so far. See you over in the other thread...
 
You must log in or register to reply here.
Top Bottom