migration failing

[Lumber4236]

hi,

i'm trying to migrate a vm between 2 hosts in my cluster, and get the following error :

()
2022-09-05 03:18:20 # /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=castor' root@192.168.100.2 /bin/true
2022-09-05 03:18:20 root@192.168.100.2: Permission denied (publickey,password).
2022-09-05 03:18:20 ERROR: migration aborted (duration 00:00:00): Can't connect to destination address using public key
TASK ERROR: migration aborted

any idea what i'm doing wrong ?

 

[gurubert]

Have you modified /root/.ssh/authorized_keys?

 

[aaron]

You can test that, by trying to SSH from the source node into the target node as root user in both cases.

root@<source node> $ ssh root@<target node>

If that fails, try to run

pvecm updatecerts

on the target node.

 

[Lumber4236]

Have you modified /root/.ssh/authorized_keys?

Not as far as i know

You can test that, by trying to SSH from the source node into the target node as root user in both cases.

root@<source node> $ ssh root@<target node>

If that fails, try to run

pvecm updatecerts

on the target node.

Does not seem tio change anything :

root@pollux:~# ssh root@castor
root@castor: Permission denied (publickey).

root@pollux:~# pvecm updatecerts
(re)generate node files
merge authorized SSH keys and known hosts
root@pollux:~# ssh root@castor
root@castor: Permission denied (publickey).

Thank you both for your help

 

[bbgeek17]

What is the IP of pollux?
What is the IP of castor? Does it match when you ping from pollux to castor?
What is the output of "pvecm status" ?
What is the context of /etc/hosts on both nodes
Can you ssh from one to another with: ssh root@<target node>
Can you ssh from one to another with: ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no root@<target node>

Does "/etc/pve/priv/authorized_keys" contain correct keys from both nodes? If it doesnt - fix it manually, keep in mind each key is a single line.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Lumber4236]

What is the IP of pollux?
What is the IP of castor? Does it match when you ping from pollux to castor?

respectively 192.168.100.2 (castor) & 192.168.100.3 (pollux), and both can ping each other, via name and IPs

What is the output of "pvecm status" ?

root@pollux:~# pvecm status
Cluster information
-------------------
Name:             dioscures
Config Version:   2
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Mon Sep  5 12:46:00 2022
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000001
Ring ID:          1.ea
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   2
Highest expected: 2
Total votes:      2
Quorum:           2  
Flags:            Quorate 

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 192.168.100.3 (local)
0x00000002          1 192.168.100.2

What is the context of /etc/hosts on both nodes

127.0.0.1 localhost.localdomain localhost
192.168.100.3 pollux.local.2027a.net pollux

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

and

root@castor:~# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
192.168.100.2 castor.local.2027a.net castor

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

Can you ssh from one to another with: ssh root@<target node>

i can ssh from castor to pollux, but not the other way around

Can you ssh from one to another with: ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no root@<target node>

from castor to pollux, yes, from pollux to castor :

root@pollux:~#  ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no root@castor
root@castor's password: 
Permission denied, please try again.

Does "/etc/pve/priv/authorized_keys" contain correct keys from both nodes? If it doesnt - fix it manually, keep in mind each key is a single line.

look like so, but i'm not sure how to check this ?

root@castor:~# cat /etc/pve/priv/authorized_keys

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDAy/Bqavea3SwgJEnMti2+yEk6hmNuBlhNcxB16kS6aJG7jMxKFFVTPl9nUsheV3rot8a8hcCvdg2EoIr/h3/zNRWAxPrxQyY65l2NCc1sgJgsRzcardJZ+v5NxghU+woNa+JcdDlk6zlCT5o1SiAX2LWbodgmfgSR3Cz+Qm7Se3GgNWrM03xxQJ09ZC99bTGAyiOxZN/0fPBK1Urhiv/lFcaeV1JxloK4616Kv4EQieMiG8m/OP0JAp+TuIbWp9wRk6RQtk+fnb8wjmTsIk0dTBE6tLD+sWxvoitiugkOtfPUx+0mhQk9L65KtPKlPoEoYnSIaklcIgv6AwBocff9 root@pollux
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC951WtatSmmSiv+vBT5XYE6lsRe4iq9nK+4FS/HAniStV6+6AkbcaE2HFP9ckLN1K1pweDe9p4cScAFgG4jl9SvQFEA/df9x97oDp9jm4EuWlr3hfsLt8JhRs1vH9tKQJPDSm8t+o1yBn5eR5kGvYvpkdAzWF/0qTKhQhUTsu44p7XAxpvUSt5ungtFS5n2BrE73HdC+N7g4gfmOtMH4/+8hPQ1r/HhAutT6/5lmsaIlPJXsQoKETap4v7ddNe+2QGB+l8shQnlooKDrPSEx7paXytUWR+K/2ebUNECP4KIOEBOuXElfKI70rgaH6tVVFx+pclX3Ae1ttkIvHCLR77 root@castor

root@pollux:~# cat /etc/pve/priv/authorized_keys

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDAy/Bqavea3SwgJEnMti2+yEk6hmNuBlhNcxB16kS6aJG7jMxKFFVTPl9nUsheV3rot8a8hcCvdg2EoIr/h3/zNRWAxPrxQyY65l2NCc1sgJgsRzcardJZ+v5NxghU+woNa+JcdDlk6zlCT5o1SiAX2LWbodgmfgSR3Cz+Qm7Se3GgNWrM03xxQJ09ZC99bTGAyiOxZN/0fPBK1Urhiv/lFcaeV1JxloK4616Kv4EQieMiG8m/OP0JAp+TuIbWp9wRk6RQtk+fnb8wjmTsIk0dTBE6tLD+sWxvoitiugkOtfPUx+0mhQk9L65KtPKlPoEoYnSIaklcIgv6AwBocff9 root@pollux
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC951WtatSmmSiv+vBT5XYE6lsRe4iq9nK+4FS/HAniStV6+6AkbcaE2HFP9ckLN1K1pweDe9p4cScAFgG4jl9SvQFEA/df9x97oDp9jm4EuWlr3hfsLt8JhRs1vH9tKQJPDSm8t+o1yBn5eR5kGvYvpkdAzWF/0qTKhQhUTsu44p7XAxpvUSt5ungtFS5n2BrE73HdC+N7g4gfmOtMH4/+8hPQ1r/HhAutT6/5lmsaIlPJXsQoKETap4v7ddNe+2QGB+l8shQnlooKDrPSEx7paXytUWR+K/2ebUNECP4KIOEBOuXElfKI70rgaH6tVVFx+pclX3Ae1ttkIvHCLR77 root@castor

 

[bbgeek17]

from castor to pollux, yes, from pollux to castor :

why are you unable to supply root password? Do you not know it? Can you login to castor from console with root/password?

look like so, but i'm not sure how to check this ?

the lines in /etc/pve/priv/authorized_keys should be matching context of "/root/.ssh/id_rsa.pub" on each node

However, giving that you are unable to login with password I suspect you have some other root related issue on your host. Check /etc/passwd and /etc/shadow. Something is broken.

try:
root@pollux:~# ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no root@192.168.100.2

or try from castor (if you are able to login to shell on it) : ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no root@localhost

ie ssh to itself, can you?

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Lumber4236]

why are you unable to supply root password? Do you not know it? Can you login to castor from console with root/password?

i was replying to you line by line when i had an illumination, and checked the sshd_config, where root login was disabled... everything works now. May i disabled password login to just keep key auth ?

 

[bbgeek17]

checked the sshd_config, where root login was disabled... everything works now. May i disabled password login to just keep key auth ?

without knowing which exact option you changed, it seems you disabled root login completely over ssh, not just password authentication.
PVE cluster relies on root account to be loginable via ssh with a key.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Lumber4236]

without knowing which exact option you changed, it seems you disabled root login completely over ssh, not just password authentication.
PVE cluster relies on root account to be loginable via ssh with a key.

yes, that was it. it's re-enabled now. From what you said, i get that i can safely disable password auth ?

 

[bbgeek17]

https://forum.proxmox.com/threads/disable-root-account.83967/

Its not an officially supported setup, afaik. Try it - see how it works out for you.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[ph0x]

You can disable password auth as soon as login via key works, yes.
You can even disable root login and enable it only for one host with a „match“ section in sshd_config.

## Block root login to every one ##
PermitRootLogin no
 
## No more password login ##
PermitEmptyPasswords no
PasswordAuthentication no
 
## Okay allow root login with public ssh key for 192.168.2.5 ##
Match Address 192.168.2.5
        PermitRootLogin yes

 

[Lumber4236]

You can disable password auth as soon as login via key works, yes.

thanks !