Possible bug with cloud-init SSH public key GUI field - VE version 8.2.2

[Cannondale]

I have created a VM template from ubuntu-22.04-minimal-cloudimg-amd64.img, which includes a cloud-init installation. The template also includes a public ssh key in the SSH public key field on the cloud-init tab. I can successfully clone a VM from the template and connect from a workstation using ssh. If I remove the public key from the SSH public key field on the cloud-init tab and reboot the VM, I can no longer ssh into the VM as expected. If I paste the same key into the SSH public key field on the cloud-init tab, I still can not ssh into the VM. Not expected. The error message is: Host key verification failed. Nothing I have tried allows me to ssh into the VM once ssh is broken on a particular VM. Is this a possible bug? ​

​

​

​

​

​

​

​

 

[bbgeek17]

Nothing I have tried allows me to ssh into the VM once ssh is broken on a particular VM.

Add an additional user when you are able to SSH. Set it either to Key or Password authentication. Perform your reproduction, then login as second user and review logs/configs/etc.

God luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Cannondale]

Thanks for your reply bbgeek17. Unfortunately, attempting to ssh with a different user produces the same result.
The error message is: Host key verification failed.

 

[bbgeek17]

I can successfully clone a VM from the template and connect from a workstation using ssh.

Create the new user at this point ^. Program the ssh-key manually, confirm that its working for new user. I cant imagine why any change to VM Cloud-Init would affect a user that Cloud-Init has no idea about.

You can also set the password for root user and login from the console for troubleshooting.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Cannondale]

Thanks for your reply bbgeek17. Attempting to ssh with a new user produces the same result.
Do you know where the public key is stored from the SSH public key field on the cloud-init tab?
The .ssh dir on the VM does not contain the key.

 

[bbgeek17]

Thanks for your reply bbgeek17. Attempting to ssh with a new user produces the same result.
Do you know where the public key is stored from the SSH public key field on the cloud-init tab?
The .ssh dir on the VM does not contain the key.

I dont think we are on the same page.
You said you can login initially after VM is powered on. Once logged in, create a new user:
sudo useradd -m newuser
sudo passwd newuser
- add ssh key to /home/newuser/.ssh/authorized_keys
- properly set permissions

Or simply :
sudo su
passwd
- then continue to login/debug via VM console.

Good luck

P.S. what you described works fine for me. The SSH-key in Cloud-Init is part of run-once.

I think there is something wrong with your procedure/testing.

root@pve-1:~# ./vm_create.sh
variables used:
STORAGE == blockbridge-nvme
VMID == 100
DEVICE == scsi0
NAME == vm100
OSUSER == blockbridge
CONSOLE == vga
CLOUDINIT == local
BIOS == default
==============
qm create 100 --memory 8192 --name vm100 --socket 1 --onboot no --cpu cputype=host
qm importdisk 100 /mnt/pve/bbnas/template/iso/ubuntu-24.04-noble-server-cloudimg-amd64.img blockbridge-nvme --format raw
qm set 100 --scsihw virtio-scsi-single --scsi0 blockbridge-nvme:vm-100-disk-0
update VM 100: -scsi0 blockbridge-nvme:vm-100-disk-0 -scsihw virtio-scsi-single
qm set 100 -net0 virtio,bridge=vmbr0,firewall=1,mtu=1
update VM 100: -net0 virtio,bridge=vmbr0,firewall=1,mtu=1
qm set 100 --scsi1 blockbridge-nvme:cloudinit
update VM 100: -scsi1 blockbridge-nvme:cloudinit
scsi1: successfully created disk 'blockbridge-nvme:vm-100-cloudinit,media=cdrom'
generating cloud-init ISO
qm set 100 --boot c --bootdisk scsi0
update VM 100: -boot c -bootdisk scsi0
qm set 100 --serial0 socket --vga virtio
update VM 100: -serial0 socket -vga virtio
qm set 100 -ipconfig0 ip=172.16.202.100/24,gw=172.16.202.1
update VM 100: -ipconfig0 ip=172.16.202.100/24,gw=172.16.202.1
qm set 100 --cipassword blockbridge --ciuser blockbridge
update VM 100: -cipassword <hidden> -ciuser blockbridge
qm set 100 -sshkeys /root/.ssh/authorized_keys
update VM 100: -sshkeys %0Assh-rsa%20AAAA#############
qm set 100 -ciupgrade 1
update VM 100: -ciupgrade 1
root@pve-1:~# qm start 100
generating cloud-init ISO

root@pve-1:~# ssh blockbridge@172.16.202.100
Welcome to Ubuntu 24.04 LTS (GNU/Linux 6.8.0-31-generic x86_64)
....

blockbridge@vm100:~$
logout
Connection to 172.16.202.100 closed.
root@pve-1:~# qm stop 100
root@pve-1:~# qm set 100 --delete sshkeys
update VM 100: -delete sshkeys
root@pve-1:~# qm start 100
generating cloud-init ISO

root@pve-1:~# ssh blockbridge@172.16.202.100
..
blockbridge@vm100:~$

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Cannondale]

bbgeek17,

I just cloned a new VM created a VM template from ubuntu-22.04-minimal-cloudimg-amd64.img to start fresh.

Here are the steps to recreate to ssh issue:
Step 1: Started new VM for the first time
Step 2: Confirm SSH public key field on the cloud-init tab
Step 3: ssh from workstation using public key. Successful
Step 4: Change SSH public key field on the cloud-init tab to new pub key
Step 5: Reboot VM
Step 6: ssh from workstation. Unsuccessful. Note: workstation has both old and new keys in .ssh
Step 7: Change SSH public key field on the cloud-init tab to old pub key
Step 8: Reboot VM
Step 9: ssh from workstation. Unsuccessful.

Note: No public key file exists in .ssh on the VM

At this point, nothing I have tried allows me to ssh into the VM once ssh is broken on a particular VM.

 

[bbgeek17]

I am not sure what else I can say besides try troubleshooting it as advised earlier several times.

A few more points:
a) The CloudInit data is provided to the VM via an ISO attached as CDrom
b) The change in PVE GUI is not automatically rebuilt into an existing ISO
c) To incorporate the change, you need to either shutdown/power-off the VM or force regenerate the ISO

Again, I cannot reproduce the problem following your steps. If you wish, convert your process to CLI and post it here, perhaps, we can spot an error.

There are two options here:
a) you found a bug in a widely used package (PVE, CloudInit, or Ubuntu)
b) you have a step that seems obvious to you but may be unexpected by most that renders your system useless

Have a good weekend.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox