terminal slow to log in after upgrade from pve 8 to 9 / pbs 3 to 4

akanealw

Member
May 11, 2022
1
0
6
I know my setup is not recommended so I appreciate any suggestions.

My setup:
  1. Server1
    1. ProxmoxVE 9.1.9 - pve kernel 7.0.0-3
    2. PBS v4.2.0
    3. zfs mirrored nvme drives as root partition
    4. zfs mirrored array on spinning disks for general storage/backups
  2. Server2
    1. ProxmoxVE 9.1.9 - pve kernel pinned to 6.14.11-5, 7.0.0-3 is also installed
    2. PBS v4.2.0
    3. zfs mirrored nvme drives as root partition
    4. zfs mirrored array on spinning disks for general storage/backups
I upgraded Server2 first as it doesn't have any VMs or CTs on it currently and the upgrade went well. I ran the pve8to9 and pbs3to4 multiple times and the only warning I was concerned about was about using kernel 6.8.12-22-pve and then again after I rebooted after the upgrade, I got the same warning again only this time for 7.0.0-3-pve. I installed 6.14.11-5-pve and pinned it and the upgrade tests passed on the kernel stage. I've been able to login to it normally.


For Server 1, I shutdown all my VMs and CTs and ran the same pve8to9 and pbs3to4 scripts. Same as before, only serious warning was about the older kernel 6.8.12-22. After the reboot is when things started to have problems.

Trying to log in locally, after i entered the username, everything else I typed was not obscured like the password should be but in plaintext. after serveral minutes, it would go back to the login prompt. Even if I typed my password in plaintext, it would not log in.

I was unable to log in to the pve webui and just got the error login failed. I double checked I was on the right realm but still no login.
I was also unable to log in to the pbs webui with a 596 timeout error.

I was unable to ssh normally into it and it would just show an empty screen in my ssh terminal for several minutes, then it would show that it accepted my ssh key, waited several more minutes, then I got the MOTD but no prompt until I waited several more minutes again. I ran the pve8to9 and pbs3to4 tests again and besides the warning about kernel 7.0.0-3, I got 500 read timeouts for my pbs backup mounts.

I restarted pve-cluster, pveproxy, pvedaemon, and pvestat as well as proxmox-backup-proxy and proxmox-backup services with no errors multiple times.

I tried installing the latest 6.14.11-5-pve kernel but it fails with a no space left on device despite the only two kernels installed being the pre-upgrade 6.8.12-22 and the post-upgrade 7.0.0-3. I then mounted the esp volumes and manually removed the partially installed 6.14.11-15 kernel and regenerated my boot partitions to only have 6.8.12-22 and 7.0.0-3 available.

I'm not even sure my issues are related to the kernel, it's just the only warning I have to go on.

TLDR, Log in is very slow taking up to 5 minutes to complete each stage of login for ssh and not able to log in at all locally. Also, my pbs mounts are not mounted and running pve8to9 test has error of 500 read timeouts for my pbs mounts.