[SOLVED] PVE-Manager not coming back after upgrade

[Andy Huang]

I've recently upgraded from 5.3 to 5.4 using:

sudo apt-get update
sudo apt-get upgrade

After rebooting, my PVE-manager isn't coming back online, though PVE daemon seems to be running.

# service pvedaemon status
● pvedaemon.service - PVE API Daemon
   Loaded: loaded (/lib/systemd/system/pvedaemon.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2019-05-03 11:31:21 PDT; 17min ago
  Process: 4675 ExecStop=/usr/bin/pvedaemon stop (code=exited, status=0/SUCCESS)
  Process: 4678 ExecStart=/usr/bin/pvedaemon start (code=exited, status=0/SUCCESS)
 Main PID: 4695 (pvedaemon)
    Tasks: 4 (limit: 9830)
   Memory: 114.5M
      CPU: 1.186s
   CGroup: /system.slice/pvedaemon.service
           ├─4695 pvedaemon
           ├─4698 pvedaemon worker
           ├─4699 pvedaemon worker
           └─4700 pvedaemon worker

May 03 11:31:20 taiyi systemd[1]: Starting PVE API Daemon...
May 03 11:31:21 taiyi pvedaemon[4695]: starting server
May 03 11:31:21 taiyi pvedaemon[4695]: starting 3 worker(s)
May 03 11:31:21 taiyi pvedaemon[4695]: worker 4698 started
May 03 11:31:21 taiyi pvedaemon[4695]: worker 4699 started
May 03 11:31:21 taiyi pvedaemon[4695]: worker 4700 started
May 03 11:31:21 taiyi systemd[1]: Started PVE API Daemon.

# service pve-manager status
● pve-guests.service - PVE guests
   Loaded: loaded (/lib/systemd/system/pve-guests.service; enabled; vendor preset: enabled)
   Active: active (exited) since Fri 2019-05-03 11:24:39 PDT; 24min ago
  Process: 2754 ExecStart=/usr/bin/pvesh --nooutput create /nodes/localhost/startall (code=exited, status=0/SUCCESS)
 Main PID: 2754 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 9830)
   Memory: 0B
      CPU: 0
   CGroup: /system.slice/pve-guests.service

May 03 11:24:24 taiyi pve-guests[2769]: <root@pam> starting task UPID:taiyi:00000C23:00001345:5CCC8758:qmstart:103:root@pam:
May 03 11:24:24 taiyi pve-guests[3107]: start VM 103: UPID:taiyi:00000C23:00001345:5CCC8758:qmstart:103:root@pam:
May 03 11:24:29 taiyi pvesh[2754]: Starting VM 108
May 03 11:24:29 taiyi pve-guests[2769]: <root@pam> starting task UPID:taiyi:00000C84:0000153E:5CCC875D:qmstart:108:root@pam:
May 03 11:24:29 taiyi pve-guests[3204]: start VM 108: UPID:taiyi:00000C84:0000153E:5CCC875D:qmstart:108:root@pam:
May 03 11:24:34 taiyi pvesh[2754]: Starting VM 110
May 03 11:24:34 taiyi pve-guests[3263]: start VM 110: UPID:taiyi:00000CBF:0000173B:5CCC8762:qmstart:110:root@pam:
May 03 11:24:34 taiyi pve-guests[2769]: <root@pam> starting task UPID:taiyi:00000CBF:0000173B:5CCC8762:qmstart:110:root@pam:
May 03 11:24:39 taiyi pve-guests[2754]: <root@pam> end task UPID:taiyi:00000AD1:00000D48:5CCC8749:startall::root@pam: OK
May 03 11:24:39 taiyi systemd[1]: Started PVE guests.

When I try to manually start pve-manager, I get a very generic error that doesn't really help my current situation:

# service pve-manager start
Failed to start pve-manager.service: Operation refused, unit pve-guests.service may be requested by dependency only.
See system logs and 'systemctl status pve-manager.service' for details.

Attempting to access the panel seems to just times out. Can someone please advise where I can find more logs to debug further on this matter, or have encountered something like this in the past and have a solution?

Thanks!

 

[janssensm]

As mentioned many times in the forum, never use apt-get upgrade.
`apt-get dist-upgrade` or `apt full-upgrade`

https://forum.proxmox.com/threads/curl-current-proxmox-release-version-from-web.53920/#post-248696

Cheers

 

[Andy Huang]

As mentioned many times in the forum, never use apt-get upgrade.
`apt-get dist-upgrade` or `apt full-upgrade`

https://forum.proxmox.com/threads/curl-current-proxmox-release-version-from-web.53920/#post-248696

Cheers

Sorry I typo'ed when I was making this post. I did dist-upgrade instead of upgrade. You physically cannot get an upgrade command to do a dist-upgrade change; and what was recorded in my .bash_history supports that.

 

[Stoiko Ivanov]

When I try to manually start pve-manager,

The pve-manager.service is not used anymore - it used to be the service which took care of starting and stopping all guests, but since this was confusing it is now an alias to pve-guests.service

According to the output you posted it seems like the guests started up fine?

Which issues are you experiencing? What is not working?

 

[Andy Huang]

The pve-manager.service is not used anymore - it used to be the service which took care of starting and stopping all guests, but since this was confusing it is now an alias to pve-guests.service

According to the output you posted it seems like the guests started up fine?

Which issues are you experiencing? What is not working?

Thanks for getting back! The KVM VMs are running as you've identified. The problem was the Web GUI manager not accessible.

I wrote a loooong wall of text about how I dug deeper and found it may be SSL related, reverting to default settings, and testing from different machines... and when I got to very end, I decided to check my iptables, and turns out, some where some thing created a good strict firewall rule of basically if not ssh/http/https ports, block it. So Proxmox's 8006 was getting blocked, and that's why things doesn't work.

I've modified the iptables rule to allow Proxmox's 8006 port, and it seem to be working now.

Now to find the ghost in the machine that's securing my firewall for an internal non-public-facing system...

 

[janssensm]

Great you got past that hurdle and thanks for sharing!