Hi all,
I'm working on a production cluster, got a trouble migrating from 6.3-3 to 6.4-13, and found a (very simple) solution.
I post it here for further notice and help, to get it crawled by search engines.
Material: 3 nodes, Threadripper 2950X - 128Go ECC DDR4, 4x2To SSD SATA, 10G-BaseT links, 25 VMs.
As a production cluster, I tested this behavior on a test cluster before deploying.
The purpose is to upgrade a v6.3-3 cluster to v6.4, to end on a mature v7 one.
Question: Why keeping the cluster on a so old v6.3-3 version ?
Response: To keep the production server as functionnal as possible. As it is working very well, keep it working as is.
NOTICE: Please migrate all your VMs/containers before upgrading any node.
So, as you can figure out, if you try to upgrade a buster proxmox node in late september, you will get an error at "apt-get update" level as stable is Bullseye now.
So, let's update, knowing that stable changed, allways with an "apt-get update", but with the "--allow-releaseinfo-change" option
So, let's try an upgrade in "pretend" mode.
Everything seems OK, so, let's download all these packages, to get them available.
Ok, then, time to upgrade them (and get an error, and workaround it).
Mama ! There is a problem! Let's try the "systemctl status pvedaemon.service"
As you can figure, pve-daemon, pve-proxy, and pve-manager are not working.
No web GUI interface, big stress.
So the workaround. We have to end the upgrade. Let's dist-upgrade.
We can check if pvedaemon is working
Web GUI is fonctionnal, node is upgraded to 6.4-13, more easyly upgradable to v7
Everything is back to production.
Hey ! There were a lot of removable packages !
Mainly kernel packages.
So let's remove them.
Thanks to the Proxmox people, for this exceptional work.
Thanks to Fabrice Bellard, for creating qemu. Fabrice Bellard homepage. QEMU homepage
I'm working on a production cluster, got a trouble migrating from 6.3-3 to 6.4-13, and found a (very simple) solution.
I post it here for further notice and help, to get it crawled by search engines.
Material: 3 nodes, Threadripper 2950X - 128Go ECC DDR4, 4x2To SSD SATA, 10G-BaseT links, 25 VMs.
As a production cluster, I tested this behavior on a test cluster before deploying.
The purpose is to upgrade a v6.3-3 cluster to v6.4, to end on a mature v7 one.
Question: Why keeping the cluster on a so old v6.3-3 version ?
Response: To keep the production server as functionnal as possible. As it is working very well, keep it working as is.
NOTICE: Please migrate all your VMs/containers before upgrading any node.
So, as you can figure out, if you try to upgrade a buster proxmox node in late september, you will get an error at "apt-get update" level as stable is Bullseye now.
This error is normal, as stable is Bullseye now.
So, let's update, knowing that stable changed, allways with an "apt-get update", but with the "--allow-releaseinfo-change" option
This time, there are juste notifications, and no errors.
So, let's try an upgrade in "pretend" mode.
Everything seems OK, so, let's download all these packages, to get them available.
Ok, then, time to upgrade them (and get an error, and workaround it).
Mama ! There is a problem! Let's try the "systemctl status pvedaemon.service"
~# systemctl status pvedaemon.service
● pvedaemon.service - PVE API Daemon
Loaded: loaded (/lib/systemd/system/pvedaemon.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2021-02-26 09:48:20 CET; 6 months 27 days ago
Main PID: 1991 (pvedaemon)
Tasks: 8 (limit: 154381)
Memory: 242.7M
CGroup: /system.slice/pvedaemon.service
├─ 1991 pvedaemon
├─1036176 pvedaemon worker
├─1045514 pvedaemon worker
├─1523990 pvedaemon worker
├─1545278 task UPIDx2:0017943E:6C413E5D:614DD7B0:vncproxy:114:root@pam:
├─1545280 /usr/bin/ssh -e none -o BatchMode=yes -o HostKeyAlias=px1 -T root@192.168.1.101 /usr/sbin/qm vncproxy 114
├─1569940 task UPID
└─1569942 /usr/bin/ssh -e none -o BatchMode=yes -o HostKeyAlias=px1 -T root@192.168.1.101 /usr/sbin/qm vncproxy 113
Sep 24 16:24:48 px2 pvedaemon[1561078]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2.pm line 15.
Sep 24 16:24:48 px2 pvedaemon[1561078]: Compilation failed in require at /usr/share/perl5/PVE/Service/pvedaemon.pm line 8.
Sep 24 16:24:48 px2 pvedaemon[1561078]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Service/pvedaemon.pm line 8.
Sep 24 16:24:48 px2 pvedaemon[1561078]: Compilation failed in require at /usr/bin/pvedaemon line 11.
Sep 24 16:24:48 px2 pvedaemon[1561078]: BEGIN failed--compilation aborted at /usr/bin/pvedaemon line 11.
Sep 24 16:24:48 px2 systemd[1]: pvedaemon.service: Control process exited, code=exited, status=2/INVALIDARGUMENT
Sep 24 16:24:48 px2 systemd[1]: Reload failed for PVE API Daemon.
As you can figure, pve-daemon, pve-proxy, and pve-manager are not working.
No web GUI interface, big stress.
So the workaround. We have to end the upgrade. Let's dist-upgrade.
We can check if pvedaemon is working
~# systemctl status pvedaemon.service
● pvedaemon.service - PVE API Daemon
Loaded: loaded (/lib/systemd/system/pvedaemon.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2021-02-26 09:48:20 CET; 6 months 27 days ago
Main PID: 1991 (pvedaemon)
Tasks: 8 (limit: 154381)
Memory: 306.6M
CGroup: /system.slice/pvedaemon.service
├─ 1991 pvedaemon
├─1545278 task UPID
├─1545280 /usr/bin/ssh -e none -o BatchMode=yes -o HostKeyAlias=px1 -T root@192.168.1.101 /usr/sbin/qm vncproxy 114
├─1569940 task UPID
├─1569942 /usr/bin/ssh -e none -o BatchMode=yes -o HostKeyAlias=px1 -T root@192.168.1.101 /usr/sbin/qm vncproxy 113
├─1582516 pvedaemon worker
├─1582517 pvedaemon worker
└─1582518 pvedaemon worker
Sep 24 16:34:02 px2 pvedaemon[1991]: starting 3 worker(s)
Sep 24 16:34:02 px2 pvedaemon[1991]: worker 1582516 started
Sep 24 16:34:02 px2 pvedaemon[1991]: worker 1582517 started
Sep 24 16:34:02 px2 pvedaemon[1991]: worker 1582518 started
Sep 24 16:34:07 px2 pvedaemon[1036176]: worker exit
Sep 24 16:34:07 px2 pvedaemon[1523990]: worker exit
Sep 24 16:34:07 px2 pvedaemon[1045514]: worker exit
Sep 24 16:34:07 px2 pvedaemon[1991]: worker 1036176 finished
Sep 24 16:34:07 px2 pvedaemon[1991]: worker 1523990 finished
Sep 24 16:34:07 px2 pvedaemon[1991]: worker 1045514 finished
Web GUI is fonctionnal, node is upgraded to 6.4-13, more easyly upgradable to v7
Everything is back to production.
Hey ! There were a lot of removable packages !
Mainly kernel packages.
So let's remove them.
Thanks to the Proxmox people, for this exceptional work.
Thanks to Fabrice Bellard, for creating qemu. Fabrice Bellard homepage. QEMU homepage
