[SOLVED] Proxmox upgrade from 5 to 6.

[m4rek11]

Hello.

I'm trying to upgrade my proxmox nodes from 5.4-13 to 6. I have two nodes (for examples node1 and node2) in cluster. One of them, node2, is not used yet, so I thought to upgrade it first. I have did all steps from https://pve.proxmox.com/wiki/Upgrade_from_5.x_to_6.0 and I stuck a little on corosync upgrade.

I have upgraded corosync to ver 3 on my node2:
corosync -v Corosync Cluster Engine, version '3.0.3' Copyright (c) 2006-2018 Red Hat, Inc.

On node1 I have stil:
corosync -v Corosync Cluster Engine, version '2.4.4-dirty' Copyright (c) 2006-2009 Red Hat, Inc.

But after upgrade to ver 3 on node2 I have lost quorum.

By tiping: pvecm status on both nodes, I got:

Quorum information ------------------ Date: Wed Apr 29 10:31:09 2020 Quorum provider: corosync_votequorum Nodes: 1 Node ID: 0x00000002 Ring ID: 2.19 Quorate: No Votequorum information ---------------------- Expected votes: 2 Highest expected: 2 Total votes: 1 Quorum: 2 Activity blocked Flags:

My question is, it is normal to lost quorum by diffrences in corosync version?

Testing next upgrade steps to proxmox 6 in node2, typing apt dist-upgrade, I got:
Do you want to continue? [Y/n] y W: (pve-apt-hook) !! WARNING !! W: (pve-apt-hook) You are attempting to remove the meta-package 'proxmox-ve'! W: (pve-apt-hook) W: (pve-apt-hook) If you really want to permanently remove 'proxmox-ve' from your system, run the following command W: (pve-apt-hook) touch '/please-remove-proxmox-ve' W: (pve-apt-hook) run apt purge proxmox-ve to remove the meta-package W: (pve-apt-hook) and repeat your apt invocation. W: (pve-apt-hook) W: (pve-apt-hook) If you are unsure why 'proxmox-ve' would be removed, please verify W: (pve-apt-hook) - your APT repository settings W: (pve-apt-hook) - that you are using 'apt full-upgrade' to upgrade your system E: Sub-process /usr/share/proxmox-ve/pve-apt-hook returned an error code (1) E: Failure running script /usr/share/proxmox-ve/pve-apt-hook

Above message a little worried me.

So, what should I do in this situation? Upgrade corosync to ver 3 on node1? I will greatfull for any informations.

Your faithfully,
Marek.

 

[aaron]

Upgrade corosync to ver 3 on node1?

Yes. In the guide, it says that this needs to be done cluster-wide because version 2 and 3 or corosync are not compatible.

Testing next upgrade steps to proxmox 6 in node2, typing apt dist-upgrade, I got:

Please check your /etc/apt/sources.list and /etc/apt/sources.list.d/*.list files.
This looks like you either missed a spot to change stretch to buster or you need to reconfigure the pve-no-subscription repo and disable the enterprise repo.

 

[m4rek11]

Yes. In the guide, it says that this needs to be done cluster-wide because version 2 and 3 or corosync are not compatible.

Yes, of course, I have to do that, but is normal to lost quorum after upgrade corosonc to ver. 3 on my nod2? This no cause problem to do upgrade proxmox to ver 6 on that node2?

Please check your /etc/apt/sources.list and /etc/apt/sources.list.d/*.list files.
This looks like you either missed a spot to change stretch to buster or you need to reconfigure the pve-no-subscription repo and disable the enterprise repo.


My repositories:

/etc/apt/sources.list:

deb http://ftp.debian.org/debian buster main contrib deb http://ftp.debian.org/debian buster-updates main contrib # PVE pve-no-subscription repository provided by proxmox.com, # NOT recommended for production use deb http://download.proxmox.com/debian/pve buster pve-no-subscription # security updates deb http://security.debian.org/debian-security buster/updates main contrib


/etc/apt/sources.list.d:
ceph.list:
deb http://download.proxmox.com/debian/ceph-luminous buster main

corosync3.list:
deb http://download.proxmox.com/debian/corosync-3/ stretch main

I don't have file pve-enterprise.list in /etc/apt/sources.list.d , because I don't have valid subscription. It is ok that I don't have that file?

Anyway if I do commands in node2:
apt-get update

apt-get upgrade Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done The following packages have been kept back: apt apt-listchanges apt-transport-https apt-utils bash bc bind9-host binutils bsd-mailx bsdmainutils busybox ca-certificates ceph-common ceph-fuse coreutils cpp criu dash debconf debconf-i18n dirmngr dmeventd dmsetup dnsutils dpkg e2fslibs e2fsprogs fuse gcc gdisk glusterfs-client glusterfs-common gnupg gnupg-agent gpgv groff-base grub-common grub-efi-amd64-bin grub-efi-ia32-bin grub-pc grub-pc-bin grub2-common iftop initramfs-tools initramfs-tools-core iproute2 ipset iptables isc-dhcp-client less libanyevent-perl libapparmor-perl libapt-pkg-perl libapt-pkg5.0 libatomic1 libauthen-pam-perl libblas3 libblkid1 libbsd0 libc-bin libc-dev-bin libc6 libc6-dev libcaca0 libcc1-0 libcilkrts5 libclone-perl libcomerr2 libcommon-sense-perl libcrypt-openssl-bignum-perl libcrypt-openssl-random-perl libcrypt-openssl-rsa-perl libcrypt-ssleay-perl libcups2 libcurl3-gnutls libdevmapper1.02.1 libedit2 libexpat1 libfdisk1 libfilesys-df-perl libfuse2 libgcc1 libglib2.0-0 libgnutls-openssl27 libgnutls30 libgnutlsxx28 libgomp1 libgoogle-perftools4 libgpg-error0 libgssapi-krb5-2 libgssrpc4 libhtml-parser-perl libibverbs1 libidn2-0 libio-socket-ssl-perl libip4tc0 libip6tc0 libiptc0 libitm1 libjson-xs-perl libk5crypto3 libkadm5clnt-mit11 libkadm5srv-mit11 libkrad0 libkrb5-3 libkrb5support0 libldap-2.4-2 libldb1 liblinux-inotify2-perl liblocale-gettext-perl liblsan0 libmount1 libmpc3 libmpx2 libncurses5 libncursesw5 libnet-dbus-perl libnet-dns-perl libnet-ssleay-perl libp11-kit0 libpam-modules libpam-modules-bin libpango-1.0-0 libpangocairo-1.0-0 libpangoft2-1.0-0 libparted2 libpopt0 libpsl5 libpulse0 libpve-access-control libpve-common-perl libpve-guest-common-perl libpve-http-server-perl libpve-storage-perl libpve-u2f-server-perl libpython-stdlib libpython2.7 libpython2.7-minimal libpython2.7-stdlib libpython3-stdlib libquadmath0 librados2 librados2-perl libradosstriper1 librbd1 librdmacm1 libreadline5 libreadline7 librgw2 librrd8 librrds-perl libsmartcols1 libsmbclient libsnmp30 libsocket6-perl libsqlite3-0 libss2 libssl-dev libssl1.1 libstdc++6 libsystemd0 libtcmalloc-minimal4 libtemplate-perl libterm-readline-gnu-perl libtext-charwidth-perl libtext-iconv-perl libtinfo5 libtokyocabinet9 libtsan0 libubsan0 libudev1 libuuid-perl libuuid1 libwbclient0 libwww-perl libxml-libxml-perl libxml-parser-perl libxml2 libxtables12 libzfs2linux libzpool2linux locales login logrotate lvm2 lxc-pve lxcfs lynx make man-db mount nano ncdu ncurses-base ncurses-bin ncurses-term nfs-common nload nmap openipmi openssh-client openssh-server openssh-sftp-server openssl parted perl perl-base pinentry-curses postfix postfix-sqlite procps proxmox-ve psmisc pve-cluster pve-container pve-firewall pve-ha-manager pve-manager pve-qemu-kvm python python-cephfs python-minimal python-protobuf python-rados python-rbd python-requests python-urllib3 python2.7 python2.7-minimal python3 python3-apt python3-minimal python3-pycurl python3-requests python3-urllib3 qemu-server rpcbind rrdcached rsyslog samba-common samba-libs screen smartmontools smbclient smbios-utils spiceterm sqlite3 ssh sudo systemd systemd-sysv udev usbutils util-linux vim-common vim-tiny vncterm wget xfsprogs zfs-initramfs zfsutils-linux 0 upgraded, 0 newly installed, 0 to remove and 261 not upgraded.

apt-get dist-upgrade Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Error! Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: pve-cluster : Breaks: pve-ha-manager (<= 3.0-3) but 2.0-9 is to be installed E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages.

I will be greatfull for any tips.

Your faithfully,
Marek.

 

[aaron]

Yes, of course, I have to do that, but is normal to lost quorum after upgrade corosonc to ver. 3 on my nod2? This no cause problem to do upgrade proxmox to ver 6 on that node2?

Yes it is normal to lose quorum until the nodes run the same major version of corosync again.
From the upgrade guide:

There are two ways to proceed with the Corosync upgrade:

• Upgrade nodes one by one. Initially, the newly upgraded node(s) will not have be quorate on their own. Once at least half of the nodes plus one have been upgraded, the upgraded partition will become quorate and the not-yet-upgraded partition will lose quorum. Once all nodes have been upgraded, they should form a healthy, quorate cluster again.
• Upgrade all nodes simultaneously, e.g. using parallel ssh/screen/tmux.

Upgrade only corosync to version 3 on the other node and you should have quorum again.

Please alway use apt dist-upgrade and not just apt upgrade. IIRC dist-upgrade will install new dependencies as well.

You can try apt install --fix-broken to solve the dependency issues you ran into.

Once you have upgraded your whole cluster, don't forget to remove the temporary corosync3 repository.

 

[m4rek11]

Yes it is normal to lose quorum until the nodes run the same major version of corosync again.
From the upgrade guide:

Upgrade only corosync to version 3 on the other node and you should have quorum again.

Please alway use apt dist-upgrade and not just apt upgrade. IIRC dist-upgrade will install new dependencies as well.

You can try apt install --fix-broken to solve the dependency issues you ran into.

Once you have upgraded your whole cluster, don't forget to remove the temporary corosync3 repository.

Thank your for reply. As I understood, it is normal to no quorum with diffrents version of corosync, ok, so I don't worry about that. After upgrade corosync to my node1, quorum should back.

You don't mention about my repository, it is correct? Should I have repository pve-enterprise.list in /etc/apt/sources.list.d even don't have valid subscription?

After command like you said, I got:
apt install --fix-broken Reading package lists... Done Building dependency tree Reading state information... Done 0 upgraded, 0 newly installed, 0 to remove and 261 not upgraded.

I still have problem:
apt dist-upgrade Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Error! Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: pve-cluster : Breaks: pve-ha-manager (<= 3.0-3) but 2.0-9 is to be installed E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages.


What should I do, to fix that problem? I will be greatfull for other tips.

Your faithfully,
Marek.

 

[aaron]

Should I have repository pve-enterprise.list in /etc/apt/sources.list.d even don't have valid subscription?

No need for that if you don't have a subscription.

Can you post the output of apt update in [ CODE ] [ /CODE ] tags (without spaces inside the brackets)?

 

[m4rek11]

No need for that if you don't have a subscription.

Can you post the output of apt update in [ CODE ] [ /CODE ] tags (without spaces inside the brackets)?

Sure, nor problem.

apt update Hit:1 http://download.proxmox.com/debian/pve buster InRelease Hit:2 http://ftp.debian.org/debian buster InRelease Hit:3 http://security.debian.org/debian-security buster/updates InRelease Hit:4 http://download.proxmox.com/debian/ceph-luminous buster InRelease Hit:5 http://ftp.debian.org/debian buster-updates InRelease Hit:6 http://download.proxmox.com/debian/corosync-3 stretch InRelease Ign:7 http://linux.dell.com/repo/community/openmanage/910/stretch stretch InRelease Hit:8 http://linux.dell.com/repo/community/openmanage/910/stretch stretch Release Reading package lists... Done Building dependency tree Reading state information... Done 261 packages can be upgraded. Run 'apt list --upgradable' to see them.

 

[aaron]

Ah, it seems you missed the step to remove the temporary corosync 3 repository after the upgrade to PVE 6.
rm /etc/apt/sources.list.d/corosync3.list

It should hopefully work afterwards.

Ign:7 http://linux.dell.com/repo/community/openmanage/910/stretch stretch InRelease
Hit:8 http://linux.dell.com/repo/community/openmanage/910/stretch stretch Release

Wherever you have those configured, you should also change them from stretch to buster.

 

[m4rek11]

Ah, it seems you missed the step to remove the temporary corosync 3 repository after the upgrade to PVE 6.
rm /etc/apt/sources.list.d/corosync3.list

It should hopefully work afterwards.


Wherever you have those configured, you should also change them from stretch to buster.

As you said, I remove corosync3.list, next my step was comment openmanage repositories and upgrade made ok. Now, I have to restore openmanage to proxmox version 6.

Anyway, thanky you for help!

 

[aaron]

Good to hear.
Could you please edit the first post in this thread and set the title prefix to solved? Thanks!