issue upgrading proxmox 6 to 7 (partialy upgraded)

[ran]

Hi

After upgrading proxmox 6 to 7 (which seems it wasn't fully upgraded), i can start only LXC containers but VMS i can't start, this is the output:

generating cloud-init ISO
qemu-img: symbol lookup error: qemu-img: undefined symbol: rbd_aio_write_zeroes

TASK ERROR: command 'set -o pipefail && genisoimage -quiet -iso-level 3 -R -V cidata /run/pve/cloudinit/126/ | qemu-img dd -n -f raw -O raw 'isize=0' 'osize=4194304' 'of=/dev/zvol/ZFS/vm-126-cloudinit'' failed: exit code 127

# pveversion --verboseproxmox-ve: 7.4-1 (running kernel: 5.15.131-2-pve)pve-manager: 7.4-17 (running version: 7.4-17/513c62be)pve-kernel-5.15: 7.4-9pve-kernel-5.4: 6.4-20pve-kernel-5.3: 6.1-6pve-kernel-5.15.131-2-pve: 5.15.131-3pve-kernel-5.4.203-1-pve: 5.4.203-1pve-kernel-5.3.18-3-pve: 5.3.18-3pve-kernel-5.3.10-1-pve: 5.3.10-1ceph-fuse: 15.2.1-0ubuntu1corosync: 3.1.7-pve1criu: 3.11-3glusterfs-client: 9.2-1ifupdown: 0.8.36+pve2ifupdown2: residual configksm-control-daemon: 1.4-1libjs-extjs: 7.0.0-1libknet1: 1.24-pve2libproxmox-acme-perl: 1.4.4libproxmox-backup-qemu0: 1.3.1-1libproxmox-rs-perl: 0.2.1libpve-access-control: 7.4.1libpve-apiclient-perl: 3.2-1libpve-common-perl: 7.4-2libpve-guest-common-perl: 4.2-4libpve-http-server-perl: 4.2-3libpve-rs-perl: 0.7.7libpve-storage-perl: 7.4-3libqb0: 1.0.5-1libspice-server1: 0.14.3-2.1lvm2: 2.03.11-2.1lxc-pve: 5.0.2-2lxcfs: 5.0.3-pve1novnc-pve: 1.4.0-1proxmox-backup-client: 2.4.6-1proxmox-backup-file-restore: 2.4.6-1proxmox-kernel-helper: 7.4-1proxmox-mail-forward: 0.1.1-1proxmox-mini-journalreader: 1.3-1proxmox-offline-mirror-helper: 0.5.2proxmox-widget-toolkit: 3.7.3pve-cluster: 7.3-3pve-container: 4.4-6pve-docs: 7.4-2pve-edk2-firmware: 3.20230228-4~bpo11+2pve-firewall: 4.3-5pve-firmware: 3.6-6pve-ha-manager: 3.6.1pve-i18n: 2.12-1pve-qemu-kvm: 7.2.0-8pve-xtermjs: 4.16.0-2pve-zsync: 2.2.3qemu-server: 7.4-4smartmontools: 7.2-pve3spiceterm: 3.2-2swtpm: 0.8.0~bpo11+3vncterm: 1.7-1zfsutils-linux: 2.1.14-pve1



root@nl1:~# apt update && apt dist-upgrade
Hit:1 http://security.debian.org bullseye-security InRelease
Hit:2 http://ftp.nl.debian.org/debian bullseye InRelease
Hit:3 http://ftp.nl.debian.org/debian bullseye-updates InRelease
Hit:4 https://enterprise.proxmox.com/debian/pve bullseye InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
28 packages can be upgraded. Run 'apt list --upgradable' to see them.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
arch-test augeas-lenses cryptsetup-bin db-util db5.3-util debootstrap exfat-fuse exfat-utils extlinux f2fs-tools hfsplus
icoutils ifenslave kpartx ldmtool libafflib0v5 libaugeas0 libbfio1 libboost-context1.71.0 libboost-program-options1.71.0
libconfig9 libdate-manip-perl libepoxy0 libewf2 libfdt1 libgbm1 libhfsp0 libhivex0 libintl-perl libintl-xs-perl libjs-qrcodejs
libjs-sencha-touch libjson-glib-1.0-0 libjson-glib-1.0-common libldm-1.0-0 libleveldb1d libnetpbm10 libntfs-3g883 liboath0
libposix-strptime-perl libproxmox-backup-qemu0 libpython3.8-minimal libpython3.8-stdlib librabbitmq4 librest-client-perl
libslirp0 libsys-virt-perl libtpms0 libtsk19 liburing1 libvhdi1 libvirglrenderer1 libvirt0 libvmdk1 libwayland-server0
libwin-hivex-perl libyajl2 libyara4 lsscsi mdadm net-tools netpbm ntfs-3g osinfo-db proxmox-archive-keyring
proxmox-backup-file-restore proxmox-backup-restore-image proxmox-kernel-helper proxmox-mail-forward
proxmox-offline-mirror-docs proxmox-offline-mirror-helper proxmox-websocket-tunnel pve-lxc-syscalld python3-ceph-argparse
python3-ceph-common python3.8 python3.8-minimal reiserfsprogs scrub sleuthkit squashfs-tools supermin swtpm swtpm-libs
swtpm-tools syslinux syslinux-common zerofree
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
ceph-common libapache2-mod-php7.3 libguestfs-hfsplus libguestfs-perl libguestfs-reiserfs libguestfs-tools libguestfs-xfs
libguestfs0 libldb1 libpve-guest-common-perl libpve-storage-perl libpython-stdlib linstor-proxmox php7.3-cli php7.3-common
php7.3-json php7.3-opcache php7.3-readline proxmox-ve pve-container pve-ha-manager pve-manager pve-qemu-kvm python
python-cephfs python-chardet python-future python-ipaddr python-minimal python-prettytable python-protobuf python-rados
python-rbd python-requests python-talloc python-urllib3 python3-cephfs python3-rados python3-rbd qemu-server spiceterm
virt-p2v
The following NEW packages will be installed:
libldb2 libmpdec3 libprotobuf23 libpython3.9 libpython3.9-minimal libpython3.9-stdlib python3-ldb python3-protobuf
python3-ruamel.yaml.clib python3-talloc python3.9 python3.9-minimal
The following packages will be upgraded:
criu libpython2-stdlib libpython2.7 libpython2.7-minimal libpython2.7-stdlib libpython3-stdlib libsmbclient libwbclient0
php-common python-six python2 python2-minimal python2.7 python2.7-minimal python3 python3-apt python3-chardet python3-dbus
python3-distutils python3-gi python3-lib2to3 python3-minimal python3-pycurl python3-ruamel.yaml python3-six samba-common
samba-libs smbclient
28 upgraded, 12 newly installed, 42 to remove and 0 not upgraded.
Need to get 0 B/22.0 MB of archives.
After this operation, 421 MB disk space will be freed.
Do you want to continue? [Y/n] y
W: (pve-apt-hook) !! WARNING !!
W: (pve-apt-hook) You are attempting to remove the meta-package 'proxmox-ve'!
W: (pve-apt-hook)
W: (pve-apt-hook) If you really want to permanently remove 'proxmox-ve' from your system, run the following command
W: (pve-apt-hook) touch '/please-remove-proxmox-ve'
W: (pve-apt-hook) run apt purge proxmox-ve to remove the meta-package
W: (pve-apt-hook) and repeat your apt invocation.
W: (pve-apt-hook)
W: (pve-apt-hook) If you are unsure why 'proxmox-ve' would be removed, please verify
W: (pve-apt-hook) - your APT repository settings
W: (pve-apt-hook) - that you are using 'apt full-upgrade' to upgrade your system
E: Sub-process /usr/share/proxmox-ve/pve-apt-hook returned an error code (1)
E: Failure running script /usr/share/proxmox-ve/pve-apt-hook





What can i do to fix the above issue?

 

[sb-jw]

Did you follow the instructions?
=> https://pve.proxmox.com/wiki/Upgrade_from_6.x_to_7.0

Hit:4 https://enterprise.proxmox.com/debian/pve bullseye InRelease

Do you have a subscription? If not, please look at the instructions above and correct the repo.

Please post qm config 127 and the content of /etc/pve/storage.cfg.

 

[ran]

Hi

I did follow the guide, also not the first node i have upgraded.

root@nl1:~# qm config 126
agent: 1
balloon: 1024
boot: cdn
bootdisk: scsi0
cipassword: **********
ciuser: root
cores: 1
cpu: host
ide2: ZFS:vm-126-cloudinit,media=cdrom
ipconfig0: ***************
memory: 2048
name: **********
net0: virtio=B6:60:2C:B2:09:6F,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
protection: 1
scsi0: ZFS:base-1116-disk-0/vm-126-disk-0,size=20G
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=c3c704cb-050a-48a8-9c6f-88034a006a58
sockets: 1
vmgenid: 7e5b929e-627f-4bc3-b29c-fffab3f52ad2
cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
content backup,iso,vztmpl

lvmthin: local-lvm
thinpool data
vgname pve
content rootdir,images
nodes nl3,nl2,nl4,nl1

nfs: DreamNFS
export /ZFS/DreamNFS
path /mnt/pve/DreamNFS
server 192.168.51.200
content iso,backup,vztmpl,images
nodes nl4,nl1,nl2,nl3
options vers=4
prune-backups keep-last=30

zfspool: ZFS
pool ZFS
content rootdir,images
mountpoint /ZFS
nodes nl1,nlbackup,nl4,nl3,nl2
sparse 1

pbs: PBS
datastore NFSBACKUP
server 192.168.51.200
content backup
fingerprint cf:9b:c8:f4:e7:4f:86:c4:61:f4:d7:e1:9c:ce:a6:67:91:1b:17:64:15:d9:82:83:63:f5:a0:3e:75:b7:53:91
prune-backups keep-all=1
username root@pam



root@nl1:/etc/apt/sources.list.d# cat /etc/apt/sources.list.d/pve-enterprise.list
deb https://enterprise.proxmox.com/debian/pve bullseye pve-enterprise

root@nl1:/etc/apt/sources.list.d# cat /etc/apt/sources.list
deb http://ftp.nl.debian.org/debian bullseye main contrib

deb http://ftp.nl.debian.org/debian bullseye-updates main contrib

# security updates
deb http://security.debian.org bullseye-security main contrib





Thanks.

 

[ran]

solved by adding repository for ceph
which is unclear i have never used ceph on this host.