[SOLVED] kvm/qemu broken after upgrade to 9.0.3

E

emilianya

New Member
Aug 9, 2025
6
1
3
I upgraded one of my Proxmox VE 8 hosts to 9.0.3 today, but after the upgrade containers start fine but no VMs.

This seems to be caused by kvm no longer functioning:
Code: 
~# kvm --version
kvm: symbol lookup error: kvm: undefined symbol: rbd_encryption_load2

All packages are up to date, could I be missing a needed package that provides rbd_encryption_load2? Did something go wrong during the upgrade?
Installing an older version (9.2.0-7) of pve-qemu-kvm allows me to start my VMs again, so must be something in the new one.

pve8to9 output seems fine, currently only getting warnings about dkms modules (nvidia), running guests and outdated pve-qemu-kvm to get VMs running until I can fix this issue.

Any ideas? Thanks in advance.
 
please provide

- pveversion -v
- apt update
- apt policy

output!
 
Outputs below
Code: 
# pveversion -v
proxmox-ve: 9.0.0 (running kernel: 6.14.8-2-pve)
pve-manager: 9.0.3 (running version: 9.0.3/025864202ebb6109)
proxmox-kernel-helper: 9.0.3
proxmox-kernel-6.14.8-2-pve-signed: 6.14.8-2
proxmox-kernel-6.14: 6.14.8-2
amd64-microcode: 3.20250311.1
ceph: 19.2.3-pve1
ceph-fuse: 19.2.3-pve1
corosync: 3.1.9-pve2
criu: 4.1.1-1
frr-pythontools: 10.3.1-1+pve4
ifupdown2: 3.3.0-1+pmx9
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libproxmox-acme-perl: 1.7.0
libproxmox-backup-qemu0: 2.0.1
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.0.3
libpve-apiclient-perl: 3.4.0
libpve-cluster-api-perl: 9.0.6
libpve-cluster-perl: 9.0.6
libpve-common-perl: 9.0.9
libpve-guest-common-perl: 6.0.2
libpve-http-server-perl: 6.0.4
libpve-network-perl: 1.1.6
libpve-rs-perl: 0.10.10
libpve-storage-perl: 9.0.13
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2
lxc-pve: 6.0.4-2
lxcfs: 6.0.4-pve1
novnc-pve: 1.6.0-3
proxmox-backup-client: 4.0.11-1
proxmox-backup-file-restore: 4.0.11-1
proxmox-backup-restore-image: 1.0.0
proxmox-firewall: 1.1.1
proxmox-kernel-helper: 9.0.3
proxmox-mail-forward: 1.0.2
proxmox-mini-journalreader: 1.6
proxmox-offline-mirror-helper: 0.7.0
proxmox-widget-toolkit: 5.0.5
pve-cluster: 9.0.6
pve-container: 6.0.9
pve-docs: 9.0.8
pve-edk2-firmware: 4.2025.02-4
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.3
pve-firmware: 3.16-3
pve-ha-manager: 5.0.4
pve-i18n: 3.5.2
pve-qemu-kvm: 10.0.2-4
pve-xtermjs: 5.5.0-2
qemu-server: 9.0.16
smartmontools: 7.4-pve1
spiceterm: 3.4.0
swtpm: 0.8.0+pve2
vncterm: 1.9.0
zfsutils-linux: 2.3.3-pve1
Code: 
# apt update
Hit:1 http://security.debian.org/debian-security trixie-security InRelease
Hit:2 http://deb.debian.org/debian trixie InRelease                                                              
Hit:3 http://deb.debian.org/debian trixie-updates InRelease                                                      
Hit:4 http://download.proxmox.com/debian/ceph-squid trixie InRelease                                             
Hit:5 http://download.proxmox.com/debian/pve trixie InRelease                              
Get:6 https://pkgs.tailscale.com/stable/debian trixie InRelease
Fetched 6,579 B in 1s (12.1 kB/s)
All packages are up to date.
Code: 
# apt policy
Package files:
 100 /var/lib/dpkg/status
     release a=now
 500 https://pkgs.tailscale.com/stable/debian trixie/main all Packages
     release o=Tailscale,n=trixie,l=Tailscale,c=main,b=all
     origin pkgs.tailscale.com
 500 https://pkgs.tailscale.com/stable/debian trixie/main amd64 Packages
     release o=Tailscale,n=trixie,l=Tailscale,c=main,b=amd64
     origin pkgs.tailscale.com
 500 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 Packages
     release o=Proxmox,a=stable,n=trixie,l=Proxmox Debian Repository,c=pve-no-subscription,b=amd64
     origin download.proxmox.com
 500 http://security.debian.org/debian-security trixie-security/main amd64 Packages
     release v=13,o=Debian,a=stable-security,n=trixie-security,l=Debian-Security,c=main,b=amd64
     origin security.debian.org
 500 http://deb.debian.org/debian trixie-updates/main amd64 Packages
     release v=13-updates,o=Debian,a=stable-updates,n=trixie-updates,l=Debian,c=main,b=amd64
     origin deb.debian.org
 500 http://deb.debian.org/debian trixie/non-free-firmware amd64 Packages
     release v=13.0,o=Debian,a=stable,n=trixie,l=Debian,c=non-free-firmware,b=amd64
     origin deb.debian.org
 500 http://deb.debian.org/debian trixie/main amd64 Packages
     release v=13.0,o=Debian,a=stable,n=trixie,l=Debian,c=main,b=amd64
     origin deb.debian.org
 500 http://download.proxmox.com/debian/ceph-squid trixie/no-subscription amd64 Packages
     release o=Proxmox,a=stable,n=trixie,l=Proxmox Ceph 19 Squid Debian Repository,c=no-subscription,b=amd64
     origin download.proxmox.com
Pinned packages:
 
could you also post `ldd /usr/bin/kvm`?
 
Code: 
# ldd /usr/bin/kvm
        linux-vdso.so.1 (0x0000782f0f0f3000)
        libgnutls.so.30 => /lib/x86_64-linux-gnu/libgnutls.so.30 (0x0000782f0d000000)
        libpixman-1.so.0 => /lib/x86_64-linux-gnu/libpixman-1.so.0 (0x0000782f0f032000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x0000782f0f012000)
        libjpeg.so.62 => /lib/x86_64-linux-gnu/libjpeg.so.62 (0x0000782f0ef7b000)
        libepoxy.so.0 => /lib/x86_64-linux-gnu/libepoxy.so.0 (0x0000782f0d2d0000)
        libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x0000782f0ef34000)
        libusb-1.0.so.0 => /lib/x86_64-linux-gnu/libusb-1.0.so.0 (0x0000782f0ef14000)
        libseccomp.so.2 => /lib/x86_64-linux-gnu/libseccomp.so.2 (0x0000782f0d2a2000)
        libfdt.so.1 => /lib/x86_64-linux-gnu/libfdt.so.1 (0x0000782f0ef09000)
        libnuma.so.1 => /lib/x86_64-linux-gnu/libnuma.so.1 (0x0000782f0d293000)
        libgio-2.0.so.0 => /lib/x86_64-linux-gnu/libgio-2.0.so.0 (0x0000782f0ce08000)
        libgobject-2.0.so.0 => /lib/x86_64-linux-gnu/libgobject-2.0.so.0 (0x0000782f0d230000)
        libglib-2.0.so.0 => /lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x0000782f0ccb0000)
        libproxmox_backup_qemu.so.0 => /lib/libproxmox_backup_qemu.so.0 (0x0000782f0c600000)
        libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x0000782f0c536000)
        libslirp.so.0 => /lib/x86_64-linux-gnu/libslirp.so.0 (0x0000782f0cc8b000)
        libgbm.so.1 => /lib/x86_64-linux-gnu/libgbm.so.1 (0x0000782f0ef01000)
        libspice-server.so.1 => /lib/x86_64-linux-gnu/libspice-server.so.1 (0x0000782f0c3f9000)
        libvirglrenderer.so.1 => /lib/x86_64-linux-gnu/libvirglrenderer.so.1 (0x0000782f0c233000)
        libusbredirparser.so.1 => /lib/x86_64-linux-gnu/libusbredirparser.so.1 (0x0000782f0d226000)
        libasound.so.2 => /lib/x86_64-linux-gnu/libasound.so.2 (0x0000782f0c112000)
        libpulse.so.0 => /lib/x86_64-linux-gnu/libpulse.so.0 (0x0000782f0c0bd000)
        libsndio.so.7 => /lib/x86_64-linux-gnu/libsndio.so.7 (0x0000782f0cc77000)
        libsystemd.so.0 => /lib/x86_64-linux-gnu/libsystemd.so.0 (0x0000782f0bfa8000)
        libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x0000782f0cc6b000)
        libiscsi.so.7 => /lib/x86_64-linux-gnu/libiscsi.so.7 (0x0000782f0bf75000)
        libaio.so.1t64 => /lib/x86_64-linux-gnu/libaio.so.1t64 (0x0000782f0cc66000)
        liburing.so.2 => /lib/x86_64-linux-gnu/liburing.so.2 (0x0000782f0cc5e000)
        libcurl-gnutls.so.4 => /lib/x86_64-linux-gnu/libcurl-gnutls.so.4 (0x0000782f0be8a000)
        librbd.so.1 => /lib/x86_64-linux-gnu/librbd.so.1 (0x0000782f0b400000)
        librados.so.2 => /lib/x86_64-linux-gnu/librados.so.2 (0x0000782f0bcfa000)
        libgmodule-2.0.so.0 => /lib/x86_64-linux-gnu/libgmodule-2.0.so.0 (0x0000782f0cc55000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x0000782f0b310000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x0000782f0b11a000)
        libp11-kit.so.0 => /lib/x86_64-linux-gnu/libp11-kit.so.0 (0x0000782f0af79000)
        libidn2.so.0 => /lib/x86_64-linux-gnu/libidn2.so.0 (0x0000782f0af46000)
        libunistring.so.5 => /lib/x86_64-linux-gnu/libunistring.so.5 (0x0000782f0ad5e000)
        libtasn1.so.6 => /lib/x86_64-linux-gnu/libtasn1.so.6 (0x0000782f0bce4000)
        libhogweed.so.6 => /lib/x86_64-linux-gnu/libhogweed.so.6 (0x0000782f0ad13000)
        libnettle.so.8 => /lib/x86_64-linux-gnu/libnettle.so.8 (0x0000782f0acbd000)
        libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x0000782f0ac33000)
        /lib64/ld-linux-x86-64.so.2 (0x0000782f0f0f5000)
        libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2 (0x0000782f0cc45000)
        libmount.so.1 => /lib/x86_64-linux-gnu/libmount.so.1 (0x0000782f0abb7000)
        libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x0000782f0ab83000)
        libffi.so.8 => /lib/x86_64-linux-gnu/libffi.so.8 (0x0000782f0ab76000)
        libatomic.so.1 => /lib/x86_64-linux-gnu/libatomic.so.1 (0x0000782f0bcda000)
        libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x0000782f0aac7000)
        libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x0000782f0a400000)
        libssl.so.3 => /lib/x86_64-linux-gnu/libssl.so.3 (0x0000782f0a2f2000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x0000782f0aa9a000)
        libdrm.so.2 => /lib/x86_64-linux-gnu/libdrm.so.2 (0x0000782f0aa83000)
        libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x0000782f0aa57000)
        libopus.so.0 => /lib/x86_64-linux-gnu/libopus.so.0 (0x0000782f09e00000)
        libgstreamer-1.0.so.0 => /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0 (0x0000782f0a189000)
        libgstapp-1.0.so.0 => /lib/x86_64-linux-gnu/libgstapp-1.0.so.0 (0x0000782f0aa41000)
        liborc-0.4.so.0 => /lib/x86_64-linux-gnu/liborc-0.4.so.0 (0x0000782f09d49000)
        liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1 (0x0000782f09d22000)
        libsasl2.so.2 => /lib/x86_64-linux-gnu/libsasl2.so.2 (0x0000782f0a16d000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x0000782f09a00000)
        libX11.so.6 => /lib/x86_64-linux-gnu/libX11.so.6 (0x0000782f098b8000)
        libvulkan.so.1 => /lib/x86_64-linux-gnu/libvulkan.so.1 (0x0000782f09c9f000)
        libva.so.2 => /lib/x86_64-linux-gnu/libva.so.2 (0x0000782f09c68000)
        libva-drm.so.2 => /lib/x86_64-linux-gnu/libva-drm.so.2 (0x0000782f0aa3a000)
        libpulsecommon-17.0.so => /usr/lib/x86_64-linux-gnu/pulseaudio/libpulsecommon-17.0.so (0x0000782f09830000)
        libdbus-1.so.3 => /lib/x86_64-linux-gnu/libdbus-1.so.3 (0x0000782f097d9000)
        libbsd.so.0 => /lib/x86_64-linux-gnu/libbsd.so.0 (0x0000782f097c3000)
        libibverbs.so.1 => /lib/x86_64-linux-gnu/libibverbs.so.1 (0x0000782f097a2000)
        librdmacm.so.1 => /lib/x86_64-linux-gnu/librdmacm.so.1 (0x0000782f09783000)
        libnghttp3.so.9 => /lib/x86_64-linux-gnu/libnghttp3.so.9 (0x0000782f09759000)
        libngtcp2_crypto_gnutls.so.8 => /lib/x86_64-linux-gnu/libngtcp2_crypto_gnutls.so.8 (0x0000782f0974d000)
        libngtcp2.so.16 => /lib/x86_64-linux-gnu/libngtcp2.so.16 (0x0000782f09704000)
        libnghttp2.so.14 => /lib/x86_64-linux-gnu/libnghttp2.so.14 (0x0000782f096d2000)
        librtmp.so.1 => /lib/x86_64-linux-gnu/librtmp.so.1 (0x0000782f096b4000)
        libssh2.so.1 => /lib/x86_64-linux-gnu/libssh2.so.1 (0x0000782f0966c000)
        libpsl.so.5 => /lib/x86_64-linux-gnu/libpsl.so.5 (0x0000782f09658000)
        libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x0000782f09602000)
        libldap.so.2 => /lib/x86_64-linux-gnu/libldap.so.2 (0x0000782f0959e000)
        liblber.so.2 => /lib/x86_64-linux-gnu/liblber.so.2 (0x0000782f0958d000)
        libbrotlidec.so.1 => /lib/x86_64-linux-gnu/libbrotlidec.so.1 (0x0000782f0957f000)
        libcryptsetup.so.12 => /lib/x86_64-linux-gnu/libcryptsetup.so.12 (0x0000782f094ec000)
        libceph-common.so.2 => /usr/lib/x86_64-linux-gnu/ceph/libceph-common.so.2 (0x0000782f00800000)
        libfmt.so.9 => /lib/x86_64-linux-gnu/libfmt.so.9 (0x0000782f094cc000)
        libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1 (0x0000782f007a0000)
        libgstbase-1.0.so.0 => /lib/x86_64-linux-gnu/libgstbase-1.0.so.0 (0x0000782f00717000)
        libxxhash.so.0 => /lib/x86_64-linux-gnu/libxxhash.so.0 (0x0000782f094b9000)
        libxcb.so.1 => /lib/x86_64-linux-gnu/libxcb.so.1 (0x0000782f006ec000)
        libsndfile.so.1 => /lib/x86_64-linux-gnu/libsndfile.so.1 (0x0000782f0065d000)
        libX11-xcb.so.1 => /lib/x86_64-linux-gnu/libX11-xcb.so.1 (0x0000782f094b4000)
        libasyncns.so.0 => /lib/x86_64-linux-gnu/libasyncns.so.0 (0x0000782f094ad000)
        libmd.so.0 => /lib/x86_64-linux-gnu/libmd.so.0 (0x0000782f0064e000)
        libnl-route-3.so.200 => /lib/x86_64-linux-gnu/libnl-route-3.so.200 (0x0000782f005bb000)
        libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x0000782f00598000)
        libkrb5.so.3 => /lib/x86_64-linux-gnu/libkrb5.so.3 (0x0000782f004c0000)
        libk5crypto.so.3 => /lib/x86_64-linux-gnu/libk5crypto.so.3 (0x0000782f00492000)
        libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x0000782f0048c000)
        libkrb5support.so.0 => /lib/x86_64-linux-gnu/libkrb5support.so.0 (0x0000782f0047e000)
        libbrotlicommon.so.1 => /lib/x86_64-linux-gnu/libbrotlicommon.so.1 (0x0000782f0045b000)
        libdevmapper.so.1.02.1 => /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1 (0x0000782f003e9000)
        libjson-c.so.5 => /lib/x86_64-linux-gnu/libjson-c.so.5 (0x0000782f003d5000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x0000782f003c3000)
        libboost_thread.so.1.74.0 => /lib/x86_64-linux-gnu/libboost_thread.so.1.74.0 (0x0000782f003a0000)
        libboost_iostreams.so.1.74.0 => /lib/x86_64-linux-gnu/libboost_iostreams.so.1.74.0 (0x0000782f00388000)
        libXau.so.6 => /lib/x86_64-linux-gnu/libXau.so.6 (0x0000782f00381000)
        libXdmcp.so.6 => /lib/x86_64-linux-gnu/libXdmcp.so.6 (0x0000782f00379000)
        libFLAC.so.14 => /lib/x86_64-linux-gnu/libFLAC.so.14 (0x0000782f0030f000)
        libvorbis.so.0 => /lib/x86_64-linux-gnu/libvorbis.so.0 (0x0000782f002e1000)
        libvorbisenc.so.2 => /lib/x86_64-linux-gnu/libvorbisenc.so.2 (0x0000782f00236000)
        libogg.so.0 => /lib/x86_64-linux-gnu/libogg.so.0 (0x0000782f0022c000)
        libmpg123.so.0 => /lib/x86_64-linux-gnu/libmpg123.so.0 (0x0000782f001cd000)
        libmp3lame.so.0 => /lib/x86_64-linux-gnu/libmp3lame.so.0 (0x0000782f00154000)
        libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x0000782f0014d000)
        libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x0000782f0013a000)
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x0000782f00109000)
 
okay - please also post "dpkg -S libceph-common librados"
 
Last edited:
fabian said:
okay - please also post "dpkg -S libceph-common librados"
Click to expand...
Output below, looks identical to another install where things are working
Code: 
# dpkg -S libceph-common librados
librados2: /usr/lib/ceph/libceph-common.so.2
librados2: /usr/lib/librados.so.2.0.0
librados2-perl: /usr/share/doc/librados2-perl/changelog.gz
libradosstriper1: /usr/share/doc/libradosstriper1/changelog.Debian.gz
libradosstriper1: /usr/share/doc/libradosstriper1/copyright
libradosstriper1: /usr/share/doc/libradosstriper1/changelog.gz
librados2: /usr/share/doc/librados2/changelog.gz
librados2-perl: /usr/share/doc/librados2-perl
librados2: /usr/share/doc/librados2/changelog.Debian.gz
libradosstriper1: /usr/lib/libradosstriper.so.1.0.0
librados2: /usr/lib/librados.so.2
librados2: /usr/lib/librados_tp.so.2
librados2-perl: /usr/share/doc/librados2-perl/copyright
librados2: /usr/lib/librados_tp.so.2.0.0
librados2: /usr/share/doc/librados2
libradosstriper1: /usr/lib/libradosstriper.so.1
librados2: /usr/share/doc/librados2/copyright
libradosstriper1: /usr/share/doc/libradosstriper1
 
yes - but it picks up various (ceph-related) libraries from /lib/x86_64-linux-gnu/ instead of directly from /lib ..

could you please do "ls -lha /lib/x86_64-linux-gnu/librados.so.2", "ls -lhHa /lib/x86_64-linux-gnu/librados.so.2" and "sha256sum /lib/x86_64-linux-gnu/librados.so.2" ?
 
fabian said:
yes - but it picks up various (ceph-related) libraries from /lib/x86_64-linux-gnu/ instead of directly from /lib ..

could you please do "ls -lha /lib/x86_64-linux-gnu/librados.so.2", "ls -lhHa /lib/x86_64-linux-gnu/librados.so.2" and "sha256sum /lib/x86_64-linux-gnu/librados.so.2" ?
Click to expand...
Code: 
# ls -lha /lib/x86_64-linux-gnu/librados.so.2
lrwxrwxrwx 1 root root 17 Feb 16  2023 /lib/x86_64-linux-gnu/librados.so.2 -> librados.so.2.0.0

# ls -lhHa /lib/x86_64-linux-gnu/librados.so.2
-rw-r--r-- 1 root root 1.6M Feb 16  2023 /lib/x86_64-linux-gnu/librados.so.2

# sha256sum /lib/x86_64-linux-gnu/librados.so.2
dcd5009e32750781efcf0f1e3df982cb7640b5ca8e11b9900e3a54173fa7476c  /lib/x86_64-linux-gnu/librados.so.2
 
okay, so you have some super old ceph libraries lying around that don't belong to any packages.. you need to find and remove them all ;) you can compare the ldd output from one of your working nodes to the problematic one, and after verifying that the file in question doesn't belong to any package, remove the files:

Code: 
+libboost_iostreams.so.1.74.0 => /lib/x86_64-linux-gnu/libboost_iostreams.so.1.74.0
+libboost_thread.so.1.74.0 => /lib/x86_64-linux-gnu/libboost_thread.so.1.74.0
+libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0
+libceph-common.so.2 => /usr/lib/x86_64-linux-gnu/ceph/libceph-common.so.2
+libfmt.so.9 => /lib/x86_64-linux-gnu/libfmt.so.9
+librados.so.2 => /lib/x86_64-linux-gnu/librados.so.2
+librbd.so.1 => /lib/x86_64-linux-gnu/librbd.so.1

I think those are all from the same source and problematic, but please double check to verify, in particular libfmt and libbz might be benign and just show up in the output by virtue of being referenced by one of the problematic ones..
 
Hi,

Thanks a lot! I removed these (except for libfmt and libbz) and kvm is no longer reporting errors, libfmt and libbz vanished from the ldd output too. The libraries don't seem to belong to anything and predate the current installation on the system, so my guess is these came from a faulty backup restore some time ago.
 
  • Like
Reactions: fabian
You must log in or register to reply here.
Top Bottom