[SOLVED] rbd_cache_policy error No such file or directory

[agnon]

After we upgraded all our clusters to PVE 7.0-11 we get the following error for all VMs using EFI / q35.

Here's the vm-config:
agent: 1
balloon: 2000
bios: ovmf
boot: order=scsi0;net0
cores: 4
cpu: host
description: super secret
efidisk0: vm-storage:vm-2005-disk-1,size=1M
hostpci0: 11:00,pcie=1,x-vga=1
machine: q35
memory: 8000
name: gpu-01
net0: e1000=1E:EF:E3:69:E4:1A,bridge=vmbr0,firewall=1,tag=2068
numa: 0
ostype: l26
parent: Backup
scsi0: vm-storage:vm-2005-disk-0,size=200G
scsihw: virtio-scsi-pci
smbios1: uuid=d7bf12b9-9d1a-41d2-9437-5b2ff48c55ed
sockets: 2
vmgenid: 808ccf21-fcf0-47c6-a52d-7379f101b9b2

And the error:

kvm: -drive if=pflash,unit=1,cache=writeback,format=raw,id=drive-efidisk0,size=131072,file=rbd:vm-storage/vm-2005-disk-1:mon_host=10.255.102.71;10.255.102.72;10.255.102.73:auth_supported=cephx:id=admin:keyring=/etc/pve/priv/ceph/vm-storage.keyring:rbd_cache_policy=writeback: invalid conf option rbd_cache_policy: No such file or directory
TASK ERROR: start failed: QEMU exited with code 1

Seems to originate from /usr/share/perl5/PVE/QemuServer.pm:3357 where this is appended if using an rbd.

vm-storage is a ceph cluster running octopus.

Legacy boot seems unaffected but that's not an option for us...

Also, enabling KRBD boots the VM just fine, but still, we want the option to not use the kernel module.

This is on enteprise repositories.

Let me know if I should post more information.

Best regards
Marcus

 

[dcsapak]

can you post the output of 'pveversion -v' ?

 

[agnon]

Sure thing!

Here it is:

proxmox-ve: 7.0-2 (running kernel: 5.11.22-5-pve)
pve-manager: 7.0-11 (running version: 7.0-11/63d82f4e)
pve-kernel-helper: 7.1-2
pve-kernel-5.11: 7.0-8
pve-kernel-5.4: 6.4-6
pve-kernel-5.11.22-5-pve: 5.11.22-10
pve-kernel-5.4.140-1-pve: 5.4.140-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve1
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-10
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-3
libpve-storage-perl: 7.0-12
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.11-1
proxmox-backup-file-restore: 2.0.11-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.1-1
pve-docs: 7.0-5
pve-edk2-firmware: 3.20200531-1
pve-firewall: 4.2-4
pve-firmware: 3.3-2
pve-ha-manager: 3.3-1
pve-i18n: 2.5-1
pve-qemu-kvm: 6.0.0-4
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-14
smartmontools: 7.2-pve2
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.5-pve1

//Marcus

 

[spirit]

are you sure that ceph librbd is correctly updated ?

#ceph-fuse: 14.2.21-1, seem nautilus.

(dpkg -l |grep librbd).

rbd_cache_policy=writeback is supported since octopus (not sure if your ceph cluster is external or not ?),
but if librbd is too old (14.X), you could have this error client side.

debian 11 use ceph 14.x by default, so you should add proxmox7 ceph octopus repo .

 

[agnon]

Thanks for the answers! Much appreciated!


It turned out to be exactly that

We only had the proxmox ceph repos added on the cluster hosting the ceph storage. On our 3 other clusters at our distribution points where we only mount that storage we used debian 11 repos and thus librbd 14.X.

All is well after adding proxmox ceph repos for octopus on the other clusters aswell.

Many thanks!

Best regards
Marcus