[SOLVED] Ceph health warning: unable to load:snappy

[cmonty14]

Hello,

after a server crash I was able to repair the cluster.
Health check looks ok, but there's this warning for 68 OSDs:
unable to load:snappy

All OSDs are located on the same cluster node.
Therefore I was checking version of related file libsnappy1v5; this was 1.1.9

Comparing this file version with other cluster nodes showed that it's newer; therefore I decided to downgrade to libsnappy1v5=1.1.7-1.

Question:
Can you please advise next steps to fix the warning for 68 OSDs?

THX

 

[fabian]

please post the full warnings, the output of "pveversion -v" and the output of "apt update".

 

[cmonty14]

2022-12-13 07:50:00.000324 mon.ld5505 [WRN] BLUESTORE_NO_COMPRESSION BlueStore compression broken on 68 OSD(s)                                                                                       [62/1968]
2022-12-13 07:50:00.000347 mon.ld5505 [WRN]      osd.0 unable to load:snappy
2022-12-13 07:50:00.000364 mon.ld5505 [WRN]      osd.1 unable to load:snappy
2022-12-13 07:50:00.000393 mon.ld5505 [WRN]      osd.8 unable to load:snappy
2022-12-13 07:50:00.000420 mon.ld5505 [WRN]      osd.9 unable to load:snappy
2022-12-13 07:50:00.000439 mon.ld5505 [WRN]      osd.10 unable to load:snappy
2022-12-13 07:50:00.000466 mon.ld5505 [WRN]      osd.31 unable to load:snappy
2022-12-13 07:50:00.000491 mon.ld5505 [WRN]      osd.32 unable to load:snappy
2022-12-13 07:50:00.000629 mon.ld5505 [WRN]      osd.33 unable to load:snappy
2022-12-13 07:50:00.000653 mon.ld5505 [WRN]      osd.34 unable to load:snappy
2022-12-13 07:50:00.000667 mon.ld5505 [WRN]      osd.35 unable to load:snappy
2022-12-13 07:50:00.000686 mon.ld5505 [WRN]      osd.36 unable to load:snappy
2022-12-13 07:50:00.000705 mon.ld5505 [WRN]      osd.37 unable to load:snappy
2022-12-13 07:50:00.000729 mon.ld5505 [WRN]      osd.38 unable to load:snappy
2022-12-13 07:50:00.000752 mon.ld5505 [WRN]      osd.39 unable to load:snappy
2022-12-13 07:50:00.000780 mon.ld5505 [WRN]      osd.40 unable to load:snappy
2022-12-13 07:50:00.000793 mon.ld5505 [WRN]      osd.41 unable to load:snappy
2022-12-13 07:50:00.000850 mon.ld5505 [WRN]      osd.59 unable to load:snappy
2022-12-13 07:50:00.000863 mon.ld5505 [WRN]      osd.60 unable to load:snappy
2022-12-13 07:50:00.000901 mon.ld5505 [WRN]      osd.62 unable to load:snappy
2022-12-13 07:50:00.000918 mon.ld5505 [WRN]      osd.63 unable to load:snappy
2022-12-13 07:50:00.000931 mon.ld5505 [WRN]      osd.64 unable to load:snappy
2022-12-13 07:50:00.000944 mon.ld5505 [WRN]      osd.76 unable to load:snappy
2022-12-13 07:50:00.000967 mon.ld5505 [WRN]      osd.78 unable to load:snappy
2022-12-13 07:50:00.001005 mon.ld5505 [WRN]      osd.79 unable to load:snappy
2022-12-13 07:50:00.001025 mon.ld5505 [WRN]      osd.80 unable to load:snappy
2022-12-13 07:50:00.001044 mon.ld5505 [WRN]      osd.81 unable to load:snappy
2022-12-13 07:50:00.001063 mon.ld5505 [WRN]      osd.82 unable to load:snappy
2022-12-13 07:50:00.001086 mon.ld5505 [WRN]      osd.83 unable to load:snappy
2022-12-13 07:50:00.001105 mon.ld5505 [WRN]      osd.84 unable to load:snappy
2022-12-13 07:50:00.001125 mon.ld5505 [WRN]      osd.85 unable to load:snappy
2022-12-13 07:50:00.001142 mon.ld5505 [WRN]      osd.86 unable to load:snappy
2022-12-13 07:50:00.001161 mon.ld5505 [WRN]      osd.87 unable to load:snappy
2022-12-13 07:50:00.001183 mon.ld5505 [WRN]      osd.88 unable to load:snappy
2022-12-13 07:50:00.001205 mon.ld5505 [WRN]      osd.89 unable to load:snappy
2022-12-13 07:50:00.001227 mon.ld5505 [WRN]      osd.90 unable to load:snappy
2022-12-13 07:50:00.001247 mon.ld5505 [WRN]      osd.91 unable to load:snappy
2022-12-13 07:50:00.001270 mon.ld5505 [WRN]      osd.92 unable to load:snappy
2022-12-13 07:50:00.001290 mon.ld5505 [WRN]      osd.93 unable to load:snappy
2022-12-13 07:50:00.001311 mon.ld5505 [WRN]      osd.94 unable to load:snappy
2022-12-13 07:50:00.001351 mon.ld5505 [WRN]      osd.95 unable to load:snappy
2022-12-13 07:50:00.001374 mon.ld5505 [WRN]      osd.96 unable to load:snappy
2022-12-13 07:50:00.001395 mon.ld5505 [WRN]      osd.97 unable to load:snappy
2022-12-13 07:50:00.001418 mon.ld5505 [WRN]      osd.98 unable to load:snappy
2022-12-13 07:50:00.001441 mon.ld5505 [WRN]      osd.99 unable to load:snappy
2022-12-13 07:50:00.001460 mon.ld5505 [WRN]      osd.100 unable to load:snappy
2022-12-13 07:50:00.001481 mon.ld5505 [WRN]      osd.101 unable to load:snappy
2022-12-13 07:50:00.001503 mon.ld5505 [WRN]      osd.102 unable to load:snappy
2022-12-13 07:50:00.001523 mon.ld5505 [WRN]      osd.103 unable to load:snappy
2022-12-13 07:50:00.001546 mon.ld5505 [WRN]      osd.104 unable to load:snappy
2022-12-13 07:50:00.001572 mon.ld5505 [WRN]      osd.105 unable to load:snappy
2022-12-13 07:50:00.001593 mon.ld5505 [WRN]      osd.106 unable to load:snappy
2022-12-13 07:50:00.001614 mon.ld5505 [WRN]      osd.107 unable to load:snappy
2022-12-13 07:50:00.001635 mon.ld5505 [WRN]      osd.108 unable to load:snappy
2022-12-13 07:50:00.001656 mon.ld5505 [WRN]      osd.109 unable to load:snappy
2022-12-13 07:50:00.001683 mon.ld5505 [WRN]      osd.110 unable to load:snappy
2022-12-13 07:50:00.001704 mon.ld5505 [WRN]      osd.111 unable to load:snappy
2022-12-13 07:50:00.001726 mon.ld5505 [WRN]      osd.112 unable to load:snappy
2022-12-13 07:50:00.001744 mon.ld5505 [WRN]      osd.113 unable to load:snappy
2022-12-13 07:50:00.001764 mon.ld5505 [WRN]      osd.114 unable to load:snappy
2022-12-13 07:50:00.001782 mon.ld5505 [WRN]      osd.115 unable to load:snappy
2022-12-13 07:50:00.001801 mon.ld5505 [WRN]      osd.116 unable to load:snappy
2022-12-13 07:50:00.001823 mon.ld5505 [WRN]      osd.117 unable to load:snappy
2022-12-13 07:50:00.001846 mon.ld5505 [WRN]      osd.118 unable to load:snappy
2022-12-13 07:50:00.001867 mon.ld5505 [WRN]      osd.119 unable to load:snappy
2022-12-13 07:50:00.001891 mon.ld5505 [WRN]      osd.120 unable to load:snappy
2022-12-13 07:50:00.001920 mon.ld5505 [WRN]      osd.121 unable to load:snappy
2022-12-13 07:50:00.001958 mon.ld5505 [WRN]      osd.122 unable to load:snappy
2022-12-13 07:50:00.001987 mon.ld5505 [WRN]      osd.123 unable to load:snappy

# pveversion -v
proxmox-ve: 6.4-1 (running kernel: 5.4.203-1-pve)
pve-manager: 6.4-15 (running version: 6.4-15/af7986e6)
pve-kernel-5.4: 6.4-20
pve-kernel-helper: 6.4-20
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.203-1-pve: 5.4.203-1
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.18-2-pve: 5.3.18-2
ceph: 14.2.22-pve1
ceph-fuse: 14.2.22-pve1
corosync: 3.1.5-pve2~bpo10+1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
libjs-extjs: 6.0.1-10
libknet1: 1.22-pve2~bpo10+1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.1.0-1
libpve-access-control: 6.4-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-5
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-5
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.14-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.6-2
pve-cluster: 6.4-1
pve-container: 3.3-6
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-4
pve-firmware: 3.3-2
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-8
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.3-pve1

# apt update
Hit:1 https://deb.debian.org/debian buster InRelease
Hit:2 https://deb.debian.org/debian-security buster/updates InRelease
Hit:3 https://deb.debian.org/debian buster-updates InRelease
Hit:4 https://deb.debian.org/debian buster-backports InRelease
Hit:5 http://download.proxmox.com/debian/ceph-nautilus buster InRelease
Hit:6 http://download.proxmox.com/debian/pve buster InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
2 packages can be upgraded. Run 'apt list --upgradable' to see them.

I'm running 7 storage nodes in this Ceph cluster, but the error is only reported by 1 node.

 

[cmonty14]

Issue fixed.

After restarting related service ceph-osd@<n>.service the warning is gone.