cephfs - bad crc/signature

[brosky]

Hi,

While copying data to a cephfs mounted folder , I encounter tons of messages like this:

[52122.119364] libceph: osd34 (1)10.0.40.95:6881 bad crc/signature
[52122.119599] libceph: osd9 (1)10.0.40.98:6801 bad crc/signature
[52122.119722] libceph: read_partial_message 00000000a1a2de9f signature check failed
[52122.119993] libceph: osd83 (1)10.0.40.38:6826 bad crc/signature
[52122.121751] libceph: read_partial_message 00000000e45aca08 signature check failed
[52122.122034] libceph: osd60 (1)10.0.40.37:6814 bad crc/signature
[52122.127937] libceph: read_partial_message 00000000e11eff1b signature check failed
[52122.127939] libceph: read_partial_message 00000000bba65e30 signature check failed
[52122.127941] libceph: read_partial_message 000000000445f988 signature check failed
[52122.127941] libceph: read_partial_message 00000000e873cd06 signature check failed
[52122.127948] libceph: osd8 (1)10.0.40.98:6819 bad crc/signature
[52122.127949] libceph: osd78 (1)10.0.40.38:6865 bad crc/signature
[52122.128044] libceph: read_partial_message 00000000dd903166 signature check failed
[52122.128054] libceph: osd38 (1)10.0.40.93:6816 bad crc/signature
[52122.128258] libceph: osd13 (1)10.0.40.97:6809 bad crc/signature
[52122.128521] libceph: osd33 (1)10.0.40.95:6873 bad crc/signature
[52122.128529] libceph: read_partial_message 00000000d5312939 signature check failed
[52122.128795] libceph: read_partial_message 0000000056365bbb signature check failed
[52122.129008] libceph: osd66 (1)10.0.40.37:6804 bad crc/signature
[52122.129251] libceph: osd17 (1)10.0.40.97:6841 bad crc/signature

Networking - the copy goes with 30-40Mbps but the backend network is up to 40Gbps.
This issue is only for CephFS - if I copy to a s3 bucket, I have no issues.

Stopping the transfers does not stop the network traffic - only a whole server reboot "solves" the issue.

Any ideas ?

 

[mira]

Please provide the output of pveversion -v and ceph versions.

 

[brosky]

Sure:

# pveversion -v
proxmox-ve: 7.2-1 (running kernel: 5.15.39-1-pve)
pve-manager: 7.2-7 (running version: 7.2-7/d0dd0e85)
pve-kernel-5.15: 7.2-6
pve-kernel-helper: 7.2-6
pve-kernel-5.15.39-1-pve: 5.15.39-1
ceph: 16.2.9-pve1
ceph-fuse: 16.2.9-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-3
libpve-storage-perl: 7.2-7
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.4-1
proxmox-backup-file-restore: 2.2.4-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-2
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.5-1
pve-ha-manager: 3.3-4
pve-i18n: 2.7-2
pve-qemu-kvm: 6.2.0-11
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.4-pve1
root@pve299:~#

ceph:

~# ceph versions
{
    "mon": {
        "ceph version 16.2.9 (a569859f5e07da0c4c39da81d5fb5675cd95da49) pacific (stable)": 3
    },
    "mgr": {
        "ceph version 16.2.9 (a569859f5e07da0c4c39da81d5fb5675cd95da49) pacific (stable)": 3
    },
    "osd": {
        "ceph version 16.2.9 (a569859f5e07da0c4c39da81d5fb5675cd95da49) pacific (stable)": 82
    },
    "mds": {
        "ceph version 16.2.9 (a569859f5e07da0c4c39da81d5fb5675cd95da49) pacific (stable)": 4
    },
    "rgw": {
        "ceph version 16.2.9 (a569859f5e07da0c4c39da81d5fb5675cd95da49) pacific (stable)": 8
    },
    "overall": {
        "ceph version 16.2.9 (a569859f5e07da0c4c39da81d5fb5675cd95da49) pacific (stable)": 100
    }
}

 

[mira]

I thought those messages were fixed after kernel 5.11, but it seems that's still not the case.
Usually those are nothing to worry about as it only affects reads and those reads are simply done again.

These can also affect iSCSI.

Regarding the performance, where and how do you copy the files?

 

[brosky]

Hi Mira,

Thank you for looking on this.

The copy is done via rsync from a TrueNAS server to the pve node, directly on the /mnt/pve/cephfs folder, on a different server.
So I have members in ceph cluster A,B,C,D,E,F - I start copy in server A and server F starts to throw bad/crc signatures.
Upload speed is at 5-700Mbps let's say however the Ceph backend goes crazy at 30Gbps.



Even if I stop the transfers, the network traffic (tcpdump shows osd chatter between nodes) stays very high.

If you need more details please let me know, I will happily help

 

[mira]

All data written requires 3 (or what you've configured) replicas. This together with cache might lead to writes lasting a lot longer than simply the network transfer.

Do you use HDDs, SSDs or NVMes?

 

[brosky]

Hi Mira,

All hdds have nvme partitions set as bluestore cache (16tb disk with 164g nvme bluestore )
I know recommended size is 4%, others suggest that due to how bluestore bucket mechanism works, 30gb will be enough... So I put 1%

For cephfs I've put replica 2, considering the data value.

 

[mira]

Network traffic on the cluster network is to be expected. Ceph scrubs PGs periodically and does other things. And with all data being written at least twice, it will also lead to additional traffic.