Proxmox + Ceph - kernel: libceph: osd3 (1)192.168.1.212:6811 bad crc/signature

C

cableguy84

New Member
May 9, 2023
1
0
1
One of my hosts is going 'grey ?' and backups are not running (freezes on LXC indefinitely). A reboot solves the issue for a while, maybe until the next backup job, but haven't confirmed. I have made some changes to CEPH on x.x.x.212 a couple of days ago but all other nodes are fine. Cluster and ceph are all healthy except for this one node. (Status and logs below)


Log of errors:
journalctl -xe
Sep 08 16:58:31 acemagic-1 kernel: libceph: osd3 (1)192.168.1.212:6811 bad crc/signature
Sep 08 16:58:31 acemagic-1 kernel: libceph: read_partial_message 00000000ef285663 signature check failed
Sep 08 16:58:31 acemagic-1 kernel: libceph: osd4 (1)192.168.1.212:6819 bad crc/signature
Sep 08 16:58:31 acemagic-1 kernel: libceph: read_partial_message 00000000f2a8c138 signature check failed
Sep 08 16:58:31 acemagic-1 kernel: libceph: osd3 (1)192.168.1.212:6811 bad crc/signature
Sep 08 16:58:31 acemagic-1 kernel: libceph: read_partial_message 00000000ef285663 signature check failed
Sep 08 16:58:31 acemagic-1 kernel: libceph: osd4 (1)192.168.1.212:6819 bad crc/signature
Sep 08 16:58:31 acemagic-1 kernel: libceph: read_partial_message 00000000f2a8c138 signature check failed
Sep 08 16:58:31 acemagic-1 kernel: libceph: osd3 (1)192.168.1.212:6811 bad crc/signature
Sep 08 16:58:31 acemagic-1 kernel: libceph: read_partial_message 00000000ef285663 signature check failed
Sep 08 16:58:31 acemagic-1 kernel: libceph: osd4 (1)192.168.1.212:6819 bad crc/signature
Sep 08 16:58:31 acemagic-1 kernel: libceph: read_partial_message 00000000f2a8c138 signature check failed
Sep 08 16:58:31 acemagic-1 kernel: libceph: osd3 (1)192.168.1.212:6811 bad crc/signature
Sep 08 16:58:31 acemagic-1 kernel: libceph: read_partial_message 00000000ef285663 signature check failed
Sep 08 16:58:31 acemagic-1 kernel: libceph: osd4 (1)192.168.1.212:6819 bad crc/signature
Sep 08 16:58:31 acemagic-1 kernel: libceph: read_partial_message 00000000f2a8c138 signature check failed





Code: 
root@acemagic-1:~# systemctl status pve-cluster
● pve-cluster.service - The Proxmox VE cluster filesystem
     Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; preset: enabled)
     Active: active (running) since Sat 2024-09-07 11:11:22 EDT; 1 day 5h ago
   Main PID: 1172 (pmxcfs)
      Tasks: 8 (limit: 38096)
     Memory: 82.7M
        CPU: 2min 58.255s
     CGroup: /system.slice/pve-cluster.service
             └─1172 /usr/bin/pmxcfs

Sep 08 16:51:08 acemagic-1 pmxcfs[1172]: [status] notice: received log
Sep 08 16:51:24 acemagic-1 pmxcfs[1172]: [status] notice: received log
Sep 08 16:51:28 acemagic-1 pmxcfs[1172]: [status] notice: received log
Sep 08 16:51:28 acemagic-1 pmxcfs[1172]: [status] notice: received log
Sep 08 16:51:42 acemagic-1 pmxcfs[1172]: [status] notice: received log
Sep 08 16:51:50 acemagic-1 pmxcfs[1172]: [status] notice: received log
Sep 08 16:51:50 acemagic-1 pmxcfs[1172]: [status] notice: received log
Sep 08 16:51:52 acemagic-1 pmxcfs[1172]: [status] notice: received log
Sep 08 16:54:16 acemagic-1 pmxcfs[1172]: [status] notice: received log
Sep 08 16:54:16 acemagic-1 pmxcfs[1172]: [status] notice: received log
root@acemagic-1:~# systemctl status pvedaemon
● pvedaemon.service - PVE API Daemon
     Loaded: loaded (/lib/systemd/system/pvedaemon.service; enabled; preset: enabled)
     Active: active (running) since Sat 2024-09-07 11:11:24 EDT; 1 day 5h ago
   Main PID: 1407 (pvedaemon)
      Tasks: 9 (limit: 38096)
     Memory: 207.6M
        CPU: 50.662s
     CGroup: /system.slice/pvedaemon.service
             ├─  1407 pvedaemon
             ├─531828 "pvedaemon worker"
             ├─641523 "pvedaemon worker"
             ├─649924 "pvedaemon worker"
             ├─689458 "task UPID:acemagic-1:000A8532:00895FB8:66DDCCBA:vzstart:102:root@pam:"
             ├─689463 lxc-info -n 102 -p
             ├─689468 lxc-info -n 102 -p
             ├─689692 "task UPID:acemagic-1:000A861C:0089858D:66DDCD1B:vzstart:110:root@pam:"
             └─689738 lxc-info -n 110 -p

Notice: journal has been rotated since unit was started, output may be incomplete.
root@acemagic-1:~# systemctl status pvestatd
● pvestatd.service - PVE Status Daemon
     Loaded: loaded (/lib/systemd/system/pvestatd.service; enabled; preset: enabled)
     Active: active (running) since Sat 2024-09-07 11:11:23 EDT; 1 day 5h ago
    Process: 691073 ExecReload=/usr/bin/pvestatd restart (code=exited, status=0/SUCCESS)
   Main PID: 1362 (pvestatd)
      Tasks: 2 (limit: 38096)
     Memory: 157.2M
        CPU: 1h 45min 6.245s
     CGroup: /system.slice/pvestatd.service
             ├─  1362 pvestatd
             └─689488 lxc-info -n 102 -p

Notice: journal has been rotated since unit was started, output may be incomplete.
root@acemagic-1:~# pvecm status
Cluster information
-------------------
Name:             prmx-cluster-1
Config Version:   15
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Sun Sep  8 16:57:23 2024
Quorum provider:  corosync_votequorum
Nodes:            5
Node ID:          0x00000001
Ring ID:          1.1c2
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   5
Highest expected: 5
Total votes:      5
Quorum:           3
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 192.168.1.210 (local)
0x00000002          1 192.168.1.211
0x00000003          1 192.168.1.212
0x00000004          1 192.168.1.213
0x00000005          1 192.168.1.214
root@acemagic-1:~# ceph osd tree
ID  CLASS  WEIGHT    TYPE NAME            STATUS  REWEIGHT  PRI-AFF
-1         15.54070  root default
-5          1.25078      host acemagic-1
 1    ssd   0.31929          osd.1            up   1.00000  1.00000
 2    ssd   0.93149          osd.2            up   1.00000  1.00000
-7          2.18228      host acemagic-2
 5    ssd   1.71649          osd.5            up   1.00000  1.00000
 6    ssd   0.46579          osd.6            up   1.00000  1.00000
-3          3.16257      host minif-1
 0   nvme   0.36809          osd.0            up   1.00000  1.00000
 3    ssd   0.93149          osd.3            up   1.00000  1.00000
 4    ssd   1.86299          osd.4            up   1.00000  1.00000
-9          8.94507      host pmox-5700g
 7    ssd   3.63869          osd.7            up   1.00000  1.00000
 8    ssd   3.63869          osd.8            up   1.00000  1.00000
10    ssd   1.66769          osd.10           up   1.00000  1.00000
root@acemagic-1:~# ceph osd status
ID  HOST         USED  AVAIL  WR OPS  WR DATA  RD OPS  RD DATA  STATE
 0  minif-1     68.3G   308G      0      819       0        0   exists,up
 1  acemagic-1  51.0G   275G      0        0       0        0   exists,up
 2  acemagic-1   203G   749G      1     10.3k      0        0   exists,up
 3  minif-1      144G   809G      0      819       0        0   exists,up
 4  minif-1      262G  1644G      2     12.0k    291     1164k  exists,up
 5  acemagic-2   338G  1419G      1     8192       0        0   exists,up
 6  acemagic-2  92.5G   384G      0     1638       0        0   exists,up
 7  pmox-5700g   261G  3465G      3     71.1k      1        0   exists,up
 8  pmox-5700g   204G  3521G      3     15.1k      1        0   exists,up
10  pmox-5700g   114G  1593G      0     5734       0        0   exists,up
root@acemagic-1:~# ceph health detail
HEALTH_OK
root@acemagic-1:~#
 
@gurubert
can you please explain how the MTU size does affect this behavior?

I am running an 5 node Cluster with a similar problem. Currently the network interfaces for ceph communication are set to mtu 9000.
 
I was able to limit the problem to an Windows VM running Server 2016. Currently it is the only Windows Server 2016 guest on the Cluster. When i migrate the VM to a other node the error messages stop appearing on the source node and a few minutes after migration they start appearing on the node where the vm was migrated to.

Code: 
Error messages:
[Tue Sep 24 11:52:53 2024] libceph: read_partial_message 00000000b9092833 data crc 3861827710 != exp. 523469708
[Tue Sep 24 11:52:53 2024] libceph: read_partial_message 000000001241abf6 data crc 736895794 != exp. 4225455992
[Tue Sep 24 11:52:53 2024] libceph: osd65 (1)10.255.240.57:6829 bad crc/signature
[Tue Sep 24 11:52:53 2024] libceph: osd7 (1)10.255.240.55:6883 bad crc/signature
[Tue Sep 24 11:52:56 2024] libceph: read_partial_message 00000000679ba0e2 data crc 1092029545 != exp. 3462758503
[Tue Sep 24 11:52:56 2024] libceph: read_partial_message 0000000038485312 data crc 2156281791 != exp. 403258642
[Tue Sep 24 11:52:56 2024] libceph: osd28 (1)10.255.240.59:6907 bad crc/signature
[Tue Sep 24 11:52:56 2024] libceph: osd23 (1)10.255.240.57:6817 bad crc/signature
[Tue Sep 24 11:52:59 2024] libceph: read_partial_message 00000000fefdbd45 data crc 2584731311 != exp. 1834767282
[Tue Sep 24 11:52:59 2024] libceph: read_partial_message 000000008e3560cd data crc 1658764332 != exp. 3282153566
[Tue Sep 24 11:52:59 2024] libceph: osd23 (1)10.255.240.57:6817 bad crc/signature
[Tue Sep 24 11:52:59 2024] libceph: osd56 (1)10.255.240.59:6830 bad crc/signature

These are the vm settings:
Code: 
agent: 1
bios: ovmf
boot: order=scsi0;ide2
cores: 24
sockets: 1
cpu: host
efidisk0: vm-compression:vm-111-disk-0,efitype=4m,pre-enrolled-keys=1,size=528K
ide2: none,media=cdrom
machine: pc-q35-8.1
memory: 98304
meta: creation-qemu=8.1.5,ctime=1726566486
numa: 1
ostype: win10
scsi0: vm-compression:vm-111-disk-1,iothread=1,size=256002M
scsi1: vm-compression:vm-111-disk-2,iothread=1,size=3145732M
scsihw: virtio-scsi-single
 
Yes compression is enabled on the pool with following settings:
compression_mode: aggressive
compression_algorithm: lz4
 
Yes I am using KRBD for this pool. There is no LVM on top of it, so the VMs are using the rbd images as raw device.

The load of the machines in the cluster is currently really low. The Cluster is new and there are just a few VMs running on it.
There is under 5% CPU load on all nodes. All 5 nodes are direct connected without an switch between them. Every connect is using 2 25Gbit Nics in lacp bond mode. The network utilization is around 40 to 60 Mbit/s so the cluster network isn't under heavy load.

What exactly do you mean with "userspace RBD" and how can I switch to it?
 
KRBD is an option in the storage configuration of Proxmox. It can be enabled or disabled on the pool and affects all VMs stored there.

You would need to create a new pool in Ceph and a new storage on that pool in Proxmox without the KRBD setting.

After that you could migrate the VM image to the new storage and start the VM from there.
 
I have similar errors on proxmox 8.2.7, ceph 8.2.2
Sep 29 16:00:28 prox-c01-n00 kernel: libceph: osd1 (1)10.2.1.223:6811 bad crc/signature
Sep 29 16:01:00 prox-c01-n00 kernel: libceph: read_partial_message 00000000f1044cf2 data crc 2362059656 != exp. 3157053822
Sep 29 16:01:00 prox-c01-n00 kernel: libceph: read_partial_message 0000000074f3249f data crc 4031080028 != exp. 1213989762
Sep 29 16:01:00 prox-c01-n00 kernel: libceph: osd3 (1)10.2.1.224:6806 bad crc/signature
Sep 29 16:01:00 prox-c01-n00 kernel: libceph: osd4 (1)10.2.1.225:6805 bad crc/signature
Sep 29 16:01:00 prox-c01-n00 kernel: libceph: read_partial_message 00000000f1044cf2 data crc 314833572 != exp. 3157053822
Sep 29 16:01:00 prox-c01-n00 kernel: libceph: osd3 (1)10.2.1.224:6806 bad crc/signature

unchecking KRBD at the datacenter storage level for the ceph pool stops the errors.
 
zrbite said:
I have similar errors on proxmox 8.2.7, ceph 8.2.2
Sep 29 16:00:28 prox-c01-n00 kernel: libceph: osd1 (1)10.2.1.223:6811 bad crc/signature
Sep 29 16:01:00 prox-c01-n00 kernel: libceph: read_partial_message 00000000f1044cf2 data crc 2362059656 != exp. 3157053822
Sep 29 16:01:00 prox-c01-n00 kernel: libceph: read_partial_message 0000000074f3249f data crc 4031080028 != exp. 1213989762
Sep 29 16:01:00 prox-c01-n00 kernel: libceph: osd3 (1)10.2.1.224:6806 bad crc/signature
Sep 29 16:01:00 prox-c01-n00 kernel: libceph: osd4 (1)10.2.1.225:6805 bad crc/signature
Sep 29 16:01:00 prox-c01-n00 kernel: libceph: read_partial_message 00000000f1044cf2 data crc 314833572 != exp. 3157053822
Sep 29 16:01:00 prox-c01-n00 kernel: libceph: osd3 (1)10.2.1.224:6806 bad crc/signature

unchecking KRBD at the datacenter storage level for the ceph pool stops the errors.
Click to expand...
There are some older discussions about KRBD and needing "rxbounce" for Windows Guests over there in redhat....
https://bugzilla.redhat.com/show_bug.cgi?id=2109455
And something in CEPH-Docs
https://docs.ceph.com/en/reef/man/8/rbd/ (rxbounce)


Maybe it has something to do with this....
 
Last edited:
  • Like
Reactions: gurubert
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back