[SOLVED] cephfs for isos wont mount. bad crc/signature.

[brad_mssw]

I'm trying to use cephfs for isos. Prior to proxmox v6, we just created a mount point with /etc/fstab:

none    /mnt/isos  fuse.ceph _netdev,defaults  0 0

That worked (and still does).

Now I noticed PVE6 has a cephfs option, so in trying to do that, I receive a timeout when trying to add the storage pool for isos in the UI.

Looking at /var/log/messages I see:

Oct 24 09:44:51 proxmox1 kernel: [ 1646.633076] libceph: mon2 10.55.30.43:6789 socket error on read
Oct 24 09:44:53 proxmox1 kernel: [ 1649.417803] libceph: mon1 10.55.30.42:6789 bad crc/signature
Oct 24 09:45:00 proxmox1 kernel: [ 1655.561284] libceph: mon2 10.55.30.43:6789 socket error on write
Oct 24 09:45:00 proxmox1 kernel: [ 1655.561897] libceph: mon1 10.55.30.42:6789 bad crc/signature
Oct 24 09:45:05 proxmox1 kernel: [ 1661.449821] libceph: mon3 10.55.30.41:6789 bad crc/signature
Oct 24 09:45:09 proxmox1 kernel: [ 1664.522317] libceph: mon0 10.55.30.41:6789 bad crc/signature
Oct 24 09:45:14 proxmox1 kernel: [ 1669.769474] libceph: mon2 10.55.30.43:6789 socket error on read
Oct 24 09:45:14 proxmox1 kernel: [ 1670.409841] libceph: mon0 10.55.30.41:6789 bad crc/signature
Oct 24 09:45:18 proxmox1 kernel: [ 1673.482244] libceph: mon1 10.55.30.42:6789 bad crc/signature
Oct 24 09:45:21 proxmox1 kernel: [ 1676.554113] libceph: mon3 10.55.30.41:6789 bad crc/signature
Oct 24 09:45:24 proxmox1 kernel: [ 1679.626050] libceph: mon0 10.55.30.41:6789 bad crc/signature
Oct 24 09:45:27 proxmox1 kernel: [ 1683.466149] libceph: mon3 10.55.30.41:6789 bad crc/signature
Oct 24 09:45:31 proxmox1 kernel: [ 1686.539180] libceph: mon1 10.55.30.42:6789 bad crc/signature
Oct 24 09:45:36 proxmox1 kernel: [ 1692.426266] libceph: mon3 10.55.30.41:6789 bad crc/signature
Oct 24 09:45:43 proxmox1 kernel: [ 1698.570359] libceph: mon0 10.55.30.41:6789 bad crc/signature
Oct 24 09:45:45 proxmox1 kernel: [ 1701.386863] libceph: mon1 10.55.30.42:6789 bad crc/signature

(ignore the socket errors, I have a mon down right now, its unrelated)

I do have signatures and authentication disabled in my ceph.conf for performance reasons:

 auth_client_required = none
 auth_cluster_required = none
 auth_service_required = none
 cephx_sign_messages = false
 cephx_require_signatures = false

So I'm assuming it is due to my config disabling signatures ... however, how do I let proxmox know this with its use of libceph? The fuse mount via fstab seems happy.

 

[brad_mssw]

I guess this is libceph in the kernel, I can reproduce it by:

mount.ceph ceph1,ceph2,ceph3:/ /mnt/isos

Looking at the kernel source code, I *should* be able to turn it off (and should by default by reading the config files):
https://git.kernel.org/pub/scm/linu...it/tree/net/ceph/ceph_common.c?h=v5.2.21#n279

However, it still doesn't work:

mount.ceph ceph1,ceph2,ceph3:/ /mnt/isos -o nocephx_sign_messages,nocephx_require_signatures,nocrc

 

[brad_mssw]

I should also mention I have

ms_crc_data = false
ms_crc_header = false

in my ceph.conf relating to disabling crcs

 

[brad_mssw]

After quite a bit of testing, setting

ms_crc_header = true

but leaving

auth_client_required = none
auth_cluster_required = none
auth_service_required = none
cephx_sign_messages = false
cephx_require_signatures = false
ms_crc_data = false

and my problem goes away.