Proxmox VE 9.2.21 – Ceph OSD Creation Fails with "Error EINVAL: invalid cephx secret" on Newly Added Cluster Nodes

[linuxxx3250]

Hi Team,

We are experiencing an issue while creating Ceph OSDs on newly added nodes in our existing Proxmox cluster.

Environment:

• Existing cluster: 5 nodes running Proxmox VE 9.2.2
• Newly added nodes: 3 nodes running Proxmox VE 9.2.21
• Total cluster nodes: 8
• Affected node: XX-XX-vmhost107
• OSD disk: /dev/sdb
• Storage: Ceph (BlueStore)
  
  We recently built three new Proxmox hosts running version 9.2.21 and successfully joined them to our existing five-node cluster running version 9.2.2.
  
  But when attempting to create Ceph OSDs on the newly added hosts, the operation fails with the following error:
  
  Error logs:
  
  create OSD on /dev/sdb (bluestore)
  wiping block device /dev/sdb
  200+0 records in
  200+0 records out
  209715200 bytes (210 MB, 200 MiB) copied, 0.10227 s, 2.1 GB/s
  
  Running command: /bin/ceph-authtool --gen-print-key
  Running command: /bin/ceph-authtool --gen-print-key
  Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 44c37adb-4a43-4524-93e6-be6bcad2f75e
  
  stderr: Error EINVAL: invalid cephx secret.
  --> RuntimeError: Unable to create a new OSD id
  
  TASK ERROR: command 'ceph-volume lvm create --data /dev/sdb' failed: exit code 1
  
  Issue:
  
  The new nodes joined the existing Proxmox cluster successfully, but Ceph OSD creation fails with Error EINVAL: invalid cephx secret.
  
  We would appreciate guidance on the following:
  
  1. Could this issue be related to the Proxmox VE version difference between the existing nodes (9.2.2) and newly added nodes (9.2.21)?
  2. Is there any Ceph authentication or keyring configuration that needs to be updated when adding new nodes to an existing cluster?
  3. Are there any recommended troubleshooting steps or commands to resolve this issue without impacting the existing Ceph OSDs and production workloads?
     
     Please advise on the safest approach to resolve this issue.
     
     Thanks in advance for your support.