I used grok and found an answer
https://x.com/i/grok/share/755SwGDfCikCNnmErBObeTycw
this worked for me on one disk. 10 more to go.
In the latest Proxmox VE (version 9.1 as of December 2025), which typically integrates Ceph Squid (19.2.x), you can physically move an existing OSD disk from one server (node) to a new server in the same Ceph cluster without destroying the OSD or migrating data over the network. This preserves the data on the disk, as Ceph OSDs (using BlueStore and LVM) store their metadata, ID, and FSID directly on the disk.This method works best for individual OSDs or groups sharing no resources (e.g., dedicated WAL/DB). If your OSDs use shared devices, you may need to move multiple disks together or fall back to destroying and recreating.Important WarningsEnsure the new server is already joined to the Proxmox cluster and has Ceph services installed/configured (mon/mgr if needed, but at least OSD capabilities).
The cluster will temporarily go degraded while the OSD is down/out during the move.
Back up critical data and test in a non-production environment if possible.
Hot-swappable bays are ideal; otherwise, shut down the old node safely.
After the move, the OSD will appear under the new host in ceph osd tree. If not, you may need to manually adjust the CRUSH map (rare in recent versions).
Step-by-Step ProcessPrepare the OSD on the Old Node
In the Proxmox GUI: Go to Ceph → OSD → select the OSD → Stop it, then mark it Out (this drains data safely).
Alternatively via CLI:
ceph osd out osd.<ID>
Wait for the cluster to reach HEALTH_OK (or at least no active backfill/recovery if you're in a hurry, but recommended to wait).
Stop the OSD Service
On the old node (via shell):
systemctl stop ceph-osd@<ID>.service
Physically Move the Disk
Shut down the old node if not hot-swappable, remove the disk, insert it into the new server, and boot the new server if needed.
Activate the OSD on the New Node
On the new node (via shell in Proxmox):
First, scan for LVM changes:
pvscan --cache
vgscan
vgchange -ay
Then activate the OSD(s):
ceph-volume lvm activate --all
(This scans and activates all moved OSDs on the node.) If activating a specific one:
ceph-volume lvm activate --osd-id <ID> --osd-fsid <FSID>
(Get <ID> and <FSID> from ceph osd metadata osd.<ID> or the GUI before moving.) Enable and start the service:
systemctl enable --now ceph-osd@<ID>.service
Bring the OSD Back In
In the Proxmox GUI: Ceph → OSD → select the OSD (now on the new node) → mark it In.
Via CLI:
ceph osd in osd.<ID>
The cluster will recover any minor inconsistencies (usually quick since data is intact).
Verify ceph status → should go back to HEALTH_OK.
ceph osd tree → OSD should show under the new host.
Monitor for brief recovery/backfill.
If It Doesn't Auto-Start or Appear CorrectlyClean up old systemd units on the old node if the node remains in the cluster: systemctl disable ceph-osd@<ID>.service.
Rarely, manually move in CRUSH map: ceph osd crush move osd.<ID> host=<new-host-name>.
Alternative (Safer but Slower: Data Migration)If the above fails (e.g., incompatible hardware or shared journals), or you prefer no risk:Mark old OSD out → let Ceph migrate data to other OSDs.
Destroy old OSD.
Move disk to new node.
Create new OSD on the (now empty) disk via GUI or pveceph osd create /dev/sdX.
This causes full data rebalance (time-consuming and I/O heavy).
The direct move method is commonly used in Proxmox/Ceph communities and avoids unnecessary data copying. Refer to Proxmox forum threads or Ceph docs for your exact setup variations.
