CEPH OSD(s) failing to initialize after a hardware change

ResidentDataHoarder · 2026-05-26T02:07:09+0200

# Overveiw of my Configuration
I am on the latest version of `Proxmox` 7 nodes total, 3 dedicated to just host `CEPH`, no `VM` or `LXC`, I have 3 `CEPH` pools divided int devices class; `HDD`, `SSD`, `NVME`. The one I need help with is the `HDD` pool in a `2/3` configuration and where I do my bulk storage via `unprivileged` passthrough over `cephfs`.
The boot SSD can be ignored completely and the NVME device can be ignored for now if they show up in any of the outputs I post.

# Overview of my issue I am trying to resolve
Been stuck running in circles trying to get one of my CEPH nodes back online. Been at it for a few days between prompting an LLM, online communities, and hardware troubleshooting.

The TL;DR of my issue is as follows
1 of my nodes from a ceph 2/3 configuration isn't picking the drives in `LVM` and `ceph` fails to set them to `UP` after a hardware change. They do, however show up in `lsblk`

The full details of my issue is as follows
A node failed to power back on after a routine update. Narrowed it down to either the motherboard or PSU. I went for the motherboard first since for the PSU; I don't know if it was the redundant modules dying at the same time (highly unlikely for anyone who doesn't have my bad luck), the the power distribution board they slot into, or possibly faulty cables to the motherboard/CPU. Turned out motherboard was the wrong choice, since I wasn't sure which part of the PSU was the issue, I just bought a new (used) chassis, and shifted my mentality from its wasting money to I need to add a fourth node in the future anyways to make it a `3/4` instead of a `2/3`, I'll just hang onto it for that. I decided to keep the new motherboard in, one other thing that changed, instead of running 4 `SFF-8643` from each `HBA` to each backplane, this chassis came with the original wiring, which only ran 2 from the `HBA` to the backplane (I think they split IIRC), mainly because those chassis are a pain in the ass to rewire. I kept both `HBA`'s though and split the backplanes 2 - 2 between the 2 `HBA`'s

# The meat and potatos
what I've found on my own so far:

[B]blkid[/B] shows the devices
Here is the output of blkid

Here is the output of `lsblk -o NAME,SIZE,TYPE,FSTYPE,PARTLABEL,UUID

dmsetup table "<device-id>" output

Bash:

device-mapper: table ioctl on ceph-b9e1fb6c-1e20-4db7-8527-a1ee295d7489  failed: No such device or address
Command failed.

[B]ceph-volume lvm list || true[/B] only shows the 1 NVME device

Bash:

====== osd.46 ======


  [block]       /dev/ceph-778c730e-e642-4c96-82ce-2fdb2e4f4082/osd-block-34d76c55-f2f6-4225-b03a-11b1a44a8e4a


      block device              /dev/ceph-778c730e-e642-4c96-82ce-2fdb2e4f4082/osd-block-34d76c55-f2f6-4225-b03a-11b1a44a8e4a

      block uuid                KcdEnZ-Wwhs-aS7X-Csob-pBNh-bQP6-6juMo1

      cephx lockbox secret     

      cluster fsid              50824a88-c4d8-40e9-aa4f-5e10c2b7c5d2

      cluster name              ceph

      crush device class        nvme

      encrypted                 0

      osd fsid                  34d76c55-f2f6-4225-b03a-11b1a44a8e4a

      osd id                    46

      osdspec affinity         

      type                      block

      vdo                       0

      with tpm                  0

      devices                   /dev/nvme0n1

I think what I need to is restore LVM from backup using

Bash:

pvcreate --force --uuid "<UUID>" --restorefile /etc/lvm/backup/<ceph-id> /dev/sdc

However I get this error as an output:

Bash:

  WARNING: Couldn't find device with uuid vg0wUM-EXBK-Uh6K-T3xK-K7P3-q2ld-3SyTb7.

  Can't initialize physical volume "/dev/sdc" of volume group "ceph-b9e1fb6c-1e20-4db7-8527-a1ee295d7489" without -ff

  /dev/sdc: physical volume not initialized.

Basically the SAS3 drives associated with CEPH are refusing to initialize for some reason, and I'm not sure why or how to get them to initialize. Any help would be massively appreciated. There is definitely some other things I've tried that I've left out, but everything I've tried so far is a jumbled mess so I apologize for that in advance, it's just been very stressful trying to resolve this issue since I don't have access to anything on there, my stress outlets like my media server are just down until I get it back up and running.

Nemesiz · 2026-05-26T11:53:08+0200

Example: I unplugged wrong OSD disk, how to reactivate it?

1. Stop OSD process ( systemctl stop ceph-osd@XX ) and unmount OSD dir ( umount /var/lib/ceph/osd/ceph-XX )

2. If OSD use encryption I need to close crypto mapper
cryptsetup close /dev/mapper/*mapper id*
cryptsetup close ceph-*LVM ID*-osd--block-*ID*

3. Let LVM scan disk for changes: vgchange -ay

4. If LVM activated volume group I needed then I need to activate OSD: systemctl start ceph-volume@lvm-XX-*ID*

Technically if you move OSD to new server (replacing) you only need to create symlinks in /etc/systemd/system/multi-user.target.wants for new OS.

Example: ceph-volume@lvm-20-2ce5b7c6-08fc-4153-bdd7-a37792a3e51a.service -> /lib/systemd/system/ceph-volume@.service

And enable services. I assume you run on the same OS and all needed systemd service links are in place.
Start from the bottom: Does LVM see and activated volume (pvdisplay)?

p.s. All those LVM, crypto, OSD ID is miserable confusion so I have list of all IDs map of hdd -> crypto -> OSD

ResidentDataHoarder · 2026-05-26T13:20:02+0200

Nemesiz said:
Example: I unplugged wrong OSD disk, how to reactivate it?

1. Stop OSD process ( systemctl stop ceph-osd@XX ) and unmount OSD dir ( umount /var/lib/ceph/osd/ceph-XX )

2. If OSD use encryption I need to close crypto mapper
cryptsetup close /dev/mapper/*mapper id*
cryptsetup close ceph-*LVM ID*-osd--block-*ID*

3. Let LVM scan disk for changes: vgchange -ay

4. If LVM activated volume group I needed then I need to activate OSD: systemctl start ceph-volume@lvm-XX-*ID*

Technically if you move OSD to new server (replacing) you only need to create symlinks in /etc/systemd/system/multi-user.target.wants for new OS.

Example: ceph-volume@lvm-20-2ce5b7c6-08fc-4153-bdd7-a37792a3e51a.service -> /lib/systemd/system/ceph-volume@.service

And enable services. I assume you run on the same OS and all needed systemd service links are in place.
Start from the bottom: Does LVM see and activated volume (pvdisplay)?

p.s. All those LVM, crypto, OSD ID is miserable confusion so I have list of all IDs map of hdd -> crypto -> OSD

Hello, and thank you for the reply!

1. Stop OSD process ( systemctl stop ceph-osd@XX ) and unmount OSD dir ( umount /var/lib/ceph/osd/ceph-XX )
As mentioned the device exists, the kernel can see it with lsblk, but LVM wont initialize the ceph volume:

Code:

root@pve5:~# lsblk
NAME                                                                                                  MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda                                                                                                     8:0    0 119.2G  0 disk
|-sda1                                                                                                  8:1    0  1007K  0 part
|-sda2                                                                                                  8:2    0     1G  0 part
`-sda3                                                                                                  8:3    0 118.2G  0 part
sdb                                                                                                     8:16   0 119.2G  0 disk
|-sdb1                                                                                                  8:17   0  1007K  0 part
|-sdb2                                                                                                  8:18   0     1G  0 part
`-sdb3                                                                                                  8:19   0 118.2G  0 part
sdc                                                                                                     8:32   0  10.9T  0 disk
`-ceph--b9e1fb6c--1e20--4db7--8527--a1ee295d7489-osd--block--aacdc12a--2f0f--4a75--8542--9a613c0f8d26 252:11   0  10.9T  0 lvm 
sdd                                                                                                     8:48   0  10.9T  0 disk
`-ceph--095653a2--d894--4713--a98b--77439400987d-osd--block--43df5e31--450f--4914--bb80--078628e85f1f 252:3    0  10.9T  0 lvm 
sde                                                                                                     8:64   0  10.9T  0 disk
sdf                                                                                                     8:80   0  10.9T  0 disk
sdg                                                                                                     8:96   0  10.9T  0 disk
`-ceph--7af30a0d--fba9--48fd--ae7d--6c7ee2d96461-osd--block--8a5813b8--c948--4e14--9861--af917047ab34 252:2    0  10.9T  0 lvm 
sdh                                                                                                     8:112  0  10.9T  0 disk
`-ceph--11d04bdd--1f69--4be7--8fc9--e0c28fb72d11-osd--block--a9b3c0d5--c437--4743--aa6f--64205d8b8633 252:7    0  10.9T  0 lvm 
sdi                                                                                                     8:128  0  10.9T  0 disk
`-ceph--d8049b27--52a7--4478--85fe--117cf8cba9d2-osd--block--31c0f65e--ad51--46df--931f--483c6c0324aa 252:12   0  10.9T  0 lvm 
sdj                                                                                                     8:144  0  10.9T  0 disk
`-ceph--3fc05ed9--59d8--49e3--bc8d--298c9a0e9b23-osd--block--b1432435--d44c--4c2b--b796--54faa4d257e7 252:8    0  10.9T  0 lvm 
sdk                                                                                                     8:160  0  10.9T  0 disk
`-ceph--7deda356--de11--4742--a773--71980f6bd84a-osd--block--0d4a4621--f087--4c1c--97f6--753b88427e5c 252:6    0  10.9T  0 lvm 
sdl                                                                                                     8:176  0  10.9T  0 disk
`-ceph--2db90f9e--74de--46a0--aaa9--7ade4746c96f-osd--block--6995b8b2--8d94--47f0--becf--820dac70e766 252:13   0  10.9T  0 lvm 
sdm                                                                                                     8:192  0  10.9T  0 disk
`-ceph--a972216d--80d7--48f1--8d85--a2a9df5b9261-osd--block--4576958b--03f6--49cd--ab0c--405b1f942c7b 252:14   0  10.9T  0 lvm 
sdn                                                                                                     8:208  0  10.9T  0 disk
sdo                                                                                                     8:224  0  10.9T  0 disk
`-ceph--d402377f--ad20--4b78--a64f--3dab06e56717-osd--block--96dbfa75--1f2d--459b--9bfe--e23332b3ce2a 252:10   0  10.9T  0 lvm 
sdp                                                                                                     8:240  0  10.9T  0 disk
`-ceph--a20745fa--d748--4c33--acc0--01361d6cbda4-osd--block--03728b9e--d559--4fcc--a7b9--1bc7abb03380 252:9    0  10.9T  0 lvm 
sdq                                                                                                    65:0    0  10.9T  0 disk
`-ceph--6a73260a--8b83--427c--a953--98de36a78060-osd--block--bdaeb37b--0491--46c5--b05c--a8770a4964c0 252:5    0  10.9T  0 lvm 
sdr                                                                                                    65:16   0  10.9T  0 disk
sds                                                                                                    65:32   0  10.9T  0 disk
sdt                                                                                                    65:48   0  10.9T  0 disk
`-ceph--74f4d41b--17bd--40fc--85db--cc911a9c62e3-osd--block--d67facbc--a33a--47ad--9149--95b29552bf50 252:1    0  10.9T  0 lvm 
sdu                                                                                                    65:64   0  10.9T  0 disk
`-ceph--5909443b--69ac--4d66--899c--f6d7345db3e5-osd--block--2f7003ec--e2cb--4ce6--8420--ca67c196bf8f 252:4    0  10.9T  0 lvm 
sdv                                                                                                    65:80   0  10.9T  0 disk
sdw                                                                                                    65:96   0  10.9T  0 disk
sdx                                                                                                    65:112  0  10.9T  0 disk
sdy                                                                                                    65:128  0  10.9T  0 disk
sdz                                                                                                    65:144  0  10.9T  0 disk
sdaa                                                                                                   65:160  0  10.9T  0 disk
sdab                                                                                                   65:176  0  10.9T  0 disk
sdac                                                                                                   65:192  0  10.9T  0 disk
`-ceph--2662cb97--3c87--49fc--9ceb--8eac077054b2-osd--block--caaf0c96--e90e--46b8--9a49--cf84260b94e4 252:15   0  10.9T  0 lvm 
sdad                                                                                                   65:208  0  10.9T  0 disk
sdae                                                                                                   65:224  0  10.9T  0 disk
sdaf                                                                                                   65:240  0  10.9T  0 disk
nvme0n1                                                                                               259:0    0 894.3G  0 disk
`-ceph--778c730e--e642--4c96--82ce--2fdb2e4f4082-osd--block--34d76c55--f2f6--4225--b03a--11b1a44a8e4a 252:0    0 894.3G  0 lvm

Notee that sda, sdb, and nvme can be ignored.

Bash:

root@pve5:~# umount /var/lib/ceph/osd/ceph-8
umount: /var/lib/ceph/osd/ceph-8: not mounted.

~~2. If OSD use encryption I need to close crypto mapper~~
[S]cryptsetup close /dev/mapper/*mapper id*[/S]
~~cryptsetup close ceph-*LVM ID*-osd--block-*ID*~~
Not using encryption through ceph so this can be skipped I assume.

3. Let LVM scan disk for changes: vgchange -ay
Only thing that shows up is the NVME device

4. If LVM activated volume group I needed then I need to activate OSD: systemctl start ceph-volume@lvm-XX-*ID*
I am under the assumption I can not proceed to this step since the volume is not being activated?

Technically if you move OSD to new server (replacing) you only need to create symlinks in /etc/systemd/system/multi-user.target.wants for new OS.

Example: [EMAIL]ceph-volume@lvm-20-2ce5b7c6-08fc-4153-bdd7-a37792a3e51a.service[/EMAIL] -> /lib/systemd/system/ceph-volume@.service

They do show up under /dev/mapper

however systemd service fails with exit code 1, it's same across the board, just using OSD 8 as an example:

ResidentDataHoarder · 2026-05-26T13:34:13+0200

Sorry I kept getting an error when trying to reply, which made me post the same reply 4-5 times when things caught up.

Nemesiz · 2026-05-26T14:11:39+0200

Which OSD device do you want to wake up ? sdc ?

ResidentDataHoarder · 2026-05-26T15:13:52+0200

Nemesiz said:
Which OSD device do you want to wake up ? sdc ?

All of them, everything on that node is an OSD, except for the ZFS boot pool. The NVME OSD device is the only one showing up with ceph-volume lvm list though.

ResidentDataHoarder · 2026-05-26T15:28:36+0200

For comparison, this is from a node where everything is working as it should:

Here is an example of the node that is not initializing the OSD's:

Nemesiz · 2026-05-26T16:33:00+0200

Maybe something is holding them from LVM activation. Could you boot host with unplugged not active HDD ? After that I would plug single HDD and look at the logs to understand what happens with LVM detection.

ResidentDataHoarder · 2026-05-26T19:14:11+0200

Nemesiz said:
Maybe something is holding them from LVM activation. Could you boot host with unplugged not active HDD ? After that I would plug single HDD and look at the logs to understand what happens with LVM detection.

That's a really good Idea, I didn't think to do that! I just got back from an errand, I'll try that now!

ResidentDataHoarder · 2026-05-26T19:36:35+0200

ResidentDataHoarder said:
For comparison, this is from a node where everything is working as it should:
View attachment 97998
View attachment 97997Here is an example of the node that is not initializing the OSD's:
View attachment 97999
View attachment 98001

Nemesiz said:
Maybe something is holding them from LVM activation. Could you boot host with unplugged not active HDD ? After that I would plug single HDD and look at the logs to understand what happens with LVM detection.

Here's what was in the logs right after plugging in the drive using journalctl --since "5 minutes ago":

Code:

May 26 12:30:14 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)

May 26 12:30:14 pve5 kernel: START_UNIT: handle(0x0027), lun(0)

May 26 12:30:14 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)

May 26 12:30:15 pve5 kernel: mpt3sas_cm0: detecting: handle(0x0027),

                             sas_address(0x5000cca27a3838c9), phy(5)

May 26 12:30:15 pve5 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0027),

                             retries(0)

May 26 12:30:15 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)

May 26 12:30:15 pve5 kernel: mpt3sas_cm0: handle(0x27) sas_address(0x5000cca27a3838c9) port_type(0x1)

May 26 12:30:15 pve5 kernel: scsi 8:0:2:0: Direct-Access     HGST     HUH721212AL4200  AB01 PQ: 0 ANSI: 6

May 26 12:30:15 pve5 kernel: scsi 8:0:2:0: SSP: handle(0x0027), sas_addr(0x5000cca27a3838c9), phy(5), device_name(0x5000cca27a3838cb)

May 26 12:30:15 pve5 kernel: scsi 8:0:2:0: enclosure logical id (0x50030480186998ff), slot(5)

May 26 12:30:15 pve5 kernel: scsi 8:0:2:0: enclosure level(0x0001), connector name(     )

May 26 12:30:15 pve5 kernel: scsi 8:0:2:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)

May 26 12:30:15 pve5 kernel: sd 8:0:2:0: Attached scsi generic sg6 type 0

May 26 12:30:15 pve5 kernel: sd 8:0:2:0: [sdc] 2929721344 4096-byte logical blocks: (12.0 TB/10.9 TiB)

May 26 12:30:15 pve5 kernel: sd 8:0:2:0: [sdc] Write Protect is off

May 26 12:30:15 pve5 kernel:  end_device-8:1:1: add: handle(0x0027), sas_addr(0x5000cca27a3838c9)

May 26 12:30:15 pve5 kernel: sd 8:0:2:0: [sdc] Mode Sense: f7 00 10 08

May 26 12:30:15 pve5 kernel: sd 8:0:2:0: [sdc] Write cache: disabled, read cache: enabled, supports DPO and FUA

May 26 12:30:15 pve5 kernel: sd 8:0:2:0: [sdc] Attached SCSI disk

May 26 12:30:15 pve5 kernel: mpt3sas_cm1: detecting: handle(0x0027),

                             sas_address(0x5000cca27a3838c9), phy(5)

May 26 12:30:15 pve5 kernel: mpt3sas_cm1: REPORT_LUNS: handle(0x0027),

                             retries(0)

May 26 12:30:15 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)

May 26 12:30:15 pve5 kernel: mpt3sas_cm1: handle(0x27) sas_address(0x5000cca27a3838c9) port_type(0x1)

May 26 12:30:15 pve5 kernel: scsi 9:0:2:0: Direct-Access     HGST     HUH721212AL4200  AB01 PQ: 0 ANSI: 6

May 26 12:30:15 pve5 kernel: scsi 9:0:2:0: SSP: handle(0x0027), sas_addr(0x5000cca27a3838c9), phy(5), device_name(0x5000cca27a3838cb)

May 26 12:30:15 pve5 kernel: scsi 9:0:2:0: enclosure logical id (0x50030480186998ff), slot(5)

May 26 12:30:15 pve5 kernel: scsi 9:0:2:0: enclosure level(0x0000), connector name(     )

May 26 12:30:15 pve5 kernel: scsi 9:0:2:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)

May 26 12:30:15 pve5 kernel: sd 9:0:2:0: Attached scsi generic sg7 type 0

May 26 12:30:15 pve5 kernel:  end_device-9:0:2: add: handle(0x0027), sas_addr(0x5000cca27a3838c9)

May 26 12:30:15 pve5 kernel: sd 9:0:2:0: [sdd] 2929721344 4096-byte logical blocks: (12.0 TB/10.9 TiB)

May 26 12:30:15 pve5 kernel: sd 9:0:2:0: [sdd] Write Protect is off

May 26 12:30:15 pve5 kernel: sd 9:0:2:0: [sdd] Mode Sense: f7 00 10 08

May 26 12:30:15 pve5 kernel: sd 9:0:2:0: [sdd] Write cache: disabled, read cache: enabled, supports DPO and FUA

May 26 12:30:15 pve5 lvm[10941]: PV /dev/sdc online, VG ceph-5909443b-69ac-4d66-899c-f6d7345db3e5 is complete.

May 26 12:30:15 pve5 kernel: sd 9:0:2:0: [sdd] Attached SCSI disk

May 26 12:30:15 pve5 lvm[10945]: PV /dev/sdd 8:48 is duplicate for PVID gTYMNVgYkgYtkGTxt7DjUT2eOFf1mAZ7 on 8:32 /dev/sdc.

May 26 12:30:15 pve5 lvm[10945]: PV /dev/sdd failed to create online file.

May 26 12:30:15 pve5 systemd[1]: Started lvm-activate-ceph-5909443b-69ac-4d66-899c-f6d7345db3e5.service - [systemd-run] /usr/sbin/lvm vgchange -aay --autoactivation event ceph-5909443b-69ac-4d66-899c-f6d7345db3e5.

May 26 12:30:15 pve5 lvm[10948]:   1 logical volume(s) in volume group "ceph-5909443b-69ac-4d66-899c-f6d7345db3e5" now active

May 26 12:30:15 pve5 systemd[1]: lvm-activate-ceph-5909443b-69ac-4d66-899c-f6d7345db3e5.service: Deactivated successfully.

Particularly this:

Code:

May 26 12:30:15 pve5 lvm[10945]: PV /dev/sdd 8:48 is duplicate for PVID gTYMNVgYkgYtkGTxt7DjUT2eOFf1mAZ7 on 8:32 /dev/sdc.


May 26 12:30:15 pve5 lvm[10945]: PV /dev/sdd failed to create online file.

makes me feel like its a mapping issue no?

ResidentDataHoarder · 2026-05-26T19:58:45+0200

Is there a way to remap everything correctly without the nightmare of doing them one by one?

Nemesiz · 2026-05-26T20:48:59+0200

Need to figure out why it say dublicate. What pvdisplay lvdisplay vgdisplay show?

ResidentDataHoarder · 2026-05-26T21:31:45+0200

Nemesiz said:
Need to figure out why it say dublicate. What pvdisplay lvdisplay vgdisplay show?

As mentioned those commands can't see anything but the NVME device. The other devices are recognized by the kernel, since lsblk can list them. To my limited knowledge; I think and am leaning towards the issue seeming to be volume mapping related. But I don't know a sane, or non destructive way to fix that without wiping and re-adding the OSDs or attempting to remap them one by one which seems tedious and not fun...

Code:

root@pve5:~# pvdisplay                  
  --- Physical volume ---
  PV Name               /dev/nvme0n1
  VG Name               ceph-778c730e-e642-4c96-82ce-2fdb2e4f4082
  PV Size               894.25 GiB / not usable <3.34 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              228928
  Free PE               0
  Allocated PE          228928
  PV UUID               ff9ut5-wZHj-Ubkw-e5I8-1Gd9-RPD5-GxIk71
 
root@pve5:~# lvdisplay        
  --- Logical volume ---
  LV Path                /dev/ceph-778c730e-e642-4c96-82ce-2fdb2e4f4082/osd-block-34d76c55-f2f6-4225-b03a-11b1a44a8e4a
  LV Name                osd-block-34d76c55-f2f6-4225-b03a-11b1a44a8e4a
  VG Name                ceph-778c730e-e642-4c96-82ce-2fdb2e4f4082
  LV UUID                KcdEnZ-Wwhs-aS7X-Csob-pBNh-bQP6-6juMo1
  LV Write Access        read/write
  LV Creation host, time pve5, 2026-02-25 20:09:47 -0600
  LV Status              available
  # open                 0
  LV Size                894.25 GiB
  Current LE             228928
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           252:0
 
root@pve5:~# vgdisplay
  --- Volume group ---
  VG Name               ceph-778c730e-e642-4c96-82ce-2fdb2e4f4082
  System ID            
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  5
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               894.25 GiB
  PE Size               4.00 MiB
  Total PE              228928
  Alloc PE / Size       228928 / 894.25 GiB
  Free  PE / Size       0 / 0  
  VG UUID               FOm1qo-sn6i-3s9C-VEuC-pKbT-StnN-cAPcYA
 
root@pve5:~#

Nemesiz · 2026-05-27T08:17:30+0200

Another try: boot host with no OSD disk and then put single HDD OSD to see does it up.

ResidentDataHoarder · 2026-05-27T14:18:29+0200

Nemesiz said:
Another try: boot host with no OSD disk and then put single HDD OSD to see does it up.

Okay. let me pull all the drives and restart it again.

ResidentDataHoarder · 2026-05-27T14:50:54+0200

here is the output of journalctl --since "5 minutes ago" before I plugged in a drive:

Code:

May 27 07:24:21 pve5 pmxcfs[1397]: [status] notice: received log
May 27 07:24:21 pve5 pmxcfs[1397]: [status] notice: received log
May 27 07:24:22 pve5 sshd-session[5026]: Accepted publickey for root from 192.168.1.31 port 49466 ssh2: RSA SHA256:/4bWNb1Uydy945OxaOeN94llU5aHMDw8qRLS9JxoFlE
May 27 07:24:22 pve5 sshd-session[5026]: pam_unix(sshd:session): session opened for user root(uid=0) by root(uid=0)
May 27 07:24:22 pve5 systemd-logind[1161]: New session 1 of user root.
May 27 07:24:22 pve5 systemd[1]: Created slice user-0.slice - User Slice of UID 0.
May 27 07:24:22 pve5 systemd[1]: Starting user-runtime-dir@0.service - User Runtime Directory /run/user/0...
May 27 07:24:22 pve5 systemd[1]: Finished user-runtime-dir@0.service - User Runtime Directory /run/user/0.
May 27 07:24:22 pve5 systemd[1]: Starting user@0.service - User Manager for UID 0...
May 27 07:24:22 pve5 (systemd)[5031]: pam_unix(systemd-user:session): session opened for user root(uid=0) by root(uid=0)
May 27 07:24:22 pve5 systemd-logind[1161]: New session 2 of user root.
May 27 07:24:22 pve5 systemd[5031]: Queued start job for default target default.target.
May 27 07:24:22 pve5 systemd[5031]: Created slice app.slice - User Application Slice.
May 27 07:24:22 pve5 systemd[5031]: Reached target paths.target - Paths.
May 27 07:24:22 pve5 systemd[5031]: Reached target timers.target - Timers.
May 27 07:24:22 pve5 systemd[5031]: Listening on dirmngr.socket - GnuPG network certificate management daemon.
May 27 07:24:22 pve5 systemd[5031]: Listening on gpg-agent-browser.socket - GnuPG cryptographic agent and passphrase cache (access for web browsers).
May 27 07:24:22 pve5 systemd[5031]: Listening on gpg-agent-extra.socket - GnuPG cryptographic agent and passphrase cache (restricted).
May 27 07:24:22 pve5 systemd[5031]: Starting gpg-agent-ssh.socket - GnuPG cryptographic agent (ssh-agent emulation)...
May 27 07:24:22 pve5 systemd[5031]: Starting gpg-agent.socket - GnuPG cryptographic agent and passphrase cache...
May 27 07:24:22 pve5 systemd[5031]: Listening on keyboxd.socket - GnuPG public key management service.
May 27 07:24:22 pve5 systemd[5031]: Starting ssh-agent.socket - OpenSSH Agent socket...
May 27 07:24:22 pve5 systemd[5031]: Listening on gpg-agent.socket - GnuPG cryptographic agent and passphrase cache.
May 27 07:24:22 pve5 systemd[5031]: Listening on ssh-agent.socket - OpenSSH Agent socket.
May 27 07:24:22 pve5 systemd[5031]: Listening on gpg-agent-ssh.socket - GnuPG cryptographic agent (ssh-agent emulation).
May 27 07:24:22 pve5 systemd[5031]: Reached target sockets.target - Sockets.
May 27 07:24:22 pve5 systemd[5031]: Reached target basic.target - Basic System.
May 27 07:24:22 pve5 systemd[5031]: Reached target default.target - Main User Target.
May 27 07:24:22 pve5 systemd[5031]: Startup finished in 236ms.
May 27 07:24:22 pve5 systemd[1]: Started user@0.service - User Manager for UID 0.
May 27 07:24:22 pve5 systemd[1]: Started session-1.scope - Session 1 of User root.
May 27 07:24:25 pve5 sh[1610]: Running command: /usr/sbin/ceph-volume lvm trigger 6-021c9850-b537-4114-8bd0-f2eaa78b3048
May 27 07:24:25 pve5 sh[1618]: Running command: /usr/sbin/ceph-volume lvm trigger 8-b17d8640-e82b-402b-ac88-e3a76d0af858
May 27 07:24:25 pve5 sh[1560]: Running command: /usr/sbin/ceph-volume lvm trigger 11-b1432435-d44c-4c2b-b796-54faa4d257e7
May 27 07:24:25 pve5 sh[1554]: Running command: /usr/sbin/ceph-volume lvm trigger 10-31c0f65e-ad51-46df-931f-483c6c0324aa
May 27 07:24:25 pve5 sh[1615]: Running command: /usr/sbin/ceph-volume lvm trigger 8-d67facbc-a33a-47ad-9149-95b29552bf50
May 27 07:24:25 pve5 sh[1593]: Running command: /usr/sbin/ceph-volume lvm trigger 36-aacdc12a-2f0f-4a75-8542-9a613c0f8d26
May 27 07:24:25 pve5 sh[1566]: Running command: /usr/sbin/ceph-volume lvm trigger 13-caaf0c96-e90e-46b8-9a49-cf84260b94e4
May 27 07:24:25 pve5 sh[1600]: Running command: /usr/sbin/ceph-volume lvm trigger 50-0d4a4621-f087-4c1c-97f6-753b88427e5c
May 27 07:24:25 pve5 sh[1569]: Running command: /usr/sbin/ceph-volume lvm trigger 14-03728b9e-d559-4fcc-a7b9-1bc7abb03380
May 27 07:24:25 pve5 sh[1575]: Running command: /usr/sbin/ceph-volume lvm trigger 16-43df5e31-450f-4914-bb80-078628e85f1f
May 27 07:24:25 pve5 sh[1603]: Running command: /usr/sbin/ceph-volume lvm trigger 28-6995b8b2-8d94-47f0-becf-820dac70e766
May 27 07:24:25 pve5 sh[1619]: Running command: /usr/sbin/ceph-volume lvm trigger 9-2f7003ec-e2cb-4ce6-8420-ca67c196bf8f
May 27 07:24:25 pve5 sh[1563]: Running command: /usr/sbin/ceph-volume lvm trigger 12-4576958b-03f6-49cd-ab0c-405b1f942c7b
May 27 07:24:25 pve5 sh[1597]: Running command: /usr/sbin/ceph-volume lvm trigger 43-8a5813b8-c948-4e14-9861-af917047ab34
May 27 07:24:25 pve5 sh[1572]: Running command: /usr/sbin/ceph-volume lvm trigger 15-bdaeb37b-0491-46c5-b05c-a8770a4964c0
May 27 07:24:25 pve5 sh[1582]: Running command: /usr/sbin/ceph-volume lvm trigger 27-a9b3c0d5-c437-4743-aa6f-64205d8b8633
May 27 07:24:25 pve5 sh[1588]: Running command: /usr/sbin/ceph-volume lvm trigger 33-96dbfa75-1f2d-459b-9bfe-e23332b3ce2a
May 27 07:24:25 pve5 sh[1578]: Running command: /usr/sbin/ceph-volume lvm trigger 16-f3aeae04-8013-45d7-a086-7fd5fb3fddb8
May 27 07:24:25 pve5 sh[1592]: Running command: /usr/sbin/ceph-volume lvm trigger 17-57b2f2ab-87e2-4578-a553-d2ac3ec00471
May 27 07:24:30 pve5 sh[1610]: Running command: /usr/sbin/ceph-volume lvm trigger 6-021c9850-b537-4114-8bd0-f2eaa78b3048
May 27 07:24:30 pve5 sh[1618]: Running command: /usr/sbin/ceph-volume lvm trigger 8-b17d8640-e82b-402b-ac88-e3a76d0af858
May 27 07:24:30 pve5 sh[1560]: Running command: /usr/sbin/ceph-volume lvm trigger 11-b1432435-d44c-4c2b-b796-54faa4d257e7
May 27 07:24:30 pve5 sh[1554]: Running command: /usr/sbin/ceph-volume lvm trigger 10-31c0f65e-ad51-46df-931f-483c6c0324aa
May 27 07:24:30 pve5 sh[1575]: Running command: /usr/sbin/ceph-volume lvm trigger 16-43df5e31-450f-4914-bb80-078628e85f1f
May 27 07:24:30 pve5 sh[1566]: Running command: /usr/sbin/ceph-volume lvm trigger 13-caaf0c96-e90e-46b8-9a49-cf84260b94e4
May 27 07:24:30 pve5 sh[1593]: Running command: /usr/sbin/ceph-volume lvm trigger 36-aacdc12a-2f0f-4a75-8542-9a613c0f8d26
May 27 07:24:30 pve5 sh[1615]: Running command: /usr/sbin/ceph-volume lvm trigger 8-d67facbc-a33a-47ad-9149-95b29552bf50
May 27 07:24:30 pve5 sh[1569]: Running command: /usr/sbin/ceph-volume lvm trigger 14-03728b9e-d559-4fcc-a7b9-1bc7abb03380
May 27 07:24:30 pve5 sh[1603]: Running command: /usr/sbin/ceph-volume lvm trigger 28-6995b8b2-8d94-47f0-becf-820dac70e766
May 27 07:24:30 pve5 sh[1563]: Running command: /usr/sbin/ceph-volume lvm trigger 12-4576958b-03f6-49cd-ab0c-405b1f942c7b
May 27 07:24:30 pve5 sh[1619]: Running command: /usr/sbin/ceph-volume lvm trigger 9-2f7003ec-e2cb-4ce6-8420-ca67c196bf8f
May 27 07:24:30 pve5 sh[1600]: Running command: /usr/sbin/ceph-volume lvm trigger 50-0d4a4621-f087-4c1c-97f6-753b88427e5c
May 27 07:24:30 pve5 sh[1597]: Running command: /usr/sbin/ceph-volume lvm trigger 43-8a5813b8-c948-4e14-9861-af917047ab34
May 27 07:24:30 pve5 sh[1572]: Running command: /usr/sbin/ceph-volume lvm trigger 15-bdaeb37b-0491-46c5-b05c-a8770a4964c0
May 27 07:24:30 pve5 sh[1588]: Running command: /usr/sbin/ceph-volume lvm trigger 33-96dbfa75-1f2d-459b-9bfe-e23332b3ce2a
May 27 07:24:30 pve5 sh[1592]: Running command: /usr/sbin/ceph-volume lvm trigger 17-57b2f2ab-87e2-4578-a553-d2ac3ec00471
May 27 07:24:30 pve5 sh[1582]: Running command: /usr/sbin/ceph-volume lvm trigger 27-a9b3c0d5-c437-4743-aa6f-64205d8b8633
May 27 07:24:30 pve5 sh[1578]: Running command: /usr/sbin/ceph-volume lvm trigger 16-f3aeae04-8013-45d7-a086-7fd5fb3fddb8
May 27 07:24:35 pve5 sh[1610]: Running command: /usr/sbin/ceph-volume lvm trigger 6-021c9850-b537-4114-8bd0-f2eaa78b3048
May 27 07:24:35 pve5 sh[1560]: Running command: /usr/sbin/ceph-volume lvm trigger 11-b1432435-d44c-4c2b-b796-54faa4d257e7
May 27 07:24:35 pve5 sh[1618]: Running command: /usr/sbin/ceph-volume lvm trigger 8-b17d8640-e82b-402b-ac88-e3a76d0af858
May 27 07:24:35 pve5 sh[1575]: Running command: /usr/sbin/ceph-volume lvm trigger 16-43df5e31-450f-4914-bb80-078628e85f1f
May 27 07:24:35 pve5 sh[1554]: Running command: /usr/sbin/ceph-volume lvm trigger 10-31c0f65e-ad51-46df-931f-483c6c0324aa
May 27 07:24:35 pve5 sh[1566]: Running command: /usr/sbin/ceph-volume lvm trigger 13-caaf0c96-e90e-46b8-9a49-cf84260b94e4
May 27 07:24:35 pve5 sh[1569]: Running command: /usr/sbin/ceph-volume lvm trigger 14-03728b9e-d559-4fcc-a7b9-1bc7abb03380
May 27 07:24:35 pve5 sh[1603]: Running command: /usr/sbin/ceph-volume lvm trigger 28-6995b8b2-8d94-47f0-becf-820dac70e766
May 27 07:24:35 pve5 sh[1600]: Running command: /usr/sbin/ceph-volume lvm trigger 50-0d4a4621-f087-4c1c-97f6-753b88427e5c
May 27 07:24:35 pve5 sh[1615]: Running command: /usr/sbin/ceph-volume lvm trigger 8-d67facbc-a33a-47ad-9149-95b29552bf50
May 27 07:24:35 pve5 sh[1593]: Running command: /usr/sbin/ceph-volume lvm trigger 36-aacdc12a-2f0f-4a75-8542-9a613c0f8d26
May 27 07:24:35 pve5 sh[1563]: Running command: /usr/sbin/ceph-volume lvm trigger 12-4576958b-03f6-49cd-ab0c-405b1f942c7b
May 27 07:24:35 pve5 sh[1619]: Running command: /usr/sbin/ceph-volume lvm trigger 9-2f7003ec-e2cb-4ce6-8420-ca67c196bf8f
May 27 07:24:35 pve5 sh[1597]: Running command: /usr/sbin/ceph-volume lvm trigger 43-8a5813b8-c948-4e14-9861-af917047ab34
May 27 07:24:35 pve5 sh[1588]: Running command: /usr/sbin/ceph-volume lvm trigger 33-96dbfa75-1f2d-459b-9bfe-e23332b3ce2a
May 27 07:24:35 pve5 sh[1572]: Running command: /usr/sbin/ceph-volume lvm trigger 15-bdaeb37b-0491-46c5-b05c-a8770a4964c0
May 27 07:24:35 pve5 sh[1592]: Running command: /usr/sbin/ceph-volume lvm trigger 17-57b2f2ab-87e2-4578-a553-d2ac3ec00471
May 27 07:24:35 pve5 sh[1582]: Running command: /usr/sbin/ceph-volume lvm trigger 27-a9b3c0d5-c437-4743-aa6f-64205d8b8633
May 27 07:24:36 pve5 sh[1578]: Running command: /usr/sbin/ceph-volume lvm trigger 16-f3aeae04-8013-45d7-a086-7fd5fb3fddb8
May 27 07:24:40 pve5 postfix/smtp[1544]: connect to mx1.simplelogin.co[185.205.70.136]:25: Connection timed out

ResidentDataHoarder · 2026-05-27T14:51:29+0200

here is after:

Code:

May 27 07:29:21 pve5 kernel: mpt3sas_cm1: REPORT_LUNS: handle(0x0027),
                             retries(0)
May 27 07:29:21 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:21 pve5 kernel: START_UNIT: handle(0x0027), lun(0)
May 27 07:29:21 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:21 pve5 kernel: mpt3sas_cm0: detecting: handle(0x0027),
                             sas_address(0x5000cca27a3838c9), phy(5)
May 27 07:29:21 pve5 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0027),
                             retries(0)
May 27 07:29:21 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:21 pve5 kernel: START_UNIT: handle(0x0027), lun(0)
May 27 07:29:21 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:22 pve5 kernel: mpt3sas_cm1: detecting: handle(0x0027),
                             sas_address(0x5000cca27a3838c9), phy(5)
May 27 07:29:22 pve5 kernel: mpt3sas_cm1: REPORT_LUNS: handle(0x0027),
                             retries(0)
May 27 07:29:22 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:22 pve5 kernel: START_UNIT: handle(0x0027), lun(0)
May 27 07:29:22 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:22 pve5 kernel: mpt3sas_cm0: detecting: handle(0x0027),
                             sas_address(0x5000cca27a3838c9), phy(5)
May 27 07:29:22 pve5 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0027),
                             retries(0)
May 27 07:29:22 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:22 pve5 kernel: START_UNIT: handle(0x0027), lun(0)
May 27 07:29:22 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:23 pve5 kernel: mpt3sas_cm1: detecting: handle(0x0027),
                             sas_address(0x5000cca27a3838c9), phy(5)
May 27 07:29:23 pve5 kernel: mpt3sas_cm1: REPORT_LUNS: handle(0x0027),
                             retries(0)
May 27 07:29:23 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:23 pve5 kernel: START_UNIT: handle(0x0027), lun(0)
May 27 07:29:23 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:23 pve5 kernel: mpt3sas_cm0: detecting: handle(0x0027),
                             sas_address(0x5000cca27a3838c9), phy(5)
May 27 07:29:23 pve5 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0027),
                             retries(0)
May 27 07:29:23 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:23 pve5 kernel: START_UNIT: handle(0x0027), lun(0)
May 27 07:29:23 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:24 pve5 kernel: mpt3sas_cm1: detecting: handle(0x0027),
                             sas_address(0x5000cca27a3838c9), phy(5)
May 27 07:29:24 pve5 kernel: mpt3sas_cm1: REPORT_LUNS: handle(0x0027),
                             retries(0)
May 27 07:29:24 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:24 pve5 kernel: START_UNIT: handle(0x0027), lun(0)
May 27 07:29:24 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:24 pve5 kernel: mpt3sas_cm0: detecting: handle(0x0027),
                             sas_address(0x5000cca27a3838c9), phy(5)
May 27 07:29:24 pve5 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0027),
                             retries(0)
May 27 07:29:24 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:24 pve5 kernel: START_UNIT: handle(0x0027), lun(0)
May 27 07:29:24 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:25 pve5 kernel: mpt3sas_cm1: detecting: handle(0x0027),
                             sas_address(0x5000cca27a3838c9), phy(5)
May 27 07:29:25 pve5 kernel: mpt3sas_cm1: REPORT_LUNS: handle(0x0027),
                             retries(0)
May 27 07:29:25 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:25 pve5 kernel: START_UNIT: handle(0x0027), lun(0)
May 27 07:29:25 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:25 pve5 kernel: mpt3sas_cm0: detecting: handle(0x0027),
                             sas_address(0x5000cca27a3838c9), phy(5)
May 27 07:29:25 pve5 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0027),
                             retries(0)
May 27 07:29:25 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:25 pve5 kernel: START_UNIT: handle(0x0027), lun(0)
May 27 07:29:25 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:26 pve5 kernel: mpt3sas_cm1: detecting: handle(0x0027),
                             sas_address(0x5000cca27a3838c9), phy(5)
May 27 07:29:26 pve5 kernel: mpt3sas_cm1: REPORT_LUNS: handle(0x0027),
                             retries(0)
May 27 07:29:26 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:26 pve5 kernel: START_UNIT: handle(0x0027), lun(0)
May 27 07:29:26 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:26 pve5 kernel: mpt3sas_cm0: detecting: handle(0x0027),
                             sas_address(0x5000cca27a3838c9), phy(5)
May 27 07:29:26 pve5 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0027),
                             retries(0)
May 27 07:29:26 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:26 pve5 kernel: START_UNIT: handle(0x0027), lun(0)
May 27 07:29:26 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:27 pve5 kernel: mpt3sas_cm1: detecting: handle(0x0027),
                             sas_address(0x5000cca27a3838c9), phy(5)
May 27 07:29:27 pve5 kernel: mpt3sas_cm1: REPORT_LUNS: handle(0x0027),
                             retries(0)
May 27 07:29:27 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:27 pve5 kernel: START_UNIT: handle(0x0027), lun(0)
May 27 07:29:27 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:27 pve5 kernel: mpt3sas_cm0: detecting: handle(0x0027),
                             sas_address(0x5000cca27a3838c9), phy(5)
May 27 07:29:27 pve5 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0027),
                             retries(0)
May 27 07:29:27 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:27 pve5 kernel: START_UNIT: handle(0x0027), lun(0)
May 27 07:29:27 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:28 pve5 kernel: mpt3sas_cm1: detecting: handle(0x0027),
                             sas_address(0x5000cca27a3838c9), phy(5)
May 27 07:29:28 pve5 kernel: mpt3sas_cm1: REPORT_LUNS: handle(0x0027),
                             retries(0)
May 27 07:29:28 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:28 pve5 kernel: START_UNIT: handle(0x0027), lun(0)
May 27 07:29:28 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:28 pve5 kernel: mpt3sas_cm0: detecting: handle(0x0027),
                             sas_address(0x5000cca27a3838c9), phy(5)
May 27 07:29:28 pve5 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0027),
                             retries(0)
May 27 07:29:28 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:28 pve5 kernel: START_UNIT: handle(0x0027), lun(0)
May 27 07:29:28 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:29 pve5 kernel: mpt3sas_cm1: detecting: handle(0x0027),
                             sas_address(0x5000cca27a3838c9), phy(5)
May 27 07:29:29 pve5 kernel: mpt3sas_cm1: REPORT_LUNS: handle(0x0027),
                             retries(0)
May 27 07:29:29 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:29 pve5 kernel: mpt3sas_cm1: handle(0x27) sas_address(0x5000cca27a3838c9) port_type(0x1)
May 27 07:29:29 pve5 kernel: scsi 9:0:2:0: Direct-Access     HGST     HUH721212AL4200  AB01 PQ: 0 ANSI: 6
May 27 07:29:29 pve5 kernel: scsi 9:0:2:0: SSP: handle(0x0027), sas_addr(0x5000cca27a3838c9), phy(5), device_name(0x5000cca27a3838cb)
May 27 07:29:29 pve5 kernel: scsi 9:0:2:0: enclosure logical id (0x50030480186998ff), slot(5)
May 27 07:29:29 pve5 kernel: scsi 9:0:2:0: enclosure level(0x0000), connector name(     )
May 27 07:29:29 pve5 kernel: scsi 9:0:2:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
May 27 07:29:29 pve5 kernel: sd 9:0:2:0: Attached scsi generic sg6 type 0
May 27 07:29:29 pve5 kernel: sd 9:0:2:0: [sdc] 2929721344 4096-byte logical blocks: (12.0 TB/10.9 TiB)
May 27 07:29:29 pve5 kernel: sd 9:0:2:0: [sdc] Write Protect is off
May 27 07:29:29 pve5 kernel:  end_device-9:0:2: add: handle(0x0027), sas_addr(0x5000cca27a3838c9)
May 27 07:29:29 pve5 kernel: sd 9:0:2:0: [sdc] Mode Sense: f7 00 10 08
May 27 07:29:29 pve5 kernel: sd 9:0:2:0: [sdc] Write cache: disabled, read cache: enabled, supports DPO and FUA
May 27 07:29:29 pve5 kernel: sd 9:0:2:0: [sdc] Attached SCSI disk
May 27 07:29:29 pve5 kernel: mpt3sas_cm0: detecting: handle(0x0027),
                             sas_address(0x5000cca27a3838c9), phy(5)
May 27 07:29:29 pve5 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0027),
                             retries(0)
May 27 07:29:29 pve5 kernel: TEST_UNIT_READY: handle(0x0027) lun(0)
May 27 07:29:29 pve5 kernel: mpt3sas_cm0: handle(0x27) sas_address(0x5000cca27a3838c9) port_type(0x1)
May 27 07:29:29 pve5 kernel: scsi 8:0:2:0: Direct-Access     HGST     HUH721212AL4200  AB01 PQ: 0 ANSI: 6
May 27 07:29:29 pve5 kernel: scsi 8:0:2:0: SSP: handle(0x0027), sas_addr(0x5000cca27a3838c9), phy(5), device_name(0x5000cca27a3838cb)
May 27 07:29:29 pve5 kernel: scsi 8:0:2:0: enclosure logical id (0x50030480186998ff), slot(5)
May 27 07:29:29 pve5 kernel: scsi 8:0:2:0: enclosure level(0x0001), connector name(     )
May 27 07:29:29 pve5 kernel: scsi 8:0:2:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
May 27 07:29:29 pve5 kernel: sd 8:0:2:0: Attached scsi generic sg7 type 0
May 27 07:29:29 pve5 kernel:  end_device-8:1:1: add: handle(0x0027), sas_addr(0x5000cca27a3838c9)
May 27 07:29:29 pve5 kernel: sd 8:0:2:0: [sdd] 2929721344 4096-byte logical blocks: (12.0 TB/10.9 TiB)
May 27 07:29:29 pve5 kernel: sd 8:0:2:0: [sdd] Write Protect is off
May 27 07:29:29 pve5 kernel: sd 8:0:2:0: [sdd] Mode Sense: f7 00 10 08
May 27 07:29:29 pve5 kernel: sd 8:0:2:0: [sdd] Write cache: disabled, read cache: enabled, supports DPO and FUA
May 27 07:29:29 pve5 lvm[10465]: PV /dev/sdc online, VG ceph-5909443b-69ac-4d66-899c-f6d7345db3e5 is complete.
May 27 07:29:29 pve5 kernel: sd 8:0:2:0: [sdd] Attached SCSI disk
May 27 07:29:29 pve5 systemd[1]: Started lvm-activate-ceph-5909443b-69ac-4d66-899c-f6d7345db3e5.service - [systemd-run] /usr/sbin/lvm vgchange -aay --autoactivation event ceph-5909443b-69ac-4d66-899c-f6d7345db3e5.
May 27 07:29:29 pve5 lvm[10482]: PV /dev/sdd 8:48 is duplicate for PVID gTYMNVgYkgYtkGTxt7DjUT2eOFf1mAZ7 on 8:32 /dev/sdc.
May 27 07:29:29 pve5 lvm[10482]: PV /dev/sdd failed to create online file.
May 27 07:29:29 pve5 lvm[10473]:   1 logical volume(s) in volume group "ceph-5909443b-69ac-4d66-899c-f6d7345db3e5" now active
May 27 07:29:29 pve5 systemd[1]: lvm-activate-ceph-5909443b-69ac-4d66-899c-f6d7345db3e5.service: Deactivated successfully.
lines 305-457/457 (END)

ResidentDataHoarder · 2026-05-27T14:58:03+0200

using journalctl --no-pager -o cat --since '30 minutes ago' | awk '!seen[$0]++' for a bit cleaner output:
here is a privatebin since it wont fit withing the limit: https://privatebin.net/?527c178abfa7ee31#AjZHSxjojsfTxN9jYV7V7Vf6RtuLHR6hUcUKCd2KRXWG

ResidentDataHoarder · 2026-05-27T15:07:06+0200

Have to run an errand be back in a bit.

CEPH OSD(s) failing to initialize after a hardware change

New Member

Renowned Member

New Member

New Member

Renowned Member

New Member

New Member

Renowned Member

New Member

New Member

New Member

Renowned Member

New Member

Renowned Member

New Member

New Member

New Member

New Member

New Member

We value your privacy