Search results

You seem to have removed the OSDs too fast and the pool was not able to migrate its PG to other OSDs. And now it cannot find the PG any more. You can try to remove the pool and recreate it. It is called ".mgr" with a dot in front.

What happend to the pool with the id 1 (usually the pool called ".mgr")? Its only placement group is in state "unknown" which is not good. Please try to restart OSDs 2, 3 and 7 with "systemctl restart ceph-osd@2.service" etc on their nodes. If that does not fix the issue please post the...

You use inages of multiple KBs in size to post text of a few bytes. why? Please attach the text files here somehow.

Please.post the output of "ceph -s", "ceph osd df tree" and "ceph pg dump".

Der Fehler wird auch hier erwähnt: https://forum.proxmox.com/threads/rbd-error-rbd-listing-images-failed-2-no-such-file-or-directory-500.120661/ Wurden Images migriert und die Migration abgebrochen?

Was sagt "ceph -s" auf den Knoten?

Are both OSDs gone? You could try to set min_size to 1 on the affected pool(s). This will allow Ceph to restore the copies from the last remaining one.

Have you tried to attach the virtual disks as SCSI disks with the virtio-scsi-single controller?

Both CPUs have 8 cores, 16 threads. In total 16 cores and 32 threads in each node. An SSD OSD should have 2 CPU threads available which is given. The 2660 is a bit slower. You will have to test the system how much this affects real workloads.

Did you add the other two hosts now? What is the output of "ceph -s"?

Use bond mode active-passive for that with the faster link as active. If you switch supports LACP you could also use that and utilize both connections.

There is enough RAM available for 9 SSD OSDs in each host. You usually calculate with 5 GB RAM per OSD.

Your placement groups are not active, meaning no data transfer (read or write) can take place. You seem to have only one host (pve01) with OSDs. With the default replication size of 3 and the default failure zone "host" Ceph is unable to place the second and third copy. You need to add at least...

The default route (gateway 192.168.1.200) goes via vmbr0, so when you pull the cable from enp87s0 the host is only reachable from within its LAN 192.168.1.0/24.

Max 67000 IOPS bzw 3,2GB/s sieht doch gar nicht so schlecht aus. Das ist dann die summierte Performance, die über alle VMs zu erwarten ist. Die einzelne VM erhält eher das Ergebnis mit 4K und 1 IO-Thread.

Ich hoffe, auf dem VM-Image vm-113-disk-0 war nichts wichtiges drauf… Für 4K Blocksize und 4 Threads sieht das ganz OK aus. Setz die iodepth mal auf 128 und danach die bs auf 4M mit einer iodepth von 8 oder 16.

Direkt auf dem Proxmox-Knoten. Fio hat auch die Möglichkeit, mit RBDs zu sprechen.

Mach den Test mit fio mal direkt auf einem RBD zum Vergleich. Und spiele mal mit den Parameter --bs und --iodepth.

Ceph is all abaout data consistency. It is not guaranteed that the single remaining copy is a valid copy. This can only be assured when there is a "majority" of copies available that are all the same. It's basically the same principle like with the quorum of the MONs. BTW: It is not the number...

This is because the data will be moved from other OSDs. Give Ceph time to do it.