Search results

I think that would make it worse. In ceph best is to use nearly identical capacities on the nodes. If you increase your "big nodes" further, ceph can not distribute data in "it´s way". I think best is to exchange the small ssd one by one with 2 TB ssd. Beginning in the node with the lowest...

hey, please use code-tags around the output. i can´t see any big mistake. Your pool seems to be near full. In ceph you can´t (by normal way) equalize all osd´s. That´s because of pg placement. It´s not on byte level or something similiar. But it doesn´t hurt. You can try to increase pg count i...

maybe your nodes are weighted unevenly (regarding disk capacity), you have hdd on your nodes as osd too. what das "ceph osd df tree" says?

i don´t exactly now where to look (maybe /var/log/ceph), but osds beeing down for itself is not an error. Any messages when you start them?

You´re right, it´s not clustered. I didn´t got the need for shared / clustered situation. I´m sorry.

It may also will be possible, to use zfs with iSCSI backed vdevs. But only for the features. Not for performance i think.

I found a hint in a german news portal, maybe it is related. https://www.golem.de/news/secure-boot-virtuelle-maschinen-mit-windows-server-2022-booten-nicht-2302-172076.html

When it comes to performance, consider using virtio. At first you should choose Hard-disk SCSI an not SATA. Network also: choose virtio instead of e1000.

I think .mgr is an ceph-internal pool. Maybe it´s contents have been erased by the update.

Hey cyp, as alyarb mentioned, i think you search in the wrong pool? Respectively using the .mgr for vm-disks may lead to problems.

i think, HA and replication (zfs) are two different things here. So far as i know you have to use shared storage for this.

Although not recommended generally, we use the HP 840 in RAID Mode with JBOD-RAID0, so we can use cache settings and battery/capacitor backuped cache. You must know what you do in this case and check your use case (e.g. data security) against this. Using the HP controller without it´s features...

We use HP G9 also. But with another HBA (P840), and we use raid mode. As far as i remember we did not had these problems with local attached storage (zfs e.g.). But we used the battery backuped write cache from the P840er. Without that, most performance was "ugly". I do not remember details...

so you are benchmarking ceph? Then there are thousands of parameters to look at. Network settings for ceph. Ceph version. Ceph storage / osd parameter, pool redundancy, number of nodes, switch settings, and many many aspects more.

If i see it right you test the ceph behavior (fsync=1) in the vm. That will be "nested" in some way i think. Normal processes in the vm will not use fsync normally. DBMS will use fsync=1 maybe, but then most sequential (redo logs e.g.).

btw., bzgl. des Monitoring Tools ist es aber ja auch genau richtig, dass der freie Wert da genommen wird. Und das genügend frei ist. "Gleich" wirst Du m.E. nicht hinbekommen.

Hey, nein, der "freie Speicher" ist bei Ceph nicht ganz trivial. Hat was mit dem Placement zu tun. Ceph schaut, wieviel es an Daten/PGs noch bei gegebener Redundanzvorgabe schreiben kann. Maßgeblich sind da die vollsten OSD und Nodes an der Berechnung beteiligt. Die STDDEV: 17.36 erscheint mir...

oh, das war hier auf Deutsch. Im Storage Tab zeigt der nur den noch verfügbaren Speicher an. Nicht die Pool _Size_. Zumindest habe ich das hier gerade quergecheckt und bei uns ist das so.

i think your first screenshot ist from storage tab? I checked this here. There you see the free space in the brackets. Look at the ceph tab, there should stand 176 TB, maybe.

Hey, for tcp connections there are a lot of parameters, that can cause such sort of trouble. I myself observed through wireshark, that a server with problems did not scale tcp window size correctly. Connection was bound to 400kbit/s therefore. It was a typo in the sysctl settings for the tcp...