Search results

Remove this OSD and redeploy it. There may be just a bit error on the disk.

Erasure coding is not usable in such small clusters. You need at least 10 nodes with enough OSDs to do anything meaningful with erasure coding.

Yes, do not mix two different device classes in one pool. You will only get HDD performance.

You need to replace the sdb drive.

Are there any signs in the kernel log about a failure on the device of this OSD?

Is data affected? Are there any PGs not active+clean?

Proxmox does not use DHCP for network configuration. You could remove the IP configuration from vmbr0 and add a local IP on vmbr1 and also point the default gateway to the opnsense VM. You just have to make sure that these changes happen so that Proxmox is still reachable (via opnsense).

Ah, I did not know about Ceph nano. It says that it only exposes S3 which is HTTP. You will not be able to access the Ceph "cluster" running inside this container with anything else.

If a MON has 127.0.0.1 as its IP there is something fundamentally wrong in the setup. The MONs need an IP from Ceph's public_network so that they are reachable from all the other Ceph daemons and the clients.

Do not use HDD only with Ceph.

Are these two separate CephFS volumes? What does "ceph fs status" show? If that's the case you need to specify the fs name with the fs= option to the mount command.

Ceph is not able to run dual stack. You have to either use IPv4 or IPv6 on both networks.

Failure donain host will not allow two copies on the same host. All PGs of the pool will be degraded in such a situation as Ceph is not able to find a location for the fourth copy.

No, the size of the pool defines the number of copies it creates (when being replicated, not erasure coded). It has nothing to do with the number of nodes. A size=3 pool will distribute the three copies over four nodes randomly. If an OSD is down the missing third copy will be recreated from...

Your datastore01 pool has a size of 4. It stores four copies of each object, hence its usable capacity is only 25% of the total capacity.

This is Ethernet and not Fibre-Channel. You cannot have two separate interfaces in the same VLAN. Create a stack out of the two switches (or MLAG or whatever the vendor calls it) and then a link aggregation group for each Ceph node with one port from each switch. On the Ceph nodes create a...

This is for the rule, it is not the size of the pool.

Why are there 8TB data stored in device_health_metrics? Do you store RBD data in this pool? You should create a separate pool for RBD data. https://pve.proxmox.com/wiki/Ceph_Pacific_to_Quincy#Guest_images_are_stored_on_pool_device_health_metrics

Please post the output of ceph df.

Are there other pools using the same OSDs?