Search results

I had bad experience when I tried HA configuration, so I stopped using it. If I am not mistaken if a node goes out of quorum just for a short while it should fence itself off, which means it needs to shutdown or something like that (and of course that brings down all the VMs). I think that...

Nope, I just remember that I used that thing a few years ago for a similar purpose. I believe I found some examples of the scripts in the Proxmox documentation, then I adapted those... No longer use that, so don't know the current status...

Can you try to use your 2.5Gb NIC just to rule out the 10Gb NIC/switch/driver issues? Also, if you have two nodes in the cluster I recommend that you assign one of the nodes (a 'primary' one) two votes, so that the cluster stays in quorum when the other node is down (it's better than having...

You can search on the forum and in the docs for the backup hook scripts and do wake-on line as the start of the backup job hook, and unmount at the end... But the backups to NAS are going to fill the space very quickly because there is no deduplication. You'd better setup a Proxmox Backup...

I believe min-size 2 applies for writes, so in 3-node cluster with 1 OSD each and 3/2 factor, if 2 OSD down you should still be able to read everything, but not able to write... So not a complete downtime, depending on the use case. But that only applies to the disks themselves going down. You...

Hi @troycarpenter Would you mind to share what kind of SSD cache configuration you are using?

There definitely should have been at least some activity during the faults (or host reboots). At the very least my test splunk server should receive a permanent inflow of data. It does looks like ceph has a way to know that the data on the OSD that stayed up and were written to need to be sent...

I am using a pool with 2/1 replication for my lab VMs in the last 5 years, and the cluster survived a couple of SSD losses and a lot of node reboots. Maybe I am just lucky, but I grew to trust that ceph does OK even wtih the non-recommended configuration. If you know what you are doing and OK...

Correct

I believe it's host mount, then the mp parameter specifies where it should be mounted within the container, so it should be something like: mp0: /mnt/pve/cephfs/export1,mp=/mnt/export1,shared=1 mp1: /mnt/pve/cephfs/export2,mp=/mnt/export2,shared=1

You would be better using a MAC learning switch on VMware rather than Promiscuous mode. You can check the following link for your options: https://williamlam.com/2023/05/refresher-on-nested-esxi-networking-requirements.html

Please don't forget that those 250MB/s throughput on HDDs are only for the large sequential operations. You can expect it to drop to 2.5MB/s or lower for random IOPS... Actually with truly random IO you should get only around 320KB/s throughput per disk (SATA drives doing 80 IOPS times 4K). I...

So, you mounted both cephfs filesystems to each of your proxmox nodes? After that you just modify the container config and add the mp0 and mp1 lines, each with the 'shared=1' option

I built my cluster 6 years ago, so don't remember which resources I used at the time... I built a cluster in a nested lab several times, and it was very straightforward. If you have your single node environment I encourage you to test first in a virtual lab, you just need the three VMs with 3GB...

Rest assured that it is entirely possible to run a Proxmox cluster on 3 nodes with a single NVMe drive. I actually run a similar configuration for several years. Note that you would basically end up with the ceph storage equivalent to the size of your single NVME drive, as the best practice is...

Hi @plastilin I played with enabling LVM cache on the OSD logical volume, and it worked but I did not do any performance comparisons, I don't believe I noticed a huge difference so eventually I decided it's not worth it to have the extra fault domain. Not sure how it's different from dm-cache...

You can test your network using iperf3. I believe the bottleneck should be your hard disks. Note that each OSD process can easily consume several GB of memory on the host, so testing of just 1GB might be a test of how fast your nodes can read or write to the memory cache. That's where the...

It definitely allows me to migrate the containers between the nodes. I use Proxmox Backup Server, so don't really use the snapshots. I just checked and indeed the snapshots are disabled for the configuration with the mount points. I guess you can use the backups as a workaround. Note that there...

You don't need those extra logical volumes if you plan to use ceph. You also don't need that much space for your root volume. I use 32GB as a root partition, and it seems to be enough. You might also want to leave 8-16GB for swap, so I think that 40-64GB for the root LVM partition should be more...

Hi @MoniPM Great job noticing that. I had an older lab cluster that was running Proxmox 7.0 and I checked that it could not add a partition as OSD, but after I updated to the current 7.4.3, I too was able to add that partition as an OSD. So it became supported somewhere between those versions...