Hi tytanick, i believe that i have the best solution for your target.
With this strategy, you will have HA without lose information, but with some cautions for have caution if you don't want to have problems of lost information when HA must be applied.
The strategy is this:
1) You should use DRBD with only two (2) PVE nodes (not more PVE nodes of according with this setup).
2) You should use HA (using rgmanager)
3) If you want avoid the lost information up last moment in that "High Availability" should be applied, you should to have in the configuration of the virtual disks of the VMs this option enabled "Write through" or "Direct sync". For better performance I prefer use "Write through" due to that "Write through" does that the information of block of disks will be in the available RAM memory of the host PVE (and obviously with cache enabled in the PVE host, the read of disk blocks in the same dishes of disks will be less frequent due to that the information will be in the RAM memory).
4) For don't have problems of quorum with only two Servers, you should add a configuration to your cluster.conf file:
In your case, that is the same case that in my test lab:
<cman expected_votes="1" keyfile="/var/lib/pve-cluster/corosync.authkey" two_node="1"/>
Note: this strategy of configuration isn't the recomended by PVE team, but for me, with some extra cautions works very well in a production enviroment.
5) I know that this configuration (explained above) for cluster.conf file isn't the best (but works), and if you are controlling all the aspects that are really important, and in the manual mode, you will not have problems.
6) In your "/etc/drbd.d/global_common.conf" file , you should to have enabled literally this lines:
split-brain "/usr/lib/drbd/notify-split-brain.sh <some address@of mail account>";
out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh <some address@of mail account>";
Notes about of these setup of DRBD:
- The target of these configurations in DRBD is for that DRBD notify by email immediately if DRBD have some problem with the synchronous replication (obviously, you need to be sure that this directives work as it is expected it to work, so my recomendation is tested before).
- If DRBD notifies by email at the same time that a server was decomposed, obviously you will not have problems of loss of information when you need apply HA.
7) When a "PVE host" be decomposed, these VMs in this "PVE host" don't will work more, but you can analyze the problem with all tranquility and patience, the purpose of analyze the problem quietly is for be sure that DRBD had no problems of replication until it broke the PVE host.
8) When are you sure that DRDB don't had any problem, and the problem only was the break down of the PVE host, obviously, you can apply HA with all tranquility.
These are the steps:
A) Manually disconnect the electric power in the physical server that is supposedly dead (it is the best practice for be sure that this Node no will work more and avoid serious problems with the information contained in your virtual disks of the VMs implicated - This step is critical)
B) For apply "HA" (and restart your VMs on the other PVE host that is alive), you should write by CLI in the secure shell of the PVE host that is alive this command:
Secure Shell of "PVE host" that is alive# /usr/sbin/fence_ack_manual <name of the PVE host that is dead>
9) Enjoy again of your VMs in the other PVE host and without lost of information.
Extra Notes:
A) PVE or Red Hat have other methods for apply "HA" with all security and without human interaction, but this type of setups requires more equipments and as minimum a PDU, so i have intent of explain you only a method that i believe that for your small infrastructure will be the better solution.
B) By other hand, Ceph (The star storage solution of PVE) speaking in basic mode have some problems of developed (but not in his concept and nor in his structure) , and for me understand "Ceph" isn't listed for use it in production environment when your information is very critical and you needs that all tools of "Ceph" works perfectly (i am sure that the problems at present of "Ceph" only are temporals, and in a short time Ceph will be really ready for use it in real environments of production to great scale without a unique point of failure).
C) In the case of DRBD, this software have many years of exist (a great difference with Ceph software), so this product since much time ago spent his first epoch of development, and actually in this epoch is considered super stable. By other hand, at present I'm using the latest version of DRDB (8.4.5) in production environments without any problem.
D) A point in very favour of DRBD is that the reads of blocks of disk are locals (and obviously with the speed of the bus of your hard disk controller installed in the same PC or Server), with the difference that "GlusterFS" or "Ceph" as are installed in other computers in a network , the speed of reads or writes of disks are determined by the hardware more slow in the path to information, or the efficiency of his software, ie, or your speed of network communication, or the speed of the hard disks in the storage layer, or the efficiency of the software involved for do these tasks, that in the majority of cases is very slow (and maybe unbearable, for example in a problem of software for do backups, Ceph is very slow compared with DRBD). It is for several reasons (including money need for purchase a good infraestructure) that i prefer to use DRBD.
For all people of this forum: If i am wrong in some thing that I have said, please feel free to correct me.
For tytanick: Awaiting that my recomendations be of great help for you, i say see you soon.
Best regards
Cesar Peschiera
