ProxVE4.2 and DRBD9 3-node setup - misc query

fortechitsolutions

Renowned Member
Jun 4, 2008
471
61
93
Hi everyone,

I am setting up a test environment for a client, to see if it is feasible for production use. So far it is 'very good' after a certain amount of time-effort-modest-frustration (learning new setup and so forth).

Briefly the environment is
- 3 servers at remote data centre, no physical access but remote IPMI web-java console for OS install / raid config etc.
- LSI hardware raid, 2 x Intel SSD in raid1, carved up to a 100gig Proxmox-OS-slice; and rest is for BCache SSD
- 2 x 2Tb Hitachi SATA in raid1, for block tank for Bcache storage
- DRBD on 3 nodes setup as per the wiki advice, https://pve.proxmox.com/wiki/DRBD9
-- except I am using a Bcache dev target instead of vanilla device. In theory for better performance. So far so good :-)
-- 1gig public interface for access to proxmox hosts
---10gig interface for DRBD communication and proxmox cluster (multicast works inside here; does not work on public interfaces)

What I've found so far,
-- had to make hostnames and IPs for proxmox cluster config, be associated with the 10gig NIC IPs
-- had to do similar for DRBD setup
-- DRBD is working and status is happy
-- an odd thing I believe is happening, the 'free disk space' in DRBD is not consistent across all 3 hosts. I believe after some review, that this is because of behaviour of thin-provision - on the host where a VM was created the thin-behaviour 'works' but when replicated to 'other host' then we have no thin behaviour.
-- I am guessing that maybe? the safer thing would be to disable the thin-tools / thin support part of the DRBD setup / (although their use - was recommended in the DRBD9 wiki proxmox howto?) - or maybe this is a self-correcting issue I need not worry about.

For reference some info is given below, not that presently there are 4 x VMs alive on the cluster, and 4 x VM Disks associated with those VMs.

Output we have back from hosts in cluster that is informative:

Code:
check drbd status:

root@dpve1:/mnt/pve/nfs3# drbdmanage list-resources

+------------------------------------------------------------------------------------------------------------+
| Name  |  | State |
|------------------------------------------------------------------------------------------------------------|
| vm-100-disk-1 |  |  ok |
| vm-101-disk-1 |  |  ok |
| vm-102-disk-1 |  |  ok |
| vm-103-disk-1 |  |  ok |
+------------------------------------------------------------------------------------------------------------+

AND ALSO

root@dpve1:/mnt/pve/nfs3# drbdmanage list-nodes
+------------------------------------------------------------------------------------------------------------+
| Name  | Pool Size | Pool Free |  | State |
|------------------------------------------------------------------------------------------------------------|
| dpve1 |  1904640 |  1776457 |  |  ok |
| dpve2 |  1904640 |  1597421 |  |  ok |
| dpve3 |  1904640 |  1597421 |  |  ok |
+------------------------------------------------------------------------------------------------------------+
 
root@dpve1:/mnt/pve/nfs3#


humm, why is POOL FREE not identical across the board ?


HAH, difference is thin - vs - not-thin I think

LOOK FIRST ON PROXMOX NODE THREE:

root@dpve3:~# vgs
  VG  #PV #LV #SN Attr  VSize  VFree
  drbdpool  1  7  0 wz--n-  1.82t  2.25g
  pve  1  3  0 wz--n- 99.87g 12.26g
root@dpve3:~# lvs
  LV  VG  Attr  LSize  Pool  Origin Data%  Meta%  Move Log Cpy%Sync Convert
  .drbdctrl_0  drbdpool -wi-ao----  4.00m
  .drbdctrl_1  drbdpool -wi-ao----  4.00m
  drbdthinpool  drbdpool twi-aotz--  1.82t  17.85  9.21
  vm-100-disk-1_00 drbdpool Vwi-aotz-- 100.02g drbdthinpool  100.00
  vm-101-disk-1_00 drbdpool Vwi-aotz-- 100.02g drbdthinpool  100.00
  vm-102-disk-1_00 drbdpool Vwi-aotz-- 100.02g drbdthinpool  100.00
  vm-103-disk-1_00 drbdpool Vwi-aotz--  32.01g drbdthinpool  100.00
  data  pve  twi-a-tz--  54.75g  0.00  0.44
  root  pve  -wi-ao----  24.75g
  swap  pve  -wi-ao----  8.00g


NOW LOOK ON PROXMOX NODE ONE:

root@dpve3:~# exit
logout
Connection to dpve3 closed.
root@dpve1:/mnt/pve/nfs3# vgs
  VG  #PV #LV #SN Attr  VSize  VFree
  drbdpool  1  7  0 wz--n-  1.82t  2.25g
  pve  1  3  0 wz--n- 99.87g 12.26g
root@dpve1:/mnt/pve/nfs3# lvs
  LV  VG  Attr  LSize  Pool  Origin Data%  Meta%  Move Log Cpy%Sync Convert
  .drbdctrl_0  drbdpool -wi-ao----  4.00m
  .drbdctrl_1  drbdpool -wi-ao----  4.00m
  drbdthinpool  drbdpool twi-aotz--  1.82t  7.49  4.09
  vm-100-disk-1_00 drbdpool Vwi-aotz-- 100.02g drbdthinpool  13.69
  vm-101-disk-1_00 drbdpool Vwi-aotz-- 100.02g drbdthinpool  23.44
  vm-102-disk-1_00 drbdpool Vwi-aotz-- 100.02g drbdthinpool  100.00
  vm-103-disk-1_00 drbdpool Vwi-aotz--  32.01g drbdthinpool  6.58
  data  pve  twi-a-tz--  54.75g  0.00  0.44
  root  pve  -wi-ao----  24.75g
  swap  pve  -wi-ao----  8.00g

AND FINALLY ON PROX NODE TWO:
root@dpve1:/mnt/pve/nfs3# ssh dpve2

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Thu Jun  9 10:38:32 2016 from dpve1

root@dpve2:~# lvs
  LV  VG  Attr  LSize  Pool  Origin Data%  Meta%  Move Log Cpy%Sync Convert
  .drbdctrl_0  drbdpool -wi-ao----  4.00m
  .drbdctrl_1  drbdpool -wi-ao----  4.00m
  drbdthinpool  drbdpool twi-aotz--  1.82t  17.85  9.21
  vm-100-disk-1_00 drbdpool Vwi-aotz-- 100.02g drbdthinpool  100.00
  vm-101-disk-1_00 drbdpool Vwi-aotz-- 100.02g drbdthinpool  100.00
  vm-102-disk-1_00 drbdpool Vwi-aotz-- 100.02g drbdthinpool  100.00
  vm-103-disk-1_00 drbdpool Vwi-aotz--  32.01g drbdthinpool  100.00
  data  pve  twi-a-tz--  54.75g  0.00  0.44
  root  pve  -wi-ao----  24.75g
  swap  pve  -wi-ao----  8.00g
root@dpve2:~# vgs
  VG  #PV #LV #SN Attr  VSize  VFree
  drbdpool  1  7  0 wz--n-  1.82t  2.25g
  pve  1  3  0 wz--n- 99.87g 12.26g
root@dpve2:~#


Anyhow. I thought I would post this query here / to ask if others using - testing DRBD9 on proxmox, have seen similar things, or have any comments.

Otherwise.

This seems to work; very nicely; more solid than I expected. I have done some 'fail tests' (hard power off a node) - and then power it back on. DRBD seems to cope quite happily, which makes me pleasantly happy-surprised.

Performance is 'pretty good' - inside a VM running on this DRBD storage I get a 'simple' disk IO benchmark thus,

Code:
[systems@localhost ~]$ dd if=/dev/zero of=./testing bs=1024M count=10
10+0 records in
10+0 records out
10737418240 bytes (11 GB) copied, 60.0275 s, 179 MB/s

Arguably this is not 100% speed for saturated 10gig, but, it is not performance to sneeze at. For a network latency based replicated filesystem with apparently decent fault tolerance.


Thanks for any comments / feedback / thoughts!

--Tim Chipman
Fortech IT Solutions
http://FortechITSolutions.ca