Hi everyone,
I am setting up a test environment for a client, to see if it is feasible for production use. So far it is 'very good' after a certain amount of time-effort-modest-frustration (learning new setup and so forth).
Briefly the environment is
- 3 servers at remote data centre, no physical access but remote IPMI web-java console for OS install / raid config etc.
- LSI hardware raid, 2 x Intel SSD in raid1, carved up to a 100gig Proxmox-OS-slice; and rest is for BCache SSD
- 2 x 2Tb Hitachi SATA in raid1, for block tank for Bcache storage
- DRBD on 3 nodes setup as per the wiki advice, https://pve.proxmox.com/wiki/DRBD9
-- except I am using a Bcache dev target instead of vanilla device. In theory for better performance. So far so good
-- 1gig public interface for access to proxmox hosts
---10gig interface for DRBD communication and proxmox cluster (multicast works inside here; does not work on public interfaces)
What I've found so far,
-- had to make hostnames and IPs for proxmox cluster config, be associated with the 10gig NIC IPs
-- had to do similar for DRBD setup
-- DRBD is working and status is happy
-- an odd thing I believe is happening, the 'free disk space' in DRBD is not consistent across all 3 hosts. I believe after some review, that this is because of behaviour of thin-provision - on the host where a VM was created the thin-behaviour 'works' but when replicated to 'other host' then we have no thin behaviour.
-- I am guessing that maybe? the safer thing would be to disable the thin-tools / thin support part of the DRBD setup / (although their use - was recommended in the DRBD9 wiki proxmox howto?) - or maybe this is a self-correcting issue I need not worry about.
For reference some info is given below, not that presently there are 4 x VMs alive on the cluster, and 4 x VM Disks associated with those VMs.
Output we have back from hosts in cluster that is informative:
Anyhow. I thought I would post this query here / to ask if others using - testing DRBD9 on proxmox, have seen similar things, or have any comments.
Otherwise.
This seems to work; very nicely; more solid than I expected. I have done some 'fail tests' (hard power off a node) - and then power it back on. DRBD seems to cope quite happily, which makes me pleasantly happy-surprised.
Performance is 'pretty good' - inside a VM running on this DRBD storage I get a 'simple' disk IO benchmark thus,
Arguably this is not 100% speed for saturated 10gig, but, it is not performance to sneeze at. For a network latency based replicated filesystem with apparently decent fault tolerance.
Thanks for any comments / feedback / thoughts!
--Tim Chipman
Fortech IT Solutions
http://FortechITSolutions.ca
I am setting up a test environment for a client, to see if it is feasible for production use. So far it is 'very good' after a certain amount of time-effort-modest-frustration (learning new setup and so forth).
Briefly the environment is
- 3 servers at remote data centre, no physical access but remote IPMI web-java console for OS install / raid config etc.
- LSI hardware raid, 2 x Intel SSD in raid1, carved up to a 100gig Proxmox-OS-slice; and rest is for BCache SSD
- 2 x 2Tb Hitachi SATA in raid1, for block tank for Bcache storage
- DRBD on 3 nodes setup as per the wiki advice, https://pve.proxmox.com/wiki/DRBD9
-- except I am using a Bcache dev target instead of vanilla device. In theory for better performance. So far so good
-- 1gig public interface for access to proxmox hosts
---10gig interface for DRBD communication and proxmox cluster (multicast works inside here; does not work on public interfaces)
What I've found so far,
-- had to make hostnames and IPs for proxmox cluster config, be associated with the 10gig NIC IPs
-- had to do similar for DRBD setup
-- DRBD is working and status is happy
-- an odd thing I believe is happening, the 'free disk space' in DRBD is not consistent across all 3 hosts. I believe after some review, that this is because of behaviour of thin-provision - on the host where a VM was created the thin-behaviour 'works' but when replicated to 'other host' then we have no thin behaviour.
-- I am guessing that maybe? the safer thing would be to disable the thin-tools / thin support part of the DRBD setup / (although their use - was recommended in the DRBD9 wiki proxmox howto?) - or maybe this is a self-correcting issue I need not worry about.
For reference some info is given below, not that presently there are 4 x VMs alive on the cluster, and 4 x VM Disks associated with those VMs.
Output we have back from hosts in cluster that is informative:
Code:
check drbd status:
root@dpve1:/mnt/pve/nfs3# drbdmanage list-resources
+------------------------------------------------------------------------------------------------------------+
| Name | | State |
|------------------------------------------------------------------------------------------------------------|
| vm-100-disk-1 | | ok |
| vm-101-disk-1 | | ok |
| vm-102-disk-1 | | ok |
| vm-103-disk-1 | | ok |
+------------------------------------------------------------------------------------------------------------+
AND ALSO
root@dpve1:/mnt/pve/nfs3# drbdmanage list-nodes
+------------------------------------------------------------------------------------------------------------+
| Name | Pool Size | Pool Free | | State |
|------------------------------------------------------------------------------------------------------------|
| dpve1 | 1904640 | 1776457 | | ok |
| dpve2 | 1904640 | 1597421 | | ok |
| dpve3 | 1904640 | 1597421 | | ok |
+------------------------------------------------------------------------------------------------------------+
root@dpve1:/mnt/pve/nfs3#
humm, why is POOL FREE not identical across the board ?
HAH, difference is thin - vs - not-thin I think
LOOK FIRST ON PROXMOX NODE THREE:
root@dpve3:~# vgs
VG #PV #LV #SN Attr VSize VFree
drbdpool 1 7 0 wz--n- 1.82t 2.25g
pve 1 3 0 wz--n- 99.87g 12.26g
root@dpve3:~# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
.drbdctrl_0 drbdpool -wi-ao---- 4.00m
.drbdctrl_1 drbdpool -wi-ao---- 4.00m
drbdthinpool drbdpool twi-aotz-- 1.82t 17.85 9.21
vm-100-disk-1_00 drbdpool Vwi-aotz-- 100.02g drbdthinpool 100.00
vm-101-disk-1_00 drbdpool Vwi-aotz-- 100.02g drbdthinpool 100.00
vm-102-disk-1_00 drbdpool Vwi-aotz-- 100.02g drbdthinpool 100.00
vm-103-disk-1_00 drbdpool Vwi-aotz-- 32.01g drbdthinpool 100.00
data pve twi-a-tz-- 54.75g 0.00 0.44
root pve -wi-ao---- 24.75g
swap pve -wi-ao---- 8.00g
NOW LOOK ON PROXMOX NODE ONE:
root@dpve3:~# exit
logout
Connection to dpve3 closed.
root@dpve1:/mnt/pve/nfs3# vgs
VG #PV #LV #SN Attr VSize VFree
drbdpool 1 7 0 wz--n- 1.82t 2.25g
pve 1 3 0 wz--n- 99.87g 12.26g
root@dpve1:/mnt/pve/nfs3# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
.drbdctrl_0 drbdpool -wi-ao---- 4.00m
.drbdctrl_1 drbdpool -wi-ao---- 4.00m
drbdthinpool drbdpool twi-aotz-- 1.82t 7.49 4.09
vm-100-disk-1_00 drbdpool Vwi-aotz-- 100.02g drbdthinpool 13.69
vm-101-disk-1_00 drbdpool Vwi-aotz-- 100.02g drbdthinpool 23.44
vm-102-disk-1_00 drbdpool Vwi-aotz-- 100.02g drbdthinpool 100.00
vm-103-disk-1_00 drbdpool Vwi-aotz-- 32.01g drbdthinpool 6.58
data pve twi-a-tz-- 54.75g 0.00 0.44
root pve -wi-ao---- 24.75g
swap pve -wi-ao---- 8.00g
AND FINALLY ON PROX NODE TWO:
root@dpve1:/mnt/pve/nfs3# ssh dpve2
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Thu Jun 9 10:38:32 2016 from dpve1
root@dpve2:~# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
.drbdctrl_0 drbdpool -wi-ao---- 4.00m
.drbdctrl_1 drbdpool -wi-ao---- 4.00m
drbdthinpool drbdpool twi-aotz-- 1.82t 17.85 9.21
vm-100-disk-1_00 drbdpool Vwi-aotz-- 100.02g drbdthinpool 100.00
vm-101-disk-1_00 drbdpool Vwi-aotz-- 100.02g drbdthinpool 100.00
vm-102-disk-1_00 drbdpool Vwi-aotz-- 100.02g drbdthinpool 100.00
vm-103-disk-1_00 drbdpool Vwi-aotz-- 32.01g drbdthinpool 100.00
data pve twi-a-tz-- 54.75g 0.00 0.44
root pve -wi-ao---- 24.75g
swap pve -wi-ao---- 8.00g
root@dpve2:~# vgs
VG #PV #LV #SN Attr VSize VFree
drbdpool 1 7 0 wz--n- 1.82t 2.25g
pve 1 3 0 wz--n- 99.87g 12.26g
root@dpve2:~#Anyhow. I thought I would post this query here / to ask if others using - testing DRBD9 on proxmox, have seen similar things, or have any comments.
Otherwise.
This seems to work; very nicely; more solid than I expected. I have done some 'fail tests' (hard power off a node) - and then power it back on. DRBD seems to cope quite happily, which makes me pleasantly happy-surprised.
Performance is 'pretty good' - inside a VM running on this DRBD storage I get a 'simple' disk IO benchmark thus,
Code:
[systems@localhost ~]$ dd if=/dev/zero of=./testing bs=1024M count=10
10+0 records in
10+0 records out
10737418240 bytes (11 GB) copied, 60.0275 s, 179 MB/sArguably this is not 100% speed for saturated 10gig, but, it is not performance to sneeze at. For a network latency based replicated filesystem with apparently decent fault tolerance.
Thanks for any comments / feedback / thoughts!
--Tim Chipman
Fortech IT Solutions
http://FortechITSolutions.ca