Best Storage Config for Two Node Cluster?

[midsize_erp]

I have a mac-mini with proxmox installed on it for the last year or so that I've been using to experiment with. It has a single 1TB internal HDD, but with two external USB HDD attached (4TB & 5TB).

I now have a second mac-mini, but this one has twin 1TB internal HDDs (so 2TB all up). I'd like to cluster the two. What's the best way to configure the storage and the LXC/VMs across them?

Spoiler: Host 1 Config

CPU(s): 8 x Intel(R) Core(TM) i7-3720QM CPU @ 2.60GHz (1 Socket)
Kernel Version: Linux 4.4.98-5-pve #1 SMP PVE 4.4.98-105 (Thu, 18 Jan 2018 10:20:04 +0100)
PVE Manager Version: pve-manager/4.4-22/2728f613

Spoiler: Host 2 Config

CPU(s): 8 x Intel(R) Core(TM) i7-3720QM CPU @ 2.60GHz (1 Socket)
Kernel Version: Linux 4.4.98-5-pve #1 SMP PVE 4.4.98-105 (Thu, 18 Jan 2018 10:20:04 +0100)
PVE Manager Version: pve-manager/4.4-22/2728f613

Spoiler: Host 1 Storage Config

Internal HDD (sda): 1TB
External HDD (sdb): 4TB
External HDD (sdc): 5TB

root@pve1:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 931.5G 0 disk
├─sda1 8:1 0 1007K 0 part
├─sda2 8:2 0 127M 0 part /boot/efi
└─sda3 8:3 0 931.4G 0 part
├─pve-root 251:0 0 96G 0 lvm /
├─pve-swap 251:1 0 8G 0 lvm [SWAP]
├─pve-data_tmeta 251:2 0 104M 0 lvm
│ └─pve-data-tpool 251:4 0 811.4G 0 lvm
│ ├─pve-data 251:5 0 811.4G 0 lvm
│ ├─pve-vm--111--disk--1 251:6 0 8G 0 lvm
│ ├─pve-vm--112--disk--1 251:7 0 8G 0 lvm
│ ├─pve-vm--113--disk--1 251:8 0 32G 0 lvm
│ ├─pve-vm--114--disk--1 251:9 0 16G 0 lvm
│ ├─pve-vm--110--disk--1 251:10 0 110G 0 lvm
│ ├─pve-vm--119--disk--1 251:11 0 8G 0 lvm
│ ├─pve-vm--120--disk--1 251:12 0 32G 0 lvm
│ ├─pve-vm--123--disk--1 251:13 0 8G 0 lvm
│ ├─pve-vm--123--disk--2 251:14 0 300G 0 lvm
│ ├─pve-vm--128--disk--1 251:15 0 80G 0 lvm
│ ├─pve-vm--129--disk--1 251:16 0 32G 0 lvm
│ ├─pve-vm--144--disk--1 251:17 0 8G 0 lvm
│ ├─pve-vm--145--disk--1 251:18 0 8G 0 lvm
│ ├─pve-vm--146--disk--1 251:19 0 8G 0 lvm
│ ├─pve-vm--147--disk--1 251:20 0 8G 0 lvm
│ ├─pve-vm--148--disk--1 251:21 0 8G 0 lvm
│ └─pve-vm--149--disk--1 251:22 0 8G 0 lvm
└─pve-data_tdata 251:3 0 811.4G 0 lvm
└─pve-data-tpool 251:4 0 811.4G 0 lvm
├─pve-data 251:5 0 811.4G 0 lvm
├─pve-vm--111--disk--1 251:6 0 8G 0 lvm
├─pve-vm--112--disk--1 251:7 0 8G 0 lvm
├─pve-vm--113--disk--1 251:8 0 32G 0 lvm
├─pve-vm--114--disk--1 251:9 0 16G 0 lvm
├─pve-vm--110--disk--1 251:10 0 110G 0 lvm
├─pve-vm--119--disk--1 251:11 0 8G 0 lvm
├─pve-vm--120--disk--1 251:12 0 32G 0 lvm
├─pve-vm--123--disk--1 251:13 0 8G 0 lvm
├─pve-vm--123--disk--2 251:14 0 300G 0 lvm
├─pve-vm--128--disk--1 251:15 0 80G 0 lvm
├─pve-vm--129--disk--1 251:16 0 32G 0 lvm
├─pve-vm--144--disk--1 251:17 0 8G 0 lvm
├─pve-vm--145--disk--1 251:18 0 8G 0 lvm
├─pve-vm--146--disk--1 251:19 0 8G 0 lvm
├─pve-vm--147--disk--1 251:20 0 8G 0 lvm
├─pve-vm--148--disk--1 251:21 0 8G 0 lvm
└─pve-vm--149--disk--1 251:22 0 8G 0 lvm
sdb 8:16 0 4.6T 0 disk
├─sdb1 8:17 0 4T 0 part /stone1
└─sdb2 8:18 0 561.5G 0 part /stone2
sdc 8:32 0 3.7T 0 disk
└─sdc1 8:33 0 3.7T 0 part /ferrite
loop0 7:0 0 300G 0 loop
loop1 7:1 0 2T 0 loop

Spoiler: Host 2 Storage Config

Internal HDD (sda): 1TB
Internal HDD (sdb): 1TB

root@pve2:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 931.5G 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 256M 0 part /boot/efi
└─sda3 8:3 0 931.3G 0 part
├─pve-root 251:0 0 96G 0 lvm /
├─pve-swap 251:1 0 8G 0 lvm [SWAP]
├─pve-data_tmeta 251:2 0 104M 0 lvm
│ └─pve-data 251:4 0 811.3G 0 lvm
└─pve-data_tdata 251:3 0 811.3G 0 lvm
└─pve-data 251:4 0 811.3G 0 lvm
sdb 8:16 0 931.5G 0 disk
├─sdb1 8:17 0 200M 0 part
├─sdb2 8:18 0 930.7G 0 part
└─sdb3 8:19 0 619.9M 0 part

 

[LnxBil]

What do you want to achieve with this setup or what should your cluster do?

If you want to use online migration (of KVM, not not work with LXC), you need a shared filesystem, e.g. glusterfs. If you want HA, you need at least 3 nodes and a shared filesystem. If one of your cluster members is down, you cannot administrate your system.

 

[midsize_erp]

Not looking at HA, need a third node for that which I don't have. Not worried about online migration. One of the VMs is pfSense, and I'd like to make that CARP across the second node.

I'm not really into complicated storage so more just asking before I add the second node, break something and then spend all my spare time over the next week backing it out and asking about 'the right way' afterwards.

 

[np86]

I would recommend some shared storage. NFS maybe for simplicity. If you dont want that. I think your only option is zfs then

 

[midsize_erp]

I would recommend some shared storage. NFS maybe for simplicity. If you dont want that. I think your only option is zfs then

Were you thinking that the dual internal HDD on the second host be ZFS, or that the external HDD on the first host be NFS shares? Can you clarify?

 

[np86]

Were you thinking that the dual internal HDD on the second host be ZFS, or that the external HDD on the first host be NFS shares? Can you clarify?

Think you misunderstood me. For a shared NFS I meant a seperate storage system. Else you would have problems when you decide to reboot the host running NFS share.