Noob friendly proxmox guide for how to setup a two node proxmox and a replica glusterfs volume,
(as well as passing through the glusterfs to a container)
Step 1-7 needs to be done on every node (server).
1. Install to the first sata/sas disk, keep only one disk inside. Make a bootable usb with win32diskimager or suse image tool, NB, this might have to be done every time it fails too, as proxmox tends to make the usb stick non bootable if it fails for some reason. (learned the hard way)
2. Setup proxmox, put easily remembered IP addresses. You cannot easily change this ip address later, so choose carefully. (I used node1.c6100.local as hostname)
3. Add the proxmox apt source to /etc/apt/sources.list skip if it already exists (important!!)
4. Apt-get update and install the glusterfs-server if it’s not installed)
Run “service glusterfs-server start”
5. Set up the extra drives or partitions with
Fdisk -l
Then select the disk you want to use with “fdisk /dev/sdx” (replace sdx with the disk you want to use)
Use “n” to create a new partition.
Select the number and sizes, etc +512G if you want to create a partition of 512GB.
If you do get stuck on fdisk a quick google search will help you
6. After this create the filesystem on the disk or partition you want to use:
mkfs.xfs -i size=512 /dev/sdbx (sdbx replace with the hdd or partition you selected earlier)
mkdir -p /data/brick1
echo '/dev/sdb1 /data/brick1 xfs defaults 1 2' >> /etc/fstab
mount -a && mount
You can also use mkfs.btrf /dev/sdb1 and echo btrfs instead of xfs to use btrfs filesystem (not
recommended for production! NB if you use sdb instead of sdb1 btrfs will setup the entire disk partitionless)
7. Make a new folder : mkdir -p /data/brick1/gv0 (I use gv0 as glustervolume 0)
8. Now create a proxmox cluster from the first node, to do this run “ pvecm create YOUR-CLUSTER-NAME” then check the state of the cluster with pvecm status
9. On the second node use : pvecm add IP-ADDRESS-CLUSTER (Use the ip-address of the node that you created the cluster on.)
10. From one server: (I personally like to use the first node and the master of proxmox cluster, also replace server1 or server2 with the either the hostname or the ip address.)
gluster volume create gv0 replica 2 server1:/data/brick1/gv0 server2:/data/brick1/gv0
gluster volume start gv0
(congrats, you have now started a replica glusterfs cluster, add hdd’s for extra space, add extra nodes for performance)
You can also make a distributed storage for performance with simply removing replica 2 from the first line.
11. To mount the glusterfs volume on proxmox, go to datacenter, then to storage, and add.
From then, add in the glusterfs server one and two in server and backup server. Select what you want to store on it, and press enter.
Congrats, you should now see the shared storage under datacenter, and both nodes should have access to it.
12. To mount an glusterfs volume on a linux container.
Edit the conf file to the container (/etc/pve/lxc/###.conf)
Add this line: (lxc.mount.entry = /host/mount /mount none bind.create=dir 0 0)
Example: (lxc.mount.entry: /mnt/pve/gv0 replicastorage none bind,create=dir 0 0)
(The second /mount is where the local directory on the container will be mounted)
I made this guide based on my own experience on what worked for me, feel free to delete my thread or come with suggestion and comments so i can edit and update it
(as well as passing through the glusterfs to a container)
Step 1-7 needs to be done on every node (server).
1. Install to the first sata/sas disk, keep only one disk inside. Make a bootable usb with win32diskimager or suse image tool, NB, this might have to be done every time it fails too, as proxmox tends to make the usb stick non bootable if it fails for some reason. (learned the hard way)
2. Setup proxmox, put easily remembered IP addresses. You cannot easily change this ip address later, so choose carefully. (I used node1.c6100.local as hostname)
3. Add the proxmox apt source to /etc/apt/sources.list skip if it already exists (important!!)
4. Apt-get update and install the glusterfs-server if it’s not installed)
Run “service glusterfs-server start”
5. Set up the extra drives or partitions with
Fdisk -l
Then select the disk you want to use with “fdisk /dev/sdx” (replace sdx with the disk you want to use)
Use “n” to create a new partition.
Select the number and sizes, etc +512G if you want to create a partition of 512GB.
If you do get stuck on fdisk a quick google search will help you
6. After this create the filesystem on the disk or partition you want to use:
mkfs.xfs -i size=512 /dev/sdbx (sdbx replace with the hdd or partition you selected earlier)
mkdir -p /data/brick1
echo '/dev/sdb1 /data/brick1 xfs defaults 1 2' >> /etc/fstab
mount -a && mount
You can also use mkfs.btrf /dev/sdb1 and echo btrfs instead of xfs to use btrfs filesystem (not
recommended for production! NB if you use sdb instead of sdb1 btrfs will setup the entire disk partitionless)
7. Make a new folder : mkdir -p /data/brick1/gv0 (I use gv0 as glustervolume 0)
8. Now create a proxmox cluster from the first node, to do this run “ pvecm create YOUR-CLUSTER-NAME” then check the state of the cluster with pvecm status
9. On the second node use : pvecm add IP-ADDRESS-CLUSTER (Use the ip-address of the node that you created the cluster on.)
10. From one server: (I personally like to use the first node and the master of proxmox cluster, also replace server1 or server2 with the either the hostname or the ip address.)
gluster volume create gv0 replica 2 server1:/data/brick1/gv0 server2:/data/brick1/gv0
gluster volume start gv0
(congrats, you have now started a replica glusterfs cluster, add hdd’s for extra space, add extra nodes for performance)
You can also make a distributed storage for performance with simply removing replica 2 from the first line.
11. To mount the glusterfs volume on proxmox, go to datacenter, then to storage, and add.
From then, add in the glusterfs server one and two in server and backup server. Select what you want to store on it, and press enter.
Congrats, you should now see the shared storage under datacenter, and both nodes should have access to it.
12. To mount an glusterfs volume on a linux container.
Edit the conf file to the container (/etc/pve/lxc/###.conf)
Add this line: (lxc.mount.entry = /host/mount /mount none bind.create=dir 0 0)
Example: (lxc.mount.entry: /mnt/pve/gv0 replicastorage none bind,create=dir 0 0)
(The second /mount is where the local directory on the container will be mounted)
I made this guide based on my own experience on what worked for me, feel free to delete my thread or come with suggestion and comments so i can edit and update it
Last edited: