GlusterFS as datastore

[davlaw]

I have a repurposed 8 node ScaleStorage appliance rebuilt with glusterFS. Have has it in use for over a year and used for small backup jobs from bareos.

Was considering using it with PBS since it appears to be more efficient method on disk space.

So, before I disturb my current setup.....

Anyone have any working knowledge using GlusterFS as a datastore?

Current have had great luck using attached scsi and iscsi disks ( not so much with NFS, but thats old news )

Thanks

 

[davlaw]

Just thought I would update

While the software seems to work, I have have had a few problems not related to PBS.

Glusterfs seems rather slow, got timeouts and such.

I just could be my gluster configuration , its not my everyday way of life and was just set up to dump rather large backup files from bareos and repurposed some hardware.

So YMMV, but with my limited knowledge of gluster guess I gonna give a thumbs down for now.

 

[Tmanok]

Just thought I would update

While the software seems to work, I have have had a few problems not related to PBS.

Glusterfs seems rather slow, got timeouts and such.

I just could be my gluster configuration , its not my everyday way of life and was just set up to dump rather large backup files from bareos and repurposed some hardware.

So YMMV, but with my limited knowledge of gluster guess I gonna give a thumbs down for now.

Hey Davlaw,

I've got a few old servers with 3.5" bays and many 4TB CMR drives to spare, my thought is that if I created a 3-way replica between them this would greatly reduce the risk of unavailability and if I created RAID10 mirrors with the drives that would greatly reduce the risk of data loss.

When you say that your observed performance was very low, can you elaborate?

For any other users, are there recommendations on one of the following options:

1. Mount GlusterFS Volume directly to PBS VM; OR
2. Mount GlusterFS Volume to PVE Cluster using GUI and then "map" the storage to the PBS VM.

I imagine:
#1 Mounting to PBS VM Directly:

• Enables highest performance with lowest overhead.
• May pose architecture issues in many cases:
  • Security risk exposing storage network to the VM - Requiring FW rules.
  • Requires that the same bridge numbers correlate to the same networks (not always true in the case of storage, many admins just care about the IP and not the VMBR#).

#2 Mounting to PVE and then mapping to PBS:

• Is more featureful:
  • Enables easy remounting to other PBS VMs if maintenance is required.
  • Offers disk performance controls.
• Is generally more stable.
• Adds overhead and reduces performance.

Cheers,


Tmanok

 

[VictorSTS]

PBS is designed to be used with local drives. There are performance issues even with a standard NFS, so I wouldn't expect it to perform correctly on a replicated network storage.

Why not just set up 2 PBS servers and replicate among them?

 

[Tmanok]

While I'm not opposed to PBS being a VM on the local storage server, PBS is not something I have ever chosen to install bare-metal for a variety of reasons. Unless you count running it as a Bhyve VM on TrueNAS at some of the off-site locations I've managed... That's still not bare-metal but it's on the machine hosting the disks, so perhaps that counts.

My considerations for GlusterFS managing the storage are in relation to storage availability at the local site without needing to do the lengthy sync and the increase in read performance. Replicating the data to multiple storage servers doesn't improve my write speed but my read speed will increase significantly both for off-site syncs and for local restoration back to the hypervisor.

Cheers,


Tmanok