[SOLVED] Spread backup over multiple datastores for faster disk restore/backup

[voarsh]

Is it possible to have backups (from a development point of view) to put backups of a VM in various datastores so different bits are stored on more than one disk, so when it comes to restoring, you're utilising more than one disk (which can be faster than reading from just one disk when restoring, which may only have 50-100 MB/s speed).

I know PBS uses datastores as "separate" disks/areas for backup sets, so using the term datastore for this isn't right and not how PBS is designed.

 

[dietmar]

This is called RAID. You can use ZFS to build fast RAID arrays from multiple disks.

 

[voarsh]

This is called RAID. You can use ZFS to build fast RAID arrays from multiple disks.

I was thinking this - but wanted to avoid that.
You're right though
I don't have enough disks with big enough space (either 2.5" 1 TB 7200 RPM drives) or slower 3.5" 16 TB+ drives to sacrifice for RAID though.

 

[oversite]

Is it possible to have backups (from a development poinit of view) to put backups of a VM in various datastores so different bits are stored on more than one disk, so when it comes to restoring, you're utilising more than one disk (which can be faster than reading from just one disk when restoring, which may only have 50-100 MB/s speed).

I know PBS uses datastores as "separate" disks/areas for backup sets, so using the term datastore for this isn't right and not how PBS is designed.

PBS is very storage agnostic, you can put the backup storage PBS need on anything you like to speed it up but the design with datastore is both right and working well. ZFS is a great help here as any fast storage but one spinning disk. If the restore seems slow, i don't think it's the problem only of reading just one disk, it's much more complicated.
However, I also think that the restore is a bit slow compared to the hardware
I do believe that PBS indeed does suffer somewhat from doing things in either a single thread or at least using to few threads meaning it can not utilize all hardware. I can be wrong but from what i see when restoring from PBS, it could be running much faster if it was much more multi threaded and streaming as many parallel paths while restoring as the hardware allows. I did not check the dev mailing list and i would not be surprised if they already are discussing this.

 

[dietmar]

it could be running much faster if it was much more multi threaded and streaming as many parallel paths while restoring as the hardware allows. I did not check the dev mailing list and i would not be surprised if they already are discussing this.

We already run restore multi-threaded. The problem is that spinning disks are simply to slow for our workload (large seek times). So you need some kind of RAID, or better use SSDs.

 

[oversite]

We already run restore multi-threaded. The problem is that spinning disks are simply to slow for our workload (large seek times). So you need some kind of RAID, or better use SSDs.

That is absolutely fantastic, thank you dietmar.