Best practices for storage replication (for a specific use case)

casparsmit

Renowned Member
Feb 24, 2015
41
2
73
Hi all,

I would like to check with the community about my findings for storage replication for a specific use case.
In short: I have a CCTV VM which stores incoming CCTV video recordings. For simplicity it's a Windows VM with a 100GB OS disk (on a ZFS RAID1 with 2x 240GB SSD's) and a 10TB data disk (on a ZFS RAID1 with 2x 10TB HDD's) for the recordings.
There's a continuous incoming data stream of around 300Mb/s (mbit, so around 30MB/s) going to the data disk. It's a kind of FIFO buffer. So the 10TB disk is always full and the oldest data is being overwritten constantly.

Now I built a cluster with the same specs and I would like to replicate this VM to that node.
I noticed that I can’t replicate this VM because ZFS immediately says out of space. Because this is a thick provisioned VM that’s expected because of the refreservation of 10TB. So it needs a 20TB+ pool (or 10TB free space in the zpool to make the snapshot).

I could turn the disk into thin provisioned by setting refreservation=none. But then I would probably corrupt the disk during the first sync (which could take a long time to sync 10TB) because there is a continuous incoming stream of data which needs to be placed somewhere and that space is not available in the pool. I know I should always use discard when thin provisioning disks but discard isn’t much helpful either in this case because all overwritten data is going into the used snapshot space.

I already have 4 times the needed storage in this setup (2x 10TB RAID1 per server) so in order to replicate thick provisioned i would need 8(!) times the actual storage which is kinda insane.

With thin provisioning I would need a large enough storage to be able to handle the delta’s between syncs (and a possible re-sync of the entire volume) and corruption being a big possibility when the storage isn’t large enough.

What would be the best way forward to achieve my goal of replicating this 10TB disk?

Is there any other way to be able to do this without risking corruption. My idea was to use ZFS to prevent corruption, not causing it.

Thanks in advance for any advice.
Kind regards,
Caspar
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!