Awesome information thanks @LnxBil really appreciate you taking the time to go into more detail.
Are you currently using Dedupe?
We are torn between having a few stand alone ZFS hosts and deploying a Ceph cluster, ZFS looks like it will be easier for our team to manage then Ceph, obviously...