8.3 CephFS use case

To store iso images, templates etc inside the Ceph cluster to leverage the benefits of Ceph (redundancy, performance, whatever).
I wonder though, why you would even have NFS as storage if you have Ceph and vice versa, because obviouvsly one of them makes the other one kind of obsolete. How is your setup (Cluster, Ceph, NAS, PBS)?
 
the NFS server acts as an emergency VM storage in the event Ceph goes south and also use it as a cheaper medium to store ISO/Templates and etc.

Our Ceph is based on NVME + 100Gbps setup so I want it to reserve for VMs data only.

our Cluster has 8 compute nodes and 5 storage node (10x OSD NVME with 2x 100Gbps). Nas is synology and PBS is on another unit synology as well.
 
Files that aren't VM images would be one use case. Archival files you want to store in an EC pool maybe. Files / subsets of files that need to be read from multiple VMs at once but you don't want to mess with NFS mounts. Media and associated metadata come to mind, especially if you wanted to remove layers between the storage and the VM reading the data. The more I can 'inform' the storage layer about the actual data I'm storing, the more the storage layer can help me.