With Ceph, you need fast networking for your Storage, 10 Gbit should be the absolute minimum, better 25 or 40 Gbit.
Your data will be on all 3 nodes for redundancy, if one node fails you can still work, if 2 Nodes fail your ceph is no longer writeable.
ZFS Replication works great for 2 Nodes...