2 Node HCI/ceph + 'witness' node

No, don't. Such devices are never meant for 24/7 use and will create more trouble then being reliable. Also the performance is so low, that it will hinder the other servers. Further, two nodes as OSD servers aren't recommended either, as on recovery the chance for data loss is higher. I recommend looking into ZFS and the storage replication, with server hardware for a third node.