I wouldn't mix ceph and gluster and, to be honest, I also wouldn't ditch ceph.
Of course the HDDs are the problem, but why do you think another cluster file system would make this any better?
Maybe you can start the replacement with 4 SSDs per node and puth a separate pool on them for the io...