Hi.
I have a 6 node setup. Each node has a 500GB ssd and two 1Gb/s nics. It's not fast but it's fun and steady.
I've created a 6 node proxmox cluster of the 6 nodes with my nas being a qdevice to maintain quorum.
I've also created a 6 node ceph cluster, each node with a single OSD. The ceph cluster runs in a 3/2 configuration.
Thinking of proxmox and ceph as independant, physical server 1 holds proxmox node 1 and ceph node 1, physical server 2 holds proxmox node 2 and ceph node 2 and so on.
If I create a VM on proxmox node 1 the ceph cluster needs to put 3 replicas of the data on 3 different OSD's.
If those 3 OSD's are on ceph node 2, 3 and 6, all reads and writes from the VM on proxmox node 1 are limited to the speed of the 1Gb/s nics.
But if the data is stored on ceph node 1, 3 and 6 instead all writes would be limited to the 1Gb/s nic speed even though the local replica could be written at the 6Gb/s speed of the local ssd, but data wouldn't be considered written until the 2 other replicas are also okay, hence the limiting factor would still be the nic speed. However read speed would be 6Gb/s because it would just read from the replica on the local ssd.
Question is, does proxmox influence which ceph OSD's holds the replicas, ensuring that one of the replicas lives on the same physical node that holds the proxmox VM or are the two completely independent?
I have a 6 node setup. Each node has a 500GB ssd and two 1Gb/s nics. It's not fast but it's fun and steady.
I've created a 6 node proxmox cluster of the 6 nodes with my nas being a qdevice to maintain quorum.
I've also created a 6 node ceph cluster, each node with a single OSD. The ceph cluster runs in a 3/2 configuration.
Thinking of proxmox and ceph as independant, physical server 1 holds proxmox node 1 and ceph node 1, physical server 2 holds proxmox node 2 and ceph node 2 and so on.
If I create a VM on proxmox node 1 the ceph cluster needs to put 3 replicas of the data on 3 different OSD's.
If those 3 OSD's are on ceph node 2, 3 and 6, all reads and writes from the VM on proxmox node 1 are limited to the speed of the 1Gb/s nics.
But if the data is stored on ceph node 1, 3 and 6 instead all writes would be limited to the 1Gb/s nic speed even though the local replica could be written at the 6Gb/s speed of the local ssd, but data wouldn't be considered written until the 2 other replicas are also okay, hence the limiting factor would still be the nic speed. However read speed would be 6Gb/s because it would just read from the replica on the local ssd.
Question is, does proxmox influence which ceph OSD's holds the replicas, ensuring that one of the replicas lives on the same physical node that holds the proxmox VM or are the two completely independent?
