E
Ein
Guest
I'm planning on building a sizable home cluster using Proxmox and Ceph, and I haven't been able to find anyone doing anything similar to what I have planned, so I wanted to run it by some users to see if it makes sense.
I'll be purchasing 3 identical servers, each with a small-ish SSD (~64 GB) and a large number of 3 TB hard drives. (Well, I'll start with a small number of them, but I can just add more later.) For each server, I plan on partitioning the SSD in half, using half for Proxmox, and using the other half for a Ceph VM. Then, I'll pass through the hard drives to the Ceph VM, which will have a Ceph Monitor and MDS running on it, along with a UDS for each hard drive. The Ceph VMs would form their own cluster (somewhat independent of Proxmox).
Then, I want to use Proxmox to create HA VMs, in particular an HA fileserver VM which runs a Samba instance, using CephFS as the backend for Samba's data. That VM would have its image allocated on the Ceph cluster (I've checked and apparently you can use RDB and CephFS on the same cluster without conflict).
The idea is that the fileserver can be migrated between hosts without issue, so hosts can go down for maintenance (if needed) without losing connections. Furthermore, there should be at most a 30-second gap in service if the host that happens to be hosting the fileserver crashes for some reason, without any loss of data. Also, I plan on adding some extra HA VMs (also allocated on Ceph) for later, like a Mumble server and a Terraria server.
My Proxmox-related questions are:
I also some have Ceph-related questions (should the MDS and monitor daemons be run in their own VMs, or would it be better to run them all on the same one?), if anyone has experience in that area, but I realize that this is not a Ceph forum.
Thanks in advance for your input.
I'll be purchasing 3 identical servers, each with a small-ish SSD (~64 GB) and a large number of 3 TB hard drives. (Well, I'll start with a small number of them, but I can just add more later.) For each server, I plan on partitioning the SSD in half, using half for Proxmox, and using the other half for a Ceph VM. Then, I'll pass through the hard drives to the Ceph VM, which will have a Ceph Monitor and MDS running on it, along with a UDS for each hard drive. The Ceph VMs would form their own cluster (somewhat independent of Proxmox).
Then, I want to use Proxmox to create HA VMs, in particular an HA fileserver VM which runs a Samba instance, using CephFS as the backend for Samba's data. That VM would have its image allocated on the Ceph cluster (I've checked and apparently you can use RDB and CephFS on the same cluster without conflict).
The idea is that the fileserver can be migrated between hosts without issue, so hosts can go down for maintenance (if needed) without losing connections. Furthermore, there should be at most a 30-second gap in service if the host that happens to be hosting the fileserver crashes for some reason, without any loss of data. Also, I plan on adding some extra HA VMs (also allocated on Ceph) for later, like a Mumble server and a Terraria server.
My Proxmox-related questions are:
- Is PVE capable of launching the Ceph VMs when it boots? Furthermore, is it capable of launching the fileserver VM as soon as the Ceph cluster becomes available (since it may be unavailable if all 3 hosts are powering up at the same time)?
- Will the fileserver VM be able to keep its IP address as it is migrated between hosts? Does the live migration keep existing TCP connections open, or will they need to be re-established?
- I'm planning on using the SSD partition of the Ceph VM to store the Ceph UDS journals. It's the same physical drive as Proxmox's OS, though - do you think contention will be an issue?
- Are there any issues or inefficiencies that you foresee with this setup? I'm probably going to get pretty decent CPUs because I'm afraid they'll be over utilized, between the multiple VMs and the fact that any IO to the cluster would incur a non-trivial CPU overhead in Ceph.
- How fast are network connections that are "internal" to a single host? That is, if two VMs on the same host communicate over a virtual network, how much overhead will that be?
- I'll be using PCIe HBAs for the hard drives (but not the SSD - that can go on the motherboard's SATA controller). Can I pass these directly through to the Ceph VM? I understand that I'll need a motherboard and processor with VT-d support for this.
I also some have Ceph-related questions (should the MDS and monitor daemons be run in their own VMs, or would it be better to run them all on the same one?), if anyone has experience in that area, but I realize that this is not a Ceph forum.
Thanks in advance for your input.