[SOLVED] Questions about Proxmox Ceph Cluster

[Razva]

Hello,

I'm looking into a solution for local storage Live Migration. As it seems Proxmox doesn't support "XCP-like" local storage migrations, it seems that I need to build a Ceph cluster. Let's consider a basic 3 machine cluster (PX1, PX2 and PX3) with 1 Gbps connectivity between them via a single NIC.

1) I want all VMs to exclusively use local storage. Aka if VM1 is on PX1, it should exclusively use PMX1's local storage and not try to use storage resources on PX2 and PX3.
2) I don't want data to be distributed between nodes. Aka if VM1 is on PX1, it should exclusively use space on PX1 and not replicate to PX2 and PX3.
3) I don't want to use this feature for redundancy, but only achieve Live Migration between nodes.

Is this doable?

Thank you!

 

[dcsapak]

you can do live migration with local storage (also offline) which should fulfill all your requirements...
where did you get that

As it seems Proxmox doesn't support "XCP-like" local storage migrations, it seems that I need to build a Ceph cluster.

?

 

[Razva]

you can do live migration with local storage (also offline) which should fulfill all your requirements...
where did you get that

?

Can Proxmox do local LVM on PX1 -> local LVM on PX2 without shutting down the VM? I might have missed this!

 

[ph0x]

Yes, it works, although migration might take long if the image is not already replicated.

 

[Razva]

Yes, it works, although migration might take long if the image is not already replicated.

WOW that sounds great! Had no idea that Proxmox implemented this recently!

Are there any limitations? I guess that there's a 2-3 second "delay" when switching from a node to another, but what about data that was changed while the migration is taking place?

 

[ph0x]

The migration subroutine takes care of that. First the image is transferred, then the RAM. Then changes that occured during that time and so on and so on until switch over is possible with a downtime of max. 100 ms.

 

[Razva]

The migration subroutine takes care of that. First the image is transferred, then the RAM. Then changes that occured during that time and so on and so on until switch over is possible with a downtime of max. 100 ms.

Thank you so much for your answers!

Here are some final questions:
- Can I do Ceph with 1 Gbps? If not, what's the minumum required bandwidth?
- Can I do #1 and #2 from the OP?

 

[ph0x]

Ceph over 1 GbE will be slow and if your vm images lie in the cluster, vms will also be slow. But possible it is, of course.
The hyperconverged Proxmox node will try to talk to the local OSDs first, though, so reading throughput might be okay.
Is that what #1 and #2 are about?

 

[Razva]

Ceph over 1 GbE will be slow and if your vm images lie in the cluster, vms will also be slow. But possible it is, of course.
The hyperconverged Proxmox node will try to talk to the local OSDs first, though, so reading throughput might be okay.
Is that what #1 and #2 are about?

Can I choose what VM gets replicated and/or not?

 

[ph0x]

Of course, you have to define the replication yourself.

See here

 

[yaboc]

not to hijack the thread but is proxmox cluster network still used for migration on shared storage? if corosync is on 1gb dedicated ring will migration on shared storage have huge performance impact? can migration network be switched to vm traffic network? or will that introduce performance issues as well?

 

[ph0x]

Usually the "first" network is used, which is the one in /etc/hosts. You can, however, define a migration network under Datacenter > Options
If this has performance issues solely depends on your network layout. If it's a completely separated network, the VM traffic will stay unaffected. If it's a VLAN on the same NIC, then VM traffic will of course be affected.