[PDM 1.1.7] Does Live Migration of a Guest between Clusters Require Shared Storage?

[SInisterPisces]

Reference: https://pdm.proxmox.com/docs/guests.html

My Setup:

• I'm currently experimenting with PDM in a VM (the "pdmVM") running on a single-node cluster C1. So, the pdmVM lives on C1.Node1. pdmVM lives on C1.Node1's local ZFS storage for guests.
• I also have a production single node cluster C2, containing only Node1, so my only other node is found at C2.Node1. C2.Node1 uses its own local ZFS mirror for guest storage.
• PDM monitors the C1 cluster and the C2 cluster, as well as a remote PBS. Everything works as expected as far as actually using the VM.
• I presently have no shared storage set up. (Part of the reason I was interested in PDM, actually, as cluster-based migration requires shared storage at the cluster level).

A Conundrum: What happens if I try to live-migrate pdmVM with no shared storage?

• As I understand it, in a cluster, this would not work.
• The above-linked portion of the documentation re: Guests doesn't proscribe it, but also doesn't say it's possible.
• The actual Migrate dialog box contains an option to destroy the "source" after the migration is complete.

I suspect that shared storage is not required, because the option to destroy the source (Delete Source), which is enabled by default.
So, I'm guessing that what happens is some variation of "copy from source to destination, verify copy, then delete original," possibly with some sort of FS freeze operation on the guest being moved.

I also suspect it uses sftp, scp, or some other modern file transfer protocol, so it can be file system agnostic (it doesn't have to care about what kind of file system is on the source or destination).

Is that what's happening? if not, how does it actually work.

I'm not just asking because I want to know. I'd also like to open an enhancement request to add additional explanation to the documentation about what's required for migration to work/how it works, so it's clear that shared storage is or is not required. I'd like to contribute a first draft of that explanation with the enhancement request, so I want to make sure I actually understand it well enough to do that.

For me personally, I'd also like to add a note about whether or not it's possible use PDM to migrate the VM PDM is running inside from one cluster to another one. Even if that somehow works, I can't imagine it's fully seamless, especially if there's a filesystem freeze.

 

[fortechitsolutions]

Hi, I am just testing out this very feature today with my first test-install of Proxmox Data Centre Manager.
I've got it linked an 8-node proxmox cluster and also a non-related proxmox standalone node
I've been able to move some LXC Containers from one to the other
there is shared-nothing here
everything goes over the wire / connectivity between (Source, Target, and assumption that PDM host has connectivity to both <> everyone can ping directly.
I get the feeling traffic is tunneled over standard proxmox SSH/8006 API port methods

Definitely (I think) it cannot be 'seamless' - there will be some downtime - either a stop-restart at the very least.

right now I'm trying to figure out side question topic, it appears my scenario might be creating some drama with preservation of LXC setting like (priv. vs unpriv). but I may have other mitigating drama factors (Ie, proxmox v8 on one side and proxmox v9 on the other)

anyhow, it will be interesting to see what other replies there are to the thread!

Hopefully I am not misleading / too far off base.

Tim

 

[fortechitsolutions]

Footnote to mention a few thoughts after a break.

PDM seems to use new method "VZMTUNNEL" to send stuff from one place to another. I believe this is API-level-based. (ie, not SCP/SFTP so much as - pushing via trusted channel from one place to another / managed by the trusted PDM as manager-of-task). The logs when this happens are very verbose so you can read it while it is chugging / or read the logs post-migration. I am pretty certain inherently this means there is no such thing as a "Live Migration" so much as it is a "Stop, copy, Start on target and delete on source" kind of situation. I have not tried on a running VM/LXC so not sure if it will try to do clever things like "Incremental first transfer of initial data, then top-up-transfer catch-up then cut-over' kind of like what happens when you do a "Live" migration in a shared-nothing-proxmox-cluster which "does work, so long as your rate of change in the migrating-VM is low enough compared to bandwidth between the migration Start>Endpoint nodes". So that is maybe something you can test and report back.

end of the day for what it is worth, proxmox has a ton of features and config options, in terms of
HA, Cluster, Shared storage, cluster (Ceph) or cluster adjescent ( ZFS simpler-so-called replication storage configs)

one of the 'challenges' of using proxmox well is to have a good understanding of
-what is the most important about what you are doing
- how much you value uptime, vs your $$, vs your own personal time invested in setup-maintenance-config of the platform
- is this a prod vs devtest / homelab fun / etc etc - there are so many deployment - use case - scenarios

I've found over the years that 'simple is good, my time is limited, a simpler config that works more reliably is worth its weight in gold'
so many of my deployments avoid 'fancier' features like HA, Shared Storage, etc.
and I have often just 'easy' config such as - single standalone proxmox node - local PBS backups - linked to reliable offsite PBS backups. That is fine.
yes, a single proxmox node is a single point of failure. etc etc. (You have to know 'what is important here, what are expectations, etc')
or nice classic "3-5 node proxmox cluster, shared nothing storage" and again linked to PBS etc.
again, it just works, it works reliably, and you can if required live-migrate things from one node to another.

end of the day - lots and lots of 'design decisions' about - how many hardware nodes to use, what is 'optimal', how fussed are you about 'optimal', using 'all the features' because-you-can vs 'what you really need, what you want', these are all different considerations. end of the day if your environment is a bit of a 'test lab' then - all good, it is great learning environment, lots of scnenarios to test/play with.

anyhoo. Otherwise after I read this actual question you have (I am conflating some of your threads into this single reply I think)

I am pretty certain you cannot 'live migrate' anything with PDM if it was not possible to live-migrate it as part of a proxmox cluster
but maybe I am wrong, and you can, and testing it is the answer, ie, (a) create a KVM VM (b) use PDM to move it from (SOURCE) to (TARGET) and see what happens. (c) try same test again with an LXC Container. (d) clearly the test will not be your PDM instance host itself (e) and as I write this all I think for certain PDM likely cannot live-self-migrate since it would inherently create a situation of "wait for me to be idle before I move me". At the best I am sure you will get a "stop" and "start" migrate. So move PDM from one place to another most likely is best done via traditional old fashioned
dump/backup > to place you have access for both (SRC-TARGET) hosts. Bring over the backup to TARGET manually. Import/spin it up. make sure turn off the old one before do that. then you have PDM moved over from SRC>TARGET. Clearly with a bit of downtime. But it should 'just work'


Tim

 

[fortechitsolutions]

footnote, for actual maybe clarity.

- doing a test of LXC and KVM migrate (live hosts, not PDM) is a good easy way to validate this (ie, live migrate possible vs only stop-start)?
- context comment, KVM shared-nothing-live-migration in proxmox showed up as a feature when upstream KVM got this functionality (basically) and ~proxmox added support for this / and it 'just works' nicely. More or less. I think. I am pretty certain there is no such thing as the same "live migration shared nothing" for LXC even on proxmox cluster. Rather those are always with a stop-start break. again, a bit of test will let you validate all this.
- I think 'migration of self' is always problematic/impossible no matter what the platform (ie, Vmware cluster managed by VSphere VM running inside said cluster - cannot do a self-live-migrate gracefully I think for example. And I'm sure PDM is basically the same core functional limitation, "I cannot copy myself realtime to anohter place and gracefully transition from here to there realtime without downtime or interruption". Since the act of looking means that the thing is active, and thus, not idle.

Tim