Hi everyone, just a quick sanity check question. I've been meaning to do some PDM testing and finally got my kick yesterday to get on with it.
after first bit of testing I must say, 'very nice' - this looks like a great tool to enhance management options for more complex setup
when a simple 'single pane of glass for a single cluster' is not sufficient (ie, multi cluter / mutli standalone proxmox management).
So - very nice.
I seem to have bumped an issue yesterday and wanted to confirm if this is a known issue
context / use case here for me. Possibly this alone will make you go 'ah, arrgh' and explain maybe the drama. (I don't know)
I'm mostly done now, with a little adventure
Upgrade on a modest sized proxmox cluster that was running proxmox.v8 / and it is a mix of similar-but-not-identical hardware nodes
they all have been installed 'gradually, over the years' growth. Cluster started as one node a bunch of years ago. Then was scaled to a 3-node cluster, Upgraded to new proxmox version in place. More incremental growth / addition of nodes / more in-place version upgrades of proxmox versions as the years marched by. Still chugging.
Drama is that being a stack with some history, there are some old containers. I had lost track of the looming drama, with upgrade from Proxmox v8 to v9 that cgroup version changes meant life was going to be a bit interesting yesterday. Sigh. (ie, old containers don't start well / network is borked / cannot work / etc)
So my clever plan, was to light up a new hardware node, running proxmox v9
run a nested proxmoxv8 inside of this, because the new node is snappy enough (NVME raid, etc vs older sata rust disk raid) that despite being slightly insane
it gives me a place to move the old VM workload until I can help client move along with container upgrades to not-ancient OS guests.
since proxmoxv8 is purely internally accessible, not attached to public internets etc in any way this lunacy seems like a decent work around
plus all the guest workload here is lxc containers basically, and from what I've seen majority of proxmox nested performance drama might arise when doing KVM VM guests on the inner most layer of this inception-like game.
So far so good with the guests working in this nested place.
So, anyhow. enough story telling. The drama appears to arise Thus.
My clever plan was to use PDM as a convenient way to move LXC Containers from the old bad place (ie, V8>V9 upgraded proxmox, where they no longer work)
over to the nested V8Proxmox where they can run happily.
What appears to happen? with this cross-proxmox-version LXC Container move
is that everything goes perfectly. except for one snag, which I am guessing maybe is due to PDM trying to be clever, or making assumptions about config file formatting, or how the config is 'copied' or 'migrated' or something like that, in my ?edge case? where PDM is talking to different version of Proxmox on the send-vs-recieve side of the migration (?)
anyhoo. short and long is that - once I move an LXC Container over
there is an unwanted line appended to the end of the LXC Config, simply reads
and this makes the LXC container start with an error if I do not catch it - but it does start - but causes some concern (* and - behaviour inside the vm is awful, due to it being not clear if this is PRIV vs UNPRIV container. So basically it is unusable actually]. The fix it seems is just to manually copy over the CONF file from old to new proxmox, or to manually edit the CONF File post-migration to ensure the unwanted line is gone / that the conf file is 'same after migration as it was before migration'. That process is slightly harder to do with confidence if you have already done a 'delete when done' migration.
but not impossible of course if you have good backups parked somewhereon a PBS box, so trivial to 'go take a look and confirm what the CONF file used to look like'.
I've found as a reference/sanity check. If I do LXC "Migrations" via PBS method (ie, I have my old proxmox host running ProxV9 linked to a PBS server and now also my new nested ProxV8 host also linked to same PBS server). Doing a restore of an LXC Container from that PBS into the new-nested place. Things go 'perfect' as one might expect
otherwise the only side-note observation, (Side topic!) is that the PDM based lxc migration seems to use 'vzmtunnel' (API based presumably) method to push data vs PBS restores - is doing something bit different - and I wonder if the vzmt method is not-sparse-aware maybe (ie, seems to have longer copy times for similar containers?) but I could be entirely mistaken (ie, my containers are different sizes and I have trouble keeping track of what is what)
anyhow. ultimately this very long winded post was just an attempt to
- give context on this use case
- observation about odd behaviour in this setup, for lxc container <> config file small drama <> sanity check if anyone else has ever seen or if this is a bug-glitch in PDM
- and other general story time thoughts.
hope if you read this far it is not too much regretted.
Thanks
Tim
after first bit of testing I must say, 'very nice' - this looks like a great tool to enhance management options for more complex setup
when a simple 'single pane of glass for a single cluster' is not sufficient (ie, multi cluter / mutli standalone proxmox management).
So - very nice.
I seem to have bumped an issue yesterday and wanted to confirm if this is a known issue
context / use case here for me. Possibly this alone will make you go 'ah, arrgh' and explain maybe the drama. (I don't know)
I'm mostly done now, with a little adventure
Upgrade on a modest sized proxmox cluster that was running proxmox.v8 / and it is a mix of similar-but-not-identical hardware nodes
they all have been installed 'gradually, over the years' growth. Cluster started as one node a bunch of years ago. Then was scaled to a 3-node cluster, Upgraded to new proxmox version in place. More incremental growth / addition of nodes / more in-place version upgrades of proxmox versions as the years marched by. Still chugging.
Drama is that being a stack with some history, there are some old containers. I had lost track of the looming drama, with upgrade from Proxmox v8 to v9 that cgroup version changes meant life was going to be a bit interesting yesterday. Sigh. (ie, old containers don't start well / network is borked / cannot work / etc)
So my clever plan, was to light up a new hardware node, running proxmox v9
run a nested proxmoxv8 inside of this, because the new node is snappy enough (NVME raid, etc vs older sata rust disk raid) that despite being slightly insane
it gives me a place to move the old VM workload until I can help client move along with container upgrades to not-ancient OS guests.
since proxmoxv8 is purely internally accessible, not attached to public internets etc in any way this lunacy seems like a decent work around
plus all the guest workload here is lxc containers basically, and from what I've seen majority of proxmox nested performance drama might arise when doing KVM VM guests on the inner most layer of this inception-like game.
So far so good with the guests working in this nested place.
So, anyhow. enough story telling. The drama appears to arise Thus.
My clever plan was to use PDM as a convenient way to move LXC Containers from the old bad place (ie, V8>V9 upgraded proxmox, where they no longer work)
over to the nested V8Proxmox where they can run happily.
What appears to happen? with this cross-proxmox-version LXC Container move
is that everything goes perfectly. except for one snag, which I am guessing maybe is due to PDM trying to be clever, or making assumptions about config file formatting, or how the config is 'copied' or 'migrated' or something like that, in my ?edge case? where PDM is talking to different version of Proxmox on the send-vs-recieve side of the migration (?)
anyhoo. short and long is that - once I move an LXC Container over
there is an unwanted line appended to the end of the LXC Config, simply reads
Code:
unprivileged:and this makes the LXC container start with an error if I do not catch it - but it does start - but causes some concern (* and - behaviour inside the vm is awful, due to it being not clear if this is PRIV vs UNPRIV container. So basically it is unusable actually]. The fix it seems is just to manually copy over the CONF file from old to new proxmox, or to manually edit the CONF File post-migration to ensure the unwanted line is gone / that the conf file is 'same after migration as it was before migration'. That process is slightly harder to do with confidence if you have already done a 'delete when done' migration.
but not impossible of course if you have good backups parked somewhereon a PBS box, so trivial to 'go take a look and confirm what the CONF file used to look like'.I've found as a reference/sanity check. If I do LXC "Migrations" via PBS method (ie, I have my old proxmox host running ProxV9 linked to a PBS server and now also my new nested ProxV8 host also linked to same PBS server). Doing a restore of an LXC Container from that PBS into the new-nested place. Things go 'perfect' as one might expect
otherwise the only side-note observation, (Side topic!) is that the PDM based lxc migration seems to use 'vzmtunnel' (API based presumably) method to push data vs PBS restores - is doing something bit different - and I wonder if the vzmt method is not-sparse-aware maybe (ie, seems to have longer copy times for similar containers?) but I could be entirely mistaken (ie, my containers are different sizes and I have trouble keeping track of what is what)
anyhow. ultimately this very long winded post was just an attempt to
- give context on this use case
- observation about odd behaviour in this setup, for lxc container <> config file small drama <> sanity check if anyone else has ever seen or if this is a bug-glitch in PDM
- and other general story time thoughts.
hope if you read this far it is not too much regretted.
Thanks
Tim
Last edited:
