I'm currently playing with Proxmox VE on a pair of ARM based SBCs, an Orange Pi 5+ (32GB) and a Raspberry Pi 5 (8GB), both sporting Cortex a76 CPUs which are pretty much on par with the Intel Atoms J5005 and N6005 I've been using for my HCI home farm using oVirt/RHV, Xcp-ng, OpenVZ, and Proxmox for some years now.
The idea was to test a resilient stack I'm using in two locations, a research lab doing lots of PoC work including ML on powerful commodity servers built as HCI and a similarly HCI-HA private home/family setup at the cost of around 50Watts total and an entry level gamer PC. Both environments hate disruptions as much as I like not to be bothered about failing VMs or systems.
I'm using JianCuo's Proxmox fork for a potential next leading edge based on ARM, which already works really well, I am even running AARCH64 VMs on storage hosted on a x86 based Ceph 3-node cluster, I can create, run and move VMs between the nodes, only live-migration fails (not across ISAs!), evidently because the two machines are not deemed compatible enough, or rather not identical.
Because the type of subclassing of CPUs we've become familiar with on x86 (e.g. x86-64-v2-AES as a CPU type that work with AMD and Intel), doesn't seem to be implemented or supported yet by KVM. E.g. even if I can set the CPU tape of a VM to A53 or A55 instead of A76, KVM will then say that such a CPU isn't available, while setting it to "max", starts the migration up to the point where the VM on the target is supposed to start with the freshly transmitted CPU state, failing back to the source machine where the VM then resumes execution.
Now, I believe this currently to be a somewhat artificially careful CPU comparison check in my case, because the compilation target on these systems is probably AARCH-8 or below, so chances of ISA differences between these CPUs are pretty minor. But it points at a larger looming issue because the Cambrian explosion of ISA extensions on ARM and RISC-V is likely to be faster than on x86, given the push or things like AI inference or security features like CHERI.
So for now I'd just like to find and snip out the check, but coming from 10.000 feet above source code and searching for the component and code which is actually executing this check (QEMU?, KVM?, Proxmox/libvirt daemons?), I'm hitting questions like: "Is Proxmox using something similar to libvirt's migration protocol V1, V2 or V3?" and then: "What is Proxmox' long-term approach to things like libvirt?"
For now, I've just read posts, that say libvirt isn't used. Understandable in a way, because to my knowledge Proxmox is a pioneer who started on a very different technical base, that included OpenVZ and perhaps even a different hypervisor and long before LXC or Libvirt came to be.
But both have been around for a while now, and maintaining proprietary APIs can be a cost that may become debilitating, when I look at how OpenVZ fared vs. Docker or Podman.
Given how Broadcom's new owners, Citrix shennanigans and Nutanix price points are creating waves in a market that is being encroached if not nearly suffucated by cloud giants I wonder if Proxmox will just continue business as usual or try to partner for scale?
What I'm trying to create personally is a hybrid private platform that includes Proxmox PVE as opertional base, Univention for IDA and integrated apps, the most important of which come from Nextcloud.
Funny, how all of them speak German, too, and may be somewhat resilient to politicians overreaching from outside Europe...
Unfortunately, only scale provides the resources required for survival and I'd like to have some hints as to how Proxmox is facing this challenge, before commiting further in your direction.
So any statements you care to make?
The idea was to test a resilient stack I'm using in two locations, a research lab doing lots of PoC work including ML on powerful commodity servers built as HCI and a similarly HCI-HA private home/family setup at the cost of around 50Watts total and an entry level gamer PC. Both environments hate disruptions as much as I like not to be bothered about failing VMs or systems.
I'm using JianCuo's Proxmox fork for a potential next leading edge based on ARM, which already works really well, I am even running AARCH64 VMs on storage hosted on a x86 based Ceph 3-node cluster, I can create, run and move VMs between the nodes, only live-migration fails (not across ISAs!), evidently because the two machines are not deemed compatible enough, or rather not identical.
Because the type of subclassing of CPUs we've become familiar with on x86 (e.g. x86-64-v2-AES as a CPU type that work with AMD and Intel), doesn't seem to be implemented or supported yet by KVM. E.g. even if I can set the CPU tape of a VM to A53 or A55 instead of A76, KVM will then say that such a CPU isn't available, while setting it to "max", starts the migration up to the point where the VM on the target is supposed to start with the freshly transmitted CPU state, failing back to the source machine where the VM then resumes execution.
Now, I believe this currently to be a somewhat artificially careful CPU comparison check in my case, because the compilation target on these systems is probably AARCH-8 or below, so chances of ISA differences between these CPUs are pretty minor. But it points at a larger looming issue because the Cambrian explosion of ISA extensions on ARM and RISC-V is likely to be faster than on x86, given the push or things like AI inference or security features like CHERI.
So for now I'd just like to find and snip out the check, but coming from 10.000 feet above source code and searching for the component and code which is actually executing this check (QEMU?, KVM?, Proxmox/libvirt daemons?), I'm hitting questions like: "Is Proxmox using something similar to libvirt's migration protocol V1, V2 or V3?" and then: "What is Proxmox' long-term approach to things like libvirt?"
For now, I've just read posts, that say libvirt isn't used. Understandable in a way, because to my knowledge Proxmox is a pioneer who started on a very different technical base, that included OpenVZ and perhaps even a different hypervisor and long before LXC or Libvirt came to be.
But both have been around for a while now, and maintaining proprietary APIs can be a cost that may become debilitating, when I look at how OpenVZ fared vs. Docker or Podman.
Given how Broadcom's new owners, Citrix shennanigans and Nutanix price points are creating waves in a market that is being encroached if not nearly suffucated by cloud giants I wonder if Proxmox will just continue business as usual or try to partner for scale?
What I'm trying to create personally is a hybrid private platform that includes Proxmox PVE as opertional base, Univention for IDA and integrated apps, the most important of which come from Nextcloud.
Funny, how all of them speak German, too, and may be somewhat resilient to politicians overreaching from outside Europe...
Unfortunately, only scale provides the resources required for survival and I'd like to have some hints as to how Proxmox is facing this challenge, before commiting further in your direction.
So any statements you care to make?