Cluster what?
- Thread starter wahmed
- Start date
Hi,
should this be an joke?
I guess it's an cluster running on an kvm baremetal using nested virtualisation.
Udo
No joke at all. This is for real. Yes it is a Virtualized Proxmox Cluster inside a Physical Proxmox Cluster. And it is fully functional without any major issue! I wanted to see if it could be done. I see some major benefits having a cluster inside a cluster. Great way to learn and test Proxmox cluster without setting up whole line of hardware. If i want to take screenshot for educational purpose i dont have to worry about masking important information i dont want whole world to know. Great for setting up Computer Lab for training cluster. Even if somebody wants to learn from long distance, i can easily provide some Real VMs and he/she can setup an entire cluster from ground up. Currently the nested cluster has a FreeNAS for NFS and iscsi. Planning to build a complex cluster using this initial setup as foundation.
I wander if i can setup another nested Cluster inside this nested Cluster inside the Physical Cluster. LOL.
AMD or INTEL CPUs?
Hi, I somewhat already played with that, to learn cluster operations, see here
http://pve.proxmox.com/wiki/Nested_Virtualization
if you have any hint or new approach, you could add to, or edit this page
Marco
http://pve.proxmox.com/wiki/Nested_Virtualization
if you have any hint or new approach, you could add to, or edit this page
Marco
I have such a cluster (2 nodes), on a MacBook pro + virtualbox! And DRBD, so live migration works fine.
BUT, you dont have hardware virtualization inside your guest proxmox : your VMs will run, slowly... But they run. Fine for educational purpose. I tried lubuntu.
Yesterday I tried new kernel 3.10.0 from pvetest, it boots but init sequence never ends for me.
3.10 is supposed to bring nested kvm acceleration, exactely what you need!
Christophe.
If you want to be able to run with nested kvm all you need is the following:
1) On each host add the following to /etc/modules :
Intel CPU: kvm-intel nested=1
AMD CPU: kvm-amd nested=1
2) Choose any CPU type for VM except kvm or qemu
3) In the VM add the following to /etc/modules :
Intel CPU: kvm-intel
AMD CPU: kvm-amd
That's it. You should now be able to install proxmox in a VM a be able to enable kvm for the VM's created in the virtualized proxmox.
1) On each host add the following to /etc/modules :
Intel CPU: kvm-intel nested=1
AMD CPU: kvm-amd nested=1
2) Choose any CPU type for VM except kvm or qemu
3) In the VM add the following to /etc/modules :
Intel CPU: kvm-intel
AMD CPU: kvm-amd
That's it. You should now be able to install proxmox in a VM a be able to enable kvm for the VM's created in the virtualized proxmox.
Nesting is not stable and not maintained or tested regularly here. (and does not work with 2.6.32 kernel anyways, use 3.10 in host and also guest).
If you really want to play with it, go for AMD cpu. My last test with Intel were not that successful.
and we are not alone with such issues, see http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=731317
If you really want to play with it, go for AMD cpu. My last test with Intel were not that successful.
and we are not alone with such issues, see http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=731317
It has been like this since pve-3.0. In pve-2.x you had to add a special argument to the vm.conf file.
And yes, this is quit stable and has been for years.
And yes, this is quit stable and has been for years.
The physical Proxmox cluster has 4 nodes with Intel i7s.
I also did not know about the nested module configuration. Going to give that a try. I did notice VMs inside Nested cluster cannot be turned on if KVM hardware virtualization option is set to Yes. No doubt because of the missing nested config.
AMD CPUs certainly gives more cores for money. But i had to go with Intel platform stability and i yet have to find a AMD low profile heatsink for 1U chassis. Besides Opteron, is AMD FX-9590 is the best AMD choice for virtualization? I heard that it creates amazing amount of heat. Anybody had experience with this?
please, can anyone explain/clarify? some (experienced) user says "full" nesting works since years, while proxmox staff says it never worked...
anyway, the procedure suggested by @mir above needs a physical host restart?
Marco
I thought I did this already. As its not stable its unstable
- that means it works for some, but not for me on latest Intel CPUs. If you want to test, use 3.10 kernel on the host and use amd cpu. I suggest you try it by yourself and report results.
I thought I did this already. As its not stable its unstable
I suggest you try it by yourself and report results.
I have no amd cpu unfortunately, just intel and not latest.
btw, to me, unstable means that it should work in a defined way, but it often fails due to bugs.
If it works on some cpu vendors and not others, to me it means it is not supported everywhere, not that is unstable.
I think following questions are still "fully" unaswered, perhaps others will be able to add:
- is "full" nested virtualization possible in pve? (probably it depends)
- if it depends, on which software/hardware requirements? (perhaps amd physical cpu, but which, and pve version?)
- any software requirement "outside" of standard pve repositories? (probably not)
- if requirements matched, how to enable? (probably just edit /etc/modules as mir suggested above; reboot required?)
- if enabled, any major drawbacks? (apart from obvious overhead?)
Marco
Agree with you there.
From the module config modification suggested by Mir in the beginning of this thread tells me that Nested Virtualization does exists and there are enough supporter there that the modification is officially recognized.
It also has been pointed out that Stable/Unstable in Nested environment somewhat depends on platform itself. The only way to truly find out is to try for ourselves.
I can only share my experience from what i have been trying lately. I have been creating a whole colony of VM Servers inside nested virtualization, including CEPH and NFS Servers, Web Servers etc and few Windows VMs. I would say the setup had been pretty stable since i did not have restart or spend any time on fixing issues. Nested VMs are working as they would on a physical machine. In my platform i only have Intel CPUs. I did not make any modification to /etc/modules to activate nested option. So obviously there are no hardware acceleration happening. Of course the I/O of disk R/W is not the greatest, but it is not so bad either. After all, it is trying to write in a VM which is inside another VM. But if we take out the slow R/W from the equation, the nested virtualization is working great. I am running the latest PVE.
just to report results so far on my servers (Intel(R) Xeon(R) CPU E5520 @ 2.27GHz)
running kernel 2.6.32-26-pve the /etc/modules modified with "kvm-intel nested=1"
does not work, in the nested pve it does not expose virt extensions.
Marco