nested proxmox

[peril]

Hi all,

I have a proxmox setup and started reading about the nested KVM possiblities.

(So I installed it ... but it seems like kvm doesn't detect the svm extensions on the qemu-cpu)

[snip]
(from proxmox on proxmox kvm)
grep svm /proc/cpuinfo
vhost3:~#

(from proxmox)
vhost01:~# grep svm /proc/cpuinfo
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow rep_good pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy 3dnowprefetch
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow rep_good pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy 3dnowprefetch
vhost01:~#

Is there some special seting to twiddle to get the nested SVM working ?

--Adrian

 

[udo]

Hi,
you must add the following line in the conf-file (/etc/qemu-server/xxx.conf):

args: -enable-nesting

Then you see svm in the vm. But thats not all - i had only a short time to try. The kvm-modul at the vm will not load

Perhaps anybody else knows why?

Udo

 

[peril]

hey - thanks - I'll try it as soon as I get home! the vm fired u - but man - everything was like 100% cpu int he proxmox vm.

(and I noticed the /dev/kvm not loaded error.)

--Adrian

 

[peril]

ICK - well - it's not any faster! it may not show - let me login to the vm.

ok - now SVM shows


[snip]

vhost3:~# grep svm /proc/cpuinfo
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm up pni svm
vhost3:~#

hmm - I think its time to lmbench.

--Adrian

 

[peril]

no go - attached is a screenshot - but when I try to migrate a karmic into the container - it works .... slowly = but with the /dev/kvm cannot be found....

Then I installed proxmox again - this time with the args: -enable-nesting

and I got the following trace out of the kernel (showed up in dmesg)

[snip from dmesg of first boot]
...
tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
general protection fault: 0000 [1] PREEMPT SMP
CPU: 0
Modules linked in: kvm_amd kvm vzethdev vznetdev simfs vzrst vzcpt tun vzdquota
vzmon vzdev xt_tcpudp xt_length ipt_ttl xt_tcpmss xt_TCPMSS iptable_mangle iptab
le_filter xt_multiport xt_limit ipt_tos ipt_REJECT ip_tables x_tables ipv6 ib_is
er rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi scsi_tran
sport_iscsi bridge virtio_balloon parport_pc parport floppy psmouse pcspkr serio
_raw e1000 joydev evdev thermal button processor sg scsi_wait_scan virtio_blk dm
_mod usbhid hid usb_storage libusual sd_mod sr_mod ide_disk ide_generic ide_cd c
drom ide_core ata_piix pata_acpi ata_generic libata scsi_mod uhci_hcd usbcore i2
c_piix4 i2c_core virtio_pci virtio_ring virtio isofs msdos fat
Pid: 2914, comm: modprobe Not tainted 2.6.24-8-pve #1 ovz005
RIP: 0010:[<ffffffff884532b0>] [<ffffffff884532b0>] :kvm_amd:svm_hardware_enabl
e+0x80/0xe0
RSP: 0018:ffff81001dcb5de8 EFLAGS: 00010006
RAX: 0000000000001d01 RBX: 0000000000000010 RCX: 00000000c0000080
RDX: 0000000000000000 RSI: ffffffff88458b26 RDI: 0000000000000000
RBP: ffff81001d49b240 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000000 R11: ffffffff88453230 R12: ffffffff88420050
R13: ffffffff8845c100 R14: ffffffff8845c100 R15: ffffc200001f8618
FS: 00007fe49ff576e0(0000) GS:ffffffff80628000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f2cc8f06098 CR3: 000000001d8d6000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process modprobe (pid: 2914, veid=0, threadinfo ffff81001dcb4000, task ffff81001
e0e08e0)
Stack: ffff806920000080 000000000000ffff ffff81001dcb5e4c 0000000000000000
0000000000000000 ffffffff80247cf6 0000000000000040 ffff81001dcb5e4c
0000000000002880 ffffffff884203f7 0000000000000001 ffffffff8845c100
Call Trace:
[<ffffffff80247cf6>] on_each_cpu+0x36/0x80
[<ffffffff884203f7>] :kvm:kvm_init+0x187/0x2d0
[<ffffffff80277532>] sys_init_module+0x192/0x1af0
[<ffffffff802c16e0>] alloc_pages_current+0x0/0x160
[<ffffffff8020c69e>] system_call+0x7e/0x83


Code: 48 8b 7d 20 e8 57 c8 e4 f7 48 8b 7d 20 48 89 c3 c1 e3 0c e8
RIP [<ffffffff884532b0>] :kvm_amd:svm_hardware_enable+0x80/0xe0
RSP <ffff81001dcb5de8>
---[ end trace b201f7be3e1028e7 ]---
note: modprobe[2914] exited with preempt_count 1
BUG: scheduling while atomic: modprobe/2914/0x10000002
Pid: 2914, comm: modprobe Tainted: G D 2.6.24-8-pve #1
peril@smokey:~$ more ofile
tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
general protection fault: 0000 [1] PREEMPT SMP
CPU: 0
Modules linked in: kvm_amd kvm vzethdev vznetdev simfs vzrst vzcpt tun vzdquota
vzmon vzdev xt_tcpudp xt_length ipt_ttl xt_tcpmss xt_TCPMSS iptable_mangle iptab
le_filter xt_multiport xt_limit ipt_tos ipt_REJECT ip_tables x_tables ipv6 ib_is
er rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi scsi_tran
sport_iscsi bridge virtio_balloon parport_pc parport floppy psmouse pcspkr serio
_raw e1000 joydev evdev thermal button processor sg scsi_wait_scan virtio_blk dm
_mod usbhid hid usb_storage libusual sd_mod sr_mod ide_disk ide_generic ide_cd c
drom ide_core ata_piix pata_acpi ata_generic libata scsi_mod uhci_hcd usbcore i2
c_piix4 i2c_core virtio_pci virtio_ring virtio isofs msdos fat
Pid: 2914, comm: modprobe Not tainted 2.6.24-8-pve #1 ovz005
RIP: 0010:[<ffffffff884532b0>] [<ffffffff884532b0>] :kvm_amd:svm_hardware_enabl
e+0x80/0xe0
RSP: 0018:ffff81001dcb5de8 EFLAGS: 00010006
RAX: 0000000000001d01 RBX: 0000000000000010 RCX: 00000000c0000080
RDX: 0000000000000000 RSI: ffffffff88458b26 RDI: 0000000000000000
RBP: ffff81001d49b240 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000000 R11: ffffffff88453230 R12: ffffffff88420050
R13: ffffffff8845c100 R14: ffffffff8845c100 R15: ffffc200001f8618
FS: 00007fe49ff576e0(0000) GS:ffffffff80628000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f2cc8f06098 CR3: 000000001d8d6000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process modprobe (pid: 2914, veid=0, threadinfo ffff81001dcb4000, task ffff81001
e0e08e0)
Stack: ffff806920000080 000000000000ffff ffff81001dcb5e4c 0000000000000000
0000000000000000 ffffffff80247cf6 0000000000000040 ffff81001dcb5e4c
0000000000002880 ffffffff884203f7 0000000000000001 ffffffff8845c100
Call Trace:
[<ffffffff80247cf6>] on_each_cpu+0x36/0x80
[<ffffffff884203f7>] :kvm:kvm_init+0x187/0x2d0
[<ffffffff80277532>] sys_init_module+0x192/0x1af0
[<ffffffff802c16e0>] alloc_pages_current+0x0/0x160
[<ffffffff8020c69e>] system_call+0x7e/0x83


Code: 48 8b 7d 20 e8 57 c8 e4 f7 48 8b 7d 20 48 89 c3 c1 e3 0c e8
RIP [<ffffffff884532b0>] :kvm_amd:svm_hardware_enable+0x80/0xe0
RSP <ffff81001dcb5de8>
---[ end trace b201f7be3e1028e7 ]---
note: modprobe[2914] exited with preempt_count 1
BUG: scheduling while atomic: modprobe/2914/0x10000002
Pid: 2914, comm: modprobe Tainted: G D 2.6.24-8-pve #1

Call Trace:
[<ffffffff804c7375>] thread_return+0x103/0x67e
[<ffffffff8023cd60>] __cond_resched+0x20/0x50
[<ffffffff804c7a45>] cond_resched+0x35/0x50
[<ffffffff802ae535>] unmap_vmas+0x835/0x960
[<ffffffff802b2da7>] exit_mmap+0x87/0x140
[<ffffffff8023e7f5>] mmput+0x25/0xe0
[<ffffffff802460c7>] do_exit+0x1b7/0xab0
[<ffffffff802376e3>] __wake_up+0x43/0x70
[<ffffffff8020e102>] die+0x52/0x70
[<ffffffff88420050>] :kvm:hardware_enable+0x0/0x30
[<ffffffff804c9b09>] error_exit+0x0/0x51
[<ffffffff88420050>] :kvm:hardware_enable+0x0/0x30
[<ffffffff88453230>] :kvm_amd:svm_hardware_enable+0x0/0xe0
[<ffffffff884532b0>] :kvm_amd:svm_hardware_enable+0x80/0xe0
[<ffffffff88453243>] :kvm_amd:svm_hardware_enable+0x13/0xe0
[<ffffffff80247cf6>] on_each_cpu+0x36/0x80
[<ffffffff884203f7>] :kvm:kvm_init+0x187/0x2d0
[<ffffffff80277532>] sys_init_module+0x192/0x1af0
[<ffffffff802c16e0>] alloc_pages_current+0x0/0x160
[<ffffffff8020c69e>] system_call+0x7e/0x83

[end dmesg]

any ideas on what to really twiddle to get nested proxmox - I'm ACHING to get this working.

 

[peril]

also posted this on KVM mailing list - I found the amd-kvm argument for the /etc/modules file
[snip]
vhost04:~# cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
# Parameters can be specified after the module name.
kvm-amd nested=1

and I get the following from dmesg
...
vm: Nested Virtualization enabled (SWEET (my comment))
general protection fault: 0000 [1] PREEMPT SMP
CPU: 0
Modules linked in: kvm_amd kvm virtio_balloon parport_pc parport floppy pcspkr p
smouse serio_raw e1000 joydev evdev thermal button processor sg scsi_wait_scan v
irtio_blk dm_mod usbhid hid usb_storage libusual sd_mod sr_mod ide_disk ide_gene
ric ide_cd cdrom ide_core ata_piix pata_acpi ata_generic libata scsi_mod uhci_hc
d usbcore i2c_piix4 i2c_core virtio_pci virtio_ring virtio isofs msdos fat
Pid: 2293, comm: modprobe Not tainted 2.6.24-8-pve #1 ovz005
RIP: 0010:[<ffffffff8829b2b0>] [<ffffffff8829b2b0>] :kvm_amd:svm_hardware_enabl
e+0x80/0xe0
RSP: 0018:ffff81001b0f9de8 EFLAGS: 00010006
RAX: 0000000000001d01 RBX: 0000000000000010 RCX: 00000000c0000080
RDX: 0000000000000000 RSI: ffffffff882a0b26 RDI: 0000000000000000
RBP: ffff810018c10a80 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000000 R11: ffffffff8829b230 R12: ffffffff88268050
R13: ffffffff882a4100 R14: ffffffff882a4100 R15: ffffc200001f2618
FS: 00007f04f28096e0(0000) GS:ffffffff80628000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fc968427098 CR3: 000000001e02e000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

...

 

[peril]

Re: nested proxmox - SOLVED

I was a little bit upside down on this so here goes.

1) in the proxmox host
in /etc/modules add
kvm-amd nested=1
2) reboot
3) create the proxmox guest
4) edit the /etc/qemu-server/CTID.conf file and add
args: -enable-nesting
5) start the guest and look for the /dev/kvm file

thanks to all the folks who helped thru their suggestions - (it was just (2) - udo from here - and Alex from the kvm list)

(Most thanks to Alex who prolly sees these questions day after day...)

-Adrian