qemu-kvm-extras / qemu-system-arm / raspberry pi under ProxMox2.x

dlasher · Sep 28, 2012

I have a small army of Raspberry Pi devices for a project, and would like to be able to set up a "dev/compile" emulated Pi within proxmox, rather than having to do it on a slow machine, or setting up a cross-compile env on a x86 machine.

http://xecdesign.com/qemu-emulating-raspberry-pi-the-easy-way/

I see directions on how to do exactly what I want, emulating a Pi, using qemu-arm, which should be, best I can tell, packaged in qemu-kvm-extras, only I can't find a good source for it, much less one I'm confident won't screw up my proxmox2.1.

Ideas?

dlasher · Oct 9, 2012

Nothing? No suggestions? ack!

dietmar · Oct 9, 2012

PVE is i386/amd64 only.

dlasher · Oct 9, 2012

dietmar said:
PVE is i386/amd64 only.

Thanks for the reply.

That's too bad considering the number of pi/android/mipsel/etc type devices there are expected to be online in the next 2-3 years.. Would make a development/testing env a lot easier, esp when kvm has support for those emulations.

dietmar · Oct 9, 2012

dlasher said:
esp when kvm has support for those emulations.

AFAIK kvm does not support that? Only qemu is able to emulate it.

Besides, the real problem is how you would configure the HW for those devices - those devices looks quite different (compared to a PC).

dlasher · Oct 9, 2012

dietmar said:
AFAIK kvm does not support that? Only qemu is able to emulate it.

Besides, the real problem is how you would configure the HW for those devices - those devices looks quite different (compared to a PC).

Correct, I mis-stated... Qemu has support for quite a list:

https://launchpad.net/ubuntu/karmic/+package/qemu-kvm-extras

QEMU is a fast processor emulator: currently the package supports ARM, CRIS, i386, M68k (ColdFire), MIPS, PowerPC, SH4, SPARc and x86-64emulation. By using dynamic translation it achieves reasonable speed while being easy to port on new host CPUs.

http://balau82.wordpress.com/2010/10/17/trying-debian-for-arm-on-qemu/

I realize this is way outside the scope of what Proxmox does, and does best, imho, compared to a lot of competitors.. I've come to think of is as my VM swiss army knife, hence the wish to add a few more blades to the tool.

Thanks for the response.. appreciate the time.

Docwyatt · Jun 30, 2014

Is it really that far out of the scope? I have currently got ArchArm and Debian running on a virtualised RPi under QEMU. And I would also like to leverage Proxmox to manage that. It is effectively the same thing - a collection of managed QEMU machines. The parameters are a little more specific to get it working. There's just no entries in the GUI for it though. I was considering trying something like knife-proxmox to create them from the command line, but it would be nice to be able to just build a collection of RPi's to use under development.

dlasher · Jun 30, 2014

I still think this is a great idea, and shouldn't be that hard to implement.. the hard parts, Framework, UI, etc, are already there.. But of course I'm not a proxmox dev either.

Docwyatt · Jul 1, 2014

Looking here (https://pve.proxmox.com/wiki/Manual:_vm.conf) shows the options for the VM, so a lot of the properties are covered.

I run my current QEMU machine with qemu-system-arm -name "Server name" -kernel ./custom-rpi-arch-kernel -cpu arm1176 -m 256 -M versatiblepb -no-reboot -serial stdio -append "selinux=0 plymouth.enable=0 smsc95xx.turbo_mode_N dwc_tog.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 elevator=noop nozswap nortc rootwait root =/dev/sda2 panic=1 rootfstype=ext4 rw" -ha ./rpi-arch.img -net nic -net bridge,br=br0

You end up with this...

Apart from the usual gotchas for emulating a RPi (memory size, changing the drive from the SD card to a hard drive, having a later QEMU installed that supports ARM1176), that side of things should be simple. It's just a matter of defining the parts in the management scripts, which taking a look at the source code isn't to complicated. When I get 5 minutes I might try it CLI, see if it works, and if I can do it, try changing the scripts.

dlasher · Sep 29, 2017

Bringing up an old thread.. are we any closer to ARM support under PVE?

LnxBil · Sep 29, 2017

I tried this a year ago and was not satisfied with the result. I used an LXC image extracted from a working pi and installed binary emulation support in linux (binfmt_misc) for arm with qemu binary emulator (qemu-arm-static). It seems to work, yet sometimes processes hang randomly so it was totally unreliable. Simple things like apt-get update images does not work due to this hangs.

stappers · Feb 17, 2018

Here another customer who loves to see that Proxmox welcomes ARM / ARM64 guests.

FWIW, I just tried to install the package qemu-system-arm but got lots of warnings about conflicts with PVE packages.

Where the others who did reproduce the ARM effort by Docwyatt in july 2014?
@Docwyatt: can you reproduce your `qemu-system-arm --many --more --parameters` on PVE version 5?

LnxBil · Feb 18, 2018

stappers said:
FWIW, I just tried to install the package qemu-system-arm but got lots of warnings about conflicts with PVE packages.

Yes, just download the dpkg manually, decompress it and install manually.

stappers · Feb 18, 2018

LnxBil said:
Yes, just download the dpkg manually, decompress it and install manually.

Code:

mkdir testingLnxBil
cd testingLnxBil
wget http : //stappers.it/do303/debian/stable/main/amd64/qemu-system-arm.deb
# partial output of  wget do303  package.deb 
# Resolving stappers.it (stappers.it)... 77.72.145.78, 2a01:7c8:aaae:103::1
# HTTP request sent, awaiting response... 303 See Other
# Location: http : //ftp.nl.debian.org/debian/pool/main/q/qemu/qemu-system-arm_2.1+dfsg12+deb8u6_amd64.deb [following]
# -timestamp-  http : //ftp.nl.debian.org/debian/pool/main/q/qemu/qemu-system-arm_2.1+dfsg12+deb8u6_amd64.deb
# Resolving ftp.nl.debian.org (ftp.nl.debian.org)...
# Connecting to ftp.nl.debian.org (ftp.nl.debian.org)
# HTTP request sent, awaiting response... 200 OK
# Saving to: ‘qemu-system-arm.deb’
dpkg -x qemu-system-arm.deb .
ls
find usr

So far, so good.
Let's give it a spin ...

Code:

yanur:~/testingLnxBil# usr/bin/qemu-system-aarch64 -M ?
usr/bin/qemu-system-aarch64: error while loading shared libraries: \
 libiscsi.so.2: cannot open shared object file: No such file or directory
yanur:~/testingLnxBil#

iscsi library is on system as version 7. I made a symbolic link libiscsi.so.2 pointing to .7.

Next run yield error about missing bluetooth library.
Installing bluetooth library by apt got me a lot warnings.
Stopped with the

Yes, just download the dpkg manually, decompress it and install manually.

advice.
Removed the libiscsi symlink to get proxmox server back to original state.

Looking forward to see further advice how to get ARM / ARM64 / MIPS / POWERPC running on Proxmox VE.

Cheers
Geert Stappers

stappers · Feb 18, 2018

Hindsight, a heads up for you.

I found this thread with keyword ARM.

With keyword MIPS I found https.//forum.proxmox.com/threads/emulating-non-x86-machine-types.35801/

Stewge · Feb 19, 2018

Just a thought, but could you used nested virtualization?

ie. Create a standard AMD64 VM, then install Qemu with relevant extensions inside the VM?

It's not ideal as it's an added layer of abstraction, but performance theoretically shouldn't suffer that much and saves the hassle of modifying PVE (which will probably break if you ever dist-upgrade it after that).

LnxBil · Feb 21, 2018

stappers said:
With keyword MIPS I found https.//forum.proxmox.com/threads/emulating-non-x86-machine-types.35801/

I also replied there too and I still use it up till today to build and test software for the PI on my PVE box. The package you tried is wrong, if you just use qemu-user-static, it works:

Code:

root@proxmox:~# dpkg -L qemu-user-static | grep arm
/usr/bin/qemu-arm-static
/usr/bin/qemu-armeb-static
/usr/share/man/man1/qemu-arm-static.1.gz
/usr/share/man/man1/qemu-armeb-static.1.gz
root@proxmox:~# ls -lh /usr/bin/qemu-arm-static
-rwxr-xr-x 1 root root 3,3M Okt  2 15:11 /usr/bin/qemu-arm-static

As an example:

Code:

root@proxmox:~# dpkg --print-architecture
amd64

root@proxmox:~# /zpool/raspberry-pi/raspbian-stretch/chroot.sh
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
(Debian CHROOT)root@raspberrypi:/# dpkg --print-architecture
armhf
(Debian CHROOT)root@raspberrypi:/# exit
exit
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

stappers · Feb 26, 2018

LnxBil said:

if you just use qemu-user-static, it works:

Code:

root@proxmox:~# dpkg --print-architecture
amd64

root@proxmox:~# /zpool/raspberry-pi/raspbian-stretch/chroot.sh
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
(Debian CHROOT)root@raspberrypi:/# dpkg --print-architecture
armhf
(Debian CHROOT)root@raspberrypi:/# exit
exit
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

That is chroot. It is the full machine virtualization that qemu-system-arm provides that I'm looking for.

LnxBil · Feb 27, 2018

stappers said:
That is chroot. It is the full machine virtualization that qemu-system-arm provides that I'm looking for.

Oh, yeah, that's not going to happen. pve-qemu-kvm is not able to do that. You can run the PI as an LXC container if you want to prefer a not so well working GUI for that, yet that's as far as you can go.

What do you want to do what a chroot cannot do? GPIO can IMHO not be emulated at the moment.

fabian · Feb 28, 2018

LnxBil said:
Oh, yeah, that's not going to happen. pve-qemu-kvm is not able to do that. You can run the PI as an LXC container if you want to prefer a not so well working GUI for that, yet that's as far as you can go.

just a slight correction - pve-qemu-kvm / qemu-server are not able to do that yet.

it is entirely possible to implement it (configuring the Qemu build to include some additional emulation targets is the small part, actually integrating those architectures into qemu-server in a meaningful way is the bigger one). of course such an ARM VM emulated in software on an x86 host is likely slower than a HW-accelerated ARM VM on an arm64 host, but it does work and for e.g., development/testing purposes it's probably fast enough

qemu-kvm-extras / qemu-system-arm / raspberry pi under ProxMox2.x

Renowned Member

Renowned Member

Proxmox Staff Member

Renowned Member

Proxmox Staff Member

Renowned Member

New Member

Renowned Member

New Member

Renowned Member

Distinguished Member

New Member

Distinguished Member

New Member

New Member

Renowned Member

Distinguished Member

New Member

Distinguished Member

Proxmox Staff Member