qemu-kvm-extras / qemu-system-arm / raspberry pi under ProxMox2.x

dlasher

Renowned Member
Mar 23, 2011
229
22
83
I have a small army of Raspberry Pi devices for a project, and would like to be able to set up a "dev/compile" emulated Pi within proxmox, rather than having to do it on a slow machine, or setting up a cross-compile env on a x86 machine.

http://xecdesign.com/qemu-emulating-raspberry-pi-the-easy-way/

I see directions on how to do exactly what I want, emulating a Pi, using qemu-arm, which should be, best I can tell, packaged in qemu-kvm-extras, only I can't find a good source for it, much less one I'm confident won't screw up my proxmox2.1.

Ideas?
 
Last edited:
  • Like
Reactions: stappers
PVE is i386/amd64 only.

Thanks for the reply.

That's too bad considering the number of pi/android/mipsel/etc type devices there are expected to be online in the next 2-3 years.. Would make a development/testing env a lot easier, esp when kvm has support for those emulations.
 
esp when kvm has support for those emulations.

AFAIK kvm does not support that? Only qemu is able to emulate it.

Besides, the real problem is how you would configure the HW for those devices - those devices looks quite different (compared to a PC).
 
AFAIK kvm does not support that? Only qemu is able to emulate it.

Besides, the real problem is how you would configure the HW for those devices - those devices looks quite different (compared to a PC).

Correct, I mis-stated... Qemu has support for quite a list:

https://launchpad.net/ubuntu/karmic/+package/qemu-kvm-extras

QEMU is a fast processor emulator: currently the package supports ARM, CRIS, i386, M68k (ColdFire), MIPS, PowerPC, SH4, SPARc and x86-64emulation. By using dynamic translation it achieves reasonable speed while being easy to port on new host CPUs.

http://balau82.wordpress.com/2010/10/17/trying-debian-for-arm-on-qemu/


I realize this is way outside the scope of what Proxmox does, and does best, imho, compared to a lot of competitors.. I've come to think of is as my VM swiss army knife, hence the wish to add a few more blades to the tool. :) Thanks for the response.. appreciate the time.
 
Last edited:
Is it really that far out of the scope? I have currently got ArchArm and Debian running on a virtualised RPi under QEMU. And I would also like to leverage Proxmox to manage that. It is effectively the same thing - a collection of managed QEMU machines. The parameters are a little more specific to get it working. There's just no entries in the GUI for it though. I was considering trying something like knife-proxmox to create them from the command line, but it would be nice to be able to just build a collection of RPi's to use under development.
 
  • Like
Reactions: stappers
I still think this is a great idea, and shouldn't be that hard to implement.. the hard parts, Framework, UI, etc, are already there.. But of course I'm not a proxmox dev either.
 
  • Like
Reactions: stappers
Looking here (https://pve.proxmox.com/wiki/Manual:_vm.conf) shows the options for the VM, so a lot of the properties are covered.

I run my current QEMU machine with qemu-system-arm -name "Server name" -kernel ./custom-rpi-arch-kernel -cpu arm1176 -m 256 -M versatiblepb -no-reboot -serial stdio -append "selinux=0 plymouth.enable=0 smsc95xx.turbo_mode_N dwc_tog.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 elevator=noop nozswap nortc rootwait root =/dev/sda2 panic=1 rootfstype=ext4 rw" -ha ./rpi-arch.img -net nic -net bridge,br=br0

You end up with this...
Arch Linux-2014-07-01-11-17-08.jpg

Apart from the usual gotchas for emulating a RPi (memory size, changing the drive from the SD card to a hard drive, having a later QEMU installed that supports ARM1176), that side of things should be simple. It's just a matter of defining the parts in the management scripts, which taking a look at the source code isn't to complicated. When I get 5 minutes I might try it CLI, see if it works, and if I can do it, try changing the scripts.
 
I tried this a year ago and was not satisfied with the result. I used an LXC image extracted from a working pi and installed binary emulation support in linux (binfmt_misc) for arm with qemu binary emulator (qemu-arm-static). It seems to work, yet sometimes processes hang randomly so it was totally unreliable. Simple things like apt-get update images does not work due to this hangs.
 
Here another customer who loves to see that Proxmox welcomes ARM / ARM64 guests.

FWIW, I just tried to install the package qemu-system-arm but got lots of warnings about conflicts with PVE packages.

Where the others who did reproduce the ARM effort by Docwyatt in july 2014?
@Docwyatt: can you reproduce your `qemu-system-arm --many --more --parameters` on PVE version 5?
 
Last edited:
Yes, just download the dpkg manually, decompress it and install manually.

Code:
mkdir testingLnxBil
cd testingLnxBil
wget http : //stappers.it/do303/debian/stable/main/amd64/qemu-system-arm.deb
# partial output of  wget do303  package.deb 
# Resolving stappers.it (stappers.it)... 77.72.145.78, 2a01:7c8:aaae:103::1
# HTTP request sent, awaiting response... 303 See Other
# Location: http : //ftp.nl.debian.org/debian/pool/main/q/qemu/qemu-system-arm_2.1+dfsg12+deb8u6_amd64.deb [following]
# -timestamp-  http : //ftp.nl.debian.org/debian/pool/main/q/qemu/qemu-system-arm_2.1+dfsg12+deb8u6_amd64.deb
# Resolving ftp.nl.debian.org (ftp.nl.debian.org)...
# Connecting to ftp.nl.debian.org (ftp.nl.debian.org)
# HTTP request sent, awaiting response... 200 OK
# Saving to: ‘qemu-system-arm.deb’
dpkg -x qemu-system-arm.deb .
ls
find usr

So far, so good.
Let's give it a spin ...

Code:
yanur:~/testingLnxBil# usr/bin/qemu-system-aarch64 -M ?
usr/bin/qemu-system-aarch64: error while loading shared libraries: \
 libiscsi.so.2: cannot open shared object file: No such file or directory
yanur:~/testingLnxBil#

iscsi library is on system as version 7. I made a symbolic link libiscsi.so.2 pointing to .7.

Next run yield error about missing bluetooth library.
Installing bluetooth library by apt got me a lot warnings.
Stopped with the
Yes, just download the dpkg manually, decompress it and install manually.
advice.
Removed the libiscsi symlink to get proxmox server back to original state.


Looking forward to see further advice how to get ARM / ARM64 / MIPS / POWERPC running on Proxmox VE.


Cheers
Geert Stappers
 
Hindsight, a heads up for you.

I found this thread with keyword ARM.

With keyword MIPS I found https.//forum.proxmox.com/threads/emulating-non-x86-machine-types.35801/
 
Just a thought, but could you used nested virtualization?

ie. Create a standard AMD64 VM, then install Qemu with relevant extensions inside the VM?

It's not ideal as it's an added layer of abstraction, but performance theoretically shouldn't suffer that much and saves the hassle of modifying PVE (which will probably break if you ever dist-upgrade it after that).
 
With keyword MIPS I found https.//forum.proxmox.com/threads/emulating-non-x86-machine-types.35801/

I also replied there too and I still use it up till today to build and test software for the PI on my PVE box. The package you tried is wrong, if you just use qemu-user-static, it works:

Code:
root@proxmox:~# dpkg -L qemu-user-static | grep arm
/usr/bin/qemu-arm-static
/usr/bin/qemu-armeb-static
/usr/share/man/man1/qemu-arm-static.1.gz
/usr/share/man/man1/qemu-armeb-static.1.gz
root@proxmox:~# ls -lh /usr/bin/qemu-arm-static
-rwxr-xr-x 1 root root 3,3M Okt  2 15:11 /usr/bin/qemu-arm-static

As an example:

Code:
root@proxmox:~# dpkg --print-architecture
amd64

root@proxmox:~# /zpool/raspberry-pi/raspbian-stretch/chroot.sh
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
(Debian CHROOT)root@raspberrypi:/# dpkg --print-architecture
armhf
(Debian CHROOT)root@raspberrypi:/# exit
exit
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
if you just use qemu-user-static, it works:

Code:
root@proxmox:~# dpkg --print-architecture
amd64

root@proxmox:~# /zpool/raspberry-pi/raspbian-stretch/chroot.sh
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
(Debian CHROOT)root@raspberrypi:/# dpkg --print-architecture
armhf
(Debian CHROOT)root@raspberrypi:/# exit
exit
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

That is chroot. It is the full machine virtualization that qemu-system-arm provides that I'm looking for.
 
That is chroot. It is the full machine virtualization that qemu-system-arm provides that I'm looking for.

Oh, yeah, that's not going to happen. pve-qemu-kvm is not able to do that. You can run the PI as an LXC container if you want to prefer a not so well working GUI for that, yet that's as far as you can go.

What do you want to do what a chroot cannot do? GPIO can IMHO not be emulated at the moment.
 
Oh, yeah, that's not going to happen. pve-qemu-kvm is not able to do that. You can run the PI as an LXC container if you want to prefer a not so well working GUI for that, yet that's as far as you can go.

just a slight correction - pve-qemu-kvm / qemu-server are not able to do that yet.

it is entirely possible to implement it (configuring the Qemu build to include some additional emulation targets is the small part, actually integrating those architectures into qemu-server in a meaningful way is the bigger one). of course such an ARM VM emulated in software on an x86 host is likely slower than a HW-accelerated ARM VM on an arm64 host, but it does work and for e.g., development/testing purposes it's probably fast enough :)
 
  • Like
Reactions: stappers

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!