[SOLVED] ARM64 VM emulation on Proxmox 5.3 AMD64

Mikus

Member
Apr 13, 2018
23
5
23
47
Hello everyone! Proxmox 5.3 added support for emulating ARM virtual machines (experimental). Anybody tried to create and run ARM-based VM using latest Proxmox 5.3? If so, can you share VM configuration file? Thanks in advance!
 
i believe the only 2 parameters necessary are
Code:
arch: aarch64
bios: ovmf
 
  • Like
Reactions: Mikus
If someone can test this option. Perhaps next week i can test this with my pine board.
 
If someone can test this option. Perhaps next week i can test this with my pine board.
AFAIU the OP wants to emulate ARM (as a VM) on PVE (amd64), not the PVE ARM build.
 
AFAIU the OP wants to emulate ARM (as a VM) on PVE (amd64), not the PVE ARM build.

I know.

The question is to know if we can create directly a arm machine on the proxmox amd64 hypervisor, or if we have to import an existing image who was create on a arm hardware.

Actually i have a lxc container who run on my pineH64, and i want to know if this lxc image can run on my proxmox amd64

Dark26
 
Tried with no success so far to install Debian on ARM64 VM. Debian iso used: debian-9.6.0-arm64-netinst.iso

Install GUI is launched, but after option "Install" selection system hangs:

EFI stub: Booting Linux Kernel
EFI stub: EFI_RNG_PROTOCOL unavailable, no randomness supplied
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...

.. and nothing happens ...

Code:
arch: aarch64
bios: ovmf
boot: dcn
bootdisk: scsi0
cores: 1
efidisk0: local-lvm:vm-110-disk-1,size=64M
memory: 2048
name: DebianARM64
net0: virtio=00:00:00:00:00:00,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local-lvm:vm-110-disk-0,size=32G
scsi2: local:iso/debian-9.6.0-arm64-netinst.iso,media=cdrom,size=203M
scsihw: virtio-scsi-pci
smbios1: uuid=ffffffff-ffff-ffff-ffff-ffffffffffff
sockets: 1
Code:
 /usr/bin/qemu-system-aarch64 -id 110 -name DebianARM64 -chardev socket,id=qmp,path=/var/run/qemu-server/110.qmp,server,nowait -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/110.pid -daemonize -smbios type=1,uuid=fde92152-18ca-4323-ba50-72ee4472bcf6 -drive if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//AAVMF_CODE.fd -drive if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/dev/pve/vm-110-disk-1 -smp 1,sockets=1,cores=1,maxcpus=1 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/110.vnc,x509,password -cpu cortex-a57 -m 2048 -device pci-bridge,id=pci.2,chassis_nr=2,bus=pcie.0,addr=0x1f -device pci-bridge,id=pci.1,chassis_nr=1,bus=pcie.0,addr=0x1e -device usb-ehci,id=ehci,bus=pcie.0,addr=0x1 -device usb-tablet,id=tablet,bus=ehci.0,port=1 -device usb-kbd,id=keyboard,bus=ehci.0,port=2 -device virtio-gpu,id=vga,bus=pcie.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pcie.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:4a918acfd3ef -device virtio-scsi-pci,id=scsihw0,bus=pcie.0,addr=0x5 -drive file=/dev/pve/vm-110-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on -device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=200 -drive file=/var/lib/vz/template/iso/debian-9.6.0-arm64-netinst.iso,if=none,id=drive-scsi2,media=cdrom,aio=threads -device scsi-cd,bus=scsihw0.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi2,id=scsi2,bootindex=100 -netdev type=tap,id=net0,ifname=tap110i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown -device virtio-net-pci,mac=D6:A7:87:68:EC:26,netdev=net0,bus=pcie.0,addr=0x12,id=net0,bootindex=300 -machine accel=tcg,type=virt
 
Last edited:
i was thinking, if it 's work some day, it's only with VM, not containers ?
 
Maybe, it will work some day ;) Yesterday I also tried to install Ubuntu 18.04 in the same way, problem persists: after selecting option in Grub system hangs.

However last night after some goooogling I was able to launch Debian installer on ARM64 VM via commandline using qemu-system-aarch64 after bootloader (linux-deb-9-6) and initrd of the Debian installer (initrd.gz) extraction:

Code:
qemu-system-aarch64 -smp 2 -M virt -cpu cortex-a57 -m 1G \
    -initrd /var/lib/vz/template/iso/initrd.gz \
    -kernel /var/lib/vz/template/iso/linux-deb-9-6 -append "root=/dev/ram console=ttyAMA0" \
    -global virtio-blk-device.scsi=off \
    -device virtio-scsi-device,id=scsi \
    -machine accel=tcg,type=virt \
    -netdev user,id=unet -device virtio-net-device,netdev=unet \
    -net user \
    -nographic
 
Tried with no success so far to install Debian on ARM64 VM. Debian iso used: debian-9.6.0-arm64-netinst.iso

Install GUI is launched, but after option "Install" selection system hangs:

EFI stub: Booting Linux Kernel
EFI stub: EFI_RNG_PROTOCOL unavailable, no randomness supplied
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...

.. and nothing happens ...

Code:
arch: aarch64
bios: ovmf
boot: dcn
bootdisk: scsi0
cores: 1
efidisk0: local-lvm:vm-110-disk-1,size=64M
memory: 2048
name: DebianARM64
net0: virtio=00:00:00:00:00:00,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local-lvm:vm-110-disk-0,size=32G
scsi2: local:iso/debian-9.6.0-arm64-netinst.iso,media=cdrom,size=203M
scsihw: virtio-scsi-pci
smbios1: uuid=ffffffff-ffff-ffff-ffff-ffffffffffff
sockets: 1
Code:
 /usr/bin/qemu-system-aarch64 -id 110 -name DebianARM64 -chardev socket,id=qmp,path=/var/run/qemu-server/110.qmp,server,nowait -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/110.pid -daemonize -smbios type=1,uuid=fde92152-18ca-4323-ba50-72ee4472bcf6 -drive if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//AAVMF_CODE.fd -drive if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/dev/pve/vm-110-disk-1 -smp 1,sockets=1,cores=1,maxcpus=1 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/110.vnc,x509,password -cpu cortex-a57 -m 2048 -device pci-bridge,id=pci.2,chassis_nr=2,bus=pcie.0,addr=0x1f -device pci-bridge,id=pci.1,chassis_nr=1,bus=pcie.0,addr=0x1e -device usb-ehci,id=ehci,bus=pcie.0,addr=0x1 -device usb-tablet,id=tablet,bus=ehci.0,port=1 -device usb-kbd,id=keyboard,bus=ehci.0,port=2 -device virtio-gpu,id=vga,bus=pcie.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pcie.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:4a918acfd3ef -device virtio-scsi-pci,id=scsihw0,bus=pcie.0,addr=0x5 -drive file=/dev/pve/vm-110-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on -device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=200 -drive file=/var/lib/vz/template/iso/debian-9.6.0-arm64-netinst.iso,if=none,id=drive-scsi2,media=cdrom,aio=threads -device scsi-cd,bus=scsihw0.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi2,id=scsi2,bootindex=100 -netdev type=tap,id=net0,ifname=tap110i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown -device virtio-net-pci,mac=D6:A7:87:68:EC:26,netdev=net0,bus=pcie.0,addr=0x12,id=net0,bootindex=300 -machine accel=tcg,type=virt

Additional observations running (actually not running ;) ) this configuration: ARM64 VM's Bios reports 0 MB RAM o_O
 
Last edited:
Tried with no success so far to install Debian on ARM64 VM. Debian iso used: debian-9.6.0-arm64-netinst.iso

Install GUI is launched, but after option "Install" selection system hangs:

EFI stub: Booting Linux Kernel
EFI stub: EFI_RNG_PROTOCOL unavailable, no randomness supplied
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...

.. and nothing happens ...

Almost none of the installer ISOs come with graphics drivers. You need to add a serial socket, and connect to that via the xterm.js button.
After installation, when using 'virtio' graphics, you'll be able to see the machine via novnc as well.
 
  • Like
Reactions: Mikus
Almost none of the installer ISOs come with graphics drivers. You need to add a serial socket, and connect to that via the xterm.js button.
After installation, when using 'virtio' graphics, you'll be able to see the machine via novnc as well.

Thank you Wolfgang, very, very useful and helpful! I connected serial console but was in stuck with xterm console message:
Code:
starting serial terminal on interface serial0
I needed only to press enter (select default "Install" option from the grub menu in "background") after this message to start installation.

So, in the end I was able to install and run and play with Debian ARM64 VM on Proxmox 5.3, this new feature works out of the box, thank you for support once more!

Code:
arch: aarch64
bios: ovmf
boot: dcn
bootdisk: scsi0
cores: 1
efidisk0: local-lvm:vm-110-disk-1,size=64M
memory: 2048
name: DebianARM64
net0: virtio=00:00:00:00:00:00,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local-lvm:vm-110-disk-0,size=32G
scsi1: local:iso/debian-9.6.0-arm64-netinst.iso,media=cdrom,size=203M
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=ffffffff-ffff-ffff-ffff-ffffffffffff
sockets: 1
vga: serial0
Code:
/usr/bin/qemu-system-aarch64
-id 110
-name DebianARM64
-chardev socket,id=qmp,path=/var/run/qemu-server/110.qmp,server,nowait
-mon chardev=qmp,mode=control
-chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5
-mon chardev=qmp-event,mode=control
-pidfile /var/run/qemu-server/110.pid
-daemonize
-smbios type=1,uuid=ffffffff-ffff-ffff-ffff-ffffffffffff
-drive if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//AAVMF_CODE.fd
-drive if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/dev/pve/vm-110-disk-1
-smp 1,sockets=1,cores=1,maxcpus=1
-nodefaults
-boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg
-nographic
-cpu cortex-a57
-m 2048
-device pci-bridge,id=pci.2,chassis_nr=2,bus=pcie.0,addr=0x1f
-device pci-bridge,id=pci.1,chassis_nr=1,bus=pcie.0,addr=0x1e
-device usb-ehci,id=ehci,bus=pcie.0,addr=0x1
-chardev socket,id=serial0,path=/var/run/qemu-server/110.serial0,server,nowait
-serial chardev:serial0
-chardev socket,path=/var/run/qemu-server/110.qga,server,nowait,id=qga0
-device virtio-serial,id=qga0,bus=pcie.0,addr=0x8
-device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0
-device virtio-balloon-pci,id=balloon0,bus=pcie.0,addr=0x3
-iscsi initiator-name=iqn.1993-08.org.debian:01:4a918acfd3ef
-device virtio-scsi-pci,id=scsihw0,bus=pcie.0,addr=0x5
-drive file=/dev/pve/vm-110-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on
-device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100
-drive file=/var/lib/vz/template/iso/debian-9.6.0-arm64-netinst.iso,if=none,id=drive-scsi1,media=cdrom,aio=threads
-device scsi-cd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1,bootindex=200
-netdev type=tap,id=net0,ifname=tap110i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown
-device virtio-net-pci,mac=00:00:00:00:00:00,netdev=net0,bus=pcie.0,addr=0x12,id=net0,bootindex=300
-machine accel=tcg,type=virt
 
Last edited:
@dcsapak I'm trying to run an arm64 VM on an amd64 Proxmox VE by using your VM configuration options.

1) I created a regular VM - everything left by default

2) I went into /etc/pve/qemu-server/107.conf and added these two lines to the very end of the file

arch: aarch64
bios: ovmf

Now, when starting the VM, I'm getting this error:

qemu-system-aarch64: -device vmgenid,guid=fe68872b-53cd-440b-8caa-ef660b50b16c: 'vmgenid' is not a valid device model name

Any ideas what can be wrong?
 
note: arm emulation is not a really greatly tested feature, so some things may not work

also try removing the 'vmgenid' line from the config
 
@dcsapak Thanks, tried it, the next error I get is with the CD-ROM drive:

-device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200: Bus 'ide.1' not found

Removing this device, as well, the VM does start indeed, however VNC is not working yet :(

1623829729775.png

My entire command looks like this:

Code:
/usr/bin/qemu-system-aarch64
-id 107
-name arm-test
-chardev 'socket,id=qmp,path=/var/run/qemu-server/107.qmp,server,nowait'
-mon 'chardev=qmp,mode=control'
-chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5'
-mon 'chardev=qmp-event,mode=control'
-pidfile /var/run/qemu-server/107.pid
-daemonize
-smbios 'type=1,uuid=57fb8c11-a5b1-4c6b-bc8d-13dbb0f73c49'
-smp '1,sockets=1,cores=1,maxcpus=1'
-nodefaults
-boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg'
-vnc unix:/var/run/qemu-server/107.vnc,password
-cpu cortex-a57
-m 512
-device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pcie.0,addr=0x1e'
-device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pcie.0,addr=0x1f'
-device 'usb-ehci,id=ehci,bus=pcie.0,addr=0x1'
-device 'usb-tablet,id=tablet,bus=ehci.0,port=1'
-device 'usb-kbd,id=keyboard,bus=ehci.0,port=2'
-device 'virtio-gpu,id=vga,bus=pcie.0,addr=0x2'
-device 'virtio-balloon-pci,id=balloon0,bus=pcie.0,addr=0x3'
-iscsi 'initiator-name=iqn.1993-08.org.debian:01:1f77b5aa2fa'
-drive 'file=/mnt/pve/ProxmoxInstall/template/iso/ubuntu-20.04.2-live-server-arm64.iso,if=none,id=drive-ide2,media=cdrom,aio=threads'
-device 'virtio-scsi-pci,id=scsihw0,bus=pcie.0,addr=0x5'
-drive 'file=/dev/pve/vm-107-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on'
-device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100'
-netdev 'type=tap,id=net0,ifname=tap107i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown'
-device 'virtio-net-pci,mac=**:**:**:**:**:**,netdev=net0,bus=pcie.0,addr=0x12,id=net0,bootindex=300'
-machine 'accel=tcg,type=virt+pve1'
 
This guide was helpful for me:
https://rotelok.com/installing-arm64-debian-10-buster-in-a-virtual-machine/

It looks to me that IDE isn't supported and you'll need to manually edit your .conf to move your CDROM from ide2 to scisi#

I didn't read this entire thread in-depth so apologies if you've already done these things. I'm currently trying to install Debian-10.9 (10.10 is bugged atm). I got the installer to boot, but am getting UEFI boot error after the install. Probably overlooking something simple. Running PVE 7.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!