[SOLVED] ARM64 VM emulation on Proxmox 5.3 AMD64

Mikus

Member
Apr 13, 2018
22
5
8
45
Hello everyone! Proxmox 5.3 added support for emulating ARM virtual machines (experimental). Anybody tried to create and run ARM-based VM using latest Proxmox 5.3? If so, can you share VM configuration file? Thanks in advance!
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
6,067
647
133
33
Vienna
i believe the only 2 parameters necessary are
Code:
arch: aarch64
bios: ovmf
 
  • Like
Reactions: Mikus

Dark26

Member
Nov 27, 2017
183
10
23
44
If someone can test this option. Perhaps next week i can test this with my pine board.
 

Alwin

Proxmox Staff Member
Staff member
Aug 1, 2017
4,617
439
88
If someone can test this option. Perhaps next week i can test this with my pine board.
AFAIU the OP wants to emulate ARM (as a VM) on PVE (amd64), not the PVE ARM build.
 

Dark26

Member
Nov 27, 2017
183
10
23
44
AFAIU the OP wants to emulate ARM (as a VM) on PVE (amd64), not the PVE ARM build.

I know.

The question is to know if we can create directly a arm machine on the proxmox amd64 hypervisor, or if we have to import an existing image who was create on a arm hardware.

Actually i have a lxc container who run on my pineH64, and i want to know if this lxc image can run on my proxmox amd64

Dark26
 

Mikus

Member
Apr 13, 2018
22
5
8
45
Tried with no success so far to install Debian on ARM64 VM. Debian iso used: debian-9.6.0-arm64-netinst.iso

Install GUI is launched, but after option "Install" selection system hangs:

EFI stub: Booting Linux Kernel
EFI stub: EFI_RNG_PROTOCOL unavailable, no randomness supplied
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...

.. and nothing happens ...

Code:
arch: aarch64
bios: ovmf
boot: dcn
bootdisk: scsi0
cores: 1
efidisk0: local-lvm:vm-110-disk-1,size=64M
memory: 2048
name: DebianARM64
net0: virtio=00:00:00:00:00:00,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local-lvm:vm-110-disk-0,size=32G
scsi2: local:iso/debian-9.6.0-arm64-netinst.iso,media=cdrom,size=203M
scsihw: virtio-scsi-pci
smbios1: uuid=ffffffff-ffff-ffff-ffff-ffffffffffff
sockets: 1
Code:
 /usr/bin/qemu-system-aarch64 -id 110 -name DebianARM64 -chardev socket,id=qmp,path=/var/run/qemu-server/110.qmp,server,nowait -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/110.pid -daemonize -smbios type=1,uuid=fde92152-18ca-4323-ba50-72ee4472bcf6 -drive if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//AAVMF_CODE.fd -drive if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/dev/pve/vm-110-disk-1 -smp 1,sockets=1,cores=1,maxcpus=1 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/110.vnc,x509,password -cpu cortex-a57 -m 2048 -device pci-bridge,id=pci.2,chassis_nr=2,bus=pcie.0,addr=0x1f -device pci-bridge,id=pci.1,chassis_nr=1,bus=pcie.0,addr=0x1e -device usb-ehci,id=ehci,bus=pcie.0,addr=0x1 -device usb-tablet,id=tablet,bus=ehci.0,port=1 -device usb-kbd,id=keyboard,bus=ehci.0,port=2 -device virtio-gpu,id=vga,bus=pcie.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pcie.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:4a918acfd3ef -device virtio-scsi-pci,id=scsihw0,bus=pcie.0,addr=0x5 -drive file=/dev/pve/vm-110-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on -device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=200 -drive file=/var/lib/vz/template/iso/debian-9.6.0-arm64-netinst.iso,if=none,id=drive-scsi2,media=cdrom,aio=threads -device scsi-cd,bus=scsihw0.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi2,id=scsi2,bootindex=100 -netdev type=tap,id=net0,ifname=tap110i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown -device virtio-net-pci,mac=D6:A7:87:68:EC:26,netdev=net0,bus=pcie.0,addr=0x12,id=net0,bootindex=300 -machine accel=tcg,type=virt
 
Last edited:

Dark26

Member
Nov 27, 2017
183
10
23
44
i was thinking, if it 's work some day, it's only with VM, not containers ?
 

Mikus

Member
Apr 13, 2018
22
5
8
45
Maybe, it will work some day ;) Yesterday I also tried to install Ubuntu 18.04 in the same way, problem persists: after selecting option in Grub system hangs.

However last night after some goooogling I was able to launch Debian installer on ARM64 VM via commandline using qemu-system-aarch64 after bootloader (linux-deb-9-6) and initrd of the Debian installer (initrd.gz) extraction:

Code:
qemu-system-aarch64 -smp 2 -M virt -cpu cortex-a57 -m 1G \
    -initrd /var/lib/vz/template/iso/initrd.gz \
    -kernel /var/lib/vz/template/iso/linux-deb-9-6 -append "root=/dev/ram console=ttyAMA0" \
    -global virtio-blk-device.scsi=off \
    -device virtio-scsi-device,id=scsi \
    -machine accel=tcg,type=virt \
    -netdev user,id=unet -device virtio-net-device,netdev=unet \
    -net user \
    -nographic
 

Mikus

Member
Apr 13, 2018
22
5
8
45
Tried with no success so far to install Debian on ARM64 VM. Debian iso used: debian-9.6.0-arm64-netinst.iso

Install GUI is launched, but after option "Install" selection system hangs:

EFI stub: Booting Linux Kernel
EFI stub: EFI_RNG_PROTOCOL unavailable, no randomness supplied
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...

.. and nothing happens ...

Code:
arch: aarch64
bios: ovmf
boot: dcn
bootdisk: scsi0
cores: 1
efidisk0: local-lvm:vm-110-disk-1,size=64M
memory: 2048
name: DebianARM64
net0: virtio=00:00:00:00:00:00,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local-lvm:vm-110-disk-0,size=32G
scsi2: local:iso/debian-9.6.0-arm64-netinst.iso,media=cdrom,size=203M
scsihw: virtio-scsi-pci
smbios1: uuid=ffffffff-ffff-ffff-ffff-ffffffffffff
sockets: 1
Code:
 /usr/bin/qemu-system-aarch64 -id 110 -name DebianARM64 -chardev socket,id=qmp,path=/var/run/qemu-server/110.qmp,server,nowait -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/110.pid -daemonize -smbios type=1,uuid=fde92152-18ca-4323-ba50-72ee4472bcf6 -drive if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//AAVMF_CODE.fd -drive if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/dev/pve/vm-110-disk-1 -smp 1,sockets=1,cores=1,maxcpus=1 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/110.vnc,x509,password -cpu cortex-a57 -m 2048 -device pci-bridge,id=pci.2,chassis_nr=2,bus=pcie.0,addr=0x1f -device pci-bridge,id=pci.1,chassis_nr=1,bus=pcie.0,addr=0x1e -device usb-ehci,id=ehci,bus=pcie.0,addr=0x1 -device usb-tablet,id=tablet,bus=ehci.0,port=1 -device usb-kbd,id=keyboard,bus=ehci.0,port=2 -device virtio-gpu,id=vga,bus=pcie.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pcie.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:4a918acfd3ef -device virtio-scsi-pci,id=scsihw0,bus=pcie.0,addr=0x5 -drive file=/dev/pve/vm-110-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on -device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=200 -drive file=/var/lib/vz/template/iso/debian-9.6.0-arm64-netinst.iso,if=none,id=drive-scsi2,media=cdrom,aio=threads -device scsi-cd,bus=scsihw0.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi2,id=scsi2,bootindex=100 -netdev type=tap,id=net0,ifname=tap110i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown -device virtio-net-pci,mac=D6:A7:87:68:EC:26,netdev=net0,bus=pcie.0,addr=0x12,id=net0,bootindex=300 -machine accel=tcg,type=virt

Additional observations running (actually not running ;) ) this configuration: ARM64 VM's Bios reports 0 MB RAM o_O
 
Last edited:

wbumiller

Proxmox Staff Member
Staff member
Jun 23, 2015
684
101
63
Tried with no success so far to install Debian on ARM64 VM. Debian iso used: debian-9.6.0-arm64-netinst.iso

Install GUI is launched, but after option "Install" selection system hangs:

EFI stub: Booting Linux Kernel
EFI stub: EFI_RNG_PROTOCOL unavailable, no randomness supplied
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...

.. and nothing happens ...

Almost none of the installer ISOs come with graphics drivers. You need to add a serial socket, and connect to that via the xterm.js button.
After installation, when using 'virtio' graphics, you'll be able to see the machine via novnc as well.
 
  • Like
Reactions: Mikus

Mikus

Member
Apr 13, 2018
22
5
8
45
Almost none of the installer ISOs come with graphics drivers. You need to add a serial socket, and connect to that via the xterm.js button.
After installation, when using 'virtio' graphics, you'll be able to see the machine via novnc as well.

Thank you Wolfgang, very, very useful and helpful! I connected serial console but was in stuck with xterm console message:
Code:
starting serial terminal on interface serial0
I needed only to press enter (select default "Install" option from the grub menu in "background") after this message to start installation.

So, in the end I was able to install and run and play with Debian ARM64 VM on Proxmox 5.3, this new feature works out of the box, thank you for support once more!

Code:
arch: aarch64
bios: ovmf
boot: dcn
bootdisk: scsi0
cores: 1
efidisk0: local-lvm:vm-110-disk-1,size=64M
memory: 2048
name: DebianARM64
net0: virtio=00:00:00:00:00:00,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local-lvm:vm-110-disk-0,size=32G
scsi1: local:iso/debian-9.6.0-arm64-netinst.iso,media=cdrom,size=203M
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=ffffffff-ffff-ffff-ffff-ffffffffffff
sockets: 1
vga: serial0
Code:
/usr/bin/qemu-system-aarch64
-id 110
-name DebianARM64
-chardev socket,id=qmp,path=/var/run/qemu-server/110.qmp,server,nowait
-mon chardev=qmp,mode=control
-chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5
-mon chardev=qmp-event,mode=control
-pidfile /var/run/qemu-server/110.pid
-daemonize
-smbios type=1,uuid=ffffffff-ffff-ffff-ffff-ffffffffffff
-drive if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//AAVMF_CODE.fd
-drive if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/dev/pve/vm-110-disk-1
-smp 1,sockets=1,cores=1,maxcpus=1
-nodefaults
-boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg
-nographic
-cpu cortex-a57
-m 2048
-device pci-bridge,id=pci.2,chassis_nr=2,bus=pcie.0,addr=0x1f
-device pci-bridge,id=pci.1,chassis_nr=1,bus=pcie.0,addr=0x1e
-device usb-ehci,id=ehci,bus=pcie.0,addr=0x1
-chardev socket,id=serial0,path=/var/run/qemu-server/110.serial0,server,nowait
-serial chardev:serial0
-chardev socket,path=/var/run/qemu-server/110.qga,server,nowait,id=qga0
-device virtio-serial,id=qga0,bus=pcie.0,addr=0x8
-device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0
-device virtio-balloon-pci,id=balloon0,bus=pcie.0,addr=0x3
-iscsi initiator-name=iqn.1993-08.org.debian:01:4a918acfd3ef
-device virtio-scsi-pci,id=scsihw0,bus=pcie.0,addr=0x5
-drive file=/dev/pve/vm-110-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on
-device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100
-drive file=/var/lib/vz/template/iso/debian-9.6.0-arm64-netinst.iso,if=none,id=drive-scsi1,media=cdrom,aio=threads
-device scsi-cd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1,bootindex=200
-netdev type=tap,id=net0,ifname=tap110i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown
-device virtio-net-pci,mac=00:00:00:00:00:00,netdev=net0,bus=pcie.0,addr=0x12,id=net0,bootindex=300
-machine accel=tcg,type=virt
 
Last edited:
Feb 17, 2020
61
3
8
41
@dcsapak I'm trying to run an arm64 VM on an amd64 Proxmox VE by using your VM configuration options.

1) I created a regular VM - everything left by default

2) I went into /etc/pve/qemu-server/107.conf and added these two lines to the very end of the file

arch: aarch64
bios: ovmf

Now, when starting the VM, I'm getting this error:

qemu-system-aarch64: -device vmgenid,guid=fe68872b-53cd-440b-8caa-ef660b50b16c: 'vmgenid' is not a valid device model name

Any ideas what can be wrong?
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
6,067
647
133
33
Vienna
note: arm emulation is not a really greatly tested feature, so some things may not work

also try removing the 'vmgenid' line from the config
 
Feb 17, 2020
61
3
8
41
@dcsapak Thanks, tried it, the next error I get is with the CD-ROM drive:

-device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200: Bus 'ide.1' not found

Removing this device, as well, the VM does start indeed, however VNC is not working yet :(

1623829729775.png

My entire command looks like this:

Code:
/usr/bin/qemu-system-aarch64
-id 107
-name arm-test
-chardev 'socket,id=qmp,path=/var/run/qemu-server/107.qmp,server,nowait'
-mon 'chardev=qmp,mode=control'
-chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5'
-mon 'chardev=qmp-event,mode=control'
-pidfile /var/run/qemu-server/107.pid
-daemonize
-smbios 'type=1,uuid=57fb8c11-a5b1-4c6b-bc8d-13dbb0f73c49'
-smp '1,sockets=1,cores=1,maxcpus=1'
-nodefaults
-boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg'
-vnc unix:/var/run/qemu-server/107.vnc,password
-cpu cortex-a57
-m 512
-device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pcie.0,addr=0x1e'
-device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pcie.0,addr=0x1f'
-device 'usb-ehci,id=ehci,bus=pcie.0,addr=0x1'
-device 'usb-tablet,id=tablet,bus=ehci.0,port=1'
-device 'usb-kbd,id=keyboard,bus=ehci.0,port=2'
-device 'virtio-gpu,id=vga,bus=pcie.0,addr=0x2'
-device 'virtio-balloon-pci,id=balloon0,bus=pcie.0,addr=0x3'
-iscsi 'initiator-name=iqn.1993-08.org.debian:01:1f77b5aa2fa'
-drive 'file=/mnt/pve/ProxmoxInstall/template/iso/ubuntu-20.04.2-live-server-arm64.iso,if=none,id=drive-ide2,media=cdrom,aio=threads'
-device 'virtio-scsi-pci,id=scsihw0,bus=pcie.0,addr=0x5'
-drive 'file=/dev/pve/vm-107-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on'
-device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100'
-netdev 'type=tap,id=net0,ifname=tap107i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown'
-device 'virtio-net-pci,mac=**:**:**:**:**:**,netdev=net0,bus=pcie.0,addr=0x12,id=net0,bootindex=300'
-machine 'accel=tcg,type=virt+pve1'
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!