proxmox ve 7.3 - freebsd 10.0

[ominsign]

For some reasons (legacy project that still needs to run "as is" for a few more months) I would need to get FreeBSD 10.0 running on a recent proxmox setup. I tried to tune the hardware (host, cpu, machine type, hotplug, aes, ballooning, etc.) in many ways according to actual and older forum posts here, but I still cannot get the VM to boot at all, it always fails with a screen like this :



Is there a way at all to achieve this, or is it hopeless ? Last time I had to something similar this I remember I used an old proxmox setup (5.x?), but if possible I would still like to stay on 7.x.

Any qemu-server/*.conf sample file welcome Proxmox host is a classical ProLiant DL360 Gen10 server, ISO image FreeBSD-10.0-RELEASE-amd64-disc1.iso (from January 2014, yes, I know.)

Thanks & best regards, Olivier

 

[fweber]

Hi, can you show your current VM config (qm config <vmid>)? There was a somewhat similar issue (but with a different backtrace and OPNsense) a while ago where memory hotplug had to be disabled [1].

[1]: https://forum.proxmox.com/threads/opnsense-vm-bsd-panic.123567/#post-538441

 

[ominsign]

Hi @fweber, thanks for your reply! I tried a lot of things, from the default VM setup to things inspired from other forum posts, including the one you mentioned (memory hotplug was already disabled by default in my case with PVE 7.3). Last try config was with:

acpi: 1
agent: 1
balloon: 0
boot: order=ide0;ide2;net0
cores: 2
cpu: host,flags=+aes
hotplug: 0
ide0: hd1:202/vm-202-disk-0.qcow2,discard=on,size=512G
ide2: local:iso/FreeBSD-10.0-RELEASE-amd64-disc1.iso,media=cdrom,size=637694K
machine: q35
memory: 8192
meta: creation-qemu=7.1.0,ctime=1679404158
name: freebsd10
net0: e1000=26:7A:C5:8F:8B:71,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: other
scsihw: virtio-scsi-pci
smbios1: uuid=2ecbdcbf-2ab3-4522-85de-807d49373e54
sockets: 2
vmgenid: d85303f2-fa08-4146-ab81-3913ee75fb30

But it is just to be seen as another attempt. I can also upgrade the setup to 7.4 later, but I don't expect much change here as we're speaking of a nearly 10 years old OS.

Best regards & to be continued I hope
Olivier

 

[fweber]

Hi, the CPU type host might be problematic -- at least there were some threads in the past where this was a problem [1]. So you could try setting it to kvm64. Also, just to be sure, have you checked the ISO for integrity?

If this doesn't help, could you post the output of pveversion -v?

[1]: https://forum.proxmox.com/threads/freebsd-opnsense-does-not-boot-with-cpu-host-on-kernel-5-11.89878/

 

[ominsign]

Switching back to kvm64 didn't help, and ISO SHA256 check was fine.

Now I upgraded the system to the latest version (7.4) and recreated the VM with basic settings (1 cpu, 4GB RAM), Interesting enough, now it's not crashing on boot, but now it is just hanging during the boot process just after the random: <Software, Yarrow> initialized line, while the VM is taking 100% of the attributed CPU power.

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                         
   2921 root      20   0 4868628   4.0g  13124 S 100.5   4.3   8:26.58 /usr/bin/kvm -id 202

qm config 202 output:

agent: 1
boot: order=ide0;ide2;net0
cores: 1
ide0: hd1:202/vm-202-disk-0.qcow2,discard=on,size=256G
ide2: local:iso/FreeBSD-10.0-RELEASE-amd64-disc1.iso,media=cdrom,size=637694K
memory: 4096
meta: creation-qemu=7.2.0,ctime=1679664774
name: freebsd10
net0: e1000=DE:25:9A:08:BA:14,bridge=vmbr0,firewall=1
numa: 0
ostype: other
scsihw: virtio-scsi-single
smbios1: uuid=c44f7fe5-bd3e-4160-81ad-97bc5601d48d
sockets: 1
vmgenid: ea0b452a-baca-40c6-9e52-50210cd742fc

pveversion -v:

proxmox-ve: 7.4-1 (running kernel: 5.15.102-1-pve)
pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a)
pve-kernel-5.15: 7.3-3
pve-kernel-5.13: 7.1-9
pve-kernel-5.15.102-1-pve: 5.15.102-1
pve-kernel-5.15.85-1-pve: 5.15.85-1
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-4.15: 5.4-19
pve-kernel-4.15.18-30-pve: 4.15.18-58
pve-kernel-4.15.18-9-pve: 4.15.18-30
ceph-fuse: 14.2.21-1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: 0.8.36+pve2
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-3
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-1
libpve-rs-perl: 0.7.5
libpve-storage-perl: 7.4-2
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.3.3-1
proxmox-backup-file-restore: 2.3.3-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.1-1
proxmox-widget-toolkit: 3.6.3
pve-cluster: 7.3-3
pve-container: 4.4-3
pve-docs: 7.4-2
pve-edk2-firmware: 3.20221111-2
pve-firewall: 4.3-1
pve-firmware: 3.6-4
pve-ha-manager: 3.6.0
pve-i18n: 2.11-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-1
qemu-server: 7.4-2
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.9-pve1

I will again try some other hardware/controller/cpu types later.

best regards, Olivier

 

[ominsign]

PS: same with q35 as machine type:



+ qm status 202 --verbose:

balloon: 4294967296
ballooninfo:
    actual: 4294967296
    max_mem: 4294967296
blockstat:
    ide0:
        account_failed: 1
        account_invalid: 1
        failed_flush_operations: 0
        failed_rd_operations: 0
        failed_unmap_operations: 0
        failed_wr_operations: 0
        flush_operations: 0
        flush_total_time_ns: 0
        idle_time_ns: 257169348381
        invalid_flush_operations: 0
        invalid_rd_operations: 0
        invalid_unmap_operations: 0
        invalid_wr_operations: 0
        rd_bytes: 512
        rd_merged: 0
        rd_operations: 1
        rd_total_time_ns: 77922
        timed_stats:
        unmap_bytes: 0
        unmap_merged: 0
        unmap_operations: 0
        unmap_total_time_ns: 0
        wr_bytes: 0
        wr_highest_offset: 0
        wr_merged: 0
        wr_operations: 0
        wr_total_time_ns: 0
    ide2:
        account_failed: 1
        account_invalid: 1
        failed_flush_operations: 0
        failed_rd_operations: 0
        failed_unmap_operations: 0
        failed_wr_operations: 0
        flush_operations: 0
        flush_total_time_ns: 0
        idle_time_ns: 237887544672
        invalid_flush_operations: 0
        invalid_rd_operations: 0
        invalid_unmap_operations: 0
        invalid_wr_operations: 0
        rd_bytes: 21520384
        rd_merged: 0
        rd_operations: 10004
        rd_total_time_ns: 129753318
        timed_stats:
        unmap_bytes: 0
        unmap_merged: 0
        unmap_operations: 0
        unmap_total_time_ns: 0
        wr_bytes: 0
        wr_highest_offset: 0
        wr_merged: 0
        wr_operations: 0
        wr_total_time_ns: 0
cpus: 1
disk: 0
diskread: 21520896
diskwrite: 0
maxdisk: 274877906944
maxmem: 4294967296
mem: 3739367764
name: freebsd10
netin: 60
netout: 0
nics:
    tap202i0:
        netin: 60
        netout: 0
pid: 5317
proxmox-support:
    backup-max-workers: 1
    pbs-dirty-bitmap: 1
    pbs-dirty-bitmap-migration: 1
    pbs-dirty-bitmap-savevm: 1
    pbs-library-version: 1.3.1 (4d450bb294cac5316d2f23bf087c4b02c0543d79)
    pbs-masterkey: 1
    query-bitmap-info: 1
qmpstatus: running
running-machine: pc-q35-7.2+pve0
running-qemu: 7.2.0
status: running
uptime: 260
vmid: 202

 

[fweber]

Hi, thanks for checking! I see the same behavior (boot hangs at random: <Software, Yarrow> initialized) on Proxmox VE 7.4 with CPU type kvm64, but on a kernel 6.2.6-1.

Here, I was able to work around it by using a serial console instead of the graphical display. For this, I added a serial port in the UI and changed the Display to "Serial terminal 0". Then I started the VM and attached to the serial console via the CLI via qm start <vmid>; qm terminal <vmid>. In the FreeBSD boot menu, I chose 3. Escape to loader prompt and entered set console=comconsole (see here [1]), and then boot. I was then able to install FreeBSD via the serial console. Maybe this is also an option for you?

[1] https://man.freebsd.org/cgi/man.cgi?loader.conf(5)

 

[ominsign]

Thanks a lot @fweber, this did the trick ! Quite an original workaround, nice work
Now let's see how stable it remains...
Best regards & a nice week-end to you, Olivier

$ uname -a
FreeBSD 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 22:34:59 UTC 2014     root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64