[SOLVED] template creation with Packer: boot options menu appears on some cluster and not other

[daks]

Hello,

We are in the process of installing and configuring 4 clusters of 3 nodes and automating templates creation using Packer.
We have a strange behaviour where for some clusters Packer template creation works fine and for others we are faced with a bug during the boot process.

First, some data:
* PVE version 7.1.10 on Debian Bullseye 11.2
* Debian and PVE installed by our servers provider
* some nodes may have been installed directly on PVE 7, others upgraded from PVE 6
* configuration should be the same on every node (done automatically or following some procedure)
* all clusters are connected to a iSCSI LUN with a LVM storage on top of it

The boot bug:
* Packer runs a QEMU virtual machine which boots on Debian ISO, then it 'types' keys to simulate a human typing on a keyboard. For Debian it means running "Esc" then "install..." to run an automated installation from a preseed file
* on some clusters, before booting on the ISO file, I see a boot menu asking to choose between Virtio-ISCSI, DVD/CD, IPXE and another one
* when this boot menu appears, key sequence break and boot on the ISO file do not occur

When I look at the VM Packer creates, I see:
* Hardware:
* Hard disk is scsi0
* SCSI Controller is Virtio SCSI
* CD/DVD Drive is ide2
* Options:
* Boot order: scsi0, ide2, net0
Boot order seems coherent with the hardware.

I see the same on clusters where Packer build is successful.

Do you have any idea what can be the reason for different behaviours between clusters which seems identically configured?
How can I solve this boot problem? I saw posts talking about setting boot order in Packer, but the option I see 'scsi0,ide2,net0' seems to be the right one.

 

[6uellerbpanda]

can you provide the packer config ?

 

[daks]

My configuration in HCL format is like this

source "proxmox-iso" "debian" {
  boot_command            = ["<esc>", "<esc><wait>", "install ", "preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/${var.debian_preseed_file} ", "debian-installer=en_US ", "auto ", "locale=en_US ", "kbd-chooser/method=fr ", "netcfg/get_hostname=${var.hostname} ", "netcfg/get_domain=
internal.domain.tld ", "fb=false ", "debconf/frontend=noninteractive ", "console-setup/ask_detect=false ", "console-keymaps-at/keymap=fr ", "keyboard-configuration/xkb-keymap=fr ", "net.ifnames=0 ", "<enter><wait>"]
  boot_wait               = "2s"
  http_directory          = "http"
  iso_checksum            = var.debian_iso_checksum
  iso_url                 = var.debian_iso_url
  iso_storage_pool        = "local"
  proxmox_url             = "https://${var.proxmox_server}:8006/api2/json"
  username                = var.proxmox_username
  password                = var.proxmox_password
  node                    = var.proxmox_node
  qemu_agent              = true
  ssh_username            = var.debian_user
  ssh_password            = var.debian_password
  ssh_port                = 22
  ssh_timeout             = "1200s"

  vm_name                 = var.output_file
  memory                  = "2048"
  cores                   = "2"
  os                      = "l26"

  template_name           = var.proxmox_template
template_description    = "Debian ${var.debian_version} version ${var.image_version} built with Packer"

  unmount_iso             = true
  scsi_controller         = "virtio-scsi-pci"
  network_adapters {
    model  = "virtio"
    bridge = "vmbr1"
  }
  disks {
    type              = "scsi"
    disk_size         = "50G"
    storage_pool      = "LVM1"
    storage_pool_type = "lvm"
  }
}

vmbr1 is a bridge connected to an internal network which provides DHCP and internet gateway.

 

[6uellerbpanda]

hmm...so just to be clear: packer creates the vm on pve node starts the vm but at the boot screen you see the boot selection menu and not the debian installer ?

 

[daks]

Yes, exactly, then after some seconds debian-installer is run (maybe after the "Esc" key by Packer).
The strange thing is that it occurs on some clusters, but on others this menu do not appear. This is my main issue: the fact that all clusters do not have the same behaviour.

 

[daks]

I tried to capture a video of what happen. It's not perfect but you can see it here https://file.re/2022/02/01/capture-video-2022-02-01-153213/ (only available for 1 day)

 

[6uellerbpanda]

did you try increasing boot_wait option ?

 

[daks]

Yes I tried and it doesn't change anything. The problem is, because of this unexpected boot menu, the key sequence supposed to configure debian-installer boot activates "Speech synthesis" instead of the automated install based on Preseed.
The installation is then blocked, see image.

 

[daks]

I have a solution which seems to work: change the boot_command first part from "<esc>", "<esc><wait>" to "<wait><esc><wait>". I'm currently testing that it still works for all clusters.

It doesn't change the fact that I don't understand why I have a different behaviour between almost identical clusters, but at least my current problem seems solved.

 

[6uellerbpanda]

I have a solution which seems to work: change the boot_command first part from "<esc>", "<esc><wait>" to "<wait><esc><wait>". I'm currently testing that it still works for all clusters.

It doesn't change the fact that I don't understand why I have a different behaviour between almost identical clusters, but at least my current problem seems solved.

yes you're boot_command is a little bit different then mine:

"boot_command": [
            "<esc><wait>",
            "install <wait>",
            "preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/{{ user `preseed` }} <wait>"
...

but strange it's working for some and some not...