Cannot Edit Boot Order with NVMe Drive

Discussion in 'Proxmox VE: Installation and configuration' started by chewie198, Nov 26, 2018.

  1. chewie198

    chewie198 New Member

    Joined:
    Nov 5, 2016
    Messages:
    4
    Likes Received:
    0
    I have a Proxmox VM with an NVMe boot drive passed through to it via PCIe passthrough and am unable to to modify the boot order to prioritize NVMe boot over the UEFI shell. This problem appears to be the same issue posted about here, but that thread is over a month old and received no responses. Has any one else encountered this problem?

    It would appear that there are two problems with the way Proxmox behaves here: it does not include passed-through drives in the boot order configuration options, and it also does not save any changes made via the OVMF graphical interface despite the presence of an attached EFI disk. Finding a way to resolve either of these shortcomings should allow for a potential workaround.
     
  2. dcsapak

    dcsapak Proxmox Staff Member
    Staff Member

    Joined:
    Feb 1, 2016
    Messages:
    3,532
    Likes Received:
    320
    it may be because ovmf only initializes devices with a 'bootindex', which we do not set for pci passthrough devices

    could you try the following:

    execute 'qm showcmd ID --pretty' and look for the line of the passthrough (should look like: -device vfio-pci,host... )
    copy that into the config under the 'args' property and add a bootindex to it
    for example: 102.conf
    Code:
    args: -device 'vfio-pci,host=02:00.0,id=hostpci0,bus,...,bootindex=50'
    
    and comment the hostpci line

    if that does work, we could maybe add an optional bootindex to the hostpci devices which then would enable that
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    HE_Cole likes this.
  3. chewie198

    chewie198 New Member

    Joined:
    Nov 5, 2016
    Messages:
    4
    Likes Received:
    0
    I was able to modify the config file successfully by adding this line,
    Code:
    -device 'vfio-pci,host=07:00.0,id=hostpci3,bus=ich9-pcie-port-4,addr=0x0,bootindex=50'
    however, I receive this error when trying to start up the VM instance:
    Code:
    kvm: -device vfio-pci,host=07:00.0,id=hostpci3,bus=ich9-pcie-port-4,addr=0x0,bootindex=50: Bus 'ich9-pcie-port-4' not found
    I'm guessing this is because Proxmox is inserting the additional arguments just before the line which defines the pcie root ports.
    Code:
    -readconfig /usr/share/qemu-server/pve-q35.cfg
    Is there any way to work around this? Here is the full output from running the qm start command:
    Code:
    kvm: -device vfio-pci,host=07:00.0,id=hostpci3,bus=ich9-pcie-port-4,addr=0x0,bootindex=50: Bus 'ich9-pcie-port-4' not found
    start failed: command '/usr/bin/kvm -id 105 -name amb47workstation -chardev 'socket,id=qmp,path=/var/run/qemu-server/105.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qemu-server/105-event.qmp,server,nowait' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/105.pid -daemonize -smbios 'type=1,uuid=abd2ccad-4438-490d-aa22-0e74baf471cb' -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/dev/zvol/rpool/data/vm-105-disk-0' -smp '8,sockets=1,cores=8,maxcpus=8' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -no-hpet -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=proxmox,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,kvm=off' -m 8192 -device 'vfio-pci,host=07:00.0,id=hostpci3,bus=ich9-pcie-port-4,addr=0x0,bootindex=50' -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=03:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on' -device 'vfio-pci,host=03:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1' -device 'vfio-pci,host=00:1b.0,id=hostpci1,bus=ich9-pcie-port-2,addr=0x0' -device 'vfio-pci,host=00:14.0,id=hostpci2,bus=ich9-pcie-port-3,addr=0x0' -chardev 'socket,path=/var/run/qemu-server/105.qga,server,nowait,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:f630670652' -drive 'if=none,id=drive-ide0,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0' -netdev 'type=tap,id=net0,ifname=tap105i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=B6:5F:55:91:78:39,netdev=net0,bus=pci.0,addr=0x12,id=net0' -rtc 'driftfix=slew,base=localtime' -machine 'type=q35' -global 'kvm-pit.lost_tick_policy=discard'' failed: exit code 1
     
  4. dcsapak

    dcsapak Proxmox Staff Member
    Staff Member

    Joined:
    Feb 1, 2016
    Messages:
    3,532
    Likes Received:
    320
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. chewie198

    chewie198 New Member

    Joined:
    Nov 5, 2016
    Messages:
    4
    Likes Received:
    0
    So I took a stab at editing and running the command standalone, but it just hangs the shell, nothing happens, and the guest remains under 1% CPU usage with no display. There are no errors returned, but I still have to Ctrl-C and run 'qm stop 105' to recover. Here's the edited command that I used to launch the VM:

    Code:
    /usr/bin/kvm -id 105 -name amb47workstation -chardev 'socket,id=qmp,path=/var/run/qemu-server/105.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qemu-server/105-event.qmp,server,nowait' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/105.pid -daemonize -smbios 'type=1,uuid=abd2ccad-4438-490d-aa22-0e74baf471cb' -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/dev/zvol/rpool/data/vm-105-disk-0' -smp '8,sockets=1,cores=8,maxcpus=8' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -no-hpet -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=proxmox,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,kvm=off' -m 8192 -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=07:00.0,id=hostpci3,bus=ich9-pcie-port-4,addr=0x0,bootindex=50' -device 'vfio-pci,host=03:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on' -device 'vfio-pci,host=03:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1' -device 'vfio-pci,host=00:1b.0,id=hostpci1,bus=ich9-pcie-port-2,addr=0x0' -device 'vfio-pci,host=00:14.0,id=hostpci2,bus=ich9-pcie-port-3,addr=0x0' -chardev 'socket,path=/var/run/qemu-server/105.qga,server,nowait,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:f630670652' -drive 'if=none,id=drive-ide0,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0' -netdev 'type=tap,id=net0,ifname=tap105i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=B6:5F:55:91:78:39,netdev=net0,bus=pci.0,addr=0x12,id=net0' -rtc 'driftfix=slew,base=localtime' -machine 'type=q35' -global 'kvm-pit.lost_tick_policy=discard'
     
  6. chewie198

    chewie198 New Member

    Joined:
    Nov 5, 2016
    Messages:
    4
    Likes Received:
    0
    @dcsapak, Any other ideas as to what I might try? I would be willing to purchase a subscription or two if necessary to help sponsor development if you feel the bootindex change you proposed would get this working. Alternatively, I'm a software developer myself, so I may be able to submit a pull request myself if you can point me in the right direction. It would be nice to find a quick way to verify that this solution works before going to all of that trouble, however.
     
  7. dcsapak

    dcsapak Proxmox Staff Member
    Staff Member

    Joined:
    Feb 1, 2016
    Messages:
    3,532
    Likes Received:
    320
    can you post the complete config and the commandline you try to execute? (preferably with the '--pretty' formatting)

    i do not want to add such an option if it is not working really, and i currently have no spare nvme to test.

    see https://pve.proxmox.com/wiki/Developer_Documentation for general developer info

    and https://git.proxmox.com/?p=qemu-server.git;a=summary for the part that generates the qemu-commandline from the config (specifically the file 'PVE/QemuServer.pm')
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. mandadila

    mandadila New Member

    Joined:
    Nov 12, 2018
    Messages:
    5
    Likes Received:
    0
    I think I'm stuck at the same Problem. Is there a solution already implemented?

    I'm passing a pcie nvme device through ... right now I manualy change the boot device everytime I start the VM.
     
  9. kedala

    kedala New Member

    Joined:
    Oct 28, 2018
    Messages:
    7
    Likes Received:
    0
    Hi everyone,

    i have the exact same issue.
    Seems to be, that args gets procecced before the qm command line generates readconfig for q35.
    Would actually be cool if someone could implement a quickfix (I'm not really a developer, more a sysadmin but would
    try it too, because i need that feature to be stable as soon as possible)

    So yeah, is there any update besides of that at early december?
     
  10. kedala

    kedala New Member

    Joined:
    Oct 28, 2018
    Messages:
    7
    Likes Received:
    0
    Hi everyone,

    a colleague of mine fixed the problem today, but because he deserves the credit for the fix he'll publish the solution by himself soon.

    EDIT:

    There seems to be a more basic problem.
    If "args" is used instead of the hostpciX configuration in the qemu config file, the device gets not initialised correctly.
    The device in /dev/vfio will not be created in that case which will prevent the VM from booting because it's missing the device.

    I only have a fix for the args issue.
    The problem seems to be in QemuServer.pm, so i'll try to debug and fix it. But honestly, i think the easiest fix would be to give hostpciX a parameter that marks the device as bootable. A entry in the webinterface for that case could be useful too.
    Seems that there is no such option.
    https://pve.proxmox.com/wiki/Manual:_qm.conf
     
    #10 kedala, Jan 11, 2019
    Last edited: Jan 12, 2019
  11. kedala

    kedala New Member

    Joined:
    Oct 28, 2018
    Messages:
    7
    Likes Received:
    0
    Sorry, the content below does not work. Why? Because the cmdline from "qm showcmd 101" is different from what is executed when pushing the "start" Button in the Web-GUI.
    "qm start 101" works fine thought.
    Sooo, i'm at a dead end at the moment with my amount of knowledge.

    Someone from PVE should have a look into this.

    Here starts the mostly useless content:
    Soooo, i fixed it. Kinda. It's a dirty and static solution, but it keeps the system running until the next update.
    It's more like a Hotfix.

    /usr/share/perl5/PVE/QemuServer.pm # I added the highlighted line because the nvme drive was configured as hostpci1 in my case.
    If yours is hostpci0 you need to change the if($i ==1) to whatever number you gave the device.
    You have to hardcode the vmid for this to work too (in my case 101)

    With that the device gets initialised correctly after a reboot too.(Which will not happen if you configured it with args:)
    Code:
    my $j=0;
    
            foreach my $pcidevice (@$pcidevices) {
    
                my $id = "hostpci$i";
                $id .= ".$j" if $multifunction;
                my $addr = $pciaddr;
                $addr .= ".$j" if $multifunction;
                my $devicestr = "vfio-pci";
                if ($sysfspath) {
                    $devicestr .= ",sysfsdev=$sysfspath";
                } else {
                    $devicestr .= ",host=$pcidevice->{id}";
                }
                $devicestr .= ",id=$id$addr";
    
                if($j == 0){
                    $devicestr .= "$rombar$xvga";
                    $devicestr .= ",multifunction=on" if $multifunction;
                    $devicestr .= ",romfile=/usr/share/kvm/$romfile" if $romfile;
                    $devicestr .= ",bootindex=50" if($i == 1 && $vmid == 101);  # <------ THIS ONE!
                }
                push @$devices, '-device', $devicestr;
                $j++;
            }
        }
     
    #11 kedala, Jan 13, 2019
    Last edited: Jan 13, 2019
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice