Can I create a 2nd entry in boot menu that starts Prox without passing GPU to VM?

[IMTheNachoMan]

I am planning on installing Prox on my box and creating a few VMs. One of the VMs will be my daily driver. I've read up on how to pass the GPU and a few USB ports to the VM so that when my prox box boots up, the keyboard/mouse/monitor connected to the prox box will show my VM.

But this means the Prox kernel won't/can't ever show anything on the monitor.

I'm thinking of some kind of fail-safe situation.

I assume the boot sequence is: system/bios > boot menu > prox with no GPU > VM with GPU.

So I'm wondering if I can add another entry to the boot menu that tells it to load prox but without passing GPU/USB (and maybe not even starting the VMs).

Is this possible?

 

[noel.]

A couple of thoughts on this:
First of all, depending on you hardware, you might have an iGPU (most intel cpus have one and some, especially the newer, amd cpus do as well) In that case, just run proxmox via that and pass through the discrete gpu to the daily driver vm.
Secondly, depending on your setup, the most comfortable thing to do might be just to get a cheap, used gt730 or r9 290 or something and build that into your machine for output.

If those two are not an option, yes, you can set up proxmox, then set up a VM and pass through your gpu to that vm. You should still be able to ssh into proxmox from a different computer while the vm is running. otherwise, if the vm is turned off, proxmox will be able to access the gpu (as long as you don't blacklist the drivers) and you can get an output from proxmox. just make sure to not set the vm to autostart on boot (autostart is off by default), that could make things annoying.

I assume the boot sequence is: system/bios > boot menu > prox with no GPU > VM with GPU.

I would be system/bios > boot menu > proxmox with GPU > VM takes over GPU and proxmox doesn't have gpu anymore.
(You might need to set the initcall_blacklist=sysfb_init kernel parameter, check the forum for more info on that)

Obviously I don't know your setup, but consider this option: since Proxmox is built on Debian, you could install a desktop environment on Proxmox [1] and just use the gpu there. that way you don't have to pass through the gpu. That's how we have our workstations set up here. If you need snapshotting as a feature, both btrfs and zfs support making and restoring from snapshots.

Hope this gives you an overview of your options

---

[1]: https://pve.proxmox.com/wiki/Developer_Workstations_with_Proxmox_VE_and_X11

 

[IMTheNachoMan]

you might have an iGPU

I do. I have an HP ProDesk 600 G4 SFF with an i5-8500 and I can't fit another GPU in the box.

get a cheap, used gt730 or r9 290 or something

Buying a new box is not an option right now.

as long as you don't blacklist the drivers

From what I read, I have to blacklist the drivers so Prox doesn't take them and they are available to the VM for passing through. Is that wrong?

just make sure to not set the vm to autostart on boot

My whole thought was that 99% of the time Prox would start, then the VM would auto start and take over the GPU/USB, so I can use the VM as my daily driver. I wanted a 2nd entery in the boot loader that would not auto start the VM or pass the GPU for break/fix scenarios.

system/bios > boot menu > proxmox with GPU > VM takes over GPU

Humm. Everything I read said you have to blacklist the GPU so Prox can't even access it?

Proxmox is built on Debian, you could install a desktop environment on Proxmox

Interesting. I had not considered that. It won't muck anything up with Prox?

 

[noel.]

From what I read, I have to blacklist the drivers so Prox doesn't take them and they are available to the VM for passing through. Is that wrong?

While it'd recommended, it's not always necessary. Proxmox tries to automagically handle drivers and releasing of cards. (But since you have an iGPU, this is not important, I'll get to that later).

My whole thought was that 99% of the time Prox would start, then the VM would auto start and take over the GPU/USB, so I can use the VM as my daily driver. I wanted a 2nd entery in the boot loader that would not auto start the VM or pass the GPU for break/fix scenarios.

that's not really possible. You have your boot manager (e.g. grub) that boots and lets you select, what operating system to hand control over the hardware to. In this case, that would be Proxmox. Then Proxmox starts services (QEMU/KVM and other stuff needed for running VMs) and boots your vm. So the bootloader has no idea how to control your VMs.

Now back to finding a good setup for you:

I do. I have an HP ProDesk 600 G4 SFF with an i5-8500 and I can't fit another GPU in the box.

Perfect! You can use your iGPU for Proxmox and pass the discrete GPU to your daily driver VM. You can plug both your iGPU and discrete GPU into your monitor and just switch input sources on the monitor, depending if you need to access the VM or the Promxmox host.

Interesting. I had not considered that. It won't muck anything up with Prox?

Nope, works fine As long as you don't start deleting important files or something. But that is true for any Workstation, I guess.

 

[IMTheNachoMan]

the discrete GPU

I don't have a discrete GPU. I only have the iGPU.

 

[noel.]

in that case you can also virtualize it and split it up [1] (even to multiple vms) at a performance cost. But that way both the host and vm will have access to the iGPU. However, iGPUs are pretty weak as is, even without splitting them up. I doubt you will get a big performance boost when passing the iGPU through.

I still think your best options are to either use the Proxmox host as your workstation or just a vm without gpu passthrough. Depending on what you need to do on your daily driver, you don't need to pass through the iGPU. If you are just browsing the web, reading emails, coding or writing, the VM without gpu passthrough is fine.
If you want to play videogames, you will get poor performance (unless you are playing old games on low settings) on an iGPU either way. In that case I'd go with using Proxmox with a Desktop environment as your daily driver, since gaming online in a vm might not work with certain anti-cheat software.
Anything more demanding, like training AI models, rendering scenes in Blender, physics simulation etc. won't benefit from an iGPU.

Hope this gives you a broad overview of things, I think the best thing is just trying out which option works best for your workflow.

---

[1]: https://wiki.archlinux.org/title/Intel_GVT-g - make sure when adding the pci device in the web interface to select an Mdev type

 

[IMTheNachoMan]

Hope this gives you a broad overview of things

It does. I've done a fair bit of research and, without trying anything yet, I think my best option is to: pass iGPU to VM, have VM autoboot, and blacklist the iGPU in Prox kernel so it doesn't try to load it. Once the iGPU is blacklisted, Prox won't be able to steal it and it'll be just the VMs.

Based on my research, this will work fine. The only time I'll have an issue is if Prox messes up and the VMs don't/can't autoboot and even Prox's SSH is not working.

That's why I was wondering if I could create a 2nd entry in Grub that would boot Prox w/o blacklisting the iGPU so that Prox can grab the iGPU and the VM won't be able to.