Dedicate whole disks to VM, make disk bootable and... Will it conflict to PVE boot?

A

Alexey Pavlyuts

Active Member
Jun 16, 2018
25
2
43
52
Hi All!

I wish to pass whole HD drives to VM and then install OS inside. My node runs on ZFS mirror, and probably the VM inside could be on ZFS mirror/stripe too.

The question: will it affect node boot? In fact, whet UEFI boot will scan for bootable media, it will find a lot of disks with MBRs (both from node itself and VM) and two ZFS pools, so theoretically it may choose VM disks for booting and boot VM image instead of host system image.

I understand that the best way is to try and test, but it is not so easy for me to unload one node for experiments like this. I wish to understand the logics behind before try.

Did anyone try to do like this?
 
Hi,

assuming you're using UEFI boot, running efibootmgr in cli on host provides information about the configured boot options and the boot order. As your VM will not have access to the hosts UEFI, boot order will stay unchanged by adding more, even bootable, hard drives.
Output could look like this, proxmox option is the default:
root@pve:~# efibootmgr
BootCurrent: 0008
Timeout: 3 seconds
BootOrder: 0008,0002,0001,0003,0004,0005,0006,0000,0007
Boot0000* UiApp
Boot0001* UEFI QEMU DVD-ROM QM00003
Boot0002* UEFI QEMU QEMU HARDDISK
Boot0003* UEFI PXEv4 (MAC:BC2411FFE33D)
Boot0004* UEFI PXEv6 (MAC:BC2411FFE33D)
Boot0005* UEFI HTTPv4 (MAC:BC2411FFE33D)
Boot0006* UEFI HTTPv6 (MAC:BC2411FFE33D)
Boot0007* EFI Internal Shell
Boot0008* proxmox
Click to expand...
 
fba said:
Hi,

assuming you're using UEFI boot, running efibootmgr in cli on host provides information about the configured boot options and the boot order. As your VM will not have access to the hosts UEFI, boot order will stay unchanged by adding more, even bootable, hard drives.
Output could look like this, proxmox option is the default:
Click to expand...
Thanks great for your clear point, think I need to go and learn UEFI in details. So far I understand that my current UEFI boot config seems like a garbage, have a look!

Code: 
pavlyuts@hp1:~$ efibootmgr -v
BootCurrent: 001B
Timeout: 0 seconds
BootOrder: 001B,001A,0019,0018,0016,0015,0014,0013,0012,0002,0001,0003,0004,0005,0006,0007,0008,000A,0000,000C,000B,0010,0009,000F,0011
Boot0000* Embedded UEFI Shell   FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(c57ad6b7-0515-40a8-9d21-551652854e37)
Boot0001  Diagnose Error        FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(0849279d-40d5-53ea-e764-2496766f9844)
Boot0002  System Utilities      FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(1fd631e5-44e0-2f91-10ab-f88f3568ef30)
Boot0003  Intelligent Provisioning      FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(4a433501-ddaa-490b-96b2-04f42d8669b8)
Boot0004  Boot Menu     FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(d3fd6286-43c5-bb8d-0793-07b70aa9de36)
Boot0005  Network Boot  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(ee8b26b0-37e9-11e1-b86c-0800200c9a66)
Boot0006  Embedded Diagnostics  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(b57fe6f1-4f49-d46e-4bba-0a8add34d2f3)
Boot0007  View Integrated Management Log        FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(93c92423-d1c6-4286-be67-b76b6671047e)
Boot0008* Generic USB Boot      UsbClass(ffff,ffff,255,255)
Boot0009* Embedded RAID 1 : Smart Array P440ar Controller - 279.40 GiB, Port:1I, Bay:1, Box:1   PciRoot(0x0)/Pci(0x1,0x0)/Pci(0x0,0x0)/SCSI(0,0)N.....YM....R,Y.
Boot000A* Embedded SATA Port 10 CD/DVD ROM : hp DVDRW GUD0N     PciRoot(0x0)/Pci(0x1f,0x2)/Sata(5,0,0)N.....YM....R,Y.
Boot000B* Embedded RAID 1 : Smart Array P440ar Controller - 372.61 GiB, Port:1I, Bay:2, Box:1   PciRoot(0x0)/Pci(0x1,0x0)/Pci(0x0,0x0)/SCSI(1,0)N.....YM....R,Y.
Boot000C* Embedded RAID 1 : Smart Array P440ar Controller - 279.40 GiB, Port:2I, Bay:5, Box:1   PciRoot(0x0)/Pci(0x1,0x0)/Pci(0x0,0x0)/SCSI(4,0)N.....YM....R,Y.
Boot000D* iLO Virtual USB 2 : HPE iLO Virtual USB CD/DVD ROM    PciRoot(0x0)/Pci(0x1d,0x0)/USB(0,0)/USB(0,0)N.....YM....R,Y.
Boot000E* Embedded LOM 1 Port 1 : HP Ethernet 1Gb 4-port 331i Adapter - NIC (PXE IPv4)  PciRoot(0x0)/Pci(0x1c,0x4)/Pci(0x0,0x0)/MAC(94188278daf8,0)/IPv4(0.0.0.00.0.0.0,0,0)N.....YM....R,Y.
Boot000F* Embedded RAID 1 : Smart Array P440ar Controller - 372.61 GiB, Port:2I, Bay:6, Box:1   PciRoot(0x0)/Pci(0x1,0x0)/Pci(0x0,0x0)/SCSI(5,0)N.....YM....R,Y.
Boot0010* Embedded RAID 1 : Smart Array P440ar Controller - 372.61 GiB, Port:1I, Bay:3, Box:1   PciRoot(0x0)/Pci(0x1,0x0)/Pci(0x0,0x0)/SCSI(2,0)N.....YM....R,Y.
Boot0011* Embedded RAID 1 : Smart Array P440ar Controller - 372.61 GiB, Port:2I, Bay:7, Box:1   PciRoot(0x0)/Pci(0x1,0x0)/Pci(0x0,0x0)/SCSI(6,0)N.....YM....R,Y.
Boot0012* Linux Boot Manager    HD(2,GPT,658d7d12-48cf-4c8e-bdc0-a56049dfd560,0x800,0x100000)/File(\EFI\systemd\systemd-bootx64.efi)
Boot0013* Linux Boot Manager    HD(2,GPT,7d315a17-65c1-45c5-a2f9-169c0b5d1cb8,0x800,0x100000)/File(\EFI\systemd\systemd-bootx64.efi)
Boot0014* Linux Boot Manager    HD(2,GPT,3b5b1d7b-1315-469f-accd-6eee15427193,0x800,0x100000)/File(\EFI\systemd\systemd-bootx64.efi)
Boot0015* Linux Boot Manager    HD(2,GPT,a2c24251-b86d-492e-a488-93ee355be4a4,0x800,0x100000)/File(\EFI\systemd\systemd-bootx64.efi)
Boot0016* Linux Boot Manager    HD(2,GPT,6e78516b-d160-43c9-9b6d-49b009b9676c,0x800,0x100000)/File(\EFI\systemd\systemd-bootx64.efi)
Boot0017* iLO Virtual USB 2 : HPE iLO Virtual USB CD/DVD ROM    PciRoot(0x0)/Pci(0x1d,0x0)/USB(0,0)/USB(0,0)N.....YM....R,Y.
Boot0018* Linux Boot Manager    HD(2,GPT,e8853c70-4822-4e59-a3c6-1081c7401e9f,0x800,0x100000)/File(\EFI\systemd\systemd-bootx64.efi)
Boot0019* Linux Boot Manager    HD(2,GPT,26327d14-d2f4-4df1-9c10-2c5ae88a3ef7,0x800,0x100000)/File(\EFI\systemd\systemd-bootx64.efi)
Boot001A* Linux Boot Manager    HD(2,GPT,ee52717e-72b0-4fc6-bc97-097bdc6c4de1,0x800,0x200000)/File(\EFI\systemd\systemd-bootx64.efi)
Boot001B* Linux Boot Manager    HD(2,GPT,9f6c4f6b-e00b-4dfe-9fe7-8bf5f70e233e,0x800,0x200000)/File(\EFI\systemd\systemd-bootx64.efi)
pavlyuts@hp1:~$
All this sounds like my UEFI records include all outdated from past PVE installs on the same server. I am veryy appreciating your sugestion: is there a way to clear unused?
In fact, I have ZFZ mirror root, so there must be two EFI partitions to boot from. Think all the other are just artefacts of old installs or, pproabaly, old system HDDs replaced in mirror.
 
Seems like entries 12-16 and 18-1A are from older installations and 1B is the currently used one, but better check before removal.
Running lsblk -o NAME,PARTUUID will list the ids on the current system. At least one of the UUIDs shown must correspond to one of the entries shown by efibootmgr. The other entries can be removed with efibootmgr -B -b <id> one by one or just use a for loop, like for i in {12..16} {18,19,1A}; do efibootmgr -B -b 00${i}; done. Don't know atm, how the mirrored ZFS is listed, so be careful and delete only entries with an UUID not found with lsblk.
If you accidently delete the current one, try using the proxmox-boot-tool to recreate it.
 
Last edited:
Thanks a lot!

When mirrored ZFS boot disk used, PVE installer creates GPT partitions like this:
Code: 
Number  Start (sector)    End (sector)  Size       Code  Name
   1              34            2047   1007.0 KiB  EF02
   2            2048         2099199   1024.0 MiB  EF00
   3         2099200       585937466   278.4 GiB   BF01
First is BIOS Boot, second is UEFI Boot, and third is ZFS data itself. Then, proxmox-boot-tool used to keep all the UEFI partitions on all of morror disks coherent and up to date, so the system may boot from any disk in the mirror set. In my case 001A and 001B are two UEFI boot partitions from two different HDD, both are the same content, created my proxmox-boot-tool.

For example, UEFI partition content from my server:
Code: 
root@hp1:/mnt# tree /mnt/
/mnt/
├── EFI
│   ├── BOOT
│   │   └── BOOTX64.EFI
│   ├── Linux
│   ├── proxmox
│   │   ├── 6.8.12-4-pve
│   │   │   ├── initrd.img-6.8.12-4-pve
│   │   │   └── vmlinuz-6.8.12-4-pve
│   │   └── 6.8.12-5-pve
│   │       ├── initrd.img-6.8.12-5-pve
│   │       └── vmlinuz-6.8.12-5-pve
│   └── systemd
│       └── systemd-bootx64.efi
└── loader
    ├── entries
    │   ├── proxmox-6.8.12-4-pve.conf
    │   └── proxmox-6.8.12-5-pve.conf
    ├── entries.srel
    ├── loader.conf
    └── random-seed
Then, kernel and initramfs loads, look for ZFS rpool and do ZFS root mount.

That's the point I am warned about, becuase if I do as I described above, it will find TWO pools, both named rpool and both are bootable. There some mechanics in ZFS preventing to use a pool you did not used before (need explicit export/import) but I am not sure about how it works. UEFI bootloader will pass the root target as root=ZFS=rpool/ROOT/pve-1 boot=zfs, it means it should scan disks, detect all ZFS pools, choose one with name rpool and then mount root at rpool/ROOT/pve-1. It is hard to say how it will handle situation with zfs pool name duplicate.

Think, experiment is required, there is a little chance to find out in theory.

Anyway, thank you so much for your valued hints!
 
Last edited:
  • Like
Reactions: fba
You must log in or register to reply here.
Top Bottom