[SoYouStart] rEFInd/Grub Boot Error (Upgrading Proxmox 6 to 7)

[eddynetweb]

Hey there,

Am going through the update process from Proxmox 6 to 7 on a SoYouStart box. When I wrapped up the install (https://pve.proxmox.com/wiki/Upgrade_from_6.x_to_7.0) and rebooted the machine, I was greeted with the following error within IPMI:

error: symbol 'grub_is_lockdown' not found.
Entering rescue mode...
grub rescue>

Fortunately, I was able to boot into the root partition normally by rebooting the machine and manually selecting it from within the boot setup. The root system booted up normally and I am able to access Proxmox/VM/SSH w/ network access/etc. When I reboot however, the grub/rEFInd error above shows up still and I need to reboot once again and manually boot from the boot setup.

What do I need to do to fix the rEFInd configuration? I'm assuming I'd need to reconfigure something with grub, but this issue doesn't appear to be very common when I was doing research and couldn't find anything.

 

[Stefan_R]

PVE does not use rEFInd... we use either grub or systemd-boot (if you use ZFS on UEFI). I'd recommend running proxmox-boot-tool refresh and grub-install if you know your bootloader configuration.

 

[Gilou]

Hi,
I had the same issue on a OVH server using nvme disks (though I think that is not related). In reFind, you can select the proper boot option, for some reason, efibootmgr or whatever is used on proxmox 7 let the debian uefi take over the proxmox one, so it fails booting (not sure why, but I didn't investigate much).

It seems to me that efibootmgr lists proxmox first, there might be something fishy on OVH's side…

 

[rabdel]

Hello,
I am encountering the same error, currently, when upgrading proxmox 6 > 7 on OVH server.
Did you find a fix ?

 

[eddynetweb]

I ended up going into the BIOS boot menu on the server via IPMI and changing the boot order to GRUB as priority. Not sure why rEFInd was even on the server to begin with, but that's just another OVH thing I'm sure with their Proxmox images.

If your server does not have IPMI (which some SYS servers don't have, if I recall), then I'm not really sure how you'd adjust it. I think the servers that don't have IPMI are the ASUS boards that SYS uses on their cheaper models. There might be someone who's more sure on how to do this though.

 

[Gilles]

Same here...

 

[Obi_Yoann]

Hey guys, i had the same issue on a couple of servers we just got from soyoustart, i found the reason there is an issue and a solution.
The issue is because the servers have multiple disk drives, in my case 2 nvmes. OVH created EFI partitions on both drives, with the same label, so you can't be sure which partition is mounted on boot on these servers. They have this in their fstab :
LABEL=EFI_SYSPART /boot/efi vfat defaults 0 1
At first when we received the servers, both partitions were identical. After upgrade the files were updated on the partition that was mounted but not the other (which is quite understandable indeed).
diff -r /boot/efi/ /mnt/efi/
Only in /boot/efi/EFI: BOOT
Binary files /boot/efi/EFI/proxmox/grubx64.efi and /mnt/efi/EFI/proxmox/grubx64.efi differ

I manually copied the content of the upgraded partition to the other efi partition on the other disk and on reboot it worked fine!

About refind, it is actually not installed on the servers, it is what is launched from pxe in normal condition, to load the efi loader from one of the partitions of the server. Setting the bios to boot from hard drive directly not only prevents your server from booting to rescue in case of issue, or from reinstalling, it can also prevent your server from booting in case one of the drives of the server fails...
My solution should be the closest to what the servers were set up initially, and be "kind of" future proof.

This is all because of what OVH/Soyoustart did to configure the servers, and this is not, i believe, documented somewhere!

Another thing, i had issue with network because a line was added to /etc/default/grub after upgrade, that caused the network devices to change their name (from eno1/eno2 to eth0/eth1), thus breaking the vmbr0 interface. I had to remove it and then update-grub.
I believe you can do most of this if not all from rescue mode if you have the issues after upgrading...
Fyi, I installed ifupdown2 before upgrading, and did not forget to add the mac address to /etc/network/interfaces as explained in the release notes.

Both servers are now happily up and running Proxmox VE 7.1.6!

 

[Gilles]

Here is my installation procedure for Proxmox 7.1 ZFS on OVH/SYS :

1. Use template PM6.4 from OVH with 3 partitions :
   1. swap set to 1/4 of the RAM (may be other)
   2. / (root) set to 20GB (may be other) on ext4 (xfs does not work)
   3. /var/lib/vz on ext4 for the remaining
2. Change :
   1. #ip -c link to see interface name and get the MAC adress of the used ethernet interface
   2. change /etc/network/interfaces to set
      1. iface vmbr0 static,
      2. address to the server ip & hwaddress to the above MAC adress
      3. gateway to the service ip with last block to .254
      4. brige-port to the corresponding ethernet name
   3. #service networking restart
   4. #upgrade-grub
   5. #lsblk to see disk config
   6. #grub-install /dev/[disks] (i did it for all disks by security)
   7. #apt update, upgrade, dist-upgrade, remove
   8. #reboot & test (you are in PM6.4 version)
3. go to PM7.1 : https://pve.proxmox.com/wiki/Upgrade_from_6.x_to_7.0 with all default options when running dist-upgrade.
   1. comment /etc/apt/sources.list.d/enterprise if not enterprise license
   2. #apt update, upgrade, dist-upgrade, remove
   3. test GUI, reboot & test GUI (your are in PM7.1)
4. go to PM7.1 zfs for the /var/lib/vz partition : https://www.ma-solution.fr/ovh-soyoustart-cree-un-partition-zfs-sur-un-serveur-dedie/
   1. but use PM GUI to create the zfs pool do it when the ext4 partition has been cleaned from the above procedure. This will create the zfs pool with all good PM options.
5. test GUI, reboot and enjoy

It seems to be better to switch to ifupdown2 manually before/after but becarefull this may lead to a loss of the remote onnection (and require use of ipmi or rescue mode). The procedure (from my own experience) is : 1.change /etc/network/interface to move to eth0 instead of any other ethernet name BEFORE upgrading to ifupdown2 2.upgrade to ifupdown2 then reboot (again any problem will required ipmi or rescue mode...)

P.S. : By the way I do not understand why OVH/SYS do not provide a PM7.1 template (with/without ZFS option like before). I have opened several tickets with OVH about it with "we don't know" answers. I suspect they do not want to promote this wonderfull european solution in order to promote their own managed hypervisor based on a US non open source solution which may leads to problems for european companies regarding the US Cloud Act and Privacy Shield even if OVH is a great european/french company.

 

[stormy27]

Hi all!
I have same trouble as you with upgrading Proxmox 6.x to 7.x with an OVH dedicated server (RISE2). Composed with 2 NVME disks, storage in ZFS.
My server was running FINE until I rebooted it. It was the 1st reboot, and -of course!- the problem arrived...

Message send by OVH team:
"Le serveur reste bloqué durant la phase de boot sur le message :
(Erreur GRUB : "grub_is_lockdown" not found)
Un redémarrage sur un noyau standard OVH ('netboot') ne corrige pas la situation. "

Server in rescue mode. Immediate action: forum.proxmox.com
Of course I read all your post, but did not find an answer to my problem.

@Obi_Yoann As you mentionned I've searched info about EFI... but in my case there's nothing comparable. In FSTAB I have similar data LABEL=EFI_SYSPART /boot/efi vfat defaults 0 1

In rescue mode there's nothing in /boot/efi so I do not understand. I'm using ZFS.
root@rescue:/mnt/test/boot# ls config-4.19.0-18-cloud-amd64 efi initrd.img-4.19.0-18-cloud-amd64 pve System.map-5.4.143-1-pve vmlinuz-5.4.143-1-pve config-5.13.19-1-pve extlinux initrd.img-5.13.19-1-pve System.map-4.19.0-18-cloud-amd64 vmlinuz-4.19.0-18-cloud-amd64 config-5.4.143-1-pve grub initrd.img-5.4.143-1-pve System.map-5.13.19-1-pve vmlinuz-5.13.19-1-pve

In EFI, GRUB and PVE directory under /boot (in rescue mode) there're following files:
root@rescue:/mnt/test/boot# ls efi/ root@rescue:/mnt/test/boot# ls grub fonts grub.cfg grubenv i386-pc locale unicode.pf2 x86_64-efi root@rescue:/mnt/test/boot# ls pve initrd.img initrd.img-5.13 initrd.img-5.4 vmlinuz vmlinuz-5.13 vmlinuz-5.4

I'm using ZFS, maybe scenario is not the same??
Of course I added the hwaddress entry with correct MACaddr in /etc/network/interfaces, etc.
In rescue mode I've checked the interfaces name and they changed from eno0/eno1 to eth0/eth1 after the reboot, so I changed the conf as well.

But the problem still present. And I don't know how to identify/solve it! Could you please help me??
What can I do in this rescue mode to change the EFI/GRUB/whatever to reboot???

root@rescue:~# df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 32G 0 32G 0% /dev 146.59.209.3:/home/pub/rescue.v8 756G 336G 382G 47% /nfs tmpfs 32G 1.7M 32G 1% /rw aufs 32G 1.7M 32G 1% / 146.59.209.3:/home/pub/pro-power 756G 336G 382G 47% /power 146.59.209.3:/home/pub/commonnfs 756G 336G 382G 47% /common tmpfs 63G 0 63G 0% /dev/shm tmpfs 32G 9.8M 32G 1% /run tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 32G 0 32G 0% /sys/fs/cgroup tmpfs 32G 12K 32G 1% /tmp

The 2 NVME disks (30GB SWAP / 40GB OS / 825GB PVE)
root@rescue:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT nvme0n1 259:0 0 894.3G 0 disk ├─nvme0n1p5 259:6 0 2M 0 part ├─nvme0n1p3 259:4 0 29.3G 0 part ├─nvme0n1p1 259:2 0 511M 0 part ├─nvme0n1p4 259:5 0 825.4G 0 part └─nvme0n1p2 259:3 0 39.1G 0 part └─md127 9:127 0 39G 0 raid1 nvme1n1 259:1 0 894.3G 0 disk ├─nvme1n1p4 259:10 0 825.4G 0 part ├─nvme1n1p2 259:8 0 39.1G 0 part │ └─md127 9:127 0 39G 0 raid1 ├─nvme1n1p3 259:9 0 29.3G 0 part └─nvme1n1p1 259:7 0 511M 0 part

The OS RAID 1 part (40GB):
root@rescue:~# mdadm --detail /dev/md127 /dev/md127: Version : 1.2 Creation Time : Fri Nov 26 18:02:25 2021 Raid Level : raid1 Array Size : 40927232 (39.03 GiB 41.91 GB) Used Dev Size : 40927232 (39.03 GiB 41.91 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Update Time : Mon Nov 29 11:27:43 2021 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Name : md2 UUID : e6765c09:e840a273:153ea75e:5402dea2 Events : 7 Number Major Minor RaidDevice State 0 259 7 0 active sync /dev/nvme0n1p2 1 259 3 1 active sync /dev/nvme1n1p2


root@rescue:~# parted -l Model: Unknown (unknown) Disk /dev/nvme0n1: 960GB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1049kB 537MB 536MB fat16 primary boot, esp 2 537MB 42.5GB 41.9GB primary raid 3 42.5GB 73.9GB 31.5GB linux-swap(v1) primary 4 73.9GB 960GB 886GB logical raid 5 960GB 960GB 2080kB logical Model: Linux Software RAID Array (md) Disk /dev/md127: 41.9GB Sector size (logical/physical): 512B/512B Partition Table: loop Disk Flags: Number Start End Size File system Flags 1 0.00B 41.9GB 41.9GB ext4 Model: Unknown (unknown) Disk /dev/nvme1n1: 960GB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1049kB 537MB 536MB fat16 primary boot, esp 2 537MB 42.5GB 41.9GB primary raid 3 42.5GB 73.9GB 31.5GB linux-swap(v1) primary 4 73.9GB 960GB 886GB logical raid

@Gilles you mentionned to applygrub-install in the 2 disks. But you've done in when your server was running fine. How to proceed when the serveur is in rescue mode?
For test when launching the command grub-install it throws an error certainly due to rescue or missing EFI??

root@rescue:/mnt/test/boot# grub-install Installing for x86_64-efi platform. grub-install: error: cannot find EFI directory.


Anybody can help please? Thanks

 

[Neox]

as per my experience, yesterday from fresh SYS server Sata drives
I had to follow the Proxmox wiki upgrade with ifupdown2 and hwaddress in /etc/network/interfaces
PLUS I had to remove net.ifnames=0 from /etc/default/grub

then reboot is fine

if using ZFS, you may need timeout option as described here :
https://forum.proxmox.com/threads/t...rade-from-promox-5-to-proxmox-6-on-ovh.60476/

 

[Obi_Yoann]

Hi all!
I have same trouble as you with upgrading Proxmox 6.x to 7.x with an OVH dedicated server (RISE2). Composed with 2 NVME disks, storage in ZFS.
My server was running FINE until I rebooted it. It was the 1st reboot, and -of course!- the problem arrived...

Message send by OVH team:
"Le serveur reste bloqué durant la phase de boot sur le message :
(Erreur GRUB : "grub_is_lockdown" not found)
Un redémarrage sur un noyau standard OVH ('netboot') ne corrige pas la situation. "

Server in rescue mode. Immediate action: forum.proxmox.com
Of course I read all your post, but did not find an answer to my problem.

@Obi_Yoann As you mentionned I've searched info about EFI... but in my case there's nothing comparable. In FSTAB I have similar data LABEL=EFI_SYSPART /boot/efi vfat defaults 0 1

In rescue mode there's nothing in /boot/efi so I do not understand. I'm using ZFS.
root@rescue:/mnt/test/boot# ls config-4.19.0-18-cloud-amd64 efi initrd.img-4.19.0-18-cloud-amd64 pve System.map-5.4.143-1-pve vmlinuz-5.4.143-1-pve config-5.13.19-1-pve extlinux initrd.img-5.13.19-1-pve System.map-4.19.0-18-cloud-amd64 vmlinuz-4.19.0-18-cloud-amd64 config-5.4.143-1-pve grub initrd.img-5.4.143-1-pve System.map-5.13.19-1-pve vmlinuz-5.13.19-1-pve

In EFI, GRUB and PVE directory under /boot (in rescue mode) there're following files:
root@rescue:/mnt/test/boot# ls efi/ root@rescue:/mnt/test/boot# ls grub fonts grub.cfg grubenv i386-pc locale unicode.pf2 x86_64-efi root@rescue:/mnt/test/boot# ls pve initrd.img initrd.img-5.13 initrd.img-5.4 vmlinuz vmlinuz-5.13 vmlinuz-5.4

I'm using ZFS, maybe scenario is not the same??
Of course I added the hwaddress entry with correct MACaddr in /etc/network/interfaces, etc.
In rescue mode I've checked the interfaces name and they changed from eno0/eno1 to eth0/eth1 after the reboot, so I changed the conf as well.

But the problem still present. And I don't know how to identify/solve it! Could you please help me??
What can I do in this rescue mode to change the EFI/GRUB/whatever to reboot???

root@rescue:~# df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 32G 0 32G 0% /dev 146.59.209.3:/home/pub/rescue.v8 756G 336G 382G 47% /nfs tmpfs 32G 1.7M 32G 1% /rw aufs 32G 1.7M 32G 1% / 146.59.209.3:/home/pub/pro-power 756G 336G 382G 47% /power 146.59.209.3:/home/pub/commonnfs 756G 336G 382G 47% /common tmpfs 63G 0 63G 0% /dev/shm tmpfs 32G 9.8M 32G 1% /run tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 32G 0 32G 0% /sys/fs/cgroup tmpfs 32G 12K 32G 1% /tmp

The 2 NVME disks (30GB SWAP / 40GB OS / 825GB PVE)
root@rescue:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT nvme0n1 259:0 0 894.3G 0 disk ├─nvme0n1p5 259:6 0 2M 0 part ├─nvme0n1p3 259:4 0 29.3G 0 part ├─nvme0n1p1 259:2 0 511M 0 part ├─nvme0n1p4 259:5 0 825.4G 0 part └─nvme0n1p2 259:3 0 39.1G 0 part └─md127 9:127 0 39G 0 raid1 nvme1n1 259:1 0 894.3G 0 disk ├─nvme1n1p4 259:10 0 825.4G 0 part ├─nvme1n1p2 259:8 0 39.1G 0 part │ └─md127 9:127 0 39G 0 raid1 ├─nvme1n1p3 259:9 0 29.3G 0 part └─nvme1n1p1 259:7 0 511M 0 part

The OS RAID 1 part (40GB):
root@rescue:~# mdadm --detail /dev/md127 /dev/md127: Version : 1.2 Creation Time : Fri Nov 26 18:02:25 2021 Raid Level : raid1 Array Size : 40927232 (39.03 GiB 41.91 GB) Used Dev Size : 40927232 (39.03 GiB 41.91 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Update Time : Mon Nov 29 11:27:43 2021 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Name : md2 UUID : e6765c09:e840a273:153ea75e:5402dea2 Events : 7 Number Major Minor RaidDevice State 0 259 7 0 active sync /dev/nvme0n1p2 1 259 3 1 active sync /dev/nvme1n1p2


root@rescue:~# parted -l Model: Unknown (unknown) Disk /dev/nvme0n1: 960GB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1049kB 537MB 536MB fat16 primary boot, esp 2 537MB 42.5GB 41.9GB primary raid 3 42.5GB 73.9GB 31.5GB linux-swap(v1) primary 4 73.9GB 960GB 886GB logical raid 5 960GB 960GB 2080kB logical Model: Linux Software RAID Array (md) Disk /dev/md127: 41.9GB Sector size (logical/physical): 512B/512B Partition Table: loop Disk Flags: Number Start End Size File system Flags 1 0.00B 41.9GB 41.9GB ext4 Model: Unknown (unknown) Disk /dev/nvme1n1: 960GB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1049kB 537MB 536MB fat16 primary boot, esp 2 537MB 42.5GB 41.9GB primary raid 3 42.5GB 73.9GB 31.5GB linux-swap(v1) primary 4 73.9GB 960GB 886GB logical raid

@Gilles you mentionned to applygrub-install in the 2 disks. But you've done in when your server was running fine. How to proceed when the serveur is in rescue mode?
For test when launching the command grub-install it throws an error certainly due to rescue or missing EFI??

root@rescue:/mnt/test/boot# grub-install Installing for x86_64-efi platform. grub-install: error: cannot find EFI directory.


Anybody can help please? Thanks

When you are in rescue mode, just try to do "fdisk -l" to check if you can see two EFI partitions.
Here's mine, while system is running from PVE though :

# fdisk -l
Disk /dev/nvme0n1: 419.19 GiB, 450098159616 bytes, 879097968 sectors
Disk model: INTEL SSDPE2MX450G7
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 4E07994C-2231-4093-941A-470353784993

Device             Start       End   Sectors   Size Type
/dev/nvme0n1p1      2048   1048575   1046528   511M EFI System
/dev/nvme0n1p2   1048576  42991615  41943040    20G Linux RAID
/dev/nvme0n1p3  42991616  45088767   2097152     1G Linux filesystem
/dev/nvme0n1p4  45088768 879093759 834004992 397.7G Linux RAID
/dev/nvme0n1p5 879093872 879097934      4063     2M Linux filesystem


Disk /dev/nvme1n1: 419.19 GiB, 450098159616 bytes, 879097968 sectors
Disk model: INTEL SSDPE2MX450G7
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 81ECC219-1FC1-4832-A839-6F9AFFC24594

Device            Start       End   Sectors   Size Type
/dev/nvme1n1p1     2048   1048575   1046528   511M EFI System
/dev/nvme1n1p2  1048576  42991615  41943040    20G Linux RAID
/dev/nvme1n1p3 42991616  45088767   2097152     1G Linux filesystem
/dev/nvme1n1p4 45088768 879093759 834004992 397.7G Linux RAID


Disk /dev/md2: 19.98 GiB, 21458059264 bytes, 41910272 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/md4: 397.56 GiB, 426876338176 bytes, 833742848 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/vg-data: 397.56 GiB, 426875289600 bytes, 833740800 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Create the folders where you are going to mount the efi partitions, since it appears they are not mounted in your rescue :

mkdir /mnt/efi1
mkdir /mnt/efi2

Then mount the efi partitions, in your case it should be as me :

mount /dev/nvme0n1p1 /mnt/efi1
mount /dev/nvme1n1p1 /mnt/efi2

Then you can compare the content of the partitions. The one that was updated most recently had more files than the other. I just copied everything from the updated efi partition to the other one.

Your error is the same as mine, refind loaded from OVH's network tried to start the wrong efi grub loader, and failed!
Hopefully the solution is the same...
Let us know how it goes!

as per my experience, yesterday from fresh SYS server Sata drives
I had to follow the Proxmox wiki upgrade with ifupdown2 and hwaddress in /etc/network/interfaces
PLUS I had to remove net.ifnames=0 from /etc/default/grub

then reboot is fine

if using ZFS, you may need timeout option as described here :
https://forum.proxmox.com/threads/t...rade-from-promox-5-to-proxmox-6-on-ovh.60476/

I don't think all SYS are based on EFI boards yet, maybe you got "lucky" and got one in CSM/MBR in which case this issue should not be relevant.
We also have another older SYS that is indeed not in EFI mode, and i followed exactly what you did and it worked fine!
There is something more to do on EFI systems though in OVH/SYS infra, to avoid this error. Thankfully we have IPMI free on these newer SYS servers (our older one does not, OVH asks for 25€ per day for a kvm device to be plugged to the server), so "debugging" is easier...

 

[stormy27]

@Obi_Yoann Thanks for your answer!

Your explanation is really clear.
I've applied your steps (/mnt than copy newest grub loader to the other one).

But at this time (since 10mn) my server is still not responding...

Update: I just received an email from OVH specifying that my server is not responding anymore.... weird!!!! what's up???

FYI my data were
Disklabel type: gpt Disk identifier: 709075CE-D84A-48A3-9396-600360D787DE Device Start End Sectors Size Type /dev/nvme0n1p1 2048 1048575 1046528 511M EFI System /dev/nvme0n1p2 1048576 82968575 81920000 39.1G Linux RAID /dev/nvme0n1p3 82968576 144408575 61440000 29.3G Linux filesystem /dev/nvme0n1p4 144408576 1875380223 1730971648 825.4G Linux RAID /dev/nvme0n1p5 1875380912 1875384974 4063 2M Linux filesystem Disk /dev/nvme1n1: 894.3 GiB, 960197124096 bytes, 1875385008 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: B6DDAF0C-01D0-482D-892C-01C129B42C6F Device Start End Sectors Size Type /dev/nvme1n1p1 2048 1048575 1046528 511M EFI System /dev/nvme1n1p2 1048576 82968575 81920000 39.1G Linux RAID /dev/nvme1n1p3 82968576 144408575 61440000 29.3G Linux filesystem /dev/nvme1n1p4 144408576 1875380223 1730971648 825.4G Linux RAID Disk /dev/md127: 39 GiB, 41909485568 bytes, 81854464 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

So I used the 2 NVMEs: /dev/nvme0n1p1 and /dev/nvme1n1p1

Giving the following results:
root@rescue:/# ls /mnt/efi1/EFI/ -l total 8 drwxr-xr-x 2 root root 8192 Nov 26 18:03 proxmox root@rescue:/# ls /mnt/efi1/EFI/proxmox/ -l total 144 -rwxr-xr-x 1 root root 147456 Nov 26 18:03 grubx64.efi root@rescue:/# ls /mnt/efi2/EFI/proxmox/ -l total 152 -rwxr-xr-x 1 root root 151552 Nov 27 08:19 grubx64.efi

and finally the update of older grubx64.efi file
root@rescue:/# cp /mnt/efi2/EFI/proxmox/grubx64.efi /mnt/efi1/EFI/proxmox/

Then I launched a REBOOT. Server is KO, not responding. ( What's wrong??

Any idea? What can I check in this situation?? thanks again

 

[Obi_Yoann]

@Obi_Yoann Thanks for your answer!

Your explanation is really clear.
I've applied your steps (/mnt than copy newest grub loader to the other one).

But at this time (since 10mn) my server is still not responding...

Update: I just received an email from OVH specifying that my server is not responding anymore.... weird!!!! what's up???

FYI my data were
Disklabel type: gpt Disk identifier: 709075CE-D84A-48A3-9396-600360D787DE Device Start End Sectors Size Type /dev/nvme0n1p1 2048 1048575 1046528 511M EFI System /dev/nvme0n1p2 1048576 82968575 81920000 39.1G Linux RAID /dev/nvme0n1p3 82968576 144408575 61440000 29.3G Linux filesystem /dev/nvme0n1p4 144408576 1875380223 1730971648 825.4G Linux RAID /dev/nvme0n1p5 1875380912 1875384974 4063 2M Linux filesystem Disk /dev/nvme1n1: 894.3 GiB, 960197124096 bytes, 1875385008 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: B6DDAF0C-01D0-482D-892C-01C129B42C6F Device Start End Sectors Size Type /dev/nvme1n1p1 2048 1048575 1046528 511M EFI System /dev/nvme1n1p2 1048576 82968575 81920000 39.1G Linux RAID /dev/nvme1n1p3 82968576 144408575 61440000 29.3G Linux filesystem /dev/nvme1n1p4 144408576 1875380223 1730971648 825.4G Linux RAID Disk /dev/md127: 39 GiB, 41909485568 bytes, 81854464 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

So I used the 2 NVMEs: /dev/nvme0n1p1 and /dev/nvme1n1p1

Giving the following results:
root@rescue:/# ls /mnt/efi1/EFI/ -l total 8 drwxr-xr-x 2 root root 8192 Nov 26 18:03 proxmox root@rescue:/# ls /mnt/efi1/EFI/proxmox/ -l total 144 -rwxr-xr-x 1 root root 147456 Nov 26 18:03 grubx64.efi root@rescue:/# ls /mnt/efi2/EFI/proxmox/ -l total 152 -rwxr-xr-x 1 root root 151552 Nov 27 08:19 grubx64.efi

and finally the update of older grubx64.efi file
root@rescue:/# cp /mnt/efi2/EFI/proxmox/grubx64.efi /mnt/efi1/EFI/proxmox/

This looks good to me, at this point i believe you've got the "grub" situation sorted, my guess is an issue with the network interfaces.
Easiest would be for your to connect to the IPMI if you can, otherwise if you are limited to rescue mode, check in /var/log if you can to see if there is anything that points to network interfaces beeing named eth0 instead of what PVE is expected.
Make sure you've got the good hwaddress in /etc/network/interfaces, and that /etc/default/grub does not contain the lines that tell PVE to use "old" naming.

EDIT : oh i just remembered, after changing the /etc/default/grub, you have to do "update-grub" so that it regenerates the grub.cfg file with the correct base, i don't quite know how to do that from rescue mode, i believe you have to mount the rootfs and boot partitions, then chroot to that but not sure *how* to do that from rescue. But if the net interfaces are named eth0 in rescue, that should not be related to your configuration in your live OS, the rescue mode has its own configuration. Only way to tell is from your PVE OS logs or from IPMI.
Anyway best way to solve this is from IPMI i'm afraid. You should have for free from your manager since you are using OVH and not SYS...
I can send my current grub.cfg if you want to compare to yours, if you want to replace it manually without doing the update-grub from rescue/chroot...

 

[stormy27]

@Obi_Yoann You Rock!
So having chance to be on OVH with IPMI access, I simply had to:
Remove net.ifnames=0 from /etc/default/grub (thanks also @Neox !
Update-grub
Reboot

My server is responding again! I'm so happy! You save my day! (so stressing situation!)
Thanks again, happy to be part of this community sharing knowledge with others

NB: in my situation, try to disable the OVH MONITORING before doing IPMI or whatever, else your server is entering in rescue mode and it's annoying when you're trying to solve directly on IPMI console

 

[Neox]

@Obi_Yoann You Rock!
So having chance to be on OVH with IPMI access, I simply had to:
Remove net.ifnames=0 from /etc/default/grub (thanks also @Neox !
Update-grub
Reboot

My server is responding again! I'm so happy! You save my day! (so stressing situation!)
Thanks again, happy to be part of this community sharing knowledge with others

NB: in my situation, try to disable the OVH MONITORING before doing IPMI or whatever, else your server is entering in rescue mode and it's annoying when you're trying to solve directly on IPMI console

nice to see many brains can find a good solution.

as I struggled with my server, I went to the point I also stop "ooh monitoring" to avoid technician going down to DC to check my PC and reboot it in rescue while I was testing my procedure...

what sound strange is that the server (SYS-1-SAT32) take nearly 4 minutes to reboot up to ping available and ssh login
well I was first afraid I made a mistake, then I know I must only wait when I asked to reboot

 

[Neox]

as per my experience, yesterday from fresh SYS server Sata drives
I had to follow the Proxmox wiki upgrade with ifupdown2 and hwaddress in /etc/network/interfaces
PLUS I had to remove net.ifnames=0 from /etc/default/grub

then reboot is fine

if using ZFS, you may need timeout option as described here :
https://forum.proxmox.com/threads/t...rade-from-promox-5-to-proxmox-6-on-ovh.60476/

An other way to achieve pve6to7 upgrade on SYS EFI machine
is to change enoXX to old ethXX name and do not change the grub settings

on my server today :
- install pve6
- add ifupdown2 as recommended per Proxmox wiki
- remove resolvconf and cloud-init package as asked per ifupdown2
- adapt the /etc/network/interfaces.new to :
-- add hwaddress (as per Proxmox wiki),
-- add interface eth0 in manual mode and eth0 to the vmbr0
-- set ip, mask, gateway and DNS to vmbr0 as static mode

restart once to check anything is OK
proceed to pve6to7 upgrade via sed to replace buster to bullseye according to Proxmox wiki
no more touching grub option

 

[p1new]

Thanks, took me a bit of time to decipher your post and figure out what to do (don't work much in Linux or whatever flavor Debian is).

I didn't have the eno1/2 issue you had, In my ifconfig/ip a -- post upgrade -- it seemed to have linked an alias for it when it booted up.

Anyway, I'll add some additional details for anyone who has this same issue.

• in the IPMI console, when the refind menu comes up, see which one of the two EFI listed allows you to boot. Just go through and select each one to boot until it doesn't bring you into rescue mode
• After booting, you want to mount both EFI System partitions and see which one has the newer grub file.
• Copy that over and reboot

root@ns56:~# fdisk -l
Disk /dev/sda: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: HGST HUS724040AL
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt

Device Start End Sectors Size Type
/dev/sda1 2048 1048575 1046528 511M EFI System
/dev/sda2 1048576 42991615 41943040 20G Linux RAID
/dev/sda3 42991616 45088767 2097152 1G Linux filesystem
/dev/sda4 45088768 7814031359 7768942592 3.6T Linux RAID
/dev/sda5 7814033072 7814037134 4063 2M Linux filesystem


Disk /dev/sdb: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: HGST HUS724040AL
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt

Device Start End Sectors Size Type
/dev/sdb1 2048 1048575 1046528 511M EFI System
/dev/sdb2 1048576 42991615 41943040 20G Linux RAID
/dev/sdb3 42991616 45088767 2097152 1G Linux filesystem
/dev/sdb4 45088768 7814031359 7768942592 3.6T Linux RAID


Disk /dev/md2: 19.98 GiB, 21458059264 bytes, 41910272 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/md4: 3.62 TiB, 3977564389376 bytes, 7768680448 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/vg-data: 3.62 TiB, 3977563340800 bytes, 7768678400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
root@ns56:~#

mkdir /mnt/efi1
mkdir /mnt/efi2

root@ns56:~# mount /dev/sda1 /mnt/efi1
root@ns56:~# mount /dev/sdb1 /mnt/efi2

root@ns56:/mnt/efi1/EFI/proxmox# ls -l
total 144
-rwxr-xr-x 1 root root 147456 Dec 3 18:56 grubx64.efi
root@ns56:/mnt/efi1/EFI/proxmox# ls -l /mnt/efi2/EFI/proxmox
total 152
-rwxr-xr-x 1 root root 151552 Dec 8 00:23 grubx64.efi
root@ns56:/mnt/efi1/EFI/proxmox#

root@ns56:/mnt/efi1/EFI/proxmox# cp grubx64.efi grubx64.efi.120721a
root@ns56:/mnt/efi1/EFI/proxmox# cp /mnt/efi2/EFI/proxmox/grubx64.efi .
root@ns56:/mnt/efi1/EFI/proxmox# ls -l
total 296
-rwxr-xr-x 1 root root 151552 Dec 8 00:55 grubx64.efi
-rwxr-xr-x 1 root root 147456 Dec 8 00:55 grubx64.efi.120721a

Hey guys, i had the same issue on a couple of servers we just got from soyoustart, i found the reason there is an issue and a solution.
The issue is because the servers have multiple disk drives, in my case 2 nvmes. OVH created EFI partitions on both drives, with the same label, so you can't be sure which partition is mounted on boot on these servers. They have this in their fstab :
LABEL=EFI_SYSPART /boot/efi vfat defaults 0 1
At first when we received the servers, both partitions were identical. After upgrade the files were updated on the partition that was mounted but not the other (which is quite understandable indeed).
diff -r /boot/efi/ /mnt/efi/
Only in /boot/efi/EFI: BOOT
Binary files /boot/efi/EFI/proxmox/grubx64.efi and /mnt/efi/EFI/proxmox/grubx64.efi differ

I manually copied the content of the upgraded partition to the other efi partition on the other disk and on reboot it worked fine!

About refind, it is actually not installed on the servers, it is what is launched from pxe in normal condition, to load the efi loader from one of the partitions of the server. Setting the bios to boot from hard drive directly not only prevents your server from booting to rescue in case of issue, or from reinstalling, it can also prevent your server from booting in case one of the drives of the server fails...
My solution should be the closest to what the servers were set up initially, and be "kind of" future proof.

This is all because of what OVH/Soyoustart did to configure the servers, and this is not, i believe, documented somewhere!

Another thing, i had issue with network because a line was added to /etc/default/grub after upgrade, that caused the network devices to change their name (from eno1/eno2 to eth0/eth1), thus breaking the vmbr0 interface. I had to remove it and then update-grub.
I believe you can do most of this if not all from rescue mode if you have the issues after upgrading...
Fyi, I installed ifupdown2 before upgrading, and did not forget to add the mac address to /etc/network/interfaces as explained in the release notes.

Both servers are now happily up and running Proxmox VE 7.1.6!

 

[jf2021]

@Obi_Yoann, @p1new &al HUUUUUGE thanks for your explanations. Copying the new grubx64.efi worked fine for me to fix the broken boot with "grub_is_lockdown" error after a 6 to 7 upgrade on OVH server.

 

[jay89]

Wow thanks you @p1new that worked perfectly.

I had to do it 4 times as I have 4 HDD's. Been searching for a fix for a few days!

Hope we don't have to keep doing it every time there is a kernal update

 

[sukerman]

I ended up going into the BIOS boot menu on the server via IPMI and changing the boot order to GRUB as priority. Not sure why rEFInd was even on the server to begin with, but that's just another OVH thing I'm sure with their Proxmox images.

If your server does not have IPMI (which some SYS servers don't have, if I recall), then I'm not really sure how you'd adjust it. I think the servers that don't have IPMI are the ASUS boards that SYS uses on their cheaper models. There might be someone who's more sure on how to do this though.

This also worked for me, go into BIOS change first boot device to OS. It also renamed my network devices. Use ip -c link to see the dev names and edit your /etc/network/interfaces accordingly.