zfs on boot drive - what did I do wrong?

[pmlearner]

Hello,

I have (7) 4 tb drives and setup Proxmox VE to use zfs raid-z2 for everything including the boot/root/etc.

I ran into an issue where I could not boot, booting had the error:

error: no such device: f9x98sdfsdfasdf8x89

A few questions:

1. Any idea of the general problem? My guess is perhaps the hardware renamed the drive?

2. Is root/boot on zfs a bad idea?

I re-installed and used ext4 on about 100 gigs of /dev/sda

I made a zpool manually on the rest of the drives including the other 3500 or so gigs unallocated on /dev/sda. Bad idea (ie. is it bad practice to mix whole "raw" drives and partitions from a drive in other use as ext4?)

Thanks for any help and advice!

 

[LnxBil]

1. Any idea of the general problem? My guess is perhaps the hardware renamed the drive?

First: What PVE version are you on?

2. Is root/boot on zfs a bad idea?

No.

Bad idea (ie. is it bad practice to mix whole "raw" drives and partitions from a drive in other use as ext4?)

Yes, it is. ZFS should have whole disks, not partitions.

 

[fabian]

I'd recommend using UEFI boot and install PVE 6.0 - the boot loader only needs to see a single bootable disk (compared to the whole set that makes the rpool readable with legacy boot/grub), since kernels and initrds are stored on all of them.

 

[pmlearner]

Hi,

Thanks all. I'm using 5.4-3 installed the "normal" way from the iso (not installed manually over Debian) with zfs selected during setup (raidz2).

Before this occurred the datacenter techs performed a hardware check on the server and, I'd guess, plugged the drives in a different order (?). I'm new to all this but I recall in the past that you could not necessarily trust that /dev/sdx mapped to the same physical drive and it was best to use the drive "ID" (again getting my terminology wrong I'm sure).

Does proxmox normally use the drive ids to create the pool or does this need to be done post install?

I really appreciate all help and corrections. In this case I just "gave up" and re-installed but I really need to make sure I understand exactly what I should be doing so that I can avoid and/or fix any future problem like this.

Btw, the server is hosted at hetzner and they have the boot set to the network first which I assume is so they can determine if I've requested the rescue boot, etc. They said not to change the bios setting. Does anyone have experience with this? Is it best to not use the network boot as first option? Will this stop the ability to use UEFI boot (I'm really not up to date on modern motherboard BIOS options). I have access to remote KVM on the server so I can always login and reset the BIOS to network if I need Hetzner's rescue tools.

Thanks again!

 

[pmlearner]

Two additional questions (I've researched these but I feel a bit better to get a quick answer from people with direct knowledge):

1. Is ZFS deduping widely in use by ProxMox users? I have 32 gigs of ram and about 20 TB of storage so seems I'm close on ram depending upon your particular view on ram to de-duping cache. I've considered perhaps adding an SSD for L2ARC

2. Are the directions from :

https://forum.proxmox.com/threads/step-by-step-install-w-efi-zfs-ssd-cache-file-server.41619/

the best way to set up UEFI boot or does PVE 6.0 change this?

from the post mentioned :

6. Fix the system to boot with UEFI
verify the drive letters with ls -ll /dev/disk/by-id
Identify the /dev/sdX of your boot drive(s) – mine were sdb and sdc
This next step re-formats the Solaris reserved partition for FAT as required by an EFS parition
mkfs.vfat /dev/sdb9
verify /boot/efi already exists or mkdir to create it
mount /dev/sdb9 /boot/efi
grub-probe -d /dev/sdb9
update-grub
grub-install -d /usr/lib/grub/x86_64-efi /dev/sdb
find /boot/efi -type f
This step adds the entry into your NVRAM (computer flash used during boot by EFI – it may or may not work on your hardware – but you can also manually enter an item in your UEFI by using a special key when your system boots, and pointing UEFI to the proper hard drive and directory)
efibootmgr -c -disk /dev/sdb -part 9
(should show a new boot named Linux)

 

[fabian]

Hi,

Thanks all. I'm using 5.4-3 installed the "normal" way from the iso (not installed manually over Debian) with zfs selected during setup (raidz2).

Before this occurred the datacenter techs performed a hardware check on the server and, I'd guess, plugged the drives in a different order (?). I'm new to all this but I recall in the past that you could not necessarily trust that /dev/sdx mapped to the same physical drive and it was best to use the drive "ID" (again getting my terminology wrong I'm sure).

Does proxmox normally use the drive ids to create the pool or does this need to be done post install?

in the installer only since PVE 6.0, but it is just cosmetic anyway.

I really appreciate all help and corrections. In this case I just "gave up" and re-installed but I really need to make sure I understand exactly what I should be doing so that I can avoid and/or fix any future problem like this.

Btw, the server is hosted at hetzner and they have the boot set to the network first which I assume is so they can determine if I've requested the rescue boot, etc. They said not to change the bios setting. Does anyone have experience with this? Is it best to not use the network boot as first option? Will this stop the ability to use UEFI boot (I'm really not up to date on modern motherboard BIOS options). I have access to remote KVM on the server so I can always login and reset the BIOS to network if I need Hetzner's rescue tools.

Thanks again!

you'd have to ask hetzner about that

 

[fabian]

Two additional questions (I've researched these but I feel a bit better to get a quick answer from people with direct knowledge):

1. Is ZFS deduping widely in use by ProxMox users? I have 32 gigs of ram and about 20 TB of storage so seems I'm close on ram depending upon your particular view on ram to de-duping cache. I've considered perhaps adding an SSD for L2ARC

dedup needs lots of RAM. like, LOTS of RAM. forever (as in, as long as you have deduplicated blocks). only enable it if you really know what you are doing.

2. Are the directions from :

https://forum.proxmox.com/threads/step-by-step-install-w-efi-zfs-ssd-cache-file-server.41619/

the best way to set up UEFI boot or does PVE 6.0 change this?

there is a completely new EFI System Partition sync mechanism and tooling around it. check the admin guide