NVME ZFS problem

[arcanatigris]

Hey all,

Im trying to install a new server with PVE4.3 with a ZFS RAID 1 configuration but it always results into a
"unable to create zfs root pool" while a standard ext4 install works just fine.

Hardware:
2x Samsung SM961 1TB

Error:

GPT data structures destroyed! You may now partition the disk using disk or other utilities.
Creating new GPT entries.
Setting name!
partNum is 1
REALLY setting name!
The operation has completed successfully.
cannot create ‘rpool’: no such pool or dataset
unable to create zfs root pool
umount: /rpool/ROOT/pve-1/var/lib/vz: mount point not found
umount: /rpool/ROOT/pve-1/tmp: mount point not found
umount: /rpool/ROOT/pve-1/proc: mount point not found
umount: /rpool/ROOT/pve-1/sys: mount point not found

I have tried adding "intel_iommu=on rootdelay=10" with different values to the grub from the install disk.
"Install Proxmox VE" press 'e' and adding them at the end of the linux line. and then press F10

Maybe I am doing something wrong here?

 

[fabian]

could you try the installer in debug mode and after it has failed, click abort and run "sgdisk -p /dev/DISK" (for each DISK, e.g. nvme0n1)?

 

[arcanatigris]

For some reason debug mode hangs and stops responding to any input.

Here is the output from a debian live disk:

 

[fabian]

I can reproduce this here, but I am afraid there is a bigger issue that will prevent you from running this setup (unless you are capable of bootstrapping it using jessie-backports): our installer does not support installing ZFS with UEFI support (because there is currently no sane way to mirror the EFI boot partition, so we simply don't), and almost all systems don't allow you to boot NVME devices in legacy/BIOS mode..

you should be able to install with ZFS on regular disks (SSD or HD) and add the NVME devices as log and/or cache or as a second pool

 

[Magneto]

I can reproduce this here, but I am afraid there is a bigger issue that will prevent you from running this setup (unless you are capable of bootstrapping it using jessie-backports): our installer does not support installing ZFS with UEFI support (because there is currently no sane way to mirror the EFI boot partition, so we simply don't), and almost all systems don't allow you to boot NVME devices in legacy/BIOS mode..

you should be able to install with ZFS on regular disks (SSD or HD) and add the NVME devices as log and/or cache or as a second pool

Has this been fixed yet?

 

[Yamnsllerty]

For some reason debug mode hangs and stops responding to any input.
Here is the output from a debian live disk:

Very detailed, hope it very helpful

 

[fabian]

Has this been fixed yet?

no, there is still no ZFS+UEFI support in our installer (as there is still no way to mirror ESPs except manually).

 

[Talion]

Any update on this issue as of today ?


Best Regards,

Talion

 

[fabian]

Any update on this issue as of today ?


Best Regards,

Talion

ZFS+UEFI is still not supported - if that is the issue you are referring to?

 

[Jorge Mesquita]

Hi, i'm running into this issue,

But i've tried installing in legacy mode and i can boot just fine from the nvme. (using ext4)
I've then tried to install the system using zfs in legacy mode. But no luck.

Based on your observations @fabian shouldnt it be possible to install zfs in legacy mode to a nvme array?

Regards

 

[Stoiko Ivanov]

Usually it's not possible to boot from an NVME-device in legacy/BIOS boot mode.

 

[Jorge Mesquita]

Hi @Stoiko Ivanov , thanks for the reply,

My Lenovo M700 boots just fine, i set the mode in the bios to 'legacy only' and installed proxmox on one of the nvme disks. Works just fine.

Any pointers on how to install proxmox using zfs in this situation?
Because i only have space for two disks and i bought 2 1TB ssds to setup as raid1, so setting a 3rd disk as root is not possible.

 

[Stoiko Ivanov]

Please check on the M700 after installing in 'legacy only' mode (with ext4) whether you have a '/sys/firmware/efi'
Else you can try setting up the system with debian (the zfs on linux wiki has instructions) on the nvme - and then install proxmox-ve on top of that

 

[fabian]

one NVME device (usually) works, its multiple that is basically impossible with legacy/MBR booting. you can either roll your own EFI setup using Debian Stretch, or move /boot and Grub to a third, small non-redundant device (e.g. USB stick) after installation and boot from that. both is not really ideal of course - we are working on some kind of solution for PVE 6.x

 

[ozgurerdogan]

Is there any news on this. Can not I install debian as zfs and proxmox on it? So any trick to install zfs on nvme?

 

[alexskysilk]

Not that I think using nvme for boot is a good use of the disks, but I believe you can accomplish it as described here:
https://outflux.net/blog/archives/2018/04/19/uefi-booting-and-raid1/

essentially, you'd need to install debian on zfs as described here: https://github.com/zfsonlinux/zfs/wiki/Debian-Stretch-Root-on-ZFS
and create an efi partition on a md RAID1 partition (~200-500MB in size), taking care to substitute the md device(s) in the sgdisk steps. Once you have a running system, you can install proxmox-ve taking care not to mix up the proxmox zfs packages.

 

[Asano]

Any update on this or maybe a workaround?

Specifically I'd like to install Proxmox on https://www.hetzner.de/dedicated-rootserver/px62-nvme but I'd like to avoid ordering them (or one to start with) and finding out Proxmox will not install

 

[ozgurerdogan]

I had to order extra ssd to install proxmox on it.

 

[Asano]

I had to order extra ssd to install proxmox on it.

Thanks for the info but that somehow would kill the purpose of a NVMe server

So I guess we have to wait. Or are there maybe other workarounds like maybe an easy way to install a non raid boot partition and still get to use most of the space for a RAID1 ZFS pool afterwards? That would at least offer some kind of redundancy and enable ZFS features for the most parts...
Unfortunately I don't have any NVMe hardware at hand with which I could test things and doing so with actual server at Hetzner requires remote hands/LARA and is therefore quite unhandy and a little costly...

 

[noname]

I have the same problem . supermicro with four NVME I try to install proxmox with zfs RAIDZ (RAID5) with no luck. any solution ?