boot off of usb with logs/db written to zfs?

[sjpotter]

Its highly recommended that proxmox not be installed onto a usb key type device, because proxmox will wear it out due to its active writing to its root file system.

This is somewhat problematic in devices like the HP DL380p (at least G8) as one cannot boot off the normal internal hard drives if its in HBA mode (as one would want for proxmox). It would seem a sane solution would be to be able to install proxmox to a usb key, while at the same time creating a zfs pool with the hard drives connected in HBA mode and for proxmox to automatically create a dataset on the pool for its active writing.

From what I can tell this isn't possible today (though perhaps I missed something). Is their a reason such functionality wouldn't be desired?

The other option (which I've done with ubuntu) is to install /boot and grub on the usb, while the root file system is on the hard drives attached to the now hba mode adapter. Is it possible to do such a setup in proxmox installation today, or will one have to jump through a bunch of hoops?

i.e. something like (off the top of my head)

1) install normally onto hba drives
2) reboot back into installation media but go into debug mode (perhaps can do this without rebooting)
3) mount the the zfs dataset where the root file system is (perhaps unneeded if dont need to reboot)
4) create a file system on one's internal usb (or whatever usb you want to boot off of). preferably something that can easily fstab mount via UUID
5) edit fstab to mount said usb as /boot
5) copy /boot from root fs to usb file system
6) install grub onto usb.

the idea would be then that the machine would boot off the usb, but would only use that for kernel/initrd while it would mount the root file system normally as if it could boot directly off of it. as the usb is also being mounted at /boot kernel/initrd updates should work as normal. as the usb is only being used for /boot, it shouldn't suffer from excessive writes.

am I off base? do people do it this way? is there a better way (perhaps lwhat i I described above where root is on usb, but heavy write area is stored on hba based zfs?)

 

[Dunuin]

You can also use the Debian ISOs installer which is more complex and will allow to put bootloader, boot and root on different drives. And then install PVE ontop of that. But Debian doesn't support ZFS, so that would be a problem if you really want that instead of mdadm or btrfs.

 

[LnxBil]

This is somewhat problematic in devices like the HP DL380p (at least G8) as one cannot boot off the normal internal hard drives if its in HBA mode (as one would want for proxmox).

Is this a hardware thing? I'm only familiar with non-p models, so I cannot say, but normal HBA IT mode should never be a problem, why should it? If you run EFI, you need to make sure that the BIOS sees the stuff, but otherwise it should just work.

The other option (which I've done with ubuntu) is to install /boot and grub on the usb, while the root file system is on the hard drives attached to the now hba mode adapter.

I ran such a setup on a machine I upgraded from non-EFI to EFI hardware and didn't want to reinstall everything. Works like it should. In the end PVE is Debian so you can do whatever you want..

 

[sjpotter]

Is this a hardware thing? I'm only familiar with non-p models, so I cannot say, but normal HBA IT mode should never be a problem, why should it? If you run EFI, you need to make sure that the BIOS sees the stuff, but otherwise it should just work.

G8 doesn't have EFI, that was only added in G9 (afaict). and afaict it just doesn't enable non logical drives (i.e. raid mode) to boot. Or at least, I haven't figured out how (and I believe googling around found many with the same issue with the response that yes that's not supported, but can't find those links right now. Have it booting off an old usb 2 8gb key for I had for plain ubuntu right now. works fine "except" anything that updates kernel or initrd takes a very long time (slow usb key for writing). thankfully, that's not really that common of an occurrence.

I ran such a setup on a machine I upgraded from non-EFI to EFI hardware and didn't want to reinstall everything. Works like it should. In the end PVE is Debian so you can do whatever you want..

I guess my point was that I know for a fact ubuntu's installer supports this and I'm guessing Debian's does too, so it be nice if proxmox could support it natively too in its installer.

 

[LnxBil]

nd afaict it just doesn't enable non logical drives (i.e. raid mode) to boot.

You could try to switch the HBA to one that supports it. The internal one is the default Pxxxx hba that is flashed with an IT firmware? I just installed an ordinary megaraid-based SAS HBA in all system that needed a working hba.

I guess my point was that I know for a fact ubuntu's installer supports this and I'm guessing Debian's does too, so it be nice if proxmox could support it natively too in its installer.

You can do that too, just install Debian as you like and then upgrade to PVE, that should always work.

 

[sjpotter]

You could try to switch the HBA to one that supports it. The internal one is the default Pxxxx hba that is flashed with an IT firmware? I just installed an ordinary megaraid-based SAS HBA in all system that needed a working hba.

not flashed, its actually has built in support for switching between hba and raid modes. run the hp utility and can select raid or hba mode. (unlike the Dell's where you have to flash it). With that said, I think I'd rather have an r720xd vs the dl380p (12 bay) at this point. ilo is in some ways nicer than idrac on the r720xd (html5 console is a big win), but overall, its less homelab friendly (i.e. its really expects to run only hp branded hw)