How to better use disks on new setup of PVE 4.1?

[alexc]

Hello,

For years I lived with the idea that the only underling storage for PVE should be hardware-backed array of disks, which can self-heal, self-cached, and which is guaranteed to be always-in-good-shape.

But now I see recent PVE versions are with ZFS out of box (but as I played with some of 3.x release, ZFS was kind of unpredictable for its speed so I given up back that time).

Now I need to set up pretty powerfull server with 2 SSD and 2 HDD disks (two of each kind to have it mirrored) but without H/W RAID card so what I see as an options are:

1. Create md-backed mirrors
or
2. Create LVM-backed mirrors
or
2. Create ZFS-backed storage

so I decided to ask you: what would you recommend?

What I need is to create host machine to host 1 or 2 web servers (both are basically LAMP), and provide backup for the VM. I'd like to boot from HDD, and store VMs on SSD, while doing backup back to HDD. The ram is like 32 or 64Gigs, so no problem of this.

 

[wolfgang]

Hi,
do you have tried ZFS with ssd cache?
This should bring the needed performance.

 

[alexc]

Thanks for the opinion!

I will try but as far the approach was a bit different: have VM stored on SSD and back it up to HDD. This is to have separate VM storage and backup storage.

In a case of ZFS, what would you recommend: I can create ZFS mirror of HDD and another ZFS mirror of SSD. Or, I can use HDD as disks and SSD as cache, keeping in mind storage's durability?

 

[vkhera]

I personally prefer to use ZFS whenever and wherever I can.

I have servers with your exact configuration: 2SSD and 2HDD. The SSDs are a mirrored pair, and the HDDs are a mirrored pair. However, the use case for those servers is not proxmox... for Proxmox, I'd see if I could squeeze in another SSD and use that as the log drive (for the ZIL) for the HDDs. It doesn't need to be so very big, maybe 5GB or 10GB. A small SATA DOM might be suitable. I do not feel safe running a production system with a single SSD as boot, so I will not recommend using one drive for boot and one for the ZIL/L2ARC.

If you have enough RAM, you likely don't need an SSD for the L2ARC at all, but it doesn't hurt to add one if you have the disk for it.

The proxmox installer does not support this for you out of the box, but you could also partition the SSDs and make room for the ZIL there.

 

[alexc]

I personally prefer to use ZFS whenever and wherever I can.

I have servers with your exact configuration: 2SSD and 2HDD. The SSDs are a mirrored pair, and the HDDs are a mirrored pair. However, the use case for those servers is not proxmox... for Proxmox, I'd see if I could squeeze in another SSD and use that as the log drive (for the ZIL) for the HDDs. It doesn't need to be so very big, maybe 5GB or 10GB. A small SATA DOM might be suitable. I do not feel safe running a production system with a single SSD as boot, so I will not recommend using one drive for boot and one for the ZIL/L2ARC.

If you have enough RAM, you likely don't need an SSD for the L2ARC at all, but it doesn't hurt to add one if you have the disk for it.

The proxmox installer does not support this for you out of the box, but you could also partition the SSDs and make room for the ZIL there.

I do love ZFS but on FreeBSD not on Linux for that very reason of being some kind of blackbox: if it fails I don't know to fix it. But ok, let's try and discuss the setup.

What I can imagine is to have part of SSD as ZIL and another part as L2ARC and another (bigger) part as drive for VMs. In fact having 64 Gb of RAM I think I won't see big slowness even without SSD cache but why not? I also hope ZFS can deal much better with trim etc that any other way of using it (but that kind of magic need to be checked).

The only thing I don't like about this setup is that Ill have 4 "connected" together drives and not 2 + 2, so it'll be a bit more tricky to deal with the setup in a case I'll need to play with.

But yes I home 4.1 is much better that 3.x branch. I tried ZFS on 3.x and it was unpredictable working, sometimes fast, sometimes slow, sometimes web gui wan't able to control disk commands - something that noone like to see in production.

 

[vkhera]

Heh... I also use FreeBSD exclusively (other than proxmox underneath). Still, I use ZFS. I asked some really smart folks (like the guys who wrote ZFS in the first place at Sun) at a conference, and they say they run ZFS on Linux with little trouble so I decided to go with it. This was last fall.

So the SSD cache for L2ARC *only* comes into play when the ARC is overflowing. When you have lots of RAM, depending on your working set size, you may never overflow your ARC, so the L2ARC will be unused, so why bother having one? It really depends on your working set size.

I'm playing with a SATA DOM now. I think I will be getting some of these to insert as ZIL's on my proxmox nodes to speed up the ZFS a bit. Since they are bigger than I need I'll also likely throw part of them as L2ARC just because I can, not because I need to. I have to do some more research on what happens if the ZIL dies (but the system is still running). I think it survives because the ZIL is never read unless there is a recovery event.

 

[alexc]

Heh... I also use FreeBSD exclusively (other than proxmox underneath). Still, I use ZFS. I asked some really smart folks (like the guys who wrote ZFS in the first place at Sun) at a conference, and they say they run ZFS on Linux with little trouble so I decided to go with it. This was last fall.

So the SSD cache for L2ARC *only* comes into play when the ARC is overflowing. When you have lots of RAM, depending on your working set size, you may never overflow your ARC, so the L2ARC will be unused, so why bother having one? It really depends on your working set size.

I'm playing with a SATA DOM now. I think I will be getting some of these to insert as ZIL's on my proxmox nodes to speed up the ZFS a bit. Since they are bigger than I need I'll also likely throw part of them as L2ARC just because I can, not because I need to. I have to do some more research on what happens if the ZIL dies (but the system is still running). I think it survives because the ZIL is never read unless there is a recovery event.

I'm afraid SATA DOM devices are often not that robust so I'd rather not to use it. Even as cache. I'm using my SSD under load only knowing I have them mirrored and have backup of VM to HDD every several hours. Kind of tradeoff between price of Intel S37xx series and hope for speed resulted from the use of SSD.

Will try and hope will see soon if its worth the mess with. But as for me I'd better create 2 mirrors, SSD+SSD and HDD+HDD, and hope plenty of RAM will (almost) do the trick.

ZFS is pure magic, you know, and we used to trust it, but hope I'll never need to revive it by hands. Sorry to see installer won't allow to do (officially supported) install over md mirror (thus pushing people to use ZFS, which is not that bad intention), since md is kind of very well-known tool for years. Surely I can add mirror by hands but ZFS looks like 'recommended' way...

 

[pizza]

ZFS with 2 SATA disks is not recommended with a machine with 4 - 8 GB Ram. The swappartition is located on the zfs pool. When swap is in use, the cpuload is skyhigh. Better to use mdraid when you have no raidcontroller.

 

[alexc]

ZFS with 2 SATA disks is not recommended with a machine with 4 - 8 GB Ram. The swappartition is located on the zfs pool. When swap is in use, the cpuload is skyhigh. Better to use mdraid when you have no raidcontroller.

Even that you see above I have 64G of RAM, I really wonder why PVE won't allow to install on md mirrors from the box.