Running a NAS on PMVE

[hacman]

Hi all,

I run quite a number of PMVE systems in production, with enterprise grade storage and such, and they all work just fine.

However, I am currently looking for a solution to use at home, and as the scale of the whole thing is a bit different, I find myself with a few questions.

I have an HP ProLiant Microserver, with an SSD for the boot drive, and 4 big SATA HDDs in hardware RAID for storage. Currently it just does one thing, and that is run as a NAS.

I want to put ProxmoxVE onto it, so that I can run two VMs for a few services we want in the home, but also need to keep the NAS running on it, likely using FreeNAS.

Has anyone ran a NAS on ProxmoxVE like this? Do you just have a giant disk image attached to one VM? Is it safe (most important) and performant (less important) to do this? Do you present the storage in a different way?

Any insight is appreciated!

Jon

 

[MoxProxxer]

I am currently in the process of moving my NAS from my non-virtualized 4-core J1900 Baytrail (OMV - openmediavault) to my Proxmox Denverton system. Because the total size of the arrays is roughly 120TB, I will most certainly not use any disk images, but pass the devices directly through to the VM which will run OMV.

I have bookmarked this as a starting point:

https://serverfault.com/questions/896027/kvm-pass-through-disk-partition-by-label-or-uuid

 

[hacman]

Thanks for the info!

120TB is a bit more than I am playing with - as it's home use I'll be looking at nothing larger than 20TB, and that's if I upgrade things in the future.

Is there any consensus on at which point it becomes ill-advised to use images?

Jon

 

[MoxProxxer]

Not aware of any consensus, but the biggest image I use in production is 100GB (a mirror site) and I consider that highly borderline already.

So my personal threshold is that - anything in the TB region is not for images. At least not until we have 100TB HDDs with 4 GB/s bandwidth in our regular off-shelf servers.

 

[vshaulsk]

In my environment I have 4 different NAS systems running.

1 - Dedicated Freenas box used for ISCSI, CIFS, etc.... (dedicated SAN/NAS) [excellent storage performance - full 10gb network saturation]

1 - OMV installed as a VM running in proxmox with an HBA passed directly to the VM (16 x 2TB disks) ... this system backs up the Freenas box [good storage performance]


1 - OMV running in proxmox with the disks as images [good storage performance]

1 - Freenas running in proxmox with the disks as images [bad storage performance]


The VM of OMV and freenas, I use as test system and to check updates. The physical hardware is an R710 (OMV) and R620 (Freenas) .... lots of ram and fast storage (R620 has all SSD). Also the network is 10gb.


From a storage performance standpoint.... OMV as a VM using disk images, performs pretty good. (Not as well as passing the HBA)

Freenas, performs terrible ..... I am not sure if it has to do with my configuration or just having ZFS on top of hardware raid w LVM-thin.

 

[hacman]

In my environment I have 4 different NAS systems running.

1 - Dedicated Freenas box used for ISCSI, CIFS, etc.... (dedicated SAN/NAS) [excellent storage performance - full 10gb network saturation]

1 - OMV installed as a VM running in proxmox with an HBA passed directly to the VM (16 x 2TB disks) ... this system backs up the Freenas box [good storage performance]


1 - OMV running in proxmox with the disks as images [good storage performance]

1 - Freenas running in proxmox with the disks as images [bad storage performance]


The VM of OMV and freenas, I use as test system and to check updates. The physical hardware is an R710 (OMV) and R620 (Freenas) .... lots of ram and fast storage (R620 has all SSD). Also the network is 10gb.


From a storage performance standpoint.... OMV as a VM using disk images, performs pretty good. (Not as well as passing the HBA)

Freenas, performs terrible ..... I am not sure if it has to do with my configuration or just having ZFS on top of hardware raid w LVM-thin.

Very interesting! Thank you!

So it's obviously not something specific to the FreeNAS code, since the instance you run that is presenting the storage from a SAN is fine, and OMV seems to perform well either way.

I'll have to conduct some tests and see if we can pinpoint where the issue is, though as we're only talking 4 x 2TB disks on a P410 I guess my use case may not see that much problematic behaviour.

With the images do you have any concerns from a data safety perspective?

Jon

 

[vshaulsk]

From, what I have learned so far in my own setup (If I wanted an all in one setup):

If possible I would have separate storage for my VM images (seperate disks for proxmox) and pass through an HBA to my NAS operation system. This way the NAS operating system has direct access to the disks. I think this way you get all of the performance from the NAS, but don't have to deal with very large disk images.

Another option is to use the native Debian which Proxmox is built on: you can setup ZFS right on proxmox and use something like webmin to manage the setup of NFS, Cifs, etc.

From a data perspective:
I don't store large data within my Proxmox VM disk images (my biggest disk image is 500 gb), my largest files maybe 10gb (only tests no critical). I am not sure how it will scale up if you have multiple terabytes.

From a proxmox data standpoint, I have had excellent performance and good luck with the system (have been running proxmox for many years)
- I do backup my VM's and have had to restore them once in a while.

* for an all in one proxmox & NAS VM, I would use OpenMediaVault as it is more lightweight vs Freenas. If you have an external USB drive you can pass the USB to VM and use the OMV usb-backup plugin to backup the critical data of the NAS. (There are many options for this kind of setup)