OMV on Proxmox VE - a few considerations

[bernardosgr]

I guys,

First of all, considering this is my first post, I just want to say that I'm loving this forum, this community and PVE itself. It's pure awesome.

Now, about the matter at hand, I've been looking far and wide for a good, comprehensive explanation on the ideal setup for a simple, good performance, fileserver.

At first I was considering the use of FreeNAS but it seems like a lot of people have complained about stability issues when virtualizing it and running it under containers. I immediately shifted to OpenMediaVault which looked fairly simple to use and setup, and had some users commending it for stability and ease of use inside Proxmox.

Good enough for me

But then, I tried to go a little more in depth and until now I've seen nothing but a huge divergence of opinions, so I want your help in planning this out.

My setup and goals:

I currently have 3 1TB drives that I want to use for a fileserver. 1 128GB SSD that I'll use for storing VM data and possibly configuration backups. I'll also use an external USB drive for housing backup data from all VMs and proxmox itself.
I'd consider ZFS but I'm probably going to go simpler: 2 drives in RAID 1 for important data and 1 drive for the irrelevant files.
BTW, I'm thinking about allocation a maximum of 8GB RAM for this fileserver.

Hardware:
- AMD FX Series FX-6300
- 16 GB DDR3 CL10 RAM
- ASRock 970M Pro.3

I guess the major questions here are the following:

• How should the VM be set up? Do I have to use KVM or can it even be a container?
• Should the disks be passed through to the VM directly? Would it be better to build the storage arrays inside Proxmox and hand them as virtual drives to the VM?
• Are there other performance considerations for drives?
• Are there alternatives to OMV that are better for this kind of setup?
• What should I consider in terms of backups?

I'm guessing this might be a lot to take in but mostly I'm focused on getting a stable, good performing fileserver that can communicate with several devices and still leave room for some VMs. Possibly an NFS share and maybe a content streaming application such as Plex or XBMC.

Thanks a lot for the help ,
Bernardo

 

[Q-wulf]

Will that Node just run proxmox + OpenMediaVault ?
Side note: we use OpenMediaVault ontop of Ceph for some 20+ OMV-VM's, serving some 70+ nodes and hundreds of clients.


How should the VM be set up? Should the disks be passed through to the VM directly? Would it be better to build the storage arrays inside Proxmox and hand them as virtual drives to the VM?If so you could:
- Setup Proxmox on the SSD (OS + VM-Data + ISO's)
- Setup OMV with 3 Virtual Disks (i prefer Virtio) each of em housed on of the 3 HDD's. Alternatively pass the HDD's directly through. Each has its up's and downs. A HDD-Passthrough does not easily allow you to backup those drives data to your USB-HDD, as Proxmox can not "see" em (since they are passed through)
- Create your "Software-Raid" config inside OMV according to your speed needs.
- Attach OMV via NFS/CIFS to proxmox

What should I consider in terms of backups?
-Consider that Raid != Backup ... Raid = increase in fault tolerance and/or speed.
-Consider the 3-2-1 strategy (or any variation thereof) Keep 3 Copies of each file. 2 Locally (OMV + USB) and 1 remotely (ColdStorage (Drive not connected to any Powersource via cable - e.g. a USB-Drive in your fire-proof Safe. / Backups on DVD/BlueRay) or on a cloudprovider ( encrypted Backups on your favourite cloud hoster)

Do I have to use KVM or can it even be a container?
- If you'd have chosen FreeNas you'd need KVM (as it is based on FreeBSD)
- OMV is based on Debian (so CT's are possible) - we do KVM on ours for no apparent (to me) reason

hope it helps.

 

[bernardosgr]

Will that Node just run proxmox + OpenMediaVault ?
Side note: we use OpenMediaVault ontop of Ceph for some 20+ OMV-VM's, serving some 70+ nodes and hundreds of clients.


How should the VM be set up? Should the disks be passed through to the VM directly? Would it be better to build the storage arrays inside Proxmox and hand them as virtual drives to the VM?If so you could:
- Setup Proxmox on the SSD (OS + VM-Data + ISO's)
- Setup OMV with 3 Virtual Disks (i prefer Virtio) each of em housed on of the 3 HDD's. Alternatively pass the HDD's directly through. Each has its up's and downs. A HDD-Passthrough does not easily allow you to backup those drives data to your USB-HDD, as Proxmox can not "see" em (since they are passed through)
- Create your "Software-Raid" config inside OMV according to your speed needs.
- Attach OMV via NFS/CIFS to proxmox

What should I consider in terms of backups?
-Consider that Raid != Backup ... Raid = increase in fault tolerance and/or speed.
-Consider the 3-2-1 strategy (or any variation thereof) Keep 3 Copies of each file. 2 Locally (OMV + USB) and 1 remotely (ColdStorage (Drive not connected to any Powersource via cable - e.g. a USB-Drive in your fire-proof Safe. / Backups on DVD/BlueRay) or on a cloudprovider ( encrypted Backups on your favourite cloud hoster)

Do I have to use KVM or can it even be a container?
- If you'd have chosen FreeNas you'd need KVM (as it is based on FreeBSD)
- OMV is based on Debian (so CT's are possible) - we do KVM on ours for no apparent (to me) reason

hope it helps.

Hi Q-wulf, thanks for your reply!

I'll be using proxmox to run OMV (with roughly 8GB RAM allocated, tops), possibly IPFire (about 512MB RAM) and an LXC with OwnCloud (I think that's around another 512MB).
The remainder will be for housing some experimental VMs.

I know the burden will mostly be on processing but I think it'll do well, considering OMV will not have to deal with intensive workloads most of the time.

About the VM setup:
I think I'm going to go your way and directly hand the disks to OMV.
Should I hand them as virtio through the VM's configuration file or is there a better way to do it?
Also, is there some additional configuration I should make, considering Proxmox is installed in a different drive from the SSD? It is currently set up in a USB drive (seems pretty stable thus far) and I left the SSD space for housing VM-Data and ISOs.

About the backup strategy:
I'm definitely considering a 3-2-1 strategy. I'll keep 2 copies locally, 1 in the storage array from OMV, 1 in a usb drive and 1 copy remotely in Amazon CLoud Drive (I'll try to get a FUSE filesystem going).

Thanks,
Bernardo

 

[Q-wulf]

What type of disk is your Proxmox OS setup on ? HDD/SSHD/SSD/SSDviaUSB ?

You stated you have
3x 1 TB HDD
1x 128GB SSD
1x USB Type unknown.

regarding passthrough of HDD's:
I personally never used it in anything production.
This might help you :

http://forum.proxmox.com/threads/24322-Hard-Drive-Passthrough?p=122551#post122551
http://pve.proxmox.com/wiki/Physical_disk_to_kvm

 

[bernardosgr]

I was looking at precisely those 2 references just yesterday

This is also not going to be a production environment.

My environment is set up on a USB flash drive (16GB USB 3.0). I know it's not recommended but I've researched it extensively and it seems that it is fairly stable with nowadays' hardware.

I guess my main concern is that, in case of a major, catastrophic, failure, I'll have the necessary configuration files to re-setup everything.

After I'd done the install on the USB drive, I had the system fail at startup but it was just a matter of tweaking the boot parameters inside GRUB.

 

[Q-wulf]

I'd do it different (especially if you do not use your SSD as some type of Cache) ..

• Use the SSD for OS + Iso's + VM-Data
• Pass the 3x 1TB HDD through to OMV
• Pass OMV as NFS through to your Proxmox (you could do backups of VM-Data on this too)
• Use the USB-Stick as a local "Backup" (call it copy) for Config-Files (keep excessive writes of from it) - i've done some test-systems based on USB-Sticks and the performance is horrendous and quite taxing on the lifetime of a USB-Stick ... different story for SSD's attached via USB.

Once everything is setup use a software like Clonezilla to create a clone of your "working" Node. You can keep that anywhere to do a quick reinstall of a "base Proxmox system". Then just keep incremental updates of the Proxmox Config-Files to completely restore stuff.

I've not done it myself yet, but it is on my to-do list, based on this:
https://forum.proxmox.com/threads/2...th-L2Arc-Slog-2nd-opinion?p=123419#post123419
Post #18 and #20.

 

[bernardosgr]

Yeah , I was considering migrating but was thrown off because USB seemed a lot stabler than complaints let show on earlier versions of the software.

I'll go precisely with that. Let's hope I can get it set up in less than 3 hours so that I can still post back today

Do you have a reference for the sort of backup data that should be in place (bsides the Clonezilla approach which I'll also follow)? Not just for the PVE node but also for VM-Data and OMV arrays.
I've researched a bit on this and noticed it being more about configuration files than data files.

 

[Q-wulf]

No personal experience, but i found a thread from '2012 where tom suggested backing up
/etc/ and having a look into http://pve.proxmox.com/wiki/Proxmox_Cluster_file_system_(pmxcfs)

As with OMV:
you obviously wanna backup the actual VM/CT via vzdump. That takes care of your your Base VM/CT data. The Data on the HDD's you either do via your internal replication (which is not a backup but fault tolerance), or a copy in another place (e.g. DVD's / cloud provider / external HDD / External NFS/ ... )

If Raid is not what you wanna rely on but you wanna do actual backups , the easiest would be to look back into post #2 where i suggested OMV on virtual Disks. That way you can use Vzdump to back it up to an external destination (4 TB USB / NAS / whatever).

 

[aivdev]

I've similar server...
I've SSD with 256GB with 20GB for pve-root and 185GB for pve-data

Furthemore I've HDD 4TB that I want to use to data storage.

I would like to create a OMV machine where SO is installed in 20 GB (of ssdve-data) anda use 3TB of 4TB disk for data. And the other TB of HDD for data in other machine.

So, What I've to do with proxmox if I want this?

THanks.