Storage design/plan thoughts?

[Pulssarman]

Ok, I'm a newbie to Proxmox only just discovering it. I've decided to virtualize my home environment and am a bit confused on how to approach my storage plan;

Hardware: Dell r710 with internal SSD (250gb) and SATA raid5 array on Perc H700 (8TB).

So far I've installed Proxmox on the internal SSD, then created the raid5 array. I couldn't figure out how to add the array to Proxmox at first. Then after some nosing around the interwebs, figured out it has to be available on Proxmox OS first. So I fdisk'd the array drive to Linux (8e), and then pvcreate /dev/sdb1 and created a volume group called Array-vg. Back in Proxmox I added "local storage" "existing volume group" and it shows up fine.

I created my first couple of test hosts on the SSD and that works fine. Now my plan is to have the array be a big media storage area for several KVM's on the server - what is the best way to do this? I was under the impression that Proxmox would be the NFS server and then NFS mount them to the KVMs, but I don't think that is correct. So now do I;

A. Install linux server on the SSD and then "add disk" for the whole of the array and then share it out via NFS, Samba etc. to the other KVMs?

B. Install another Linux NAS type solution distro like FreeNAS or OpenMediaVault etc.?

C. something else?

I'd like the ability to reboot any client KVM without knocking down the storage array for other KVMs, but just not sure how to proceed. One of the KVMs will be Plex and it will need nice fast access to the array. Any thoughts are VERY welcome!

Thanks!

 

[Vito Reiter]

I have kind of a biased opinion when it comes to this because I'm a FreeNAS junkie. If you run FreeNAS on a separate box with the appropriate specs it'll do great things for your ProxMox cluster. ZFS (although usable by ProxMox) works incredibly well and intelligently in FreeNAS and it comes with several bonuses:

• Assign NFS storage to the ProxMox cluster accessible by all nodes
• High-Availability will become so fast there is almost no downtime
• Better data safety - You'll have a dedicated box for storage that is not being used by VM's etc. and ZFS will be running to its full potential with no bottlenecks and ProxMox will only have to handle config files.
• VM migrations will take seconds while running, and if your node literally turns off, when everything is back on the VM will be right where it was because FreeNAS will keep it spinning the whole time (In most cases this will work vice-versa as well)

There are several things you should look into prior to setting this all up, tons of documentation on here and on the FreeNAS forums as well. Pairing these two systems together has done me and my business so well recently and really increased performance. In this all, though I did run into hiccups so make sure you do research and feel free to inbox me if there are any serious questions I'd love to help

Good luck and do research!
Vito Reiter
vito@compcorrect.cc

 

[jim.bond.9862]

I have kind of a biased opinion when it comes to this because I'm a FreeNAS junkie. If you run FreeNAS on a separate box with the appropriate specs it'll do great things for your ProxMox cluster. ZFS (although usable by ProxMox) works incredibly well and intelligently in FreeNAS and it comes with several bonuses:

• Assign NFS storage to the ProxMox cluster accessible by all nodes
• High-Availability will become so fast there is almost no downtime
• Better data safety - You'll have a dedicated box for storage that is not being used by VM's etc. and ZFS will be running to its full potential with no bottlenecks and ProxMox will only have to handle config files.
• VM migrations will take seconds while running, and if your node literally turns off, when everything is back on the VM will be right where it was because FreeNAS will keep it spinning the whole time (In most cases this will work vice-versa as well)

There are several things you should look into prior to setting this all up, tons of documentation on here and on the FreeNAS forums as well. Pairing these two systems together has done me and my business so well recently and really increased performance. In this all, though I did run into hiccups so make sure you do research and feel free to inbox me if there are any serious questions I'd love to help

Good luck and do research!
Vito Reiter
vito@compcorrect.cc

vito, I will try to not to sound overly obtuse or anything, but where in OP did you see that he is trying to build a Proxmox cluster?
what relevance do your post have to OP question at all?

let me elaborate a bit here,

OP asks for help in how to better setup/use a local storage on a SINGLE HOST.
he/she has single(one, 1, uno) machine with a bunch of disks that he wants to setup proxmox on and use the Local storage on.
HOW does your post on setting up and using FreeNas on a dedicated hardware helps OP with his/hers situations ?

I have very same needs as OP. I have ONE server that I want to use for Proxmox.
nothing else. no additional hardware exists or planed.
and just like OP I also have , in that server , a bunch of drives that I want to use for data storage.
and Just like OP, I want to use my ONE server, as a Virtualization server AND a File Server.
will/can your post help me and OP do what we want/need to do?


PS>> I really not trying to be snooty or anything , I am just tired of seeing and reading posts like this one, that have absolutely no relevancy to the question asked and no help at all, everywhere. I got tons of them on my posts and have seen tons of them all over the forums I have been doing my research and info gathering on.

 

[Vito Reiter]

He asked if he should set-up a Linux server such as FreeNAS to do what he is trying to do. Furthermore, everything I mentioned works both in a single ProxMox host as well as in a ProxMox cluster (Allowing for future expansion) I don't believe it had no relevance:

So now do I;

A. Install linux server on the SSD and then "add disk" for the whole of the array and then share it out via NFS, Samba etc. to the other KVMs?

B. Install another Linux NAS type solution distro like FreeNAS or OpenMediaVault etc.?

C. something else?

Obviously, I took the information that was provided and I myself gave a solution that works for what is set-up and allows for expansion in the future. That in itself is the point of setting up systems such as these; Reliability, Functionality, and a strong ability to adapt and expand.

Hope you understand,
Vito

 

[jim.bond.9862]

He asked if he should set-up a Linux server such as FreeNAS to do what he is trying to do. Furthermore, everything I mentioned works both in a single ProxMox host as well as in a ProxMox cluster (Allowing for future expansion) I don't believe it had no relevance:



Obviously, I took the information that was provided and I myself gave a solution that works for what is set-up and allows for expansion in the future. That in itself is the point of setting up systems such as these; Reliability, Functionality, and a strong ability to adapt and expand.

Hope you understand,
Vito

OK, maybe I am misunderstanding the OP wants here.
the way I read it was
#1 I want to setup Proxmox as VM host
#2 I want to use the Proxmox host as both virtualization server and File server for the whole setup.
#3 I got the Proxmox host up and going and have an array on raid5 ready to be shared
how can I do File server setup ?

but you are right , now that I have reread the OP it seams a bit confusing on what the goals is, at least to me.

didn't mean to offend.

 

[Pulssarman]

Sorry, I really wasn't very clear in how I wrote that. Let me clarify a few things and tell you about my approach now;

I have a single physical machine, running Proxmox as the primary OS (no KVM etc for Proxmox);

r710 - internal CD rom has been replaced with a holder that has a Samsung EVO 250gig SSD installed (I know not full speed but this works quite well and keeps the bays free). I then have 4 of the 6, 3.5" drives, populated on the server with 4TB drives. These are raid5 so 12TB available. What I have done right now is I installed Ubuntu 16.04 (KVM) on the array itself, with most of the space taken by this server. It has SMB and NFS and will share to the rest of the KVM's that need access. The SSD has Proxmox on it and is a storage area for some other KVM's installed there. I'm aware there is no redundancy with that.

I may be able to make a private network on the server itself and do all the drive sharing over that for greater speed/efficiency but need more time to play.

Thanks for the responses, I definitely know now how to progress this beyond home setup if I ever need too, but this is primarily a home media server with a few other vm's needed for some other stuff.

Thanks!!

 

[jim.bond.9862]

Sorry, I really wasn't very clear in how I wrote that. Let me clarify a few things and tell you about my approach now;

I have a single physical machine, running Proxmox as the primary OS (no KVM etc for Proxmox);

r710 - internal CD rom has been replaced with a holder that has a Samsung EVO 250gig SSD installed (I know not full speed but this works quite well and keeps the bays free). I then have 4 of the 6, 3.5" drives, populated on the server with 4TB drives. These are raid5 so 12TB available. What I have done right now is I installed Ubuntu 16.04 (KVM) on the array itself, with most of the space taken by this server. It has SMB and NFS and will share to the rest of the KVM's that need access. The SSD has Proxmox on it and is a storage area for some other KVM's installed there. I'm aware there is no redundancy with that.

I may be able to make a private network on the server itself and do all the drive sharing over that for greater speed/efficiency but need more time to play.

Thanks for the responses, I definitely know now how to progress this beyond home setup if I ever need too, but this is primarily a home media server with a few other vm's needed for some other stuff.

Thanks!!

so I was mostly correct in reading the OP.
you have a single physical machine that you want to use as a Proxmox Host and a File server.
on your HOST you have setup a Proxmox OS using SSD for OS.
than build out a fully virtualized(KVM) Ubuntu setup using your RAID-5 volume passed/attached to VM as both OS and data volume, which you than share using NFS/SAMBA from within the VM setup.

BTW, I am struggling in figure out how to do the very same setup myself.

well if you do this why not just use the host for sharing?
what do you use to manage the shares and data volume (the raid-5 volume)
Is your Ubuntu install a desktop or CLI only setup?
if you use CLI to work with your configs than you do not need an extra overhead

there is no redundancy because you only have one machine so that is not an issue you can solve, unless you want to setup multiple physical servers. than , if you do , you can follow the Vitos advice and have a segregated File server and Proxmox server.


no as I was saying, and bare in mind that this is the solution I am leaning to with my own setup,
if you ok with CLI ,or maybe using a third party UI on the Proxmox server. you can (and BTW some of the steps maybe already done )

#1. build a Proxmox setup (already done that)
#2. using SSH or login directly to host CLI install WebMin for Debian.
there is a plenty howtos on how to do this. just be carefull when you run install and I have found that a webmin install may hung up at the end for a while , so just let it sit there. on many of my test runs it can look like it is just sitting there for 15-20 minutes doing nothing, but trust me it does something there.
PS> I recommend WebMin because I found it this one of the best WebUI that actually works very well and not interfere with Proxmox install if you take the percussions not to manage or edit things that are managed by proxmox with it. i.e. only use webmin to setup/manage NFS and Samba and maybe your data volume.
I also use to edit some config files when proxmox calls for ssh cli use. (I am still in research mode so all my tests and trials are on a test setup not production )
newer webmin has a very good file manager plugin that also have a file editor built-in. almost never need to use ssh/cli. you can essentially just run both UI (Proxmox and WebMin )
in browser sessions side by side, and manage all via Web GUI with only very limited CLI needs. (I find some things are just easier or faster to do vai CLI. but not much)

#3. go into WebMin and setup Samba and NFS there. you may need to install samba and nfs modules for webmin
and maybe the server modules for proxmox using cli and " apt-get install " (I believe clients for nfs and samba are already there.) once there the webmin can manage them easily.
also the data drives mounts and such.

just take care not to touch anything that is managed by Proxmox. networking, primary volume, and other things.
only use webmin to manage nfs,samba and your data drive(s).
that is the setup I am leaning towards myself. it has some drawbacks but have some good points as well.

unless you need some more advanced management it would work well.
Proxmox box would have access to the physical drives so it can run SMART monitoring on them for you,
the host will have the access to the data directly, and you can mount the shares on all and any VM using either smb or nfs and manage data on them as needed.

 

[Pulssarman]

Jim, that's a fine idea.... *slaps forehead*!! I'm going to look into that now. Rearranging basement at the moment so the server isn't up but should be in the next couple days. I'll try that and go from there. Eliminating another layer can only help right?!

Thanks much!!

 

[jim.bond.9862]

you are welcome
if you want to separate the 2 anyway you can always try the container approach as well.
add a container VM and bind mount the data drive(s) to it. than you can install webmin there and manage samba/nfs from within the container. this should protect the host , but adds some complexity to the setup.
my next try is to setup a turnkey file server container and see if it works well.