First real setup seeking confirmation of concept.

[jim.bond.9862]

OK,
I have an OMV server running on SM-SC846(24 bay)| H8DME-2 |2x AMD Opteron Hex Core 2431 @ 2.4Ghz |49GB RAM machine at home.
I want to load a Proxmox setup on it and run OMV in VM for better and easier management and upgrades.
This will be free license as it is my home setup single machine.

here is where I am not sure how to proceed.

I only have 1 server with all storage on it. the server is 24 bay sata box with about 8 disks.
OS: 2x120GB SSDs
DATA: 4x3T| 3x2T | 2x1T


I need/want a Host (Proxmox) obviously.
I also need/want to share the data storage disks from the very same host.

DO I :

(A) . configure Proxmox as needed using 2 SSDs in ZFS Raid-1
with minimal local storage, say using 2x1TB drives in raid-1 config for system needs and VM configs and drives
create an OMV VM and pass-through the rest of the disks into it for all management needs.

OR

(B) configure Proxmox as option (A) plus setup a BTRFS raid-10 pool using the rest of the data disks (4x3T| 3x2T ) and yes I want to use BTRFS for my data pool as it works better with mismatched disk..
setup SMART tools on Proxmox (it maybe there already in V4.3+) and just manage the DAS via CLI on Host.

now, if I go with option "B"
how do I share the DAS with all VMs and network PCs easily ?

my main reason I wanted OMM is the ease of managing Samba and NFS shares with WebUI.
option B does not give me this.

my main needs are:

#1. disk/hardware monitoring. (I know that Proxmox have some hardware monitoring build in.)
#2. allowing access to main data pool (the one I will build using BTRFS raid-10) as it was a file server setup.
I need a samba shares and NFS shares.

one way maybe it to have the pool shared as NFS, mount that in OMV VM and do all samba and staff from there. is it sounds OK?

thanks.

 

[LnxBil]

I'm unclear why you want to use Proxmox VE inside this stack. If you hard pinpoint disks to the VM, you cannot do "easier upgrades", because everything is hard wired. Virtualization is only "worth" using if you virtualize everything. If you do not, you have only more work and a more complex setup, but less benefits and features.

Technically both setups work and you can use a kind of storage VM with directly attached disks in Proxmox VE, but as I already said, you cannot "test" a upgrade because everything is hard wired. I do not thing that mixing ZFS and BTRFS (or any other filesystem) is a good idea with respect to such limited RAM (every OS has its own block cache).

Running ZFS for your OS only is also overkill - without using benefits from ZFS, why bother? If you want to do everything by yourself, why don't you create a big raid10 BTRFS pool and attach both SSDs as cache to BTRFS so that you only have one pool (or whatever it is called in BTRFS). You have a much simpler disk setup and will have probably more speed. Of course you need to setup your BTRFS before on Debian Jessie and then install Proxmox VE on top of it.

 

[jim.bond.9862]

I'm unclear why you want to use Proxmox VE inside this stack. If you hard pinpoint disks to the VM, you cannot do "easier upgrades", because everything is hard wired. Virtualization is only "worth" using if you virtualize everything. If you do not, you have only more work and a more complex setup, but less benefits and features.

Technically both setups work and you can use a kind of storage VM with directly attached disks in Proxmox VE, but as I already said, you cannot "test" a upgrade because everything is hard wired. I do not thing that mixing ZFS and BTRFS (or any other filesystem) is a good idea with respect to such limited RAM (every OS has its own block cache).

Running ZFS for your OS only is also overkill - without using benefits from ZFS, why bother? If you want to do everything by yourself, why don't you create a big raid10 BTRFS pool and attach both SSDs as cache to BTRFS so that you only have one pool (or whatever it is called in BTRFS). You have a much simpler disk setup and will have probably more speed. Of course you need to setup your BTRFS before on Debian Jessie and then install Proxmox VE on top of it.

OK, I am not sure I understand what you are saying here.

first of all, what do you mean when you say "easier upgrades" ? or "test" a upgrade ?
upgrades of what?
the Host?
the VM?
can you please clarify it for me.


also can you clarify meaning
"Virtualization is only "worth" using if you virtualize everything. If you do not, you have only more work and a more complex setup, but less benefits and features."

I do plan to virtualise everything, I just not sure what is the best way of doing it. hence my post here.
the only reason I am looking into VM with direct-attached disks is that I am not sure how to share and manage this disks via Host.
Proxmox does not have the tools and option in the WebUI for me to manage the disks in the fashion I need to
and/or at least manage a setup similar to file server configuration. it only has option to manage the Host local storage used for VM infrastructure.

I know I can just pool all of this disks together on the host using CLI (the option "B" in my OP)
and than build out a File server VM like OMV using an expanding virtual disk. But is it a good way for a home server to do this?
also how would I back all of this up on an external USB drive?

this is what I am trying to figure out here. and what I am asking help on.

Second, this is a home server with maybe 5-6 users,so I think I can afford a bit a non-optimal config.

Third, it maybe an old school, but I prefer to separate OS from Data. hence I use SSD for OS and other drives for Data. also I want to use ZFS for OS only because it allows me to setup a bootable RAID-1 OS disk right from the Proxmox installer, and it works straight forward. no mess, no fuss, it just works.
I would like to be able to have a raid-1 setup option available for other OS including BTRFS, but it is not there.
doing a 2 part install (Debian+Proxmox) is a bother I want to avoid, if it comes to the fact that I have no choice I would do it but right this moment I do not plan on it.
the only benefit ZFS give me in this setup is that I have a simple option of creating a bootable raid-1 OS disk with in an official system installer. if ZFS was as flexible with using mixed size disks as BTRFS is I would be using it as described, but it isn't, and it is more cost prohibitive in future updates as well.

the laydown is, if I use ZFS as you proposing,
let say I still want to separate OS from data, so a raid-1 ZFS pool for OS and the rest is for data.
my data drives are "4x3T| 3x2T | 2x1T "
I will have to setup 3 ZFS pools at one with 3TB drive , one with 2TB drives and one with 1TB drives.
none of them is easily updatable or expandable . none of them is easily replaceable either as they are separate vdevs and can not intermingle.
and to expand the pool I will have to add drives in pairs. leading me to higher costs.

now look at this from the BTRFS setup perspective.
again, will use separate OS drive as before
but I can in BTRFS build a single raid-10 pool using all of the disks regardless of size.
I can add a disk of any size one at a time. I can remove any one disk at any time providing I have space on the pool to move the data from that disk to it.
so updates/upgrades/expansion is easier and less costly.

and with my setup I can install /reinstall or even replace Host OS at any time without disturbing the data config at all.

Is my logistic flawed?

Please look at my question in a "small home use setup" POV not as an enterprise config and use.
I run a Hyper-V setup at work and it is configured using practices you described, no I am not using ZFS or BTRFS in windows, but I do use virtualize everything and all methodology. but it is an enterprise SMB setup.
we have a good real time offsite backup and all . I am running a 2 node failover cluster there.

I do not have the resources to do this at home. so I am trying to figure out how to do the best and easiest setup I can put together using what I have.

Again, my question is not what is the best use of specific file systems here,
my question is how to have a single host virtualization server and File server on the same hardware and make it work , and how to best administer all of that using remote GUI.

also, I just though of it right now, I might have an option C as well. just not sure if it any good.

I can setup Proxmox as proposed in Option "B", OS on ZFS raid, and an extra local storage pool for VMs using the 1TB drives.
a BTRFS raid10 DAS pool using the rest of data drives.

than install WebAdmin and use that to setup an NFS/SAMBA file server on the host.
I can manage VM side in proxmox, a file server and DAS in WebMin
and mount the NFS shares in all VMs and PC clients falling back on SAMBA shares for windows and Mobile clients on limited basis.

what do you think?

Thanks sincerely VL .







'

 

[LnxBil]

OK, I am not sure I understand what you are saying here.

first of all, what do you mean when you say "easier upgrades" ? or "test" a upgrade ?
upgrades of what?
the Host?
the VM?
can you please clarify it for me.

With direct attached disks to a VM, there is not way to snapshot it, therefore no test of an upgrade, therefore not using features a virtualization provides.

Running BTRFS works, but is not supported in any way Proxmox VE (you cannot snapshot etc. and you also do not get support if you subscribed it). If you can live with that .. do it. Hopefully there is some send/receive technology in BTRFS which enables you to do incremental backups like ZFS can.

BTW: You can create raid setups with different sizes in ZFS too, but yes, it's not as dynamic as BTRFS. As long as you do not use RAID5/6 with BTRFS, everything is fine (both are still not marked as stable).

 

[jim.bond.9862]

With direct attached disks to a VM, there is not way to snapshot it, therefore no test of an upgrade, therefore not using features a virtualization provides.

Running BTRFS works, but is not supported in any way Proxmox VE (you cannot snapshot etc. and you also do not get support if you subscribed it). If you can live with that .. do it. Hopefully there is some send/receive technology in BTRFS which enables you to do incremental backups like ZFS can.

BTW: You can create raid setups with different sizes in ZFS too, but yes, it's not as dynamic as BTRFS. As long as you do not use RAID5/6 with BTRFS, everything is fine (both are still not marked as stable).

ok, that is a bit clearer now.

coincidently, "Hopefully there is some send/receive technology in BTRFS which enables you to do incremental backups like ZFS can"
there is options for send/receive snapshots built into BTRFS , just like in ZFS.
there are some limitations there but it exist. I do not plan to use it though for now anyway.

also I do not see an option to snapshot ZFS pool in Proxmox GUI either, did I miss something?
if not than I am in the same boat with ZFS as with BTRFS , CLI management at it best

I do not plan to use anything but raid-1 or raid-10, so issues there.
and I had test out a setup,using VM in Hyper-V, with Proxmox as main host OS + Webmin added
used 2 disk in ZFS raid=1 for host.
added 5 small disks and build a raid-10 btrfs pool on it.
mounted in /media/data
using webmin, installed NFS server and Samba server on the host.

seams to be a viable setup.
I can keep all VM related staff on local storage which might be on small ZFS pool (I have 2 1TB disk I can use.) and keep all my data files, media, PC backup etc. on shared BTRFS pool.

only thing need to test is setting up OMV in VM and remote mount and manage the NFS share properly. I do not want the host to be full fledge file server, want some easy use distro with webUI for management of shares (or in this case the actual data on them.)

 

[LnxBil]

ok, that is a bit clearer now.

coincidently, "Hopefully there is some send/receive technology in BTRFS which enables you to do incremental backups like ZFS can"
there is options for send/receive snapshots built into BTRFS , just like in ZFS.
there are some limitations there but it exist. I do not plan to use it though for now anyway.

also I do not see an option to snapshot ZFS pool in Proxmox GUI either, did I miss something?
if not than I am in the same boat with ZFS as with BTRFS , CLI management at it best

It's just the ordinary snapshot dialog of the VM.

 

[jim.bond.9862]

It's just the ordinary snapshot dialog of the VM.

well than, how it is related to ZFS or BTRFS for that matter?
you could and still can do a VM snapshot on any File system.
what is the difference?
why use of ZFS special?

 

[fabian]

well than, how it is related to ZFS or BTRFS for that matter?
you could and still can do a VM snapshot on any File system.
what is the difference?
why use of ZFS special?

guest disk snapshots are only possible if the underlying storage offers snapshots (like Ceph, ZFS, LVM-thin, ..) or if the disk image format itself does (QCow2). if you e.g. use raw images on NFS, you have no snapshot support. if you use ZFS, you do. if you use BTRFS but PVE does not know how to handle it, you don't have snapshot support.

of course you can always do snapshots yourself if your storage supports it, but those snapshots won't be managed or respected by PVE.

 

[jim.bond.9862]

guest disk snapshots are only possible if the underlying storage offers snapshots (like Ceph, ZFS, LVM-thin, ..) or if the disk image format itself does (QCow2). if you e.g. use raw images on NFS, you have no snapshot support. if you use ZFS, you do. if you use BTRFS but PVE does not know how to handle it, you don't have snapshot support.

of course you can always do snapshots yourself if your storage supports it, but those snapshots won't be managed or respected by PVE.

I see, so what you are saying is that all this time I have been swimming in the dark
until now I had not run Proxmox setup for real, it was only testing builds , but since I was testing mostly with default setup, which is for Proxmox before v4.x was always LVM
and since I mainly use QCoW2 disk format for my VMs, I expected to have snapshot ability.
But if I had used anything else I would be up the creek without a paddle.

I just tested the assumption and you are right.
VM1 - QCow2 image --can take snapshot
VM2- Raw Image -- no option to do so in UI.
both vms are on localstore BTRFS based volume directory

moved the VM2 disk to localstore-zfs and got the snapshot option instantly.

good to know thanks