Combining a Proxmox and a Freenas installation in home setup

[johnnychicago]

Hello --

I've been a happy proxmox user for a few months now, running a handful of VM's in my basement. I am just as happy using FreeNAS for a bit longer, living on the computer right next to it.

I've now been thinking about combining the work done by those two into a single physical machine (for power saving, and liberating one of the computers currently running). I have a bit of a hard time figuring out what the best approach is.

I like ZFS in Freenas. It currently serves files to a number of the VM's running in Proxmox, as well as other devices on NFS and SMB. It is being backed up to another Freenas using ZFS replication, and I'd really want to keep this going. I've been burnt and since am backing up religiously.

First thing I thought was taking the pool from the Freenas machine, physically transferring it into the Proxmox server and running Freenas virtualized. I've learnt that my Hardware doesn't support VT-d and this might therefore not be a great idea.

Now I am wondering if I couldn't leverage the fact that Proxmox can run ZFS. Could I import my pool and use Proxmox to do snapshotting, and ZFS replication? And then maybe share the whole pool to a VM running things like NFS and SMB to share out to the other VM's and the network? What should I be looking for in terms of software/setup to make my life easiest here? Some tool to manage the snapshotting would be very welcome.

I'd be sad to loose Freenas, but using Freenas' virtualization (bhyve) would be hard as well. So I am really looking for a reasonably smart option to combine ZFS file serving and virtualizing in Proxmox...

 

[alexskysilk]

Could I import my pool and use Proxmox to do snapshotting, and ZFS replication?

Yes.

And then maybe share the whole pool to a VM running things like NFS and SMB to share out to the other VM's and the network?

Yes, although in your case thats probably not necessary; you can run all the same services on proxmox directly. if you really want to compartmentalize, you can have a filer container.

What should I be looking for in terms of software/setup to make my life easiest here?

depends on what you're trying to accomplish (eg, what services is freenas performing for you?)

Some tool to manage the snapshotting would be very welcome.

thats already built in to proxmox- no additional software or setup required! If you're referring the filer pool, there are a number of scripts available but really its a simple matter of putting the commands in your crontab.

I'd be sad to loose Freenas, but using Freenas' virtualization (bhyve) would be hard as well. So I am really looking for a reasonably smart option to combine ZFS file serving and virtualizing in Proxmox...

As should become clear in the answers, both freenas and proxmox are good, multi purpose tools (incidentally, Freenas jails could probably accomplish most if not all of what you're using virtualization for anyway.) I would start by making a list of what your desired services are and tailor the solution accordingly. it is safe to say that you should be able to combine them into one server regardless of which server software you choose to use.

 

[johnnychicago]

Yes, although in your case thats probably not necessary; you can run all the same services on proxmox directly. if you really want to compartmentalize, you can have a filer container.

Would it not be discouraged to install tons of additional software on the proxmox host? I do understand it's debian, and it's not like I hadn't added some tools to my existing install, but still...

-v on the filer? I was thinking having a setup where Proxmox would really only cover the ZFS basics, export the whole tree via NFS to some sort of NAS VM that would then offer shares by NFS, SMB and whatnot, complete with user management and stuff.

How big is the performance hit going to be? I'd guess not dramatic, since the physical way out of the box is always only going to be 1GBit and I assume the virtual bridges to be faster than the HDD's.

depends on what you're trying to accomplish (eg, what services is freenas performing for you?)

On the storage end the whole ZFS shebang. Snapshotting, replicating to the backup, scrubbing etc. Then some rsync backup tasks inbound from remote servers, and on the sharing end offering SMB and NFS to clients inside and outside of the virtual environment.

(snapshotting)
thats already built in to proxmox- no additional software or setup required! If you're referring the filer pool, there are a number of scripts available but really its a simple matter of putting the commands in your crontab.

Like different schedules per dataset, automatically expiring and all? Cool. Can you enlighten me about what the filer pool is?

As should become clear in the answers, both freenas and proxmox are good, multi purpose tools (incidentally, Freenas jails could probably accomplish most if not all of what you're using virtualization for anyway.) I would start by making a list of what your desired services are and tailor the solution accordingly. it is safe to say that you should be able to combine them into one server regardless of which server software you choose to use.

I am hesitant about using Freenas for virtualization. It looks considerably less straightforward than Proxmox in that area, and it doesn't look like it'll support some legacy Windows (XP) VM that I desperately need to cater for old hardware.

 

[alexskysilk]

Would it not be discouraged to install tons of additional software on the proxmox host?

for "realz", yes. for home its fine.

-v on the filer? I was thinking having a setup where Proxmox would really only cover the ZFS basics, export the whole tree via NFS to some sort of NAS VM that would then offer shares by NFS, SMB and whatnot, complete with user management and stuff.

you can do it like that, but the overhead is a lot more then it seems. To get the maximal benefit from ZFS for virtualization, you'll map zvols as raw block devices to the virtual machines, which in turn will put their own file system on the zvols for their own use. you gain snapshot and replication at the block level. If you pass a block device to a filer virtual machine, you'll be creating another file system for sharing purposes- which means if you were expecting file snapshots and replication offered by ZFS you'd have to put ANOTHER zfs file system on the zvol. zfs eats memory and resources. Instead, just use the pool underneath, create a dataset(s), and share it out. much lower overhead.

Can you enlighten me about what the filer pool is?

poor choice of words; I was referring to the dataset you're using for shared data.

You CAN run virtualbox on freenas, and it works reasonably well- but in general I agree with your assessment; Proxmox does this by design.

 

[johnnychicago]

you can do it like that, but the overhead is a lot more then it seems. To get the maximal benefit from ZFS for virtualization, you'll map zvols as raw block devices to the virtual machines, which in turn will put their own file system on the zvols for their own use. you gain snapshot and replication at the block level. If you pass a block device to a filer virtual machine, you'll be creating another file system for sharing purposes- which means if you were expecting file snapshots and replication offered by ZFS you'd have to put ANOTHER zfs file system on the zvol. zfs eats memory and resources. Instead, just use the pool underneath, create a dataset(s), and share it out. much lower overhead.

Yeah, I'd want to do a proper distinction between the block devices used for VM's storage and use the rest as one/multiple datasets shared out to clients. I'm hesitant as to if I want to do the latter all manually directly from the proxmox host (and install/configure NFS, SMB and friends on it) or if I'd prefer just giving it to a VM/container as an NFS mount and install some sort of NASsy software on that one, to manage the sharing.

Anyway. Things starting to clear up and assumptions validated. Thanks a lot for the opinions.

 

[Wouter Samaey]

Is there a way to avoid having a ZFS filesystem in FreeNAS on top of the ZFS that Proxmox has? Or is this not possible?

Or in other words have FreeNAS use the hypervisors ZFS directly? I know this goes against the idea of virtualization, but to me this looks like the only way to combine both optimally.

 

[jim.bond.9862]

If you only using freenas for file sharing you maybe able to approximate the same setup using proxmox and zfs pools which you than bindmount into a file server container. Not sure what the equivalent of bind mount is for full VMs though.
With bind mount you connect the zfs pool to container and it will share it as if it was a real drive. Works well.
Check out proxmox turnkey container file server template.

 

[Wouter Samaey]

I think I may have found a better solution. Just add more drives and use some drives for FreeNAS directly, and use other drives for Proxmox.

Wouldn’t this be better?

 

[Quintin Cardinal]

I'd like to advocate for a single node Proxmox on ZFS setup. I took my old gaming PC hardware, bought 4x 1TB spinning disks, threw my old 256GB SATA SSD in for ZIL and L2ARC (I know, single disk no RAID isn't recommended for ZIL but this is not production, just my home server). In a ZFS RAID10, I have just under 2TB usable space and can get 1GB/s sequential write with over 25k IOPS in FIO testing. You could also, if you still wish to use both, use ZFS over iSCSI to the FreeNAS pool for VM storage and use Proxmox on ZFS for backups/LXC storage. FreeNAS can be added as an option for ZFS over iSCSI using the procedure here.

 

[Wouter Samaey]

if you still wish to use both, use ZFS over iSCSI to the FreeNAS pool for VM storage and use Proxmox on ZFS for backups/LXC storage.

Not sure if I follow, but you mean to say that I:
- Setup Proxmox with ZFS on 2 small disks with mirroring (boot only)
- Put in several spinning disks + dual SSD for ZFS + ZIL + L2ARC and use these directly for FreeNAS
- Create iSCSI disks in FreeNAS and use these for other VMs on Proxmox?

If this is what you mean, why is patching Proxmox needed? Proxmox can connect to any iSCSI without patching, right?

Also, if the server fails, the FreeNAS VM would need to boot first before any other VMs, as they would need the storage FreeNAS provides.

This is the setup I was considering, but am having second thoughts as this gets more complicated.
I do like the simplicity and tools that FreeNAS offers and it saves me from setting up everything in separate VMs without GUI etc.

This does mean that I would have 2 ZFS systems (1 for Proxmox and 1 for FreeNAS) which is the a downside I can't fully estimate in terms of risk due to lack of experience

 

[Quintin Cardinal]

If this is what you mean, why is patching Proxmox needed? Proxmox can connect to any iSCSI without patching, right?

Proxmox can connect to iSCSI, sure, but you would have to use iSCSI full-stop to a zvol on the FreeNAS box. This patch allows you to connect ZFS over iSCSI to the entire pool, and your VM's disks would each be a separate zvol on the ZPool. Supports snapshots, HA, all the cool guy stuff.

- Setup Proxmox with ZFS on 2 small disks with mirroring (boot only)
- Put in several spinning disks + dual SSD for ZFS + ZIL + L2ARC and use these directly for FreeNAS
- Create iSCSI disks in FreeNAS and use these for other VMs on Proxmox?

If you want to use a separate disks for FreeNAS, I would run them in different hardware and network connect the two devices. Personally, I think Proxmox on ZFS is awesome, and I like working with ZPools in Linux (especially Debian) much more than I like doing anything at all in FreeBSD.

If you're looking to have one physical server, I'd vote Proxmox as the host OS and install the OS on ZFS. It is available as an installation filesystem in the installer. If you want to use FreeNAS, I'd say you should probably keep a hardware RAID1 on your Proxmox server (or zfs RAID1 if you're using consumer grade hardware) and put all the go-fast bits in the FreeNAS box. If you go that route, definitely do the patch and use ZFS over iSCSI for the best possible experience out of your iSCSI connection to the FreeNAS server!

Hope I was clear enough, I'm a couple coffees deep and it's nearly the end of the work day!

 

[Wouter Samaey]

Hi Quentin, thanks for the thorough reply. Made a lot of things clear.

Basically, I'm trying too hard to combine the 2. The reason I wanted to use FreeNAS is because I'm replacing an old Synology, which I actually loved, but hated the fact that it does 101 different things in a Non-VM way, making me depend on the Synology hardware. Upgrading / replacing the Synology is not so easy, so their products are out for me.

I was looking at FreeNAS, because it has the same 101 tools on board that saved me so much time, like setting up time machine, different apps, WebDAV, etc.

Recreating all these tools in separate VMs would be a pain, so I was looking for an easy package that offers Synology-like easy of use, but in a virtualized way.

So maybe, FreeNAS is too complex because of the double ZFS filesystem, and I should look for another NAS-like OS, that does everything except the storage.

I'm also looking at Docker, but there setting things up like Time Machine gets quite complicated again. The complexity here differs greatly per container and how mature it is.

Would you by any chance have an idea for another tool that would combine well with the Proxmox on ZFS setup?

Whatever VM I end up with, it would again be a new filesystem on top of the great ZFS, so maybe there's a way to virtualize without the disks? This again comes down to Docker. KVM will have new FS and for LXC I'm not sure how disks work... Haven't explored them before.

 

[Quintin Cardinal]

KVM will have new FS and for LXC I'm not sure how disks work... Haven't explored them before.

LXC Containers are fun, and I much prefer them over docker. It acts like it's own VM, but really it works kind of like a BSD Jail. It has a mac address, an ip address, and an OS, but no kernel. It uses the Proxmox host kernel. This means extremely low overhead, I'm talking like 20MB of RAM and absolutely 0 cpu if this thing is just sitting idle. It will use a zvol for the disk image, and the LXC will see *zpool*/*zvol* mounted on / type zfs. There are some turnkey container templates available, I haven't messed with them though.

Would you by any chance have an idea for another tool that would combine well with the Proxmox on ZFS setup?

I unfortunately have not tried to use external shares storage on Proxmox other than FreeNAS. I do love it, I just think it is really going to require an additional piece of hardware.

 

[Wouter Samaey]

It will use a zvol for the disk image, and the LXC will see *zpool*/*zvol* mounted on / type zfs.

If this zvol can be created with thin provisioning, that would be perfect! Much like docker, you can use the main filesystem and don't waste any disk space or need to grow/shrink as the VMs disks fill.

Have you tried this in practise? Did a quick Google search and noticed that these are officially called Sparse ZVols?

 

[Quintin Cardinal]

If this zvol can be created with thin provisioning, that would be perfect!

It is thin provisioned! The raw file if you view in your VM storage shows the configured disk size, but it only uses what the container uses. Most of my LXC containers (like my bind dns servers) are less than 1GB in size. Space can be added to a disk while the container is running, and so can cpu and memory resources. I would probably shut down to reduce resources, but I don't think you have to!

Did a quick Google search and noticed that these are officially called Sparse ZVols?

I am not familiar with the term "sparse zvols", I'll have to research that when I get some time!