Storage Idea

Astraea

Renowned Member
Aug 25, 2018
223
40
68
I have been using Proxmox for a few years now to run the various servers I have on my network and have been using whitebox NAS solutions made by QNAP for my storage solution. Right now I run 3 QNAP boxes, 1 for personal storage, the second for VM storage for my 4 Proxmox Nodes and the other as shared storage for my nextcloud install and other random storage needs.

I am wanting to as reasonably as possible eliminate any single points of failure to minimize downtime while understanding I am operating a more or less hobby/home setup and on the budget to match.

I am wondering which is the better way to go for my eventual move to FreeNAS as my new storage solution. I know that FreeNAS does not support HA but was wondering if the following would be possible and be a suitable setup. Should I just run FreeNAS on bare metal hardware and build out my solution similar to how I do now with various boxes running my NAS solution or should I virtualize FreeNAS to create a sudo HA solution.

I am planning to purchase either 2 old Dell R720XDs or build a new setup. My goal is to have a storage solution that would be a 4U rackmount case that would support 12 drives not including 2 drives for the OS and at least 4 gigabit ethernet ports plus IPMI.

My question though is if I were to go the virtualize route could I pickup say 3 smaller 1U servers and and 2 DAS boxes to start and could I in theory virtualize FreeNAS and get something close to HA by having the 2 DAS attached to each Proxmox node. Could I then passthrough the DAS boxes through to FreeNAS in such a way that it would not matter which node it was running on the DAS boxes would be connected. This would allow for say 12 drives for data storage from on DAS and other 12 drives from the other DAS as a backup to the first DAS so that as long as one DAS is operating I will have my data. Of course I would still be able to loose drives in either DAS as they would be setup in probably RAIDZ3. My theory is that this setup using Proxmox to run the freeNAS as a VM on any of the 3 nodes would make the VM have HA.

I know that it would obviously be more hardware than just running a FreeNAS box on bare metal but I am curious if this is possible and also if I could then extend the storage solution when the time comes by say creating another VM after adding more DAS boxes to the setup.

I would say the maximum I would need in either setup would be 3 FreeNAS setups, 1 for data storage, the second for VM storage and the third for critical data replication. Which I am missing the replication part now in my current setup.
 
You should use ceph rbd with a target of 2 for the rbd pool.
Each proxmox node of your setup would be hypervisor AND storage.
 
  • Like
Reactions: AlexLup
I am wanting to as reasonably as possible eliminate any single points of failure to minimize downtime while understanding I am operating a more or less hobby/home setup and on the budget to match.

Hmm ... I think this is already the end. You cannot eliminate any single point of failure in a home setup. You need a HA storage box or CEPH with an HA network and redundant power supplies. I don't think that is inside your budget.
 
He explained that he can have multiple 1U or 2 old R720XDs. With that he can make an HA setup by spreading the disk between the 2 boxes enable ceph and make a proxmox cluster in a 2 node setup.
 
Hmm ... I think this is already the end. You cannot eliminate any single point of failure in a home setup. You need a HA storage box or CEPH with an HA network and redundant power supplies. I don't think that is inside your budget.

I'm not familiar with current Dell gear but couldn't an option be two compute nodes, two data store nodes and use an aggressive VM replication schedule? This is assuming you could get enough storage out of two 1U servers and essentially mirror. Not true HA by any means ... but CEPH has a bit of overhead especially as your OSDs grow -- you really only get about 85% of your total storage to not overfill them in the safety range. I wouldn't suggest that 2:1 setup is good for anything critical.
 
He explained that he can have multiple 1U or 2 old R720XDs. With that he can make an HA setup by spreading the disk between the 2 boxes enable ceph and make a proxmox cluster in a 2 node setup.

2-node cluster with an additional third quorum node. Yes, sure he can build a CEPH cluster ... but still ... there are multiple SPOFs left:
- network (multiple switches required and for a good setup at least 4 nics cluster/storage and outside)
- power supply lines (UPS)
- internet uplink
 
I have considered CEPH for holding VM images but I would still want a separate solution for holding all my other data. Most of my servers don't use the internet so a dual internet link isn't needed. All my servers have dual power supplies and are on UPS. I also run 6 switches that all support spanning tree so thats also covered.

My goal is to have a reliable and close to HA NAS storage as possible for my data, ideally, it would be a SAN but that's out of my budget.
 
Install ceph, activate RGW (S3) and multisite sync with another ceph cluster is my plan..
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!