Best Storage Practices for Small One, Two, and Three Node Installations

[uberdome]

I'm interested to know if there are accepted best-practices for storage on small or limited installations, or to hear personal thoughts on best-practices. I recognize the benefits of Ceph on larger installations with more drives available, but in this case I'm thinking specifically about limited space / limited budget sort of installations.

If a location only warrants one or two nodes, what sort of minimal storage configuration would be acceptable for function and backup? These are some configurations / questions I'm thinking, but I'm open to any suggestions:

One-Node

• Single internal HDD or SSD, with external on-network backup?
• RAID 1 internal HDD or SSD, with external on-network backup?
• Shared network storage for VMs with a single device or replicated device, with external on-network backup?

Two-Node

• Single internal HDD or SSD in each node, with external on-network backup?
• Shared network storage for VMs with a single device or replicated device, with external on-network backup?
• PVE-zsync?
• Storage Replication?

​

Three-Node

• Is it worth doing Ceph if there is only one drive in each node?

I actually have one of each of these scenarios at this time. One site that we can only justify the cost of a single node that will be running several network monitoring tools. It would be good if it didn't go down, but the network would continue to function without it if necessary. A second site that is more critical but is tight on space and power, so we were looking at two nodes for manual failover. A third site where it would be nice to have high reliability through a cluster of 3 nodes, but with limited power consumption and limited budget.

I'm not sure I posed the question clearly, but in general, I'm looking for the accepted methods for maximizing reliability when space, power, or money (and thus, nodes) are constrained.

Thank you, Chris

 

[Astraea]

Here is my take on your situation, not sure these are "Best Practices" but from what I have learned through my years, this is what I would probably do given this information:

Single Node:
-Dual PSUs in the node
-2 SSDs in RAID 1 for Proxmox (Host OS)
-As many HDDs or SSDs that can fit in that node in an appropriate RAID
-A separate device used to backup not only the VMs but also Proxmox as well.

Dual Node:
-Dual PSUs in the nodes
-2 SSDs in RAID 1 for Proxmox (Host OS) in each node
-A separate device used with as many HDDs or SSDs that can fit appropriate RAID for shared VM Storage
-A separate device used to backup not only the VMs but also Proxmox as well.

Three Node:
-Dual PSUs in the nodes
-2 SSDs in RAID 1 for Proxmox (Host OS) in each node
-A separate device used with as many HDDs or SSDs that can fit appropriate RAID for shared VM Storage
-A separate device used to backup not only the VMs but also Proxmox as well.

A few notes about this plan and a little more explanation:

1 Node: Depending on if this is a tabletop setup or rack-mounted you should be able to fit all this equipment into 2Us of space. Though you won't have HA you would have some redundancies in the PSUs, HDDs/SSDs but and of course backups to help reduce downtime.

2 Node: Depending on if this is a tabletop setup or rack-mounted you should be able to fit all this equipment into 4Us of space. Though you won't have HA you would have some redundancies in the PSUs, HDDs/SSDs but and of course backups to help reduce downtime. However, the storage device for the VMs is a SPOF in this design, though it is one that I have used before with great success.

3 Node: Depending on if this is a tabletop setup or rack-mounted you should be able to fit all this equipment into 5Us of space. However, the storage device for the VMs is a SPOF as well.

I currently run 6 nodes with 2 NAS servers for VM storage and 2 other NAS devices to backup VMs. I do not back up the Node OS as it is fairly quick to spin up a new installation and add it into the cluster. I can also run on only 4 nodes so my biggest weakness is the fact my VM storage is a single point of failure as the 2 NAS devices are used for different VMs. Though the backups are saved to both of those NAS devices.

 

[guletz]

Hi,

I think you ask the wrong question. You ask how to do a task, but you need to ask yourself what are your goals, like:
- no data lose/acceptable to lose data writen in last x hours;
- service disponibility;
- time to recover after a indisponibility/crash event;
- whatever;

After you define your own goals, then you can try to think how to achive this targets using your limited budget(belive me that I had have a low budget most of the time)!

 

[uberdome]

Hi,

I think you ask the wrong question. You ask how to do a task, but you need to ask yourself what are your goals, like:
- no data lose/acceptable to lose data writen in last x hours;
- service disponibility;
- time to recover after a indisponibility/crash event;
- whatever;

After you define your own goals, then you can try to think how to achive this targets using your limited budget(belive me that I had have a low budget most of the time)!

guletz, I suppose this is an iterative process for me. Most of these boxes will be running network monitoring software, so downtime is not preferred, but not a functional / major problem... but I would prefer not to deal with downtime when possible. There are some other more important VMs, but I suppose from this process I'm hoping to identify where I would run them (it may be preferable to run on-site, but I could run them at the main office if a 2-server on-site cluster just won't meet the requirements).


Perhaps I have some particular [backwards] questions:

1. When are you too small for ceph (Assuming ceph is always a good option when possible.)?
2. What are preferred storage methods when too small for ceph?
3. Do the preferred methods vary significantly with uptime need, or are they fairly consistent?

Understanding those concepts would help me to understand the needs. In my experience, it seems the most failures I have are related to hard drives (although that is still not many). That suggests to me that storage should be my primary interest when trying to understand reliability with a new system.

Here is my take on your situation, not sure these are "Best Practices" but from what I have learned through my years, this is what I would probably do given this information:

...Dual PSUs in the node
...2 SSDs in RAID 1 for Proxmox (Host OS)

in each node​

...A separate device used with as many HDDs or SSDs that can fit appropriate RAID for shared VM Storage
...A separate device used to backup not only the VMs but also Proxmox as well.

... I can also run on only 4 nodes so my biggest weakness is the fact my VM storage is a single point of failure as the 2 NAS devices are used for different VMs. Though the backups are saved to both of those NAS devices.

​

Astraea, in every scenario, you suggested to use separate VM storage, and didn't mention ceph. Do you have a specific reason for not using it / trying it? Based on my reading, it seems like your 6 node arrangement would be ideal for it. [I'm really interested to know the reasoning.]

Have you tried running your NAS devices in a high-availability configuration to eliminate the SPOF?

The separate storage seems like a good / easy way to handle scaling since you can add nodes without having to fundamentally change storage. If I used a separate device for the 1 node location, I could add a second node without much hassle.

Regarding running RAID for the host OS, in a larger cluster, if you can handle losing nodes, why would you take the extra expense of a second SSD and RAID card instead of just replacing a failed OS drive? Same question for dual PSUs.

Conceptually, if I'm starting with 3 nodes, it seems a 3-disc ceph setup would be less expensive and potentially more reliable than external storage (even though there is a speed limitation of doing so).​

 

[Astraea]

When I was designing the implementation I already had the NAS units in place, though they were serving a different purpose at the time. I did look a ceph but decided against it as it would require me to purchase more hardware. I originally started with 2 nodes each with a small SSD (250GB) for the OS and decided that for me it was not worth the expense at the time to use properly implement ceph. When I later upgraded to new nodes the cost to add more than 1 drive(currently each node as 1 250GB SSD) to the systems was the prohibitive factor to not move to ceph at that point as I would have needed to install enough drives in each server to replicate my 20TB of existing NAS storage.

I have recently gotten 2 Dell R510s which I will be using as NAS servers and I am considering running them in an HA configuration I am not too worried about them being a SPOF as they are backed up by 2 additional NAS units that contain backups of the VMs of both servers so I essentially have a live copy of the VM and 2 backups. Those backup NAS units are also capable and will be configured to be able to run the storage for the VMs should I have issues with either or both of the R510 units.

If and When I replace the existing nodes with newer machines I will look at ceph again and its impacts on hardware costs.

I currently do not run the node OS drives in a raid as I can afford to lose a node or 2, though it would be a better implementation in my option but not worth the added cost for my situation.

I try to make each node as redundant as possible and since they are all rack mounted enterprise servers they come with dual PSUs which is another layer of protection at the node level. It's also why I have each node on two different UPSs.


My implementation is not my ideal method but was put together on a very small budget as this is still a home-based setup in a matter of speaking. It has slowly evolved as funds become available and I can find used and new equipment that meets my needs. Ideally, I would be running an odd number of nodes (probably 7 or 9) with a true SAN implementation and supporting infrastructure that would allow for no SPOF but I have looked into the cost and it is just not feasible at this time.

 

[uberdome]

Since I'm continuing to research this in an attempt to better understand the options, I'll continue to share my findings here, and possibly build a spreadsheet that might be helpful to someone in the future.

First, backups are critical to minimizing downtime in the even of a major failure. Proxmox has built-in backup capability, and it seems an on-network drive share (likely basic NAS) is a good idea for a destination. Since this is relatively straightforward and appears to be universal for every option, I won't keep listing it unless necessary.

Since I am most heavily researching single nodes right now, here are some ordered lists of important components (PSU, Drive Count and Configuration):

Single Node [Item 1 is intended to be the best, and usually most expensive]

Drive Count and Configuration:

1. 4 drives - 2 for PVE on ZFS RAID, 2 for VMs on ZFS RAID
2. 4 drives in ZFS RAID - PVE and VMs
   
3. 3 drives - 1 for PVE, 2 for VMs on ZFS RAID (Is it better to protect PVE or VMs?)
   
4. 3 drives - 2 for PVE on ZFS RAID, 1 for VMs (Is it better to protect PVE or VMs?)
   
5. 2 drives in ZFS RAID + NAS - PVE internal, VMs on NAS
6. 2 drives in ZFS RAID - PVE and VMs
   
7. 2 drives - PVE on #1, VMs on #2 (Allows for experimentation with OS without losing VMs)
8. 1 drive + NAS - PVE internal, VMs on NAS
   
9. 1 drive - PVE and VMs

*No hardware RAID, use ZFS for any RAID necessary when internal.
*No significant benefit to SSD for PVE install.
*NAS for VMs is only used with 2 internal drives or less

Power:

1. Dual power supplies, dual UPSs
2. Dual power supplies, single UPS
3. Single power supply, single UPS

*UPS is considered a requirement

Network w/o NAS:

1. 2 Gbe - LAG to stacked switches
2. 2 Gbe - 1 port with failover to 2nd port on different switch
   
3. 2 Gbe - LAG to single switch
   
4. 2 Gbe - 1 port with failover to 2nd port on same switch
   
5. 2 Gbe - 1 port for all traffic

*Not considering speeds requiring more than 2 LAG ports

NAS:

1. Dual NAS, 4 drives in RAID10 in each, configured for HA
2. Dual NAS, 2 drives in RAID1 in each, configured for HA
3. Single NAS, 4 drives in RAID10
   
4. Single NAS, 2 drives in RAID1

*If NAS is used for VM storage
*RAID required
*Only listing RAID1 and RAID10 for simplicity

Please comment or add suggestions if you have any. I'd like to know if these are conceptually out of order. I continue to be interested in the original questions and feedback here.
​

 

[guletz]

When are you too small for ceph (Assuming ceph is always a good option when possible.)?

At minimum you need 3 nodes in the same location with 10 Gb for ceph.

What are preferred storage methods when too small for ceph?

It is not exist a preferred storage. Like I said, define your goals and then you can find what you cand get using your own budget.

Questions to ask yourself:

- how many VM do you want to have
- what VM are critical (and what is the acceptable downtime for each of them)
- for any VM what operation is acceptable to lose (for VM x1 if I lose 5 min of data, is ok, so if I can restore the vdisk as it was 5 min ago, is ok)
- what is the policy retention backup (try to make bakup and/or async replication, I want to keep last 7 daily backups... and so on)


As you see, is nothing about any concept or system or what ever tool I must / need to use.

So if do not put the wright questions, you will go in wrong directions. After you will define your own goals, then you start to find a technical solutions that is usable for you (storage, servers, nas, and so on)
And finally you will chose from multiple solutions what is ok for your budget(we have money for a external nas or not for example). Sometimes, with a low budget is very hard to solve all objectives (functionality, data safety, performance, service restore capabilites, scalability, and so on). You will make a compromise. .. and must be very creative.

In the end I would say that the most important phase in this process is the design phase. A bad design with any budget is a good path to /dev/null

Good luck !