Single server storage recommendations

kresten

New Member
Sep 14, 2014
7
0
1
Hi

I'm going to set up a single machine used for VM's soon. I think i have all the hardware figured out except for the hard-drives which i would like some recommendations for.

As this is going to be a quite simple setup i would prefer local storage - easier and cheaper. My initial plan was just to buy 2*X TB disks and mirror them somehow, but the following questions arised.

1) Is it recommended to use separate discs for proxmox itself and the VM's ?
2) Are there any real performance gains of placing proxmox on SSD's ?
3) Should i use ZFS to mirror the discs or buy some cheap LSI/adaptec/something controller ?

Any advise is much appreciated.
Br
 
1) Is it recommended to use separate discs for proxmox itself and the VM's ?

Depends on how firm you are if problems arise and how much raw IOPS power you need. Split if you want to reinstall Proxmox VE without loosing the VM-disk data (not the configuration) which is stored on the other disk. Four Drives yield more performance than two if they're combined in a RAID, so separation is strictly limited to available Ports and physical space in the server to mount drives.

2) Are there any real performance gains of placing proxmox on SSD's ?

Besides beeing faster, no. If the system is running, there is very little IO. Most IO is (also dependend on the used VMs) on the disk where your data is stored.

3) Should i use ZFS to mirror the discs or buy some cheap LSI/adaptec/something controller ?

Depends on the rest of the hardware. ZFS is a very good filesystem including compression, snapshots and incremental, asynchronous replication or duplication features, yet it needs RAM to be performant. I'd always choose ZFS if you also want to use the disk for VMs.

Depending on your (yet unknown) workload, a ZFS system with e.g. 4x SATA disks is a good way to start. It is cheap and works out of the box (Proxmox VE installer) and you should use raidz (similar to RAID-5). Depending on your budget and space needs, you can also go with 4x SSD.

Could you please elaborate on the rest of your hardware?
 
Thank you for your reply LnxBil

My plan is to make a cheap box to host different game servers. Content is not really important so if everything blows it doesn't matter too much. It will only be a few CSGO servers, apache, ebot, trackmania - all simple things.
I'm planning to use something like i7-7700 with 32gb of DDR4 memory. Based on what you said i think i will stick with 3 x 1tb drives in raidz (raid-5)

Are the 32gb of memory enough to do zfs - and do i need to make any speciel settings based on my amount of memory ?

Br
 
Yes, 32 GB in total is fine for that workload and disk size. Per default, ZFS will use at most 16 GB of RAM. I'd change only if needed. Run your server and if everything is running smoothly, don't change anything.
 
By now I would recommend Intel .. Ryzen is still pretty fresh and well supported only on very recent kernel releases such as 4.12 but ProxMox is based on 4.4 which is a bit old for Ryzen.
That might change in a near future but right now, I would stick with Intel.
 
intel or ryzen
I don't think ryzen is supported on proxmox 4.x and for 5.x you should wait for the next kernel release since the current 5.x installer cannot boot on ryzen. The bug is known and a fix is ready and have been applied to the next kernel for 5.x.
 
Are the 32gb of memory enough to do zfs - and do i need to make any speciel settings based on my amount of memory ?

Yes, 32 GB in total is fine for that workload and disk size. Per default, ZFS will use at most 16 GB of RAM. I'd change only if needed. Run your server and if everything is running smoothly, don't change anything.

You will in fact need to change a few things, because the default Proxmox ZFS setup is far from being optimized, and can lead to unstable servers (during high memory pressure and/or when vzdump backups are running).

1. Limit the ZFS ARC size
We aggressively limit the ZFS ARC size, as it has led to several spontaneous reboots in the past when left unlimited. Basically, we add up all the memory the system uses without caches and buffers (like all the KVM guest's maximum RAM combined), subtract that from total host RAM, and set the ARC to something a bit less than that, so it has to compete with system cache only. For example: if on a 32GB server the maximum RAM allocation of KVM guests is 25 GB, we set the ARC upper limit to 5GB (leaving 2GB for anything else). We also set a lower limit of 1GB to the ARC, as it has been reported that it helps performance for some reason. Note that the values are in bytes, so 5GB = 5x1024x1024x1024

To do that, you have add the following lines to /etc/modprobe.d/zfs.conf
Code:
options zfs zfs_arc_max=5368709120
options zfs zfs_arc_min=1073741824
and after that run the following command and reboot:
Code:
update-initramfs -u

Looking at the ARC of this very server with arc_summary.py you can see it stays between the limits:
Code:
ARC Size:                               30.72%  1.54    GiB
       Target Size: (Adaptive)         30.72%  1.54    GiB
       Min Size (Hard Limit):          20.00%  1.00    GiB
       Max Size (High Water):          5:1     5.00    GiB

ARC Size Breakdown:
       Recently Used Cache Size:       35.27%  554.85  MiB
       Frequently Used Cache Size:     64.73%  1018.10 MiB

2. SWAP on ZFS zvol

You also have to make sure that swap behaves well if it resides on a ZFS zvol (default installation places it there). Most important is disabling ARC caching back the swap volume, but the other tweaks are important as well (and endorsed by the ZFS on Linux community):
https://github.com/zfsonlinux/zfs/wiki/FAQ

Execute these commands in your shell:
Code:
zfs set primarycache=metadata rpool/swap
zfs set secondarycache=metadata rpool/swap
zfs set compression=zle rpool/swap
zfs set checksum=off rpool/swap
zfs set sync=always rpool/swap
zfs set logbias=throughput rpool/swap

You can verify these settings by running:
Code:
zfs get all rpool/swap


3. Virtual memory settings

Set these on the Proxmox host to increase stability and performance of your VMs during vzdump backups, by adding these lines to the end of /etc/sysctl.conf and reboot:
Code:
vm.min_free_kbytes = 262144
vm.dirty_ratio = 5
vm.dirty_background_ratio = 1
vm.swappiness = 1

You should use these values in your KVM Linux guests as well, although for min_free_kbytes the approximate rule is to use 65536 kbytes for every 4GB of system memory. (So for a 1-4GB RAM VM use 65536, 4-8 GB use 131072, more than that 262144. You don't need to use higher values.)
 
Last edited:
gkovacs - thank you for your feedback regarding ZFS.
Sounds like i should take a bit care about the settings after installation.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!