Setup Guidance...

[musasabi]

I've got a "brand new" server I'm going to be setting up as a home NAS/media/GitLab/etc server this week. The hardware is the following:

• Dell R510 (8-bay chassis)
  • 2x E5640 2.66GHz quad-core hyperthreaded
  • 32GB RAM
  • PERC H700 RAID controller
• Sun F40 400GB flash accelerator
• 6x WD Red 3TB drives

My objective is to have my server suspend during inactivity (say, 30 minutes) and resume via WOL. My instinct for that is to have the H700 present a RAID5 (or 6) to Proxmox installed on the F40. I'll store the VM's on the RAID volume.

Other than that... how should I set this up? It seems ridiculous to me to store Proxmox on a 400GB volume, but that's what I've got. Also, is this a recipe for 'speed' with regards to resuming from a suspended server? If not, what would be better?

Thanks in advance, all. =)

 

[udo]

Hi,
don't know how good supend/wakeup work with dell-server. preboot is not very fast with this servers...

And Perc + F40 don't fit for me. If i see the F40 right, it's an HBA with 400 GB ssd-cache? I assume for zfs?
Remark - found another picture without SAS-connectors - it's an "much better" SSD.


Or simply create two raid-volumes on your hdds (perc) - one smaller one for OS/templates/iso and an bigger one for VM-storage.

Udo

PS: It's intresting how you like to measure, that the system is not in use to go in sleep-modus?!

 

[guletz]

But for zfs is 32GB not realy much...

it is only a pure speculations, if you do not know the load of this server. Why 32GB is not relly much? But 32 + 1 it will be OK?maybe\the owner it will use this server once in a week to copy one file. In this case 32 GB will be ""not really much"? Or let say he will have 64 Gb. It will be OK? Why?

ZFS memory consumtion depends on your load. You can use zfs with only 1GB of RAM, with good results if you make the proper zfs settings and you have a light load. Also it could be bad, with the same load but with the bad settings.


I run zfs/proxmox with a lot of services, with only 16 GB RAM. ANd I have 1-4 GB RAM free ..... samba4-AD/webmail/mysql/rsync/dns/file-server/nfs-server/glusterfs(4 lxc guests) and others. ..... but if "32 GB is not realy much", how is 16 GB? Or I am a luky guy?

So, Udo please do not give false directions, without any strong reasons. Why a lots of guys have a fixed ideea about zfs= at least 32-64 GB RAM, despite whatever load it is?

 

[udo]

it is only a pure speculations, if you do not know the load of this server. Why 32GB is not relly much? But 32 + 1 it will be OK?maybe\the owner it will use this server once in a week to copy one file. In this case 32 GB will be ""not really much"? Or let say he will have 64 Gb. It will be OK? Why?

ZFS memory consumtion depends on your load. You can use zfs with only 1GB of RAM, with good results if you make the proper zfs settings and you have a light load. Also it could be bad, with the same load but with the bad settings.


I run zfs/proxmox with a lot of services, with only 16 GB RAM. ANd I have 1-4 GB RAM free ..... samba4-AD/webmail/mysql/rsync/dns/file-server/nfs-server/glusterfs(4 lxc guests) and others. ..... but if "32 GB is not realy much", how is 16 GB? Or I am a luky guy?

So, Udo please do not give false directions, without any strong reasons. Why a lots of guys have a fixed ideea about zfs= at least 32-64 GB RAM, despite whatever load it is?

Hi,
yes it's speculation, but the original poster wrote about six 3TB-disk with raid6 (or raid5). E.g. zfs-raidz2.

And even with such an raid I have made test and the performance are not realy high (the server load indeed yes) - and this with only one VM, where an fileserver was filled with rsync. Perhaps there was tuning posibilities, but people, which are more familiar with zfs mean I need more Ram (and SSD-cache)...

Of course, I don't know what musasabi run on the server, but we speak about virtualisation and app. 10TB storage - so I asume he don't copy single files...

But I like to learn - what are your proper zfs settings to run zfs (raidz2? how many disks?) with 16GB and get light load (and good performance?).

Udo

 

[guletz]

Hi Udo, and my respect for your good sense.

Now at the subject I tell only the most usefull settings (I guess will be ok for most of the cases, but not for all):
- I use a proper volblocksize(16/32 k)
- primarycache=metadata

With primarycache=all you will do a doubles memory usage for KVM. Anything is caching in VM memory is also again caching by zfs.

If you ask me, I think that by default Proxmox need to use this, if you want to avoid double caching (at VM level and at zfs level).


As a general speaking, each server have it its own load/specifics. So each admin need to know how it is happening on this server. It is not so important what quantity of TB do you have. Spend time and see what is happening on your server. Watch some tools like iotop/atop/zfs statistics.

Have a nice day.

 

[Pablo Alcaraz]

I am running a PVE 5.0 instance in a Dell T110II with 32 Gb of RAM. It has 1 container with postgresql (150Gb of databases heavily used) on SDD and one 1Gb Virtual machine with almost not use on HDD and it is fast.

BTRFS use VFS caching provided by kernel and it is fast. At the same time it is not so reliable as ZFS with 'exotic' configurations. It is unsafe with RAID 5 and 6. Safe with DUP, RAID 0 and 1 and 10. Booting from broken RAID 1 could give you problems because the scripts in your linux distribution must execute a command before mount the volume.

But it is fast and reliable if you use it under described scenarios. If you memory restrictions, perhaps you want to consider it.