ZFS Ram and VM trade off?

[Chummers]

Hi Folks,

I am very new to proxmox, and finding it a nice experience, but I have appeared to run into something I need help on.

As far as i understand ZFS uses 8Gb for starters + 1Gb per Tb..

My host has 32Gb of RAM (I can not put any more in).

Performed a default install with ZFS in a RZ1 and everything was looked well.

Then I started loading up the host with VM's and got really close to the RAM limit, that's when things started going sideways..

All my VM's started going really slow, ~1 minute to ssh into a VM ect.

I was using 5Tb of my 8Tb of storage.


I have now created a cluster with another node (32Gb also) and moved all my VM's off onto that, but on my original node it still says I am using 56% of the RAM. (is this ZFS?)

So my question is: Does the 8Gb of RAM used for ZFS come out of the RAM used for the hosts displayed in the UI or is the UI displaying the total RAM in the unit. (if you follow me?)

If so, does that mean I only have 24Gb for my VMs? (or 75% of the RAM on the UI?) - I really need it


I have a HW RAID controller I can use, but that will use the only slot on the board (currently there is a Quad-1Gbe NW Card in there), hence using ZFS.

Is there any way I can squeeze more RAM out of this unit? maybe not using ZFS?

I really don't want to go back to VMware, as there's no really good way to back up the VM's apart from third party applications) - and proxmox is far better!


I would appreciate any advice or recommendations

 

[LnxBil]

As far as i understand ZFS uses 8Gb for starters + 1Gb per Tb..

Where did you read this? ZFS uses at most half your RAM on Linux if you have not changed it manually.

I have now created a cluster with another node (32Gb also) and moved all my VM's off onto that, but on my original node it still says I am using 56% of the RAM. (is this ZFS?)

Can be, you have to look with e.g. arcstat.

If so, does that mean I only have 24Gb for my VMs? (or 75% of the RAM on the UI?) - I really need it

As I already pointed out, default is for ZFS to use at max half your RAM, so 16 GB out of your 32 GB, leaving your with - you may have guessed it - also 16 GB for VMs.

Is there any way I can squeeze more RAM out of this unit? maybe not using ZFS?

You could restrict the maximum amount of RAM ZFS uses.

 

[leesteken]

I've set my ZFS ARC limits to between 2% and 10% of my total memory and it worked fine for me when I had only 16GB and now when I have 64GB.
Maybe just try limiting ZFS memory usage and see if your VM responsiveness improves. Try to keep memory usage on average to not much higher than 80% for best performance.
If your disk IO are slow(er) after that, try increasing memory for ZFS based on cache hit-rate and the amount of free memory left on the host.

 

[Chummers]

Thanks for the replies, I ended up putting a RAID card in and reinstalling Proxmox onto that. That way I get to utilise the full amount of RAM on the server, since putting any more RAM in would have ment a new server..

Oh, the 8Gb thing? I got that from the FreeNAS site, I must have read it there and it kinda stuck, went through the Proxmox documentation and it does state half your RAM.

I did not know about setting the hard limit, and that may come in handy one day, so thanks for that too!.

All the best!