Memory usage more than summary of VMs

M

Manny Vazquez

Well-Known Member
Jul 12, 2017
107
2
58
Miami, FL USA
Hi,


again, rookie here so maybe stupid question.

I have a 3 node cluster, each 128 gb ram and 3 tb hard drvie with 16 cores each.

So far I have only started a few VMs, all windows 2008 standard (just want to create a RDP farm) each one of them with 8gb ram and 4 cpus.

I I check each machine independently the report the right memory used (Matches the reported inside windows), but if I look a the "summary' in Proxmox, it is totally wrong, reporting way more memory used than that allocated to the VMs.

As you can see in these images, the reporting in the VM web page is not consistent with the usage required.
I am afraid of launching the rest of the VMs I need since I may run out of resources,

What did I do wrong?

vm report.png report vm.png summary.png
 
What kind of storage do you have? You said "each has 3TB drive", so is it local on every node? Do not forget some memory is used for cache (be it "traditional" or ARC in case of ZFS)...
 
1. Then remember, ZFS can (per default) use up to half RAM you have...
2. It is VERY BAD IDEA to put ZFS on top of HW-raid...
 
Filesystem is cached, of course! Without caching you'd get terrible performance. And concerning HW-raid and ZFS, I'd recommend to read wiki:

Do not use ZFS on top of hardware controller which has its own cache management. ZFS needs to directly communicate with disks.
 
Thanks,
At this point, I guess my only option is to try to limit the memory usage by ZFS.

I have tried to do what the article recommends
"
Limit ZFS Memory Usage
It is good to use at most 50 percent (which is the default) of the system memory for ZFS ARC to prevent performance shortage of the host. Use your preferred editor to change the configuration in /etc/modprobe.d/zfs.conf and insert:

options zfs zfs_arc_max=8589934592
"

But that file does not exist in my system. So I searched for it and found it at /lib/modules-load/zfs.conf , I edited the file and added the recommended line, did not seem to do anything. (I did the update-initramfs -u as recommended)

I will reformat each one of the nodes, and remove the hw raid, will recreate with ZFS letting it manage the drives directly. But, that will not fix my memory problem. Any suggestion?
 
You have to create that file in given location (/etc/modprobe.d/zfs.conf), update initram, and reboot (not sure if this is necessary, but I think so).

You can monitor ARC-size in /proc/spl/kstat/zfs/arcstats: Look for "size" (current ARC size in bytes) and "c_max" (maximum ARC-size).

And you can use ZFS to create raid5 (called raidz1 in zfs)...
 
I did that, that's how I started, but changed to zfs in order to do live migration.

The raid was default set up, I'll have to redo the cluster not using the hw raid.
 
dcsapak said:
you do not need zfs for live migration, or do you mean replication?
Click to expand...

And of course, you are right, Did I mention this is all new to me? 3 weeks into this test :)

I love it, I come from large enterprise (300k plus employees) with vmware contract (actual vmware engineer on site) and there are many things that I just did not have to know, manage the servers, allocations, migrations, etc is very clear, but the actual configuration of the esx environment is a mystery to us in PMO.

I have done the config on 2 of the servers and it seems to be holding now. the Third one I rebooted and also got the sage down.

memory.png


The wiki says ZFS will use up to 50% of the ram available, which is ok, you want ram to be used, my concern is will it free up ram for the VMs or will it starve the VMs? in other words, if I do not actively take care of the memory and just let it go, when I launch a new vm, will it adjust to the new 50% 'free' or to the 50% of the total of the server?
 
Using memory for ZFS-cache has low priority. So if some app (or VM) needs more memory then there is currently available, ZFS should reduce ARC-size automatically down to minimum value (can be configured too) and release it. This is similar to standard linux disk-cache.

So if you have plenty of memory, do not worry about it. Leave the default config and remember: any meaningful using or memory is better than not using it at all. And when we are at it: maybe memory diagram in pve web-interface could graphically split used memory to that used by pve/apps/vm and cache/arc...
 
  • Like
Reactions: GadgetPig
Rhinox said:
Using memory for ZFS-cache has low priority. So if some app (or VM) needs more memory then there is currently available, ZFS should reduce ARC-size automatically down to minimum value (can be configured too) and release it. This is similar to standard linux disk-cache.

So if you have plenty of memory, do not worry about it. Leave the default config and remember: any meaningful using or memory is better than not using it at all. And when we are at it: maybe memory diagram in pve web-interface could graphically split used memory to that used by pve/apps/vm and cache/arc...
Click to expand...

I Have 128 gb on each of the 3 servers and was planning on using about 84 (12gb *7 vm) to 108 (12 gb *9 vm) on each one, leaving plenty for management and in case one of the servers goes down, be able to split the load to the others.

Rhinox, and the others, I really appreciate all the help :)
 
You do not need free RAM, because free RAM is the worst situation you can have. In any server (that has storage >> memory), you system is only fast if all RAM is used (and not overcommited by VMs), because it is used as cache.

Please familiarize yourself with linux memory management. You're not the first one to ask about this and most probably not the millionth even more people ask this all the time. Therefore this nice website exists:

http://www.linuxatemyram.com/
 
  • Like
Reactions: GadgetPig
LnxBil said:
....
Please familiarize yourself with linux memory management. You're not the first one to ask about this and most probably not the millionth even more people ask this all the time. Therefore this nice website exists:

http://www.linuxatemyram.com/
Click to expand...

I do appreciate the answer, not the condescending tone.
If you do not feel like 'helping' please just don't.
Thanks for the answer, please work on your delivery.
 
Sorry if it was too direct. I answered this hundreds of times and I did not intend to sound condescending. I just wanted to say that the question was literally asked millions of times and therefore the aforementioned website exists since 2009.
 
  • Like
Reactions: Manny Vazquez
Rhinox said:
That web-site should be updated to take ZFS into account. Standard linux tools (free, top, etc) are not sorting ARC properly, they do not put it into "buffers/cache" value. So user really can see "available" value close to zero...
Click to expand...

Unfortunately yes, but ZFS is not officially in the Linux kernel (strictly speaking) - but the website lacks also BTRFS which is officially in the Linux kernel.

ZoL is also not a good implementation memory wise due to the solaris portability layer (SPL) in between which does things really different from "normal" filesystems and memory allocation. Therefore it does not show up as cache.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back