[SOLVED] ZFS memory usage in NFS share

[Alanl96]

Hi everyone, I am new to Proxmox and I have been searching around on how to solve this issue. I have learned that ZFS requires ECC memory, I have been using ZFS on Ubuntu server for a year now with standard DDR4. I ordered an Epyc 7402 and Supermicro H12SSL-i, I will buy ECC memory now that I know I am doing this wrong.

I set up Proxmox and made an Ubuntu server VM. I wanted the VM to be small because I can have small backups. The movies will be stored on the host machine and shared with the guest using NFS. I also want other VMs to use my ZFS volume. I created a ZFS RAID10 volume with eight 8TB HDDs, and copied my movies over to it (14.01TB used with compression enabled). My old setup used 100GB of memory, the Proxmox host uses 65GB of memory and my Plex VM wants more than 50GB, but I can't physically give it more memory.

Does NFS cause the guest VM to consume the same amount of memory as the host? I would like to decrease memory usage, but if that isn't reasonable I can purchase higher capacity DIMMs since I will have 8 DIMM slots on my motherboard. I just want to know if I need to change settings in the host or guest to decrease ZFS memory consumption, or if both devices actually need the memory allocated. If they do need the memory, then I can buy more. Is there a better way to handle my storage between guest VMs and the host machine? Thanks in advance!

 

[leesteken]

I have learned that ZFS requires ECC memory. ... I will buy ECC memory now that I know I am doing this wrong.

ZFS does not require ECC memory. However, if you go to the trouble to use a filesystem that checksums the data to detect bit rot (and you better have another copy of the data like mirror or raidz, otherwise it cannot be fixed), it would be a shame if your data got corrupted in memory. Still, it remains a (expensive) choice and not a requirement.
Using ECC when you have a large amount of memory is always a good idea (also with other filesystems like ext4) because the likelihood of a bit corrupting increases with the size (and uptime). But it is never mandatory.

 

[Alanl96]

ZFS does not require ECC memory. However, if you go to the trouble to use a filesystem that checksums the data to detect bit rot (and you better have another copy of the data like mirror or raidz, otherwise it cannot be fixed), it would be a shame if your data got corrupted in memory. Still, it remains a (expensive) choice and not a requirement.
Using ECC when you have a large amount of memory is always a good idea (also with other filesystems like ext4) because the likelihood of a bit corrupting increases with the size (and uptime). But it is never mandatory.

Good to know. I was looking at other ZFS related posts on this forum and a lot of comments said ECC is a requirement. I'm not sure why my memory usage is so high. Perhaps I have outgrown my current setup and I need a server.

 

[leesteken]

Good to know. I was looking at other ZFS related posts on this forum and a lot of comments said ECC is a requirement. I'm not sure why my memory usage is so high. Perhaps I have outgrown my current setup and I need a server.

ZFS will take 50% of your host memory, unless you limit it. Every VM will also use as much memory for filesystem cache as there is available (but it can also be released when something else needs it). How much memory you really need for ZFS or as cache in VMs depends on the I/O work-load of your VMs and how fast you want it to do that work. Just reduce it until it becomes too slow or you feel that memory is being wasted by going unused.

 

[Alanl96]

ZFS will take 50% of your host memory, unless you limit it. Every VM will also use as much memory for filesystem cache as there is available (but it can also be released when something else needs it). How much memory you really need for ZFS or as cache in VMs depends on the I/O work-load of your VMs and how fast you want it to do that work. Just reduce it until it becomes too slow or you feel that memory is being wasted by going unused.

Do I limit for the guest or the host?

 

[leesteken]

Do I limit for the guest or the host?

The ZFS pools and drivers are on the Proxmox host (and not on the guests), right? So you limit it on the host.

 

[Alanl96]

The ZFS pools and drivers are on the Proxmox host (and not on the guests), right? So you limit it on the host.

lol, good point. I just noticed the guest spikes in memory when I share a ZFS volume, so I thought there might be a setting for that. It just copies the host. Thanks for your help! I will apply these changes and I that should get my Plex VM running without crashing. I need this to last until December 5th. Supposedly that's when my mobo and CPU arrive.

 

[Dunuin]

when I share a ZFS volume

I hope you don't share a zvol (ZFS Volume) between guests. That will corrupt data. Only sharing a filesystem on it using SMB/NFS would be ok.

 

[Alanl96]

I hope you don't share a zvol (ZFS Volume) between guests. That will corrupt data. Only sharing a filesystem on it using SMB/NFS would be ok.

Good to know, I made folders within the ZFS volume and shared it from the Proxmox host to the individual VMs. In my case I would be using my Proxmox host as the NFS server and my VMs are the clients.