VM out of Memory despite no memory used

DavHau

New Member
Aug 13, 2019
1
0
1
31
Hello, this is my first post here.

I'm currently trying to move from freenas to proxmox 6.0.
I also want to move from the two mirrored 8TB HDDs freenas was using to a new set of 11x 8TB HDDs running in a z3 raid.
The rest of the hardware will stay the same. Only the hard drives will be exchanged.

My mission is now to copy the data from the old HDDs managed by freenas to the new HDDs managed by proxmox. Therefore I booted freenas as a VM ontop of the new proxmox isntallation and pass the old HDDs directly to the freenas VM. Then i used 'zfs send' to copy the data from the freenas VM to the proxmox host.

So far so good. Everything worked fine, but after 15-20 hours of copying, the network connection breaks and i see some log messages on freenas which say that the interface cannot get an ip address or somehting like this. Also ifconfig shows that the network adapter lost it's addresses.

I noticed that the kvm process on proxmox takes up a lot of memory. Basically all the memory which is assigned to the vm is fully used (14GB). But inside the VM htop is showing low memory usage. Is this normal?
If not, then my first guess would be that there are memory leaks somewhere.

I'm not very experienced with debugging things on proxmox. It would be great if you guys could give some advice.

Honestly i wouldn't care so much about the problem since it always manages to copy 1,5TB before it crashes and i could restart it from time to time. But since i copy from an 'unencrypted' pool to an encrypted dataset, resuming zfs receive is not supported. It has to work in one go without interruptions. If i manages to migrate the data, I have no necessity anymore to keep the freenas VM running. But at the moment it looks like the only viable option to me to do the migration. I also cannot mount the old HDDs in proxmox directly since the HDDs are encrypted using geli and i couldn't find any debian tool to mount these.

Edit: OK, the high memory usage of KVM is probably because of caching inside the freenas VM. Forgot about that. So i have no idea what could be the reason that the adapter loses it's ip address after some time. I will collect some logs when the problem occurs the next time and post them here.
 
Last edited:
Limit ZFS on Linux Memory Usage
echo "options zfs zfs_arc_max=1000000000" >> /etc/modprobe.d/zfs.conf
echo 1000000000 > /sys/module/zfs/parameters/zfs_arc_max
update-initramfs -u
1073741824 is 1 GiB. It is good to use at most 50 percent (which is the default) of the system memory for ZFS ARC to prevent performance shortage of the host.
So in your case you want to use some thing like 6000000000 on the system that runs ZFS.

Also you reap what you saw. If you use crap'ola like Broad'com you might want to do some fine tuning with ethtool on the host.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!