[SOLVED] vm running rsync hangs proxmox

[marceloprimon]

Hello, I am new to this forum and I have been using Proxmox for a few months now. I am used to run backups on my vms, mostly debian, using rsync. Since I migrated a few physical servers to Proxmox, all of them freeze during these backups. I searched the forum and found some threads saying that the problem could be on ZFS or that if I use the opiton bwlimit on rsync the problem would be solved, but that didn't happen in my case. Even running rsync on a vm with the option --bwlimit=5m the whole server hangs, and I need to restart it. I changed the hard disk cache on the vm to write through and the problem wasn't solved. Is there anyone experiencing this, or anyone that could help me? Thanks in advance.

I run rsync using these options:
rsync -avzu --delete --bwlimit=5m

I use Proxmox 7.0

 

[marceloprimon]

I still have this problem. I raised the amount of RAM to 8GB, disabled balooning and created another VM with 4 cores and 16GB of swap. Still the problem persists. I ran htop during the copy process and noticed the cache memory of the VM was full when the whole server hanged. Should I still increase the swap partition?

 

[apoc]

I am using rsync as well and never saw these kind of issues. How large are your filesystems and how many files does rsync need to handle?
I only saw issues with memory consumption on fsck running out of memory (1gb memory per 1TB disk was a good value for me) but never on rsync.

Since you are running zfs on the host: have you limited the zfs arc/cache?
By default thus can eat up to 50% of your memory. Maybe your issue lies there?

 

[marceloprimon]

Thank you, apoc. I made a mistake. Please consider that when I said swap I meant cache memory. I usually copy no more than 20 tar.gz files from remote servers, and some are big, like 10GB or more. I also copy around 2000 files from local servers. The problem occurs on internal and USB disks. I used to run rsync on bare metal with the same variables and everything was fine. I tried disabling the --delete option and using the --no-inc-recursive option, but still the same. I reduced the amount of memory on zfs to 8GB (this particular server has 48GB of RAM), and the host still hanged. As you can see in the attachments, the VM cache memory is full when the host crashes. Oh, and you can see that I increased the --bwlimit to 25m. I noticed it makes no difference, because even reducing to 5m the host crashes. I also have another server without zfs disks, and it crashes the same. Maybe, if you could post the options you use on rsync, I could try them. Thank you again.

 

[marceloprimon]

I upgraded Proxmox to version 7.1-10 and the problem didn't show up anymore. I even ran 2 instances of rsync and the backup jobs completed successfully. Thank you, apoc, for your time.

 

[apoc]

Make sure that your ARC-limit is honored. I have had some issues with this in the past. My system has 32 GB arc limit and it should look like this then:
sudo arcstat time read miss miss% dmis dm% pmis pm% mmis mm% size c avail 13:57:02 0 0 0 0 0 0 0 0 0 [B]32G 32G [/B] 51G

if your host is under memory pressure all sort of things can happen.
I am not sure if I understand you correctly. According to the second screenshot your memory is full (likely through caching) but the SWAP is completely empty. That would indicate that there is something going on with memory pressure.

my rsync options are:
/usr/bin/rsync /home/user/ backup:/datastore/backup/2022-03-15_17-00 -v --exclude-from=/home/user/exclude.list --rsh="ssh -l rsync-user -p 22" --links --copy-dirlinks --times --delete --delete-excluded --delete-after --ignore-errors --hard-links --ipv4 --link-dest=../2022-03-15_16-00 --recursive --progress --exclude 'backupIsReferringTo' --exclude 'backupFailure' --exclude 'backupSuccess'

HTH

 

[marceloprimon]

Thanks again, apoc. My arcstat shows this:
sudo arcstat time read miss miss% dmis dm% pmis pm% mmis mm% size c avail 13:50:19 0 0 0 0 0 0 0 0 0 2.0G 2.9G 77G

Even with all vms online the memory consumption is only 19%. I upgraded another server with the same issue to version 7.1-10 and the problem was also solved there.

 

[apoc]

Keep an eye on the cache size / usage. If I am not mistaken these 2.9 GB should be the upper limit and this indicates to me that your 8GB limit is not honored. I have not booted my rig for a while and could be mistaken but I think if there is a limit specified, this should be the upper boundary.
This is how my config looks like:

#Limit ZFS Memory and RAM usage
sudo vi /etc/modprobe.d/zfs.conf

# This is for 48 GB / 49152 MB
#options zfs zfs_arc_min=51539607551
#options zfs zfs_arc_max=51539607552
# This is for 32 GB / 32768 MB
options zfs zfs_arc_min=34359738367
options zfs zfs_arc_max=34359738368
# This is for 16 GB / 16384 MB
#options zfs zfs_arc_min=17179869183
#options zfs zfs_arc_max=17179869184
# This is for 8 GB / 8192 MB
#options zfs zfs_arc_min=8589934591
#options zfs zfs_arc_max=8589934592
# This is for 4 GB / 4096 MB
#options zfs zfs_arc_min=4294967295
#options zfs zfs_arc_max=4294967296
# This is for 2 GB / 2048 MB
#options zfs zfs_arc_min=2147483647
#options zfs zfs_arc_max=2147483648
# This is for 1 GB / 1024 MB
#options zfs zfs_arc_min=1073741823
#options zfs zfs_arc_max=1073741824
# This is for 512 MB
#options zfs zfs_arc_min=536870911
#options zfs zfs_arc_max=536870912
# This is for 256 MB
#options zfs zfs_arc_min=268435455
#options zfs zfs_arc_max=268435456