Rsync inside VM crashing - zfs

[Saahib]

Hi,
I have ZFS based system, there is a ubuntu based VM and I am trying to rsync remote data to this VM.
After a while the VM starts eating all RAM, no matter if you have 8GB RAM VM or 96 GB, it eats up all and things start crawling and eventually Rsync crashes.
Disk IO wait get very high, I have seen other threads with similar problem but no solution so far.
If you look at RAM usage, it appears as if whatever is coming in through Rsync, is getting stored in RAM and after a while fills it up and system starts to struggle, Due to low RAM, on host KSMD kicks in and I think it further slows down. I don't remember having such problem non-zfs proxmox nodes.
Have already reduced arc size to 25% of system memory, still no gain.
For rsync, I also added --no-inc-recursive argument as it is supposed save rsync memory usage.

# pveversion
pve-manager/7.3-3/c3928077 (running kernel: 5.15.74-1-pve)
root@ugi-nl-ds-hc:~# pveversion -v
proxmox-ve: 7.3-1 (running kernel: 5.15.74-1-pve)
pve-manager: 7.3-3 (running version: 7.3-3/c3928077)
pve-kernel-5.15: 7.2-14
pve-kernel-helper: 7.2-14
pve-kernel-5.15.74-1-pve: 5.15.74-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-8
libpve-guest-common-perl: 4.2-3
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.2-12
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.7-1
proxmox-backup-file-restore: 2.2.7-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.3
pve-cluster: 7.3-1
pve-container: 4.4-2
pve-docs: 7.3-1
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-7
pve-firmware: 3.5-6
pve-ha-manager: 3.5.1
pve-i18n: 2.8-1
pve-qemu-kvm: 7.1.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.3-1
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1

Not sure where to start

 

[Saahib]

Its now very troublesome, I can't use the machine, whenever there is large files or copy, the cached RAM fills up everything and disturbs and crashes other services on VM rendering it useless. I have tested disks, they appears absolutely fine . Its Raid 1 zfs mirror.



Does this mean if any VM users does any rsync related task, it would bring down whole node ? Should i ditch ZFS or what I am missing here ?

 

[Saahib]

Hoping someone can help, further its indeed problem of filesystem which is zfs. The VM is simple cPanel based server, what I am trying to do is rsync cpanel accounts from remote server to this server and then restore them.
1. During rsync it starts to crawl and disk io goes very high as already mentioned
2. Even with restoration, it starts to crawl again with high disk io, eg. there is account with just 10GB data, if I restore, cpanel will first extract files and then restore it as well restore mysql, but by the time we go to mysql restoration, due to very high resource usage (read high RAM usage), mysql get crashed and then restoration fails.

Now when I tried same with remote Filesystem, there is no issue because data is being read from remote share and extracted data is also on remote share, once extraction is done, then cpanel will start copying it to local filesystem.

Can any one help me what I am doing wrong ?

 

[Saahib]

Is this something only happening with me ?

 

[Neobin]

What is(/are) the exact model number(s) of the disks used?

 

[Max Carrara]

Does this mean if any VM users does any rsync related task, it would bring down whole node ? Should i ditch ZFS or what I am missing here ?

Could you go into your host's shell and post the output of lsblk -o +MODEL please?

 

[Saahib]

Here, is the output as asked

sda        8:0    0 931.5G  0 disk            Samsung_SSD_870_QVO_1TB
├─sda1     8:1    0  1007K  0 part
├─sda2     8:2    0   512M  0 part
└─sda3     8:3    0   931G  0 part
sdb        8:16   0 931.5G  0 disk            Samsung_SSD_870_QVO_1TB
├─sdb1     8:17   0  1007K  0 part
├─sdb2     8:18   0   512M  0 part
└─sdb3     8:19   0   931G  0 part
zd0      230:0    0     8G  0 disk [SWAP]
zd16     230:16   0 649.8G  0 disk
├─zd16p1 230:17   0     1M  0 part

 

[Neobin]

Samsung_SSD_870_QVO_1TB

Those are your problem. This topic comes up over and over again...
Please read about the (performance-)behavior of QLC-nand-SSDs in general, as well as about ZFS on consumer SSDs without PLP, even more on those QLC-nand-SSDs and have a look at: [1], especially the comparison table on page 2.
Also search the forum for e.g.: "QVO": [2] and "QLC": [3], as well as "ZFS consumer SSD": [4].

[1] https://forum.proxmox.com/threads/proxmox-ve-zfs-benchmark-with-nvme.80744
[2] https://forum.proxmox.com/search/5733251/?q=QVO&t=post&o=date
[3] https://forum.proxmox.com/search/5733274/?q=QLC&t=post&o=date
[4] https://forum.proxmox.com/search/5733277/?q=ZFS+consumer+SSD&t=post&o=date

 

[Saahib]

I do understand that they are consumer grade SSD but they are faster than HDDs, aren't they ?
I tried to went through all those threads, I I understand that should not use consumer grade SSD, but why is not explained anywhere, is this just speed of consumer ssd vs enteprise ssd ?

If its about thorough output, why it gradually starts getting slow and then very slow, shouldn't be like if its slow speed, then should slow right from start. What I observed that there is some kind of race condition happening and then multiplying the effect.
I can redo with LVM but wan't ZFS designed for much slower HDDs, then why its creating trouble with comparatively faster SSDs (even if they are consumer grade) ?

 

[Dunuin]

I do understand that they are consumer grade SSD but they are faster than HDDs, aren't they ?

Depends on the workload. For sequential writes a CMR HDD for example is faster than a QLC SSD.

I understand that should not use consumer grade SSD, but why is not explained anywhere, is this just speed of consumer ssd vs enteprise ssd ?

Both speed and durability. ZFS is a copy-on-write filesystem with focus on data integrity. Therefore it got massive overhead and is doing a lot of sync writes. Consumer SSDs are missing the power-loss protection, so they are horrible doing sync writes, as they can't cache them. And because of the overhead the SSD will wear multiple times faster than with normal consumer usage, resuling in potentially very short life spans (ZFS can kill conusmer SSDs in months doing the right workloads). Have a look at the TBW or DWPD of decent Enterprise SSDs. They usually can handle a multiple or even magnitues more writes.

If its about thorough output, why it gradually starts getting slow and then very slow, shouldn't be like if its slow speed, then should slow right from start.

No, thats normal. First you are writing to the cache in your hosts RAM. This is very fast. As soon as your hosts RAM is full performance will drop.
Then you are writing to the DRAM cache of your SSD. First fast again and then slow as your SSDs DRAM cache gets full.
Then it will write to the SLC cache of your SSD. As soon as this SLC cache also gets full it will drop to the real terrible performance of the QLC NAND.
Those consumer SSDs are just designed for short bursts of writes. They can't handle 24/7 usage with continuous writes. Enterprise SSDs are designed to provide moderate performance all the time.

 

[Saahib]

Thanks Dunuin for the explanation, so it means one should get HDDs if they are using HDDs or else go for ultra fast NVMes and that too enterprise grade ?
I thought ZFS will improve things with in-RAM cache. So there is no way I can make things work using ZFS on these QLC SSDs ? Would not mind little compromise but doesn't want system to crawl on basic file transfer or sync.

 

[Max Carrara]

In my opinion its best if you just got CMR HDDs. There are a lot of vendors offering them. Just really make sure they're CMR drives, not SMR drives (look up the difference if you don't know it already). Usually CMR HDDs will give you more storage for the money you spend anyway.

What you could do in your case is get two CMR HDDs - same size or bigger than those QVOs you got there - and then replace them with your SSDs one by one. I assume you have a mirror zpool, so you can take out one SSD, replace it with a new HDD, resilver, and then take out the other and resilver again. Good luck!

 

[Max Carrara]

So there is no way I can make things work using ZFS on these QLC SSDs ?

Also, to answer your question: No, unfortunately not. Not for your use case at least.

I'm sure there's some kind of workload / setup combination where it's possible to use them effectively, but it's best if you just don't bother. Good old spinning rust never let anyone down.

 

[Saahib]

Ok, here is alternate offered by provider SAMSUNG MZQLB960HAJR-00007, ira TLC, any experience with those drives and ZFS ?
Since this is not a colo, I don't have much choices here.
On the other hand, I am thinking to have two more HDDs and use them for mirror ZFS with above SSDs in soft raid 1 as L2ARC cache and SLOG, will that make any difference if I use ZFS on HDDs without SSD cache ? I have on read about, not really practically used this though. Is it helpful in real use case ?
My use case is simple, there would be multiple webserers running on multiple VMs, mostly cpanel based. There will not be any high traffic sites but regular backup etc. will be there. Where I think my current setup will cause issue.

 

[Dunuin]

SLOG will only help with sync writes, nothing else. Unless you got a lot of sync writes in your workload, like when primarily running DBs, using a SLOG won'T really help.

L2ARC is only useful in specific situations. It can even make your server slower. Keep in mind that you are sacrificing fast RAM, that could be used as a fast read cache, to get some more but slower read cache on a SSD. Rule of thumb usually is: If you need more read cache, buy more RAM. If you maxed out the supported RAM of your platform and you still need more read cache, get an L2ARC.

A special device SSD mirror might be a better alround choice, as it will boost all reads and writes a bit, as the HDDs aren't hit by all the metadata IO anymore.

But if you want SSD performance, you need to replace your HDDs with SSDs. No way to get near to SSD performance by adding SSDs to a HDD pool.

 

[RolandK]

>I do understand that they are consumer grade SSD but they are faster than HDDs, aren't they ?

no.
i have seen consumer grade ssd go down <1MB/s with zfs. i have never seen such with consumer grade hdd.

 

[Saahib]

SAMSUNG MZQLB960HAJR-00007, can we consider this SSD for zfs ?

 

[RolandK]

if this is Samsung PM983 i.e. datacenter/enterprise disk (looks like) , then yes