Brutally slow local storage volume move

[HrvojeS]

I have a SuperMicro 5028D-TN4T with Xeon D-1541 and 128GB ram. Running Proxmox 6.4-13.

Basic storage config:

NAME                                          STATE     READ WRITE CKSUM
    hdd-raid                                      ONLINE       0     0     0
      raidz1-0                                    ONLINE       0     0     0
        ata-WDC_WD40EFAX-68JH4N0_WD-WX4444444444  ONLINE       0     0     0
        ata-WDC_WD40EFAX-68JH4N0_WD-WX62D106T1P3  ONLINE       0     0     0
        ata-WDC_WD40EFAX-68JH4N0_WD-WX12D20N1CA8  ONLINE       0     0     0
        ata-WDC_WD40EFAX-68JH4N0_WD-WX12D10JSDFX  ONLINE       0     0     0
    logs 
      nvme0n1p5                                   ONLINE       0     0     0

NAME           STATE     READ WRITE CKSUM
    ssd-mirror     ONLINE       0     0     0
      mirror-0     ONLINE       0     0     0
        sdf4       ONLINE       0     0     0
        nvme0n1p4  ONLINE       0     0     0

My SSD's above are Samsung 850 (sata) and Intel 600p (nvme). I do have ZFS mirrors on SSDs for boot partitions and rpool. Leftover mirror SSD is above for container/VM storage.

My WD raid is primarily used for a container running Samba, and occasional container needing more storage. One of my nginx containers on WD raid (based on ubuntu 20.04 template) had a very poor response time. I know I don't have the read cache and working on redoing hdd-raid. For time-being I've decided to move the container storage to SSD mirror, but 1.7GB of data has been transferring over 11hrs now and I'm puzzled.

Job output:
Number of files: 33,052 (reg: 26,421, dir: 3,
441, link: 3,159, special: 31)
Number of created files: 33,051 (reg: 26,421, dir: 3,440, link: 3,159, special: 31)
Number of deleted files: 0
Number of regular files transferred: 26,401
Total file size: 1,783,468,051 bytes
Total transferred file size: 1,480,027,438 bytes
Literal data: 1,480,027,438 bytes
Matched data: 0 bytes
File list size: 1,114,015
File list generation time: 0.004 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 1,482,398,291
Total bytes received: 531,055

sent 1,482,398,291 bytes  received 531,055 bytes  6,270,314.36 bytes/sec
total size is 1,783,468,051  speedup is 1.20

Status:


If anyone has any advice or observation I would greatly appreciate it.

Hrv

 

[Dunuin]

Your WD40EFAX HDDs are SMR drives and because of that not suitable for ZFS. SMR drives are really crappy by design because data is overlapping so you can store more data in the same space sacrificing write performance. So if you want to write something to it, it also needs to rewrite all data next to it resulting in very high latencies and abyssimal write performance as soon as the RAM and CMR cache has filled up. Its basically the same just like with QLC SSD and SLC cache but for HDDs. With SMR HDDs the latencies can get so bad, that ZFS thinks the HDD is faulted because it is too slow to answer in time resulting in ZFS errors and degraded pools.

So I would guess the HDDs are your bottleneck.

 

[spirit]

also : "SSD's above are Samsung 850 (sata) and Intel 600p (nvme)."

you really need datacenter grade ssd/nvme for zfs to have fast write. (at least for zfs journal). Zfs journal need fast sync write.

 

[HrvojeS]

Yeah, SMR I didn't know about to be honest. However I did burn-in tests and rebuild tests when I deployed raid and rebuild when full and disk erased and put it was only about day and a half. Didn't think it was problematic. Am def concerned about SMR now and will revisit.

For SSD/NVME speed I'll have to look but I'm not running datacenter loads here so this is just to get improvement over HDD and I couldn't justify the cost.

Question here is 1.7gb transfer job for moving a volume from HDD raid (read) to SSD mirror (write) is now taking over 20hrs, which I can't comprehend.