Installation with ZFS mirror on SSD super slow

A

Amari

Active Member
May 12, 2020
7
0
41
33
Hi,
I just did a new installation of Proxmox and selected ZFS mirror with 2 SSDs during the installation. I used the default values with ashift 12.

Unfortunately, the 4k read/write performance is increadable slow compared to mdadm raid 1.

On mdadm1 I had around 100k IOPS with 4k block size. Now with zfs it's 10k and I can't figure out why. I always use the default settings from proxmox.

Disks are:
SAMSUNG MZQLB960HAJR-00007
SAMSUNG MZQLW960HMJP-00003

This is on mdadm:
Code: 
fio Disk Speed Tests (Mixed R/W 50/50) (Partition /dev/md2):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ----
Read       | 398.70 MB/s  (99.6k) | 466.46 MB/s   (7.2k)
Write      | 399.75 MB/s  (99.9k) | 468.92 MB/s   (7.3k)
Total      | 798.45 MB/s (199.6k) | 935.39 MB/s  (14.6k)
           |                      |
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ----
Read       | 449.28 MB/s    (877) | 419.24 MB/s    (409)
Write      | 473.15 MB/s    (924) | 447.16 MB/s    (436)
Total      | 922.43 MB/s   (1.8k) | 866.41 MB/s    (845)

This is with ZFS
Code: 
fio Disk Speed Tests (Mixed R/W 50/50) (Partition rpool/ROOT/pve-1):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ----
Read       | 55.19 MB/s   (13.7k) | 1.46 GB/s    (22.9k)
Write      | 55.28 MB/s   (13.8k) | 1.47 GB/s    (23.0k)
Total      | 110.48 MB/s  (27.6k) | 2.93 GB/s    (45.9k)
           |                      |
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ----
Read       | 4.49 GB/s     (8.7k) | 2.99 GB/s     (2.9k)
Write      | 4.73 GB/s     (9.2k) | 3.19 GB/s     (3.1k)
Total      | 9.23 GB/s    (18.0k) | 6.18 GB/s     (6.0k)


I read that some SSDs have ashift 13. Could this make such a big difference? If I wouldn't need to book a KVM console in the datacenter, I would just reinstall and try it out. That is why I am trying to understand the problem first.
 
Last edited:
Thanks, I can try with your parameters.
My results are from yabs.sh benchmark. Not sure which default parameters they use.
 
@ghusson those are the results.
I do not understand all of it. What do you think? It is only testing a rather big blocksize, do I see this correct?
Code: 
fio_test: (g=0): rw=randrw, bs=(R) 8192B-16.0KiB, (W) 8192B-16.0KiB, (T) 8192B-16.0KiB, ioengine=libaio, iodepth=64
...
fio-3.33
Starting 50 processes
fio_test: Laying out IO file (1 file / 10240MiB)
Jobs: 50 (f=50): [m(50)][100.0%][r=633MiB/s,w=643MiB/s][r=57.5k,w=58.4k IOPS][eta 00m:00s]
fio_test: (groupid=0, jobs=50): err= 0: pid=253569: Fri Jan 31 08:29:02 2025
  read: IOPS=55.5k, BW=629MiB/s (660MB/s)(73.8GiB/120033msec)
   bw (  KiB/s): min=243238, max=1073205, per=100.00%, avg=645108.81, stdev=2972.01, samples=11900
   iops        : min=20855, max=91868, avg=55499.77, stdev=255.10, samples=11900
  write: IOPS=55.5k, BW=630MiB/s (660MB/s)(73.8GiB/120033msec); 0 zone resets
   bw (  KiB/s): min=241454, max=1058477, per=100.00%, avg=645367.56, stdev=2943.61, samples=11900
   iops        : min=20874, max=90768, avg=55518.90, stdev=252.57, samples=11900
  cpu          : usr=0.85%, sys=10.92%, ctx=3821228, majf=6, minf=435
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=101.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=6656666,6658992,0,132882 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=629MiB/s (660MB/s), 629MiB/s-629MiB/s (660MB/s-660MB/s), io=73.8GiB (79.2GB), run=120033-120033msec
  WRITE: bw=630MiB/s (660MB/s), 630MiB/s-630MiB/s (660MB/s-660MB/s), io=73.8GiB (79.2GB), run=120033-120033msec


Code: 
~# fio --filename=testfile --size=100GB --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1
iops-test-job: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.33
Starting 4 processes
iops-test-job: Laying out IO file (1 file / 102400MiB)
Jobs: 4 (f=4): [m(4)][2.5%][r=11.8MiB/s,w=11.9MiB/s][r=3021,w=3049 IOPS][eta 01m:57s]
Jobs: 4 (f=4): [m(4)][4.2%][r=19.7MiB/s,w=19.4MiB/s][r=5054,w=4966 IOPS][eta 01m:55s]
Jobs: 4 (f=4): [m(4)][5.8%][r=12.0MiB/s,w=11.5MiB/s][r=3077,w=2952 IOPS][eta 01m:53s]
Jobs: 4 (f=4): [m(4)][7.5%][r=19.7MiB/s,w=19.5MiB/s][r=5054,w=4989 IOPS][eta 01m:51s]
Jobs: 4 (f=4): [m(4)][9.2%][r=13.6MiB/s,w=13.9MiB/s][r=3472,w=3560 IOPS][eta 01m:49s]
Jobs: 4 (f=4): [m(4)][10.8%][r=19.6MiB/s,w=19.7MiB/s][r=5009,w=5047 IOPS][eta 01m:47s]
Jobs: 4 (f=4): [m(4)][12.5%][r=15.4MiB/s,w=15.5MiB/s][r=3932,w=3974 IOPS][eta 01m:45s]
Jobs: 4 (f=4): [m(4)][14.2%][r=16.9MiB/s,w=17.2MiB/s][r=4339,w=4415 IOPS][eta 01m:43s]
Jobs: 4 (f=4): [m(4)][15.8%][r=18.7MiB/s,w=18.8MiB/s][r=4786,w=4824 IOPS][eta 01m:41s]
Jobs: 4 (f=4): [m(4)][17.5%][r=16.2MiB/s,w=16.4MiB/s][r=4140,w=4203 IOPS][eta 01m:39s]
Jobs: 4 (f=4): [m(4)][19.2%][r=18.6MiB/s,w=19.3MiB/s][r=4758,w=4946 IOPS][eta 01m:37s]
Jobs: 4 (f=4): [m(4)][20.8%][r=13.5MiB/s,w=13.6MiB/s][r=3447,w=3490 IOPS][eta 01m:35s]
Jobs: 4 (f=4): [m(4)][22.5%][r=19.2MiB/s,w=19.5MiB/s][r=4917,w=4990 IOPS][eta 01m:33s]
Jobs: 4 (f=4): [m(4)][24.2%][r=11.4MiB/s,w=11.6MiB/s][r=2922,w=2978 IOPS][eta 01m:31s]
Jobs: 4 (f=4): [m(4)][25.8%][r=19.6MiB/s,w=19.0MiB/s][r=5015,w=4873 IOPS][eta 01m:29s]
Jobs: 4 (f=4): [m(4)][27.5%][r=12.4MiB/s,w=12.0MiB/s][r=3182,w=3063 IOPS][eta 01m:27s]
Jobs: 4 (f=4): [m(4)][29.2%][r=19.9MiB/s,w=19.1MiB/s][r=5092,w=4885 IOPS][eta 01m:25s]
Jobs: 4 (f=4): [m(4)][30.8%][r=14.7MiB/s,w=15.3MiB/s][r=3764,w=3909 IOPS][eta 01m:23s]
Jobs: 4 (f=4): [m(4)][32.5%][r=17.5MiB/s,w=17.1MiB/s][r=4489,w=4365 IOPS][eta 01m:21s]
Jobs: 4 (f=4): [m(4)][34.2%][r=18.4MiB/s,w=18.2MiB/s][r=4703,w=4666 IOPS][eta 01m:19s]
Jobs: 4 (f=4): [m(4)][35.8%][r=15.8MiB/s,w=15.7MiB/s][r=4053,w=4024 IOPS][eta 01m:17s]
Jobs: 4 (f=4): [m(4)][37.5%][r=19.3MiB/s,w=19.3MiB/s][r=4932,w=4953 IOPS][eta 01m:15s]
Jobs: 4 (f=4): [m(4)][39.2%][r=12.8MiB/s,w=13.1MiB/s][r=3274,w=3341 IOPS][eta 01m:13s]
Jobs: 4 (f=4): [m(4)][40.8%][r=18.7MiB/s,w=19.3MiB/s][r=4793,w=4949 IOPS][eta 01m:11s]
Jobs: 4 (f=4): [m(4)][42.5%][r=11.7MiB/s,w=12.0MiB/s][r=2988,w=3061 IOPS][eta 01m:09s]
Jobs: 4 (f=4): [m(4)][44.2%][r=19.5MiB/s,w=19.2MiB/s][r=4982,w=4923 IOPS][eta 01m:07s]
Jobs: 4 (f=4): [m(4)][45.8%][r=12.4MiB/s,w=12.4MiB/s][r=3164,w=3165 IOPS][eta 01m:05s]
Jobs: 4 (f=4): [m(4)][47.5%][r=19.5MiB/s,w=19.2MiB/s][r=4983,w=4921 IOPS][eta 01m:03s]
Jobs: 4 (f=4): [m(4)][49.2%][r=16.3MiB/s,w=15.9MiB/s][r=4185,w=4077 IOPS][eta 01m:01s]
Jobs: 4 (f=4): [m(4)][50.8%][r=16.4MiB/s,w=16.3MiB/s][r=4206,w=4169 IOPS][eta 00m:59s]
Jobs: 4 (f=4): [m(4)][52.5%][r=19.0MiB/s,w=19.3MiB/s][r=4870,w=4948 IOPS][eta 00m:57s]
Jobs: 4 (f=4): [m(4)][54.2%][r=14.6MiB/s,w=14.7MiB/s][r=3748,w=3754 IOPS][eta 00m:55s]
Jobs: 4 (f=4): [m(4)][55.8%][r=19.9MiB/s,w=19.2MiB/s][r=5107,w=4904 IOPS][eta 00m:53s]
Jobs: 4 (f=4): [m(4)][57.5%][r=11.5MiB/s,w=11.8MiB/s][r=2935,w=3019 IOPS][eta 00m:51s]
Jobs: 4 (f=4): [m(4)][59.2%][r=19.4MiB/s,w=19.3MiB/s][r=4961,w=4940 IOPS][eta 00m:49s]
Jobs: 4 (f=4): [m(4)][60.8%][r=11.8MiB/s,w=12.0MiB/s][r=3025,w=3064 IOPS][eta 00m:47s]
Jobs: 4 (f=4): [m(4)][62.5%][r=19.6MiB/s,w=19.5MiB/s][r=5009,w=4989 IOPS][eta 00m:45s]
Jobs: 4 (f=4): [m(4)][64.2%][r=13.9MiB/s,w=14.4MiB/s][r=3555,w=3687 IOPS][eta 00m:43s]
Jobs: 4 (f=4): [m(4)][65.8%][r=18.8MiB/s,w=18.7MiB/s][r=4805,w=4793 IOPS][eta 00m:41s]
Jobs: 4 (f=4): [m(4)][67.5%][r=17.2MiB/s,w=16.7MiB/s][r=4413,w=4278 IOPS][eta 00m:39s]
Jobs: 4 (f=4): [m(4)][69.2%][r=15.7MiB/s,w=15.9MiB/s][r=4023,w=4066 IOPS][eta 00m:37s]
Jobs: 4 (f=4): [m(4)][70.8%][r=19.5MiB/s,w=19.4MiB/s][r=4981,w=4974 IOPS][eta 00m:35s]
Jobs: 4 (f=4): [m(4)][72.5%][r=13.6MiB/s,w=13.6MiB/s][r=3469,w=3483 IOPS][eta 00m:33s]
Jobs: 4 (f=4): [m(4)][74.2%][r=20.4MiB/s,w=20.1MiB/s][r=5218,w=5152 IOPS][eta 00m:31s]
Jobs: 4 (f=4): [m(4)][75.8%][r=12.7MiB/s,w=12.8MiB/s][r=3255,w=3289 IOPS][eta 00m:29s]
Jobs: 4 (f=4): [m(4)][77.5%][r=19.1MiB/s,w=19.2MiB/s][r=4893,w=4909 IOPS][eta 00m:27s]
Jobs: 4 (f=4): [m(4)][79.2%][r=11.7MiB/s,w=11.7MiB/s][r=2992,w=2986 IOPS][eta 00m:25s]
Jobs: 4 (f=4): [m(4)][80.8%][r=19.4MiB/s,w=19.7MiB/s][r=4964,w=5033 IOPS][eta 00m:23s]
Jobs: 4 (f=4): [m(4)][82.5%][r=12.2MiB/s,w=11.6MiB/s][r=3128,w=2961 IOPS][eta 00m:21s]
Jobs: 4 (f=4): [m(4)][84.2%][r=19.3MiB/s,w=19.4MiB/s][r=4947,w=4974 IOPS][eta 00m:19s]
Jobs: 4 (f=4): [m(4)][85.8%][r=14.1MiB/s,w=14.3MiB/s][r=3621,w=3652 IOPS][eta 00m:17s]
Jobs: 4 (f=4): [m(4)][87.5%][r=19.1MiB/s,w=18.9MiB/s][r=4900,w=4849 IOPS][eta 00m:15s]
Jobs: 4 (f=4): [m(4)][89.2%][r=16.2MiB/s,w=16.2MiB/s][r=4160,w=4156 IOPS][eta 00m:13s]
Jobs: 4 (f=4): [m(4)][90.8%][r=16.7MiB/s,w=17.0MiB/s][r=4271,w=4343 IOPS][eta 00m:11s]
Jobs: 4 (f=4): [m(4)][92.5%][r=18.2MiB/s,w=18.0MiB/s][r=4660,w=4620 IOPS][eta 00m:09s]
Jobs: 4 (f=4): [m(4)][94.2%][r=16.0MiB/s,w=16.2MiB/s][r=4100,w=4153 IOPS][eta 00m:07s]
Jobs: 4 (f=4): [m(4)][95.8%][r=18.8MiB/s,w=19.2MiB/s][r=4816,w=4921 IOPS][eta 00m:05s]
Jobs: 4 (f=4): [m(4)][97.5%][r=14.3MiB/s,w=14.2MiB/s][r=3673,w=3642 IOPS][eta 00m:03s]
Jobs: 4 (f=4): [m(4)][99.2%][r=19.0MiB/s,w=19.4MiB/s][r=4866,w=4964 IOPS][eta 00m:01s]
Jobs: 4 (f=4): [m(4)][100.0%][r=18.3MiB/s,w=19.1MiB/s][r=4696,w=4879 IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=264889: Fri Jan 31 09:24:50 2025
  read: IOPS=4207, BW=16.4MiB/s (17.2MB/s)(1972MiB/120001msec)
    slat (usec): min=3, max=15952, avg=467.75, stdev=629.61
    clat (usec): min=4, max=366619, avg=120908.66, stdev=73151.15
     lat (usec): min=593, max=367152, avg=121376.41, stdev=73407.01
    clat percentiles (msec):
     |  1.00th=[   56],  5.00th=[   66], 10.00th=[   67], 20.00th=[   69],
     | 30.00th=[   70], 40.00th=[   71], 50.00th=[   72], 60.00th=[   75],
     | 70.00th=[  165], 80.00th=[  211], 90.00th=[  239], 95.00th=[  259],
     | 99.00th=[  292], 99.50th=[  305], 99.90th=[  326], 99.95th=[  330],
     | 99.99th=[  347]
   bw (  KiB/s): min= 6696, max=31160, per=99.75%, avg=16790.27, stdev=1801.42, samples=956
   iops        : min= 1674, max= 7790, avg=4197.56, stdev=450.35, samples=956
  write: IOPS=4216, BW=16.5MiB/s (17.3MB/s)(1976MiB/120001msec); 0 zone resets
    slat (usec): min=5, max=39899, avg=475.52, stdev=630.92
    clat (usec): min=4, max=366683, avg=121112.35, stdev=73230.92
     lat (usec): min=1179, max=367253, avg=121587.87, stdev=73487.81
    clat percentiles (msec):
     |  1.00th=[   56],  5.00th=[   66], 10.00th=[   67], 20.00th=[   69],
     | 30.00th=[   70], 40.00th=[   71], 50.00th=[   72], 60.00th=[   75],
     | 70.00th=[  165], 80.00th=[  211], 90.00th=[  241], 95.00th=[  259],
     | 99.00th=[  292], 99.50th=[  300], 99.90th=[  326], 99.95th=[  330],
     | 99.99th=[  342]
   bw (  KiB/s): min= 6832, max=31024, per=99.72%, avg=16819.68, stdev=1790.11, samples=956
   iops        : min= 1708, max= 7756, avg=4204.91, stdev=447.53, samples=956
  lat (usec)   : 10=0.01%, 750=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=63.10%, 250=30.04%, 500=6.83%
  cpu          : usr=0.79%, sys=6.70%, ctx=962017, majf=0, minf=51
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=504943,505967,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=16.4MiB/s (17.2MB/s), 16.4MiB/s-16.4MiB/s (17.2MB/s-17.2MB/s), io=1972MiB (2068MB), run=120001-120001msec
  WRITE: bw=16.5MiB/s (17.3MB/s), 16.5MiB/s-16.5MiB/s (17.3MB/s-17.3MB/s), io=1976MiB (2072MB), run=120001-120001msec
 
Last edited:
You must log in or register to reply here.
Top Bottom