I/O issues with RAW

T

Tanguy Bourdet

Member
Aug 6, 2018
21
2
8
31
Hi,
I'm using LXC container on NFS target
when i do directly inside Proxmox /mnt/pve/free-storage
I got this:

Code: 
fio --name=rand-write --ioengine=libaio --iodepth=32 --rw=randwrite --invalidate=1 --bsrange=4k:4k,4k:4k --size=512m --runtime=60 --time_based --do_verify=1 --direct=1 --group_reporting --numjobs=1

rand-write: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32

fio-2.16

Starting 1 process

rand-write: Laying out IO file(s) (1 file(s) / 512MB)

Jobs: 1 (f=1): [w(1)] [4.9% done] [0KB/43968KB/0KB /s] [0/10.1K/0 iops] [eta 00mJobs: 1 (f=1): [w(1)] [6.6% done] [0KB/48440KB/0KB /s] [0/12.2K/0 iops] [eta 00mJobs: 1 (f=1): [w(1)] [8.2% done] [0KB/50706KB/0KB /s] [0/12.7K/0 iops] [eta 00mJobs: 1 (f=1): [w(1)] [9.8% done] [0KB/50268KB/0KB /s] [0/12.6K/0 iops] [eta 00mJobs: 1 (f=1): [w(1)] [11.5% done] [0KB/46148KB/0KB /s] [0/11.6K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [13.1% done] [0KB/53052KB/0KB /s] [0/13.3K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [14.8% done] [0KB/50124KB/0KB /s] [0/12.6K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [16.4% done] [0KB/51216KB/0KB /s] [0/12.9K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [18.0% done] [0KB/52940KB/0KB /s] [0/13.3K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [19.7% done] [0KB/49021KB/0KB /s] [0/12.3K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [21.3% done] [0KB/44724KB/0KB /s] [0/11.2K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [23.3% done] [0KB/52744KB/0KB /s] [0/13.2K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [24.6% done] [0KB/53748KB/0KB /s] [0/13.5K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [26.2% done] [0KB/49896KB/0KB /s] [0/12.5K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [27.9% done] [0KB/46732KB/0KB /s] [0/11.7K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [29.5% done] [0KB/54156KB/0KB /s] [0/13.6K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [31.1% done] [0KB/52632KB/0KB /s] [0/13.2K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [32.8% done] [0KB/53996KB/0KB /s] [0/13.5K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [34.4% done]

...


[0KB/46838KB/0KB /s] [0/11.8K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [82.0% done] [0KB/47892KB/0KB /s] [0/11.1K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [83.6% done] [0KB/48336KB/0KB /s] [0/12.9K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [85.2% done] [0KB/47880KB/0KB /s] [0/11.1K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [88.3% done] [0KB/49296KB/0KB /s] [0/12.4K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [88.5% done] [0KB/47080KB/0KB /s] [0/11.8K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [90.2% done] [0KB/49444KB/0KB /s] [0/12.4K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [91.8% done] [0KB/45557KB/0KB /s] [0/11.4K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [93.4% done] [0KB/49276KB/0KB /s] [0/12.4K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [95.1% done] [0KB/50512KB/0KB /s] [0/12.7K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [96.7% done] [0KB/55172KB/0KB /s] [0/13.8K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [98.4% done] [0KB/50240KB/0KB /s] [0/12.6K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [100.0% done] [0KB/53201KB/0KB /s] [0/13.3K/0 iops] [eta 00m:00s]

rand-write: (groupid=0, jobs=1): err= 0: pid=10936: Thu Feb 27 10:25:35 2020

  write: io=2846.9MB, bw=48583KB/s, iops=12145, runt= 60003msec

    slat (usec): min=2, max=3575, avg=20.57, stdev=24.86

    clat (usec): min=242, max=55224, avg=2606.43, stdev=1321.59

     lat (msec): min=1, max=55, avg= 2.63, stdev= 1.32

    clat percentiles (usec):

     |  1.00th=[ 2024],  5.00th=[ 2096], 10.00th=[ 2160], 20.00th=[ 2224],

     | 30.00th=[ 2288], 40.00th=[ 2352], 50.00th=[ 2416], 60.00th=[ 2512],

     | 70.00th=[ 2640], 80.00th=[ 2800], 90.00th=[ 3088], 95.00th=[ 3472],

     | 99.00th=[ 4512], 99.50th=[ 5280], 99.90th=[23680], 99.95th=[40192],

     | 99.99th=[45824]

    lat (usec) : 250=0.01%, 750=0.01%

    lat (msec) : 2=0.63%, 4=97.37%, 10=1.79%, 20=0.04%, 50=0.15%

    lat (msec) : 100=0.01%

  cpu          : usr=5.77%, sys=14.86%, ctx=780890, majf=0, minf=10

  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%

     issued    : total=r=0/w=728784/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0

     latency   : target=0, window=0, percentile=100.00%, depth=32


Run status group 0 (all jobs):

  WRITE: io=2846.9MB, aggrb=48583KB/s, minb=48583KB/s, maxb=48583KB/s, mint=60003msec, maxt=60003msec



But when i do it inside lxc container with raw disk on free-storage i got this :


Code: 
fio --name=rand-write --ioengine=libaio --iodepth=32 --rw=randwrite --invalidate=1 --bsrange=4k:4k,4k:4k --size=512m --runtime=60 --time_based --do_verify=1 --direct=1 --group_reporting --numjobs=1
rand-write: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.16
Starting 1 process
Jobs: 1 (f=1): [w(1)] [5.0% done] [0KB/339.3MB/0KB /s] [0/86.9K/0 iops] [eta 00mJobs: 1 (f=1): [w(1)] [6.7% done] [0KB/334.6MB/0KB /s] [0/85.7K/0 iops] [eta 00mJobs: 1 (f=1): [w(1)] [8.3% done] [0KB/331.6MB/0KB /s] [0/84.9K/0 iops] [eta 00mJobs: 1 (f=1): [w(1)] [10.0% done] [0KB/40180KB/0KB /s] [0/10.5K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [11.7% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:53s]  Jobs: 1 (f=1): [w(1)] [25.0% done] [0KB/238.2MB/0KB /s] [0/60.1K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [26.7% done] [0KB/364.9MB/0KB /s] [0/93.3K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [28.3% done] [0KB/342.5MB/0KB /s] [0/87.6K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [30.0% done] [0KB/341.1MB/0KB /s] [0/87.6K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [31.7% done] [0KB/342.6MB/0KB /s] [0/87.7K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [33.3% done] [0KB/87836KB/0KB /s] [0/21.1K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [35.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:39s]  Jobs: 1 (f=1): [w(1)] [48.3% done] [0KB/216.9MB/0KB /s] [0/55.6K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [50.0% done] [0KB/197.5MB/0KB /s] [0/50.6K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [52.5% done] [0KB/12KB/0KB /s] [0/3/0 iops] [eta 00m:28s] Jobs: 1 (f=1): [w(1)] [60.0% done] [0KB/165.7MB/0KB /s] [0/42.5K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [62.7% done] [0KB/333.1MB/0KB /s] [0/85.5K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [63.3% done] [0KB/335.9MB/0KB /s] [0/85.1K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [66.1% done] [0KB/323.1MB/0KB /s] [0/82.1K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [66.7% done] [0KB/331.9MB/0KB /s] [0/84.1K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [68.3% done] [0KB/7955KB/0KB /s] [0/1988/0 iops] [eta 00m:Jobs: 1 (f=1): [w(1)] [70.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:18s]  Jobs: 1 (f=1): [w(1)] [83.6% done] [0KB/295.4MB/0KB /s] [0/75.6K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [85.0% done] [0KB/333.4MB/0KB /s] [0/85.3K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [88.3% done] [0KB/348.6MB/0KB /s] [0/89.3K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [88.5% done] [0KB/350.5MB/0KB /s] [0/89.8K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [90.2% done] [0KB/379.3MB/0KB /s] [0/97.4K/0 iops] [eta 00Jobs: 1 (f=1): [w(1)] [91.8% done] [0KB/29660KB/0KB /s] [0/7415/0 iops] [eta 00mJobs: 1 (f=1): [w(1)] [93.4% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:04s]  Jobs: 1 (f=1): [w(1)] [100.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:00s]
rand-write: (groupid=0, jobs=1): err= 0: pid=24128: Thu Feb 27 11:29:33 2020
  write: io=7052.8MB, bw=114730KB/s, iops=28682, runt= 62948msec
    slat (usec): min=3, max=191804, avg= 7.35, stdev=142.86
    clat (usec): min=216, max=9048.8K, avg=1105.84, stdev=78307.23
     lat (usec): min=221, max=9048.8K, avg=1113.18, stdev=78307.36
    clat percentiles (usec):
     |  1.00th=[  298],  5.00th=[  306], 10.00th=[  314], 20.00th=[  330],
     | 30.00th=[  338], 40.00th=[  346], 50.00th=[  350], 60.00th=[  354],
     | 70.00th=[  358], 80.00th=[  366], 90.00th=[  382], 95.00th=[  422],
     | 99.00th=[  564], 99.50th=[  756], 99.90th=[ 1272], 99.95th=[ 1528],
     | 99.99th=[1564672]
    lat (usec) : 250=0.02%, 500=98.37%, 750=1.10%, 1000=0.20%
    lat (msec) : 2=0.26%, 4=0.01%, 10=0.02%, 20=0.01%, 50=0.01%
    lat (msec) : 250=0.01%, 500=0.01%, 750=0.01%, 2000=0.01%, >=2000=0.01%
  cpu          : usr=7.20%, sys=25.23%, ctx=5094, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=1805505/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=7052.8MB, aggrb=114729KB/s, minb=114729KB/s, maxb=114729KB/s, mint=62948msec, maxt=62948msec

Disk stats (read/write):
  loop25: ios=0/1805527, merge=0/0, ticks=0/1197716, in_queue=1451468, util=77.03%
As you can see there is drop to 0 ios

When i mount the raw file inside a mount point he problem is the same

Can you help me ?
 
hi,

it might have to do with the way that fio gathers the disk stats, sometimes it happens in LXC when a tool uses a syscall or similar

maybe you can test it some other way to confirm
 
You must log in or register to reply here.
Top Bottom