Recent changes

Hi Jens,
I am seeing a serious problem with fio testing. I tried running fio test on ext3 partition with Sequential Reads and Writes. I keep getting the I/O write error for Sequential writes. I tried with multiple servers with different controllers and I still get the same error. With Sequential Reads and Random Reads/Writes there is no issue seen. Please find the attached fio results file for Sequential Reads/Writes which has the error details pertaining to Sequential writes on ext3 file system. Thanks, Girish

512B_MaxIO_Read: (g=0): rw=read, bs=512-512/512-512, ioengine=libaio, iodepth=1 512B_MaxIO_Read: (g=1): rw=read, bs=512-512/512-512, ioengine=libaio, iodepth=2 512B_MaxIO_Read: (g=2): rw=read, bs=512-512/512-512, ioengine=libaio, iodepth=4 512B_MaxIO_Read: (g=3): rw=read, bs=512-512/512-512, ioengine=libaio, iodepth=8 512B_MaxIO_Read: (g=4): rw=read, bs=512-512/512-512, ioengine=libaio, iodepth=16 512B_MaxIO_Read: (g=5): rw=read, bs=512-512/512-512, ioengine=libaio, iodepth=32 512B_MaxIO_Read: (g=6): rw=read, bs=512-512/512-512, ioengine=libaio, iodepth=64 512B_MaxIO_Write: (g=7): rw=write, bs=512-512/512-512, ioengine=libaio, iodepth=1 512B_MaxIO_Write: (g=8): rw=write, bs=512-512/512-512, ioengine=libaio, iodepth=2 512B_MaxIO_Write: (g=9): rw=write, bs=512-512/512-512, ioengine=libaio, iodepth=4 512B_MaxIO_Write: (g=10): rw=write, bs=512-512/512-512, ioengine=libaio, iodepth=8 512B_MaxIO_Write: (g=11): rw=write, bs=512-512/512-512, ioengine=libaio, iodepth=16 512B_MaxIO_Write: (g=12): rw=write, bs=512-512/512-512, ioengine=libaio, iodepth=32 512B_MaxIO_Write: (g=13): rw=write, bs=512-512/512-512, ioengine=libaio, iodepth=64
Idle: (g=14): rw=write, bs=64K-64K/64K-64K, ioengine=libaio, iodepth=1 4KB_Sequential_Read: (g=15): rw=read, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=1 4KB_Sequential_Read: (g=16): rw=read, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=2 4KB_Sequential_Read: (g=17): rw=read, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=4 4KB_Sequential_Read: (g=18): rw=read, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=8 4KB_Sequential_Read: (g=19): rw=read, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=16 4KB_Sequential_Read: (g=20): rw=read, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32 4KB_Sequential_Read: (g=21): rw=read, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64 4KB_Sequential_Write: (g=22): rw=write, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=1 4KB_Sequential_Write: (g=23): rw=write, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=2 4KB_Sequential_Write: (g=24): rw=write, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=4 4KB_Sequential_Write: (g=25): rw=write, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=8 4KB_Sequential_Write: (g=26): rw=write, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=16 4KB_Sequential_Write: (g=27): rw=write, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32 4KB_Sequential_Write: (g=28): rw=write, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
Idle: (g=29): rw=write, bs=64K-64K/64K-64K, ioengine=libaio, iodepth=1 64KB_Sequential_Read: (g=30): rw=read, bs=64K-64K/64K-64K, ioengine=libaio, iodepth=1 64KB_Sequential_Read: (g=31): rw=read, bs=64K-64K/64K-64K, ioengine=libaio, iodepth=2 64KB_Sequential_Read: (g=32): rw=read, bs=64K-64K/64K-64K, ioengine=libaio, iodepth=4 64KB_Sequential_Read: (g=33): rw=read, bs=64K-64K/64K-64K, ioengine=libaio, iodepth=8 64KB_Sequential_Read: (g=34): rw=read, bs=64K-64K/64K-64K, ioengine=libaio, iodepth=16 64KB_Sequential_Read: (g=35): rw=read, bs=64K-64K/64K-64K, ioengine=libaio, iodepth=32 64KB_Sequential_Read: (g=36): rw=read, bs=64K-64K/64K-64K, ioengine=libaio, iodepth=64 64KB_Sequential_Write: (g=37): rw=write, bs=64K-64K/64K-64K, ioengine=libaio, iodepth=1 64KB_Sequential_Write: (g=38): rw=write, bs=64K-64K/64K-64K, ioengine=libaio, iodepth=2 64KB_Sequential_Write: (g=39): rw=write, bs=64K-64K/64K-64K, ioengine=libaio, iodepth=4 64KB_Sequential_Write: (g=40): rw=write, bs=64K-64K/64K-64K, ioengine=libaio, iodepth=8 64KB_Sequential_Write: (g=41): rw=write, bs=64K-64K/64K-64K, ioengine=libaio, iodepth=16 64KB_Sequential_Write: (g=42): rw=write, bs=64K-64K/64K-64K, ioengine=libaio, iodepth=32 64KB_Sequential_Write: (g=43): rw=write, bs=64K-64K/64K-64K, ioengine=libaio, iodepth=64
Idle: (g=44): rw=write, bs=64K-64K/64K-64K, ioengine=libaio, iodepth=1 512KB_Sequential_Read: (g=45): rw=read, bs=512K-512K/512K-512K, ioengine=libaio, iodepth=1 512KB_Sequential_Read: (g=46): rw=read, bs=512K-512K/512K-512K, ioengine=libaio, iodepth=2 512KB_Sequential_Read: (g=47): rw=read, bs=512K-512K/512K-512K, ioengine=libaio, iodepth=4 512KB_Sequential_Read: (g=48): rw=read, bs=512K-512K/512K-512K, ioengine=libaio, iodepth=8 512KB_Sequential_Read: (g=49): rw=read, bs=512K-512K/512K-512K, ioengine=libaio, iodepth=16 512KB_Sequential_Read: (g=50): rw=read, bs=512K-512K/512K-512K, ioengine=libaio, iodepth=32 512KB_Sequential_Read: (g=51): rw=read, bs=512K-512K/512K-512K, ioengine=libaio, iodepth=64 512KB_Sequential_Write: (g=52): rw=write, bs=512K-512K/512K-512K, ioengine=libaio, iodepth=1 512KB_Sequential_Write: (g=53): rw=write, bs=512K-512K/512K-512K, ioengine=libaio, iodepth=2 512KB_Sequential_Write: (g=54): rw=write, bs=512K-512K/512K-512K, ioengine=libaio, iodepth=4 512KB_Sequential_Write: (g=55): rw=write, bs=512K-512K/512K-512K, ioengine=libaio, iodepth=8 512KB_Sequential_Write: (g=56): rw=write, bs=512K-512K/512K-512K, ioengine=libaio, iodepth=16 512KB_Sequential_Write: (g=57): rw=write, bs=512K-512K/512K-512K, ioengine=libaio, iodepth=32 512KB_Sequential_Write: (g=58): rw=write, bs=512K-512K/512K-512K, ioengine=libaio, iodepth=64
Idle: (g=59): rw=write, bs=64K-64K/64K-64K, ioengine=libaio, iodepth=1 1MB_Sequential_Read: (g=60): rw=read, bs=1M-1M/1M-1M, ioengine=libaio, iodepth=1 1MB_Sequential_Read: (g=61): rw=read, bs=1M-1M/1M-1M, ioengine=libaio, iodepth=2 1MB_Sequential_Read: (g=62): rw=read, bs=1M-1M/1M-1M, ioengine=libaio, iodepth=4 1MB_Sequential_Read: (g=63): rw=read, bs=1M-1M/1M-1M, ioengine=libaio, iodepth=8 1MB_Sequential_Read: (g=64): rw=read, bs=1M-1M/1M-1M, ioengine=libaio, iodepth=16 1MB_Sequential_Read: (g=65): rw=read, bs=1M-1M/1M-1M, ioengine=libaio, iodepth=32 1MB_Sequential_Read: (g=66): rw=read, bs=1M-1M/1M-1M, ioengine=libaio, iodepth=64 Starting 67 processes

512B_MaxIO_Read: (groupid=0, jobs=1): err= 0: pid=10239
read : io=294MiB, bw=10,279KiB/s, iops=20,077, runt= 30001msec

    slat (usec): min=8, max=260, avg= 9.03, stdev= 2.29
    clat (usec): min=2, max=7,842, avg=37.93, stdev=25.59
    bw (KiB/s) : min= 9850, max=10397, per=100.00%, avg=10278.71, stdev=142.01
  cpu          : usr=7.70%, sys=25.72%, ctx=602331, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=602334/0, short=0/0
     lat (usec): 4=0.01%, 50=97.69%, 100=1.77%, 250=0.46%, 500=0.04%
     lat (usec): 750=0.03%, 1000=0.01%
     lat (msec): 2=0.01%, 4=0.01%, 10=0.01%
512B_MaxIO_Read: (groupid=1, jobs=1): err= 0: pid=10240
  read : io=416MiB, bw=14,546KiB/s, iops=28,411, runt= 30001msec
    slat (usec): min=7, max=264, avg= 8.62, stdev= 2.38
    clat (usec): min=15, max=1,459, avg=55.57, stdev=16.99
    bw (KiB/s) : min=13876, max=14614, per=100.01%, avg=14546.83, stdev=151.00
  cpu          : usr=9.98%, sys=42.34%, ctx=433157, majf=0, minf=13
  IO depths    : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=852385/0, short=0/0
     lat (usec): 20=0.01%, 50=44.97%, 100=53.27%, 250=1.65%, 500=0.10%
     lat (usec): 750=0.01%, 1000=0.01%
     lat (msec): 2=0.01%
512B_MaxIO_Read: (groupid=2, jobs=1): err= 0: pid=10241
  read : io=537MiB, bw=18,752KiB/s, iops=36,625, runt= 30001msec
    slat (usec): min=7, max=278, avg=12.35, stdev= 7.53
    clat (usec): min=40, max=5,098, avg=92.40, stdev=31.58
    bw (KiB/s) : min=17766, max=18894, per=100.00%, avg=18751.71, stdev=248.35
  cpu          : usr=13.25%, sys=59.93%, ctx=287573, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=1098810/0, short=0/0
     lat (usec): 50=0.02%, 100=88.36%, 250=11.46%, 500=0.10%, 750=0.03%
     lat (usec): 1000=0.03%
     lat (msec): 2=0.01%, 4=0.01%, 10=0.01%
512B_MaxIO_Read: (groupid=3, jobs=1): err= 0: pid=10242
  read : io=543MiB, bw=18,976KiB/s, iops=37,063, runt= 30001msec
    slat (usec): min=7, max=288, avg=12.30, stdev= 7.47
    clat (usec): min=49, max=12,835, avg=199.08, stdev=54.58
    bw (KiB/s) : min=18082, max=19119, per=100.00%, avg=18975.69, stdev=237.93
  cpu          : usr=13.06%, sys=61.39%, ctx=282600, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=1111948/0, short=0/0
     lat (usec): 50=0.01%, 100=0.02%, 250=90.49%, 500=9.40%, 750=0.02%
     lat (usec): 1000=0.01%
     lat (msec): 2=0.06%, 4=0.01%, 10=0.01%, 20=0.01%
512B_MaxIO_Read: (groupid=4, jobs=1): err= 0: pid=10243
  read : io=543MiB, bw=18,976KiB/s, iops=37,062, runt= 30001msec
    slat (usec): min=7, max=318, avg=12.34, stdev= 7.56
    clat (usec): min=99, max=19,794, avg=414.94, stdev=116.41
    bw (KiB/s) : min=17968, max=19156, per=100.00%, avg=18975.14, stdev=293.15
  cpu          : usr=13.21%, sys=60.92%, ctx=281679, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=1111915/0, short=0/0
     lat (usec): 100=0.01%, 250=0.06%, 500=92.17%, 750=7.64%, 1000=0.01%
     lat (msec): 2=0.03%, 4=0.07%, 10=0.01%, 20=0.01%
512B_MaxIO_Read: (groupid=5, jobs=1): err= 0: pid=10244
  read : io=543MiB, bw=18,977KiB/s, iops=37,064, runt= 30001msec
    slat (usec): min=7, max=397, avg=12.29, stdev= 7.89
    clat (usec): min=127, max=19,269, avg=846.60, stdev=198.53
    bw (KiB/s) : min=17707, max=19159, per=99.99%, avg=18975.32, stdev=329.36
  cpu          : usr=13.78%, sys=60.71%, ctx=281899, majf=0, minf=16
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued r/w: total=1111976/0, short=0/0
     lat (usec): 250=0.01%, 500=0.10%, 750=0.69%, 1000=98.54%
     lat (msec): 2=0.53%, 4=0.04%, 10=0.09%, 20=0.01%
512B_MaxIO_Read: (groupid=6, jobs=1): err= 0: pid=10245
  read : io=543MiB, bw=18,992KiB/s, iops=37,095, runt= 30001msec
    slat (usec): min=7, max=713, avg=12.20, stdev= 7.85
    clat (usec): min=897, max=20,933, avg=1708.60, stdev=319.15
    bw (KiB/s) : min=17952, max=19176, per=100.00%, avg=18992.56, stdev=324.62
  cpu          : usr=12.95%, sys=63.51%, ctx=256924, majf=0, minf=20
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued r/w: total=1112897/0, short=0/0
     lat (usec): 1000=0.01%
     lat (msec): 2=99.60%, 4=0.22%, 10=0.12%, 20=0.06%, 50=0.01%
512B_MaxIO_Write: (groupid=7, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10246
  cpu          : usr=0.00%, sys=0.00%, ctx=1, majf=0, minf=39
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/1, short=0/0

512B_MaxIO_Write: (groupid=8, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10247

  cpu          : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=39
  IO depths    : 1=50.0%, 2=50.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/2, short=0/0

512B_MaxIO_Write: (groupid=9, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10248

  cpu          : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=39
  IO depths    : 1=25.0%, 2=50.0%, 4=25.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/4, short=0/0

512B_MaxIO_Write: (groupid=10, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10249

  cpu          : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=39
  IO depths    : 1=12.5%, 2=25.0%, 4=50.0%, 8=12.5%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/8, short=0/0

512B_MaxIO_Write: (groupid=11, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10250

  cpu          : usr=0.00%, sys=0.00%, ctx=2, majf=0, minf=39
  IO depths    : 1=6.2%, 2=12.5%, 4=25.0%, 8=50.0%, 16=6.2%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/16, short=0/0

512B_MaxIO_Write: (groupid=12, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10251

  cpu          : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=39
  IO depths    : 1=3.1%, 2=6.2%, 4=12.5%, 8=25.0%, 16=50.0%, 32=3.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/32, short=0/0

512B_MaxIO_Write: (groupid=13, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10252

  cpu          : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=39
  IO depths    : 1=1.6%, 2=3.1%, 4=6.2%, 8=12.5%, 16=25.0%, 32=50.0%, >=64=1.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/64, short=0/0

Idle: (groupid=14, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10253

  cpu          : usr=0.00%, sys=0.00%, ctx=4, majf=0, minf=39
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/1, short=0/0

4KB_Sequential_Read: (groupid=15, jobs=1): err= 0: pid=10254
read : io=2,041MiB, bw=71,333KiB/s, iops=17,415, runt= 30000msec

    slat (usec): min=8, max=262, avg= 9.04, stdev= 2.21
    clat (usec): min=2, max=7,258, avg=45.52, stdev=36.43
    bw (KiB/s) : min=67158, max=71892, per=100.00%, avg=71330.00, stdev=919.87
  cpu          : usr=6.63%, sys=22.14%, ctx=522448, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=522463/0, short=0/0
     lat (usec): 4=0.01%, 50=83.40%, 100=10.97%, 250=5.56%, 500=0.05%
     lat (usec): 750=0.01%, 1000=0.01%
     lat (msec): 2=0.01%, 4=0.01%, 10=0.01%
4KB_Sequential_Read: (groupid=16, jobs=1): err= 0: pid=10255
  read : io=2,694MiB, bw=94,155KiB/s, iops=22,987, runt= 30001msec
    slat (usec): min=7, max=273, avg= 9.69, stdev= 3.65
    clat (usec): min=15, max=8,991, avg=72.16, stdev=45.69
    bw (KiB/s) : min=89309, max=95092, per=100.02%, avg=94170.07, stdev=1141.17
  cpu          : usr=8.51%, sys=35.25%, ctx=388015, majf=0, minf=14
  IO depths    : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=689639/0, short=0/0
     lat (usec): 20=0.01%, 50=1.25%, 100=85.94%, 250=12.67%, 500=0.10%
     lat (usec): 750=0.02%, 1000=0.01%
     lat (msec): 2=0.01%, 4=0.01%, 10=0.01%
4KB_Sequential_Read: (groupid=17, jobs=1): err= 0: pid=10256
  read : io=3,235MiB, bw=110MiB/s, iops=27,603, runt= 30001msec
    slat (usec): min=7, max=273, avg=12.15, stdev= 6.57
    clat (usec): min=40, max=9,024, avg=129.19, stdev=61.94
    bw (KiB/s) : min=106176, max=113991, per=100.06%, avg=113134.92, stdev=1396.37
  cpu          : usr=9.92%, sys=43.60%, ctx=330705, majf=0, minf=16
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=828128/0, short=0/0
     lat (usec): 50=0.01%, 100=34.74%, 250=64.91%, 500=0.28%, 750=0.02%
     lat (usec): 1000=0.03%
     lat (msec): 2=0.02%, 4=0.01%, 10=0.01%
4KB_Sequential_Read: (groupid=18, jobs=1): err= 0: pid=10257
  read : io=3,245MiB, bw=111MiB/s, iops=27,690, runt= 30001msec
    slat (usec): min=7, max=293, avg=12.05, stdev= 6.94
    clat (usec): min=64, max=11,538, avg=273.15, stdev=101.30
    bw (KiB/s) : min=100458, max=115048, per=100.06%, avg=113491.71, stdev=3045.06
  cpu          : usr=9.99%, sys=44.00%, ctx=335355, majf=0, minf=20
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=830743/0, short=0/0
     lat (usec): 100=0.01%, 250=35.80%, 500=63.86%, 750=0.22%, 1000=0.02%
     lat (msec): 2=0.05%, 4=0.02%, 10=0.02%, 20=0.01%
4KB_Sequential_Read: (groupid=19, jobs=1): err= 0: pid=10258
  read : io=3,225MiB, bw=110MiB/s, iops=27,520, runt= 30001msec
    slat (usec): min=7, max=333, avg=11.73, stdev= 6.76
    clat (usec): min=39, max=16,859, avg=565.66, stdev=144.59
    bw (KiB/s) : min=95461, max=114851, per=100.04%, avg=112768.17, stdev=3989.65
  cpu          : usr=9.75%, sys=43.47%, ctx=326966, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=825638/0, short=0/0
     lat (usec): 50=0.01%, 100=0.04%, 250=0.01%, 500=0.68%, 750=97.28%
     lat (usec): 1000=1.63%
     lat (msec): 2=0.24%, 4=0.08%, 10=0.03%, 20=0.01%
4KB_Sequential_Read: (groupid=20, jobs=1): err= 0: pid=10259
  read : io=3,219MiB, bw=110MiB/s, iops=27,467, runt= 30001msec
    slat (usec): min=7, max=362, avg=11.82, stdev= 6.82
    clat (usec): min=40, max=11,055, avg=1149.27, stdev=213.30
    bw (KiB/s) : min=96829, max=114466, per=100.04%, avg=112553.86, stdev=3443.50
  cpu          : usr=9.53%, sys=43.25%, ctx=328117, majf=0, minf=44
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued r/w: total=824064/0, short=0/0
     lat (usec): 50=0.01%, 100=0.01%, 250=0.01%, 500=0.02%, 750=0.06%
     lat (usec): 1000=1.33%
     lat (msec): 2=98.01%, 4=0.46%, 10=0.11%, 20=0.01%
4KB_Sequential_Read: (groupid=21, jobs=1): err= 0: pid=10260
  read : io=3,203MiB, bw=109MiB/s, iops=27,326, runt= 30003msec
    slat (usec): min=7, max=360, avg=11.83, stdev= 6.80
    clat (usec): min=44, max=15,044, avg=2326.10, stdev=363.68
    bw (KiB/s) : min=96354, max=113729, per=100.02%, avg=111955.27, stdev=3448.84
  cpu          : usr=9.46%, sys=43.52%, ctx=318324, majf=0, minf=76
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued r/w: total=819877/0, short=0/0
     lat (usec): 50=0.01%, 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%
     lat (usec): 1000=0.01%
     lat (msec): 2=1.70%, 4=97.47%, 10=0.75%, 20=0.03%
4KB_Sequential_Write: (groupid=22, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10261
  cpu          : usr=0.00%, sys=0.00%, ctx=1, majf=0, minf=39
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/1, short=0/0

4KB_Sequential_Write: (groupid=23, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10262

  cpu          : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=39
  IO depths    : 1=50.0%, 2=50.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/2, short=0/0

4KB_Sequential_Write: (groupid=24, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10263

  cpu          : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=39
  IO depths    : 1=25.0%, 2=50.0%, 4=25.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/4, short=0/0

4KB_Sequential_Write: (groupid=25, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10264

  cpu          : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=39
  IO depths    : 1=12.5%, 2=25.0%, 4=50.0%, 8=12.5%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/8, short=0/0

4KB_Sequential_Write: (groupid=26, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10265

  cpu          : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=39
  IO depths    : 1=6.2%, 2=12.5%, 4=25.0%, 8=50.0%, 16=6.2%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/16, short=0/0

4KB_Sequential_Write: (groupid=27, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10266

  cpu          : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=39
  IO depths    : 1=3.1%, 2=6.2%, 4=12.5%, 8=25.0%, 16=50.0%, 32=3.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/32, short=0/0

4KB_Sequential_Write: (groupid=28, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10267

  cpu          : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=39
  IO depths    : 1=1.6%, 2=3.1%, 4=6.2%, 8=12.5%, 16=25.0%, 32=50.0%, >=64=1.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/64, short=0/0

Idle: (groupid=29, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10268

  cpu          : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=39
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/1, short=0/0

64KB_Sequential_Read: (groupid=30, jobs=1): err= 0: pid=10269
read : io=3,867MiB, bw=132MiB/s, iops=2,062, runt= 30001msec

    slat (usec): min=19, max=267, avg=19.82, stdev= 2.43
    clat (usec): min=170, max=7,775, avg=462.19, stdev=67.66
    bw (KiB/s) : min=130416, max=140247, per=100.06%, avg=135223.64, stdev=1249.55
  cpu          : usr=0.86%, sys=5.05%, ctx=61870, majf=0, minf=28
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=61867/0, short=0/0
     lat (usec): 250=1.50%, 500=96.45%, 750=1.97%, 1000=0.07%
     lat (msec): 2=0.01%, 4=0.01%, 10=0.01%
64KB_Sequential_Read: (groupid=31, jobs=1): err= 0: pid=10270
  read : io=7,024MiB, bw=240MiB/s, iops=3,745, runt= 30001msec
    slat (usec): min=18, max=279, avg=20.13, stdev= 3.83
    clat (usec): min=170, max=444K, avg=509.84, stdev=2124.70
    bw (KiB/s) : min=  851, max=263716, per=100.68%, avg=247160.27, stdev=37824.66
  cpu          : usr=1.41%, sys=9.04%, ctx=88506, majf=0, minf=44
  IO depths    : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=112379/0, short=0/0
     lat (usec): 250=0.01%, 500=77.32%, 750=20.94%, 1000=1.67%
     lat (msec): 2=0.05%, 4=0.01%, 10=0.01%, 20=0.01%, 250=0.01%
     lat (msec): 500=0.01%
64KB_Sequential_Read: (groupid=32, jobs=1): err= 0: pid=10271
  read : io=7,525MiB, bw=257MiB/s, iops=4,012, runt= 30002msec
    slat (usec): min=18, max=291, avg=20.35, stdev= 4.22
    clat (usec): min=162, max=12,394, avg=973.41, stdev=507.76
    bw (KiB/s) : min=257163, max=263847, per=100.04%, avg=263098.66, stdev=896.20
  cpu          : usr=1.51%, sys=9.63%, ctx=97357, majf=0, minf=76
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=120397/0, short=0/0
     lat (usec): 250=0.01%, 500=28.19%, 750=18.68%, 1000=4.03%
     lat (msec): 2=48.90%, 4=0.19%, 10=0.01%, 20=0.01%
64KB_Sequential_Read: (groupid=33, jobs=1): err= 0: pid=10272
  read : io=7,527MiB, bw=257MiB/s, iops=4,013, runt= 30003msec
    slat (usec): min=18, max=293, avg=20.13, stdev= 4.39
    clat (usec): min=158, max=5,714, avg=1969.89, stdev=1422.31
    bw (KiB/s) : min=260833, max=263454, per=100.04%, avg=263160.90, stdev=380.36
  cpu          : usr=1.46%, sys=9.37%, ctx=93458, majf=0, minf=140
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=120429/0, short=0/0
     lat (usec): 250=0.01%, 500=26.94%, 750=17.04%, 1000=1.80%
     lat (msec): 2=4.34%, 4=49.77%, 10=0.09%
64KB_Sequential_Read: (groupid=34, jobs=1): err= 0: pid=10273
  read : io=7,587MiB, bw=259MiB/s, iops=4,045, runt= 30008msec
    slat (usec): min=18, max=287, avg=20.26, stdev= 3.89
    clat (usec): min=144, max=16,453, avg=3931.35, stdev=3092.75
    bw (KiB/s) : min=261881, max=374341, per=100.06%, avg=265293.56, stdev=14523.76
  cpu          : usr=1.44%, sys=9.50%, ctx=97732, majf=0, minf=268
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=121399/0, short=0/0
     lat (usec): 250=0.04%, 500=22.24%, 750=13.03%, 1000=1.48%
     lat (msec): 2=4.85%, 4=8.61%, 10=49.73%, 20=0.02%
64KB_Sequential_Read: (groupid=35, jobs=1): err= 0: pid=10274
  read : io=7,591MiB, bw=259MiB/s, iops=4,046, runt= 30015msec
    slat (usec): min=18, max=289, avg=20.14, stdev= 3.64
    clat (usec): min=136, max=32,545, avg=7884.74, stdev=5840.24
    bw (KiB/s) : min=    0, max=384172, per=66.67%, avg=176809.00, stdev=126369.08
  cpu          : usr=1.46%, sys=9.43%, ctx=101580, majf=0, minf=524
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued r/w: total=121451/0, short=0/0
     lat (usec): 250=0.02%, 500=13.17%, 750=7.44%, 1000=1.23%
     lat (msec): 2=4.36%, 4=9.46%, 10=22.40%, 20=41.89%, 50=0.03%
64KB_Sequential_Read: (groupid=36, jobs=1): err= 0: pid=10275
  read : io=7,604MiB, bw=259MiB/s, iops=4,051, runt= 30029msec
    slat (usec): min=18, max=319, avg=20.00, stdev= 4.46
    clat (usec): min=155, max=56,162, avg=15771.63, stdev=6261.03
    bw (KiB/s) : min=    0, max=407896, per=49.18%, avg=130570.46, stdev=133920.16
  cpu          : usr=1.64%, sys=9.47%, ctx=109713, majf=0, minf=1036
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued r/w: total=121664/0, short=0/0
     lat (usec): 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
     lat (msec): 2=0.01%, 4=0.57%, 10=21.54%, 20=49.28%, 50=28.57%
     lat (msec): 100=0.01%
64KB_Sequential_Write: (groupid=37, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10278
  cpu          : usr=0.00%, sys=0.00%, ctx=1, majf=0, minf=39
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/1, short=0/0

64KB_Sequential_Write: (groupid=38, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10279

  cpu          : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=39
  IO depths    : 1=50.0%, 2=50.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/2, short=0/0

64KB_Sequential_Write: (groupid=39, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10280

  cpu          : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=40
  IO depths    : 1=25.0%, 2=50.0%, 4=25.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/4, short=0/0

64KB_Sequential_Write: (groupid=40, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10281

  cpu          : usr=0.00%, sys=0.00%, ctx=2, majf=0, minf=39
  IO depths    : 1=12.5%, 2=25.0%, 4=50.0%, 8=12.5%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/8, short=0/0

64KB_Sequential_Write: (groupid=41, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10282

  cpu          : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=39
  IO depths    : 1=6.2%, 2=12.5%, 4=25.0%, 8=50.0%, 16=6.2%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/16, short=0/0

64KB_Sequential_Write: (groupid=42, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10283

  cpu          : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=39
  IO depths    : 1=3.1%, 2=6.2%, 4=12.5%, 8=25.0%, 16=50.0%, 32=3.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/32, short=0/0

64KB_Sequential_Write: (groupid=43, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10284

  cpu          : usr=0.00%, sys=200.00%, ctx=2, majf=0, minf=39
  IO depths    : 1=1.6%, 2=3.1%, 4=6.2%, 8=12.5%, 16=25.0%, 32=50.0%, >=64=1.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/64, short=0/0

Idle: (groupid=44, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10285

  cpu          : usr=0.00%, sys=0.00%, ctx=1, majf=0, minf=39
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/1, short=0/0

512KB_Sequential_Read: (groupid=45, jobs=1): err= 0: pid=10286
read : io=7,570MiB, bw=258MiB/s, iops=504, runt= 30002msec

    slat (usec): min=105, max=477, avg=107.03, stdev= 9.12
    clat (usec): min=823, max=14,212, avg=1871.73, stdev=364.26
    bw (KiB/s) : min=256901, max=346030, per=100.10%, avg=264824.61, stdev=11162.12
  cpu          : usr=0.21%, sys=5.54%, ctx=15142, majf=0, minf=140
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=15139/0, short=0/0
     lat (usec): 1000=1.45%
     lat (msec): 2=55.88%, 4=42.61%, 10=0.04%, 20=0.02%
512KB_Sequential_Read: (groupid=46, jobs=1): err= 0: pid=10287
  read : io=7,599MiB, bw=259MiB/s, iops=506, runt= 30004msec
    slat (usec): min=105, max=522, avg=108.47, stdev= 7.84
    clat (usec): min=874, max=7,918, avg=3836.69, stdev=368.62
    bw (KiB/s) : min=260574, max=415236, per=100.11%, avg=265856.08, stdev=19788.90
  cpu          : usr=0.20%, sys=5.65%, ctx=15201, majf=0, minf=268
  IO depths    : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=15198/0, short=0/0
     lat (usec): 1000=0.01%
     lat (msec): 2=1.47%, 4=97.03%, 10=1.50%
512KB_Sequential_Read: (groupid=47, jobs=1): err= 0: pid=10288
  read : io=7,605MiB, bw=260MiB/s, iops=506, runt= 30007msec
    slat (usec): min=107, max=522, avg=110.17, stdev=10.07
    clat (usec): min=973, max=15,177, avg=7778.19, stdev=724.03
    bw (KiB/s) : min=259527, max=424673, per=100.11%, avg=266016.07, stdev=21020.07
  cpu          : usr=0.21%, sys=5.76%, ctx=15210, majf=0, minf=524
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=15209/0, short=0/0
     lat (usec): 1000=0.01%
     lat (msec): 2=0.09%, 4=1.39%, 10=98.45%, 20=0.06%
512KB_Sequential_Read: (groupid=48, jobs=1): err= 0: pid=10289
  read : io=7,609MiB, bw=260MiB/s, iops=507, runt= 30015msec
    slat (usec): min=107, max=529, avg=110.35, stdev=11.99
    clat (msec): min=1, max=27, avg=15.66, stdev= 1.39
    bw (KiB/s) : min=258998, max=431151, per=100.10%, avg=266098.98, stdev=21867.96
  cpu          : usr=0.21%, sys=5.77%, ctx=15220, majf=0, minf=1036
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=15218/0, short=0/0

     lat (msec): 2=0.02%, 4=0.04%, 10=1.47%, 20=98.44%, 50=0.03%
512KB_Sequential_Read: (groupid=49, jobs=1): err= 0: pid=10290
  read : io=7,619MiB, bw=260MiB/s, iops=507, runt= 30032msec
    slat (usec): min=97, max=538, avg=106.09, stdev=16.14
    clat (usec): min=736, max=1,524K, avg=31420.02, stdev=12463.84
    bw (KiB/s) : min=261620, max=441450, per=100.11%, avg=266291.53, stdev=23200.48
  cpu          : usr=0.22%, sys=5.58%, ctx=15216, majf=0, minf=2061
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=15237/0, short=0/0
     lat (usec): 750=0.01%
     lat (msec): 2=0.03%, 4=0.13%, 10=1.30%, 20=0.09%, 50=98.39%
     lat (msec): 100=0.03%, 2000=0.01%
512KB_Sequential_Read: (groupid=50, jobs=1): err= 0: pid=10291
  read : io=7,478MiB, bw=255MiB/s, iops=497, runt= 30064msec
    slat (usec): min=99, max=791, avg=104.88, stdev=22.09
    clat (usec): min=649, max=188K, avg=64193.71, stdev=13521.52
    bw (KiB/s) : min=126877, max=466616, per=100.09%, avg=261039.81, stdev=37852.91
  cpu          : usr=0.21%, sys=5.41%, ctx=14938, majf=0, minf=4108
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.8%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued r/w: total=14956/0, short=0/0
     lat (usec): 750=0.02%, 1000=0.01%
     lat (msec): 2=0.03%, 4=0.08%, 10=0.45%, 20=0.90%, 50=0.12%
     lat (msec): 100=97.17%, 250=1.22%
512KB_Sequential_Read: (groupid=51, jobs=1): err= 0: pid=10292
  read : io=7,468MiB, bw=254MiB/s, iops=495, runt= 30126msec
    slat (usec): min=59, max=158K, avg=1698.92, stdev=6183.61
    clat (msec): min=1, max=393, avg=127.31, stdev=23.65
    bw (KiB/s) : min=125664, max=474748, per=100.06%, avg=260087.88, stdev=40168.34
  cpu          : usr=0.18%, sys=5.03%, ctx=3251, majf=0, minf=8203
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued r/w: total=14936/0, short=0/0

     lat (msec): 2=0.02%, 4=0.02%, 10=0.21%, 20=0.23%, 50=1.00%
     lat (msec): 100=1.31%, 250=96.85%, 500=0.35%
512KB_Sequential_Write: (groupid=52, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10294
  cpu          : usr=0.00%, sys=0.00%, ctx=1, majf=0, minf=39
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/1, short=0/0

512KB_Sequential_Write: (groupid=53, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10295
write: io=512KiB, bw=256MiB/s, iops=1,000, runt= 2msec

    slat (usec): min=12, max=118, avg=65.00, stdev=74.95
    clat (usec): min=1,043, max=1,043, avg=1043.00, stdev= 0.00
  cpu          : usr=0.00%, sys=0.00%, ctx=1, majf=0, minf=40
  IO depths    : 1=50.0%, 2=50.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=33.3%, 4=66.7%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/2, short=0/0

     lat (msec): 2=50.00%
512KB_Sequential_Write: (groupid=54, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10296
  write: io=1,536KiB, bw=512MiB/s, iops=1,333, runt=     3msec
    slat (usec): min=12, max=116, avg=85.50, stdev=49.21
    clat (usec): min=2,547, max=2,765, avg=2655.33, stdev=109.01
  cpu          : usr=0.00%, sys=0.00%, ctx=4, majf=0, minf=40
  IO depths    : 1=25.0%, 2=50.0%, 4=25.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=33.3%, 4=66.7%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/4, short=0/0

     lat (msec): 4=75.00%
512KB_Sequential_Write: (groupid=55, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10297
  write: io=3,584KiB, bw=149MiB/s, iops=333, runt=    24msec
    slat (usec): min=16, max=123, avg=100.12, stdev=34.34
    clat (usec): min=22,430, max=23,107, avg=22766.14, stdev=244.35
  cpu          : usr=0.00%, sys=4.35%, ctx=8, majf=0, minf=40
  IO depths    : 1=12.5%, 2=25.0%, 4=50.0%, 8=12.5%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=33.3%, 4=33.3%, 8=33.3%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/8, short=0/0

     lat (msec): 50=87.50%
512KB_Sequential_Write: (groupid=56, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10298
  cpu          : usr=0.00%, sys=100.00%, ctx=1, majf=0, minf=39
  IO depths    : 1=6.2%, 2=12.5%, 4=25.0%, 8=50.0%, 16=6.2%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/16, short=0/0

512KB_Sequential_Write: (groupid=57, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10299

  cpu          : usr=0.00%, sys=66.67%, ctx=1, majf=0, minf=39
  IO depths    : 1=3.1%, 2=6.2%, 4=12.5%, 8=25.0%, 16=50.0%, 32=3.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/32, short=0/0

512KB_Sequential_Write: (groupid=58, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10300

  cpu          : usr=0.00%, sys=14.29%, ctx=3, majf=0, minf=39
  IO depths    : 1=1.6%, 2=3.1%, 4=6.2%, 8=12.5%, 16=25.0%, 32=50.0%, >=64=1.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/64, short=0/0

Idle: (groupid=59, jobs=1): err= 5 (file:io_u.c:879, func=io_u error, error=Input/output error): pid=10301

  cpu          : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=39
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/1, short=0/0

1MB_Sequential_Read: (groupid=60, jobs=1): err= 0: pid=10302
read : io=7,590MiB, bw=259MiB/s, iops=252, runt= 30004msec

    slat (usec): min=206, max=1,027, avg=208.98, stdev=12.66
    clat (usec): min=1,387, max=9,705, avg=3740.99, stdev=356.79
    bw (KiB/s) : min=259010, max=398458, per=100.09%, avg=265490.02, stdev=17619.44
  cpu          : usr=0.10%, sys=5.37%, ctx=7592, majf=0, minf=268
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=7590/0, short=0/0

     lat (msec): 2=1.46%, 4=97.11%, 10=1.42%
1MB_Sequential_Read: (groupid=61, jobs=1): err= 0: pid=10303
  read : io=7,602MiB, bw=259MiB/s, iops=253, runt= 30006msec
    slat (usec): min=205, max=1,049, avg=209.13, stdev=17.17
    clat (usec): min=1,324, max=11,399, avg=7681.35, stdev=691.71
    bw (KiB/s) : min=261099, max=419430, per=100.10%, avg=265916.47, stdev=20333.78
  cpu          : usr=0.11%, sys=5.37%, ctx=7603, majf=0, minf=524
  IO depths    : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=7602/0, short=0/0

     lat (msec): 2=0.07%, 4=1.41%, 10=98.50%, 20=0.03%
1MB_Sequential_Read: (groupid=62, jobs=1): err= 0: pid=10304
  read : io=7,606MiB, bw=259MiB/s, iops=253, runt= 30015msec
    slat (usec): min=205, max=1,046, avg=210.28, stdev=21.76
    clat (msec): min=1, max=23, avg=15.57, stdev= 1.34
    bw (KiB/s) : min=261099, max=423182, per=100.09%, avg=265944.44, stdev=20826.31
  cpu          : usr=0.10%, sys=5.44%, ctx=7608, majf=0, minf=1036
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=7606/0, short=0/0

     lat (msec): 2=0.01%, 4=0.04%, 10=1.46%, 20=98.47%, 50=0.01%
1MB_Sequential_Read: (groupid=63, jobs=1): err= 0: pid=10305
  read : io=7,617MiB, bw=260MiB/s, iops=253, runt= 30032msec
    slat (usec): min=194, max=1,078, avg=207.05, stdev=29.76
    clat (msec): min=1, max=55, avg=31.32, stdev= 2.91
    bw (KiB/s) : min=260580, max=438304, per=100.10%, avg=266227.58, stdev=22792.71
  cpu          : usr=0.10%, sys=5.36%, ctx=7616, majf=0, minf=2060
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=7617/0, short=0/0

     lat (msec): 2=0.04%, 4=0.14%, 10=1.25%, 20=0.12%, 50=98.42%
     lat (msec): 100=0.03%
1MB_Sequential_Read: (groupid=64, jobs=1): err= 0: pid=10306
  read : io=7,638MiB, bw=260MiB/s, iops=254, runt= 30062msec
    slat (usec): min=192, max=1,112, avg=202.62, stdev=41.97
    clat (msec): min=1, max=119, avg=62.74, stdev= 6.79
    bw (KiB/s) : min=259527, max=466960, per=100.09%, avg=266668.73, stdev=26530.70
  cpu          : usr=0.11%, sys=5.18%, ctx=7621, majf=0, minf=4108
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=7638/0, short=0/0

     lat (msec): 2=0.03%, 4=0.18%, 10=0.45%, 20=0.84%, 50=0.29%
     lat (msec): 100=98.15%, 250=0.07%
1MB_Sequential_Read: (groupid=65, jobs=1): err= 0: pid=10307
  read : io=7,652MiB, bw=260MiB/s, iops=254, runt= 30125msec
    slat (usec): min=111, max=38,547, avg=2527.24, stdev=6275.75
    clat (msec): min=1, max=241, avg=123.36, stdev=17.70
    bw (KiB/s) : min=    0, max=480247, per=66.13%, avg=176143.49, stdev=128614.59
  cpu          : usr=0.08%, sys=4.87%, ctx=4642, majf=0, minf=8204
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.6%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued r/w: total=7652/0, short=0/0

     lat (msec): 2=0.04%, 4=0.12%, 10=0.16%, 20=0.13%, 50=1.10%
     lat (msec): 100=2.35%, 250=96.11%
1MB_Sequential_Read: (groupid=66, jobs=1): err= 0: pid=10308
  read : io=7,638MiB, bw=260MiB/s, iops=253, runt= 30089msec
    slat (usec): min=111, max=61,024, avg=3923.41, stdev=8579.21
    clat (msec): min=51, max=335, avg=247.61, stdev=23.33
    bw (KiB/s) : min=192472, max=448512, per=99.57%, avg=265028.88, stdev=26072.12
  cpu          : usr=0.08%, sys=4.09%, ctx=1470, majf=0, minf=16396
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.2%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued r/w: total=7638/0, short=0/0

     lat (msec): 100=0.94%, 250=15.29%, 500=83.77%

Run status group 0 (all jobs):
READ: io=294MiB, aggrb=10,279KiB/s, minb=10,279KiB/s, maxb=10,279KiB/s, mint=30001msec, maxt=30001msec

Run status group 1 (all jobs):
READ: io=416MiB, aggrb=14,546KiB/s, minb=14,546KiB/s, maxb=14,546KiB/s, mint=30001msec, maxt=30001msec

Run status group 2 (all jobs):
READ: io=537MiB, aggrb=18,752KiB/s, minb=18,752KiB/s, maxb=18,752KiB/s, mint=30001msec, maxt=30001msec

Run status group 3 (all jobs):
READ: io=543MiB, aggrb=18,976KiB/s, minb=18,976KiB/s, maxb=18,976KiB/s, mint=30001msec, maxt=30001msec

Run status group 4 (all jobs):
READ: io=543MiB, aggrb=18,976KiB/s, minb=18,976KiB/s, maxb=18,976KiB/s, mint=30001msec, maxt=30001msec

Run status group 5 (all jobs):
READ: io=543MiB, aggrb=18,977KiB/s, minb=18,977KiB/s, maxb=18,977KiB/s, mint=30001msec, maxt=30001msec

Run status group 6 (all jobs):
READ: io=543MiB, aggrb=18,992KiB/s, minb=18,992KiB/s, maxb=18,992KiB/s, mint=30001msec, maxt=30001msec

Run status group 7 (all jobs):

Run status group 8 (all jobs):

Run status group 9 (all jobs):

Run status group 10 (all jobs):

Run status group 11 (all jobs):

Run status group 12 (all jobs):

Run status group 13 (all jobs):

Run status group 14 (all jobs):

Run status group 15 (all jobs):
READ: io=2,041MiB, aggrb=71,333KiB/s, minb=71,333KiB/s, maxb=71,333KiB/s, mint=30000msec, maxt=30000msec

Run status group 16 (all jobs):
READ: io=2,694MiB, aggrb=94,155KiB/s, minb=94,155KiB/s, maxb=94,155KiB/s, mint=30001msec, maxt=30001msec

Run status group 17 (all jobs):
READ: io=3,235MiB, aggrb=110MiB/s, minb=110MiB/s, maxb=110MiB/s, mint=30001msec, maxt=30001msec

Run status group 18 (all jobs):
READ: io=3,245MiB, aggrb=111MiB/s, minb=111MiB/s, maxb=111MiB/s, mint=30001msec, maxt=30001msec

Run status group 19 (all jobs):
READ: io=3,225MiB, aggrb=110MiB/s, minb=110MiB/s, maxb=110MiB/s, mint=30001msec, maxt=30001msec

Run status group 20 (all jobs):
READ: io=3,219MiB, aggrb=110MiB/s, minb=110MiB/s, maxb=110MiB/s, mint=30001msec, maxt=30001msec

Run status group 21 (all jobs):
READ: io=3,203MiB, aggrb=109MiB/s, minb=109MiB/s, maxb=109MiB/s, mint=30003msec, maxt=30003msec

Run status group 22 (all jobs):

Run status group 23 (all jobs):

Run status group 24 (all jobs):

Run status group 25 (all jobs):

Run status group 26 (all jobs):

Run status group 27 (all jobs):

Run status group 28 (all jobs):

Run status group 29 (all jobs):

Run status group 30 (all jobs):
READ: io=3,867MiB, aggrb=132MiB/s, minb=132MiB/s, maxb=132MiB/s, mint=30001msec, maxt=30001msec

Run status group 31 (all jobs):
READ: io=7,024MiB, aggrb=240MiB/s, minb=240MiB/s, maxb=240MiB/s, mint=30001msec, maxt=30001msec

Run status group 32 (all jobs):
READ: io=7,525MiB, aggrb=257MiB/s, minb=257MiB/s, maxb=257MiB/s, mint=30002msec, maxt=30002msec

Run status group 33 (all jobs):
READ: io=7,527MiB, aggrb=257MiB/s, minb=257MiB/s, maxb=257MiB/s, mint=30003msec, maxt=30003msec

Run status group 34 (all jobs):
READ: io=7,587MiB, aggrb=259MiB/s, minb=259MiB/s, maxb=259MiB/s, mint=30008msec, maxt=30008msec

Run status group 35 (all jobs):
READ: io=7,591MiB, aggrb=259MiB/s, minb=259MiB/s, maxb=259MiB/s, mint=30015msec, maxt=30015msec

Run status group 36 (all jobs):
READ: io=7,604MiB, aggrb=259MiB/s, minb=259MiB/s, maxb=259MiB/s, mint=30029msec, maxt=30029msec

Run status group 37 (all jobs):

Run status group 38 (all jobs):

Run status group 39 (all jobs):

Run status group 40 (all jobs):

Run status group 41 (all jobs):

Run status group 42 (all jobs):

Run status group 43 (all jobs):

Run status group 44 (all jobs):

Run status group 45 (all jobs):
READ: io=7,570MiB, aggrb=258MiB/s, minb=258MiB/s, maxb=258MiB/s, mint=30002msec, maxt=30002msec

Run status group 46 (all jobs):
READ: io=7,599MiB, aggrb=259MiB/s, minb=259MiB/s, maxb=259MiB/s, mint=30004msec, maxt=30004msec

Run status group 47 (all jobs):
READ: io=7,605MiB, aggrb=260MiB/s, minb=260MiB/s, maxb=260MiB/s, mint=30007msec, maxt=30007msec

Run status group 48 (all jobs):
READ: io=7,609MiB, aggrb=260MiB/s, minb=260MiB/s, maxb=260MiB/s, mint=30015msec, maxt=30015msec

Run status group 49 (all jobs):
READ: io=7,619MiB, aggrb=260MiB/s, minb=260MiB/s, maxb=260MiB/s, mint=30032msec, maxt=30032msec

Run status group 50 (all jobs):
READ: io=7,478MiB, aggrb=255MiB/s, minb=255MiB/s, maxb=255MiB/s, mint=30064msec, maxt=30064msec

Run status group 51 (all jobs):
READ: io=7,468MiB, aggrb=254MiB/s, minb=254MiB/s, maxb=254MiB/s, mint=30126msec, maxt=30126msec

Run status group 52 (all jobs):

Run status group 53 (all jobs):
WRITE: io=512KiB, aggrb=256MiB/s, minb=256MiB/s, maxb=256MiB/s, mint=2msec, maxt=2msec

Run status group 54 (all jobs):
WRITE: io=1,536KiB, aggrb=512MiB/s, minb=512MiB/s, maxb=512MiB/s, mint=3msec, maxt=3msec

Run status group 55 (all jobs):
WRITE: io=3,584KiB, aggrb=149MiB/s, minb=149MiB/s, maxb=149MiB/s, mint=24msec, maxt=24msec

Run status group 56 (all jobs):

Run status group 57 (all jobs):

Run status group 58 (all jobs):

Run status group 59 (all jobs):

Run status group 60 (all jobs):
READ: io=7,590MiB, aggrb=259MiB/s, minb=259MiB/s, maxb=259MiB/s, mint=30004msec, maxt=30004msec

Run status group 61 (all jobs):
READ: io=7,602MiB, aggrb=259MiB/s, minb=259MiB/s, maxb=259MiB/s, mint=30006msec, maxt=30006msec

Run status group 62 (all jobs):
READ: io=7,606MiB, aggrb=259MiB/s, minb=259MiB/s, maxb=259MiB/s, mint=30015msec, maxt=30015msec

Run status group 63 (all jobs):
READ: io=7,617MiB, aggrb=260MiB/s, minb=260MiB/s, maxb=260MiB/s, mint=30032msec, maxt=30032msec

Run status group 64 (all jobs):
READ: io=7,638MiB, aggrb=260MiB/s, minb=260MiB/s, maxb=260MiB/s, mint=30062msec, maxt=30062msec

Run status group 65 (all jobs):
READ: io=7,652MiB, aggrb=260MiB/s, minb=260MiB/s, maxb=260MiB/s, mint=30125msec, maxt=30125msec

Run status group 66 (all jobs):
READ: io=7,638MiB, aggrb=260MiB/s, minb=260MiB/s, maxb=260MiB/s, mint=30089msec, maxt=30089msec

Disk stats (read/write):
cciss!c1d0: ios=13674968/445, merge=21891/0, ticks=32034605/16709, in_queue=32049193, util=96.88%

That's pretty strange. Your test job(s) seem pretty massive, you don't happen to have a cut down version that exhibits the problem?

There also seems to be parts of the error missing from io_u_log_error(), like the file associated with the error and the io_u details (offset= and buflen=, for instance).

What version of fio are you using?

Jens Axboe

Hi Jens,
This build is the latest build after version 1.24. This build was the outcome of the request for adding blockalign/ba option. Can you please help in resolving this issue of sequential writes on ext3 file system?

Thanks,
Girish

I'd love to help you, but you have to help me a bit as well. I asked you for a job file that reproduces the problem. If you can send me such a job file (preferably cut down to just a single section that exhibits the problem), then I can probably find the error quickly.

Jens Axboe

Hi Jens,
Here is the job file. I am facing problem with Sequential Writes on ext3 file system.
Thanks,
 Girish

--- On Mon, 4/20/09, Jens Axboe <[email protected]> wrote:

From: Jens Axboe <[email protected]>
Subject: Re: Recent changes
To: "Girish Satihal" <[email protected]>
Cc: [email protected]
Date: Monday, April 20, 2009, 11:35 PM

I'd love to help you, but you have to help me a bit as well. I asked you for a job file that reproduces the problem. If you can send me such a job file (preferably cut down to just a single section that exhibits the problem), then I can probably find the error quickly.

Jens Axboe

--
To unsubscribe from this list: send the line "unsubscribe fio" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

You are starting to annoy me a bit... Are you reading my emails? What did I ask you for? A cut down version that exhibits the problem. You are using a piece of free software and expecting help/support with your issue, the VERY least you can do is pay a bit of attention and provide what is asked (or at least say WHY you didn't provide a cut down version, perhaps you couldn't get it working etc).

Jens Axboe

Your job file works fine for me. I'm not sure why you are talking about ext3, your job file works on the full device, not a file system.

Jens Axboe

Hi Jens,
I guess I did not understand your question for which I am very sorry.

As per my understanding I have sent you the job file which was input file to fio
for the performance testing.

As only the problem with Seqential Writes, I have cut down the job file to only
Write parameters where the issue is being seen. The problem is seen with all block size and IO depths of Sequential Writes.
Please let me know if this would help you or still you are looking for anything else.

Thanks,
 Girish

--- On Tue, 4/21/09, Jens Axboe <[email protected]> wrote:

From: Jens Axboe <[email protected]>
Subject: Re: Recent changes
To: "Girish Satihal" <[email protected]>
Cc: [email protected]
Date: Tuesday, April 21, 2009, 1:17 PM

Your job file works fine for me. I'm not sure why you are talking about ext3, your job file works on the full device, not a file system.

was the

please

system?

asked you

the

<[email protected]>

fio" in

in

Jens Axboe

--
To unsubscribe from this list: send the line "unsubscribe fio" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Don't apologize for that. But if there's something I write you don't understand, do just ask for an explanation.

So my current question is why you are mentioning ext3? The test case you provided is reading and writing directly to the CCISS device, not a file inside a file system hosted on that device.

Also, please try a make clean && make && make install in your fio directory. Sometimes things get hosed when updates are pulled, which may explain why yours is acting up.

Jens Axboe

Hi Jens,
The reason for saying ext3, is I run fdisk on /dev/cciss/c1d0 and created a primary partition.

Then I created a ext3 filesystem on this partition by running mkefs and mounted this partition on /mnt folder. Then I run fio on this partition.

We earlier use to run fio on raw partition. But now we tried running fio on a partition which has ext3 filesystem.

When I run fio on raw filesystem I dont get any errors for Sequential Writes, but when I run fio test on this ext3 filesystem then I am getting the errors for Sequential Writes what I mentioned u earlier.
Thanks,
 Girish

--- On Wed, 4/22/09, Jens Axboe <[email protected]> wrote:

From: Jens Axboe <[email protected]>
Subject: Re: Recent changes
To: "Girish Satihal" <[email protected]>
Cc: [email protected]
Date: Wednesday, April 22, 2009, 12:45 AM

Don't apologize for that. But if there's something I write you don't
understand, do just ask for an explanation.

So my current question is why you are mentioning ext3? The test case you provided is reading and writing directly to the CCISS device, not a file inside a file system hosted on that device.

Also, please try a make clean && make && make install in your fio
directory. Sometimes things get hosed when updates are pulled, which may explain why yours is acting up.

Jens Axboe

You run fio on the partition, or on the mount point? Your fio job file indicates that you run fio on the full device, not even just the partition. This has _nothing_ to do with ext3. Irregardless of whether you run fio on the partition or the device, your file system will be hosed. Perhaps things go bad because ext3 turns your device read-only on errors. Did you get any errors logged in dmesg?

Why would the content of the device matter?

Lets back up a bit... What are you trying to test? Running fio on a device/partition that is mounted with a file system is just a crazy thing to do. Why would you do that?

Jens Axboe

Girish,

you run it on disk directly(/dev/cciss/c1d0) without mounting it on /mnt.

Thanks,
-Guru

Hi Jens,
Running the fio on the partition mounted with filesystem was the intention to compare the functinality of fio with other performance benchmarking tool called IOmeter which is used for Windows. We use two different performance test tools, one is IOmeter for Windows and fio for Linux.

Also, recently we got a customer issue who had very bad performance numbers for the test they conducted on ext2 filesystem and wanted us to try the same. They dont use fio. They have thier own proprietery tool for performance measure. So it was just an investigation for us to run fio on partition with filesystem instead of keep it raw. Since we did not find any problem with Random Writes but only problem with Seqential Writes thought of reporting to you and get the solution.
I am not bothered about the contents of the device. I was just wanted to compare the performance numbers between the Raw partition and the filesystem partition.

If you want I can look into the erroe messages logged in dmesg.


Thanks,
 Girish

--- On Wed, 4/22/09, Jens Axboe <[email protected]> wrote:

From: Jens Axboe <[email protected]>
Subject: Re: Recent changes
To: "Girish Satihal" <[email protected]>
Cc: [email protected]
Date: Wednesday, April 22, 2009, 11:16 AM

a primary partition.

mounted this partition on /mnt folder. Then I run fio on this partition.

You run fio on the partition, or on the mount point? Your fio job file indicates that you run fio on the full device, not even just the partition. This has _nothing_ to do with ext3. Irregardless of whether you run fio on the partition or the device, your file system will be hosed. Perhaps things go bad because ext3 turns your device read-only on errors. Did you get any errors logged in dmesg?

on a partition which has ext3 filesystem.

Why would the content of the device matter?

Writes, but when I run fio test on this ext3 filesystem then I am getting the errors for Sequential Writes what I mentioned u earlier.

Lets back up a bit... What are you trying to test? Running fio on a device/partition that is mounted with a file system is just a crazy thing to do. Why would you do that?

all

your

Jens Axboe

--
To unsubscribe from this list: send the line "unsubscribe fio" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

OK, but that doesn't make a lot of sense. The file system is going to be very unhappy if you scribble random data to the device while it is mounted. So either you should keep the device unmounted and run the test on the raw device (or partition), OR you mount the device and run the tests on files hosted on that device. With fio, that would either be:

filename=/dev/cciss/c0d0p1

OR

directory=/where/c0d0p1/is/mounted

Thanks for reporting, you should always do that. I'm just trying to make sense of what you want to accomplish.

Typically the only difference will be if the device defaults to a small block sizes. For buffered IO, one may find that the umounted partition has 1kb default block size, while once it's mounted, the file system will set the soft block size to it's own size (typically 4kb). So if that makes your difference, you should just mount and immediately umount the partition. That will accomplish the same, and you don't risk panic'ing the file system due to it becoming corrupt while you run the test.

Please do!

Jens Axboe