IB/srp: use multiple CPU cores more effectively

SRP I/O with small block sizes causes a high CPU load. Processing IB completions on the context of a kernel thread instead of in interrupt context allows to process up to 25% more I/O operations per second. This patch does add a kernel parameter 'thread' that allows to specify whether to process IB completions in interrupt context or in kernel thread context. Also, the IB receive notification processing loop is rewritten as proposed earlier by Ralph Campbell (see also https://patchwork.kernel.org/patch/89426/). As the measurement results below show, rewriting the IB receive notification processing loop did not have a measurable impact on performance. Processing IB receive notifications in thread context however does have a measurable
impact: workloads with I/O depth one are processed at most 10% slower and workloads with larger I/O depths are processed up to 25% faster.

block size number of IOPS IOPS IOPS
in bytes threads without with with
($bs) ($numjobs) this patch thread=n thread=y

    512           1        25,400      25,400    23,100
    512         128       122,000     122,000   153,000
   4096           1        25,000      25,000    22,700
   4096         128       122,000     121,000   157,000
  65536           1        14,300      14,400    13,600
  65536           4        36,700      36,700    36,600
524288           1         3,470       3,430     3,420
524288           4         5,020       5,020     4,990

performance test used to gather the above results:

   fio --bs=${bs} --ioengine=sg --buffered=0 --size=128M --rw=read \
       --thread --numjobs=${numjobs} --loops=100 --group_reporting \
       --gtod_reduce=1 --name=${dev} --filename=${dev}

other ib_srp kernel module parameters: srp_sg_tablesize=128

Quoted 26 lines · Bart Van Assche, on 08/02/2010 12:15 PM wrote:

SRP I/O with small block sizes causes a high CPU load. Processing IB completions on the context of a kernel thread instead of in interrupt context allows to process up to 25% more I/O operations per second. This patch does add a kernel parameter 'thread' that allows to specify whether to process IB completions in interrupt context or in kernel thread context. Also, the IB receive notification processing loop is rewritten as proposed earlier by Ralph Campbell (see also https://patchwork.kernel.org/patch/89426/). As the measurement results below show, rewriting the IB receive notification processing loop did not have a measurable impact on performance. Processing IB receive notifications in thread context however does have a measurable
impact: workloads with I/O depth one are processed at most 10% slower and workloads with larger I/O depths are processed up to 25% faster.

block size  number of    IOPS        IOPS      IOPS in bytes    threads     without     with      with ($bs)     ($numjobs)  this patch  thread=n  thread=y 512           1        25,400      25,400    23,100 512         128       122,000     122,000   153,000 4096           1        25,000      25,000    22,700 4096         128       122,000     121,000   157,000 65536           1        14,300      14,400    13,600 65536           4        36,700      36,700    36,600 524288           1         3,470       3,430     3,420 524288           4         5,020       5,020     4,990

Quoted 5 lines

performance test used to gather the above results:
  fio --bs=${bs} --ioengine=sg --buffered=0 --size=128M --rw=read \
      --thread --numjobs=${numjobs} --loops=100 --group_reporting \
      --gtod_reduce=1 --name=${dev} --filename=${dev} other ib_srp kernel module parameters: srp_sg_tablesize=128

How about results of "dd Xflags=direct" in different modes to find out the lowest latency the driver can process 512 and 4K packets? Sorry, I don't trust fio, when it comes to precise latency measurements.

Quoted 31 lines

Quoted 25 lines

SRP I/O with small block sizes causes a high CPU load. Processing IB completions on the context of a kernel thread instead of in interrupt context allows to process up to 25% more I/O operations per second. This patch does add a kernel parameter 'thread' that allows to specify whether to process IB completions in interrupt context or in kernel thread context. Also, the IB receive notification processing loop is rewritten as proposed earlier by Ralph Campbell (see also https://patchwork.kernel.org/patch/89426/). As the measurement results below show, rewriting the IB receive notification processing loop did not have a measurable impact on performance. Processing IB receive notifications in thread context however does have a measurable
impact: workloads with I/O depth one are processed at most 10% slower and workloads with larger I/O depths are processed up to 25% faster.

block size number of IOPS IOPS IOPS
in bytes threads without with with
($bs) ($numjobs) this patch thread=n thread=y

    512           1        25,400      25,400    23,100
    512         128       122,000     122,000   153,000
   4096           1        25,000      25,000    22,700
   4096         128       122,000     121,000   157,000
  65536           1        14,300      14,400    13,600
  65536           4        36,700      36,700    36,600
524288           1         3,470       3,430     3,420
524288           4         5,020       5,020     4,990

Quoted 5 lines

performance test used to gather the above results:

   fio --bs=${bs} --ioengine=sg --buffered=0 --size=128M --rw=read \
       --thread --numjobs=${numjobs} --loops=100 --group_reporting \
       --gtod_reduce=1 --name=${dev} --filename=${dev}

other ib_srp kernel module parameters: srp_sg_tablesize=128

Quoted 3 lines

How about results of "dd Xflags=direct" in different modes to find out the lowest latency the driver can process 512 and 4K packets? Sorry, I don't trust fio, when it comes to precise latency measurements.

It would be interesting to compare such results, but unfortunately, dd does not provide a way to perform I/O from multiple threads simultaneously. I have tried to run multiple dd processes in parallel, but that resulted in much lower IOPS results than a comparable multithreaded fio test.

Quoted 12 lines · Bart Van Assche, on 08/02/2010 07:57 PM wrote:

Quoted 12 lines

Quoted 4 lines

block size number of IOPS IOPS IOPS
in bytes threads without with with
($bs) ($numjobs) this patch thread=n thread=y

Quoted 8 lines

    512           1        25,400      25,400    23,100
    512         128       122,000     122,000   153,000
   4096           1        25,000      25,000    22,700
   4096         128       122,000     121,000   157,000
  65536           1        14,300      14,400    13,600
  65536           4        36,700      36,700    36,600
524288           1         3,470       3,430     3,420
524288           4         5,020       5,020     4,990

I'm interested to see how much your changes affected processing latency, i.e. to measure execution latency before and after changes. You can't do that with several threads, because latency = 1/bandwidth only if you always have only one command at time. So, all those sophisticated measurements can't substitute a plane old:

Quoted 14 lines · Vladislav Bolkhovitin

Quoted 12 lines · Bart Van Assche, on 08/02/2010 07:57 PM wrote:

Quoted 12 lines

Quoted 12 lines

block size  number of    IOPS        IOPS      IOPS in bytes    threads     without     with      with ($bs)     ($numjobs)  this patch  thread=n  thread=y 512           1        25,400      25,400    23,100 512         128       122,000     122,000   153,000 4096           1        25,000      25,000    22,700 4096         128       122,000     121,000   157,000 65536           1        14,300      14,400    13,600 65536           4        36,700      36,700    36,600 524288           1         3,470       3,430     3,420 524288           4         5,020       5,020     4,990

Quoted 5 lines

I'm interested to see how much your changes affected processing latency, i.e. to measure execution latency before and after changes. You can't do that with several threads, because latency = 1/bandwidth only if you always have only one command at time. So, all those sophisticated measurements can't substitute a plane old:

If my assumption that --numjobs=1 puts fio into a single-threaded mode is correct, it seems that using this patch hurts individual command latency, at least in a gross sense. The table listed above shows a ~9% hit for single-threaded 0.5 KB and 4 KB requests, ~4.8% for 64 KB requests, and ~1.4% for 512 KB requests. It seems to win @ lots of requests and small block sizes, but still seems to hurt performance at larger request sizes, though it seems they were tested with smaller thread counts.

I've not reviewed the patch yet, but that's how I read the table above. I'm assuming latency is hurt by the need to schedule the kernel thread, but the batching helps increase the IOPS for low request sizes.

Bart, you could also try xdd as a benchmark tool.

Quoted 31 lines · David Dillow

Quoted 14 lines · Vladislav Bolkhovitin

Quoted 12 lines · Bart Van Assche, on 08/02/2010 07:57 PM wrote:

Quoted 12 lines

Quoted 12 lines

block size number of IOPS IOPS IOPS
in bytes threads without with with

    ($bs)     ($numjobs)  this patch  thread=n  thread=y
     512           1        25,400      25,400    23,100
     512         128       122,000     122,000   153,000
    4096           1        25,000      25,000    22,700
    4096         128       122,000     121,000   157,000
   65536           1        14,300      14,400    13,600
   65536           4        36,700      36,700    36,600
524288           1         3,470       3,430     3,420
524288           4         5,020       5,020     4,990

Quoted 5 lines

I'm interested to see how much your changes affected processing latency, i.e. to measure execution latency before and after changes. You can't do that with several threads, because latency = 1/bandwidth only if you always have only one command at time. So, all those sophisticated measurements can't substitute a plane old:

If my assumption that --numjobs=1 puts fio into a single-threaded mode is correct, it seems that using this patch hurts individual command latency, at least in a gross sense. The table listed above shows a ~9% hit for single-threaded 0.5 KB and 4 KB requests, ~4.8% for 64 KB requests, and ~1.4% for 512 KB requests. It seems to win @ lots of requests and small block sizes, but still seems to hurt performance at larger request sizes, though it seems they were tested with smaller thread counts.

Quoted 3 lines

I've not reviewed the patch yet, but that's how I read the table above. I'm assuming latency is hurt by the need to schedule the kernel thread, but the batching helps increase the IOPS for low request sizes.

Please note that the user has to enable mode thread=y explicitly. The default mode is thread=n and in that mode neither latency nor throughput is affected by this patch.

Quoted 1 line

Bart, you could also try xdd as a benchmark tool.

I'm familiar with xdd. However, I consider fio both as more powerful and easier to user than xdd.