ISCSI-SCST performance (with also IET and STGT data)

Quoted 18 lines · Vladislav Bolkhovitin

As part of 1.0.1 release preparations I made some performance tests to make sure there are no performance regressions in SCST overall and iSCSI-SCST particularly. Results were quite interesting, so I decided to publish them together with the corresponding numbers for IET and STGT iSCSI targets. This isn't a real performance comparison, it includes only few chosen tests, because I don't have time for a complete comparison. But I hope somebody will take up what I did and make it complete.

Setup:

Target: HT 2.4GHz Xeon, x86_32, 2GB of memory limited to 256MB by kernel command line to have less test data footprint, 75GB 15K RPM SCSI disk as backstorage, dual port 1Gbps E1000 Intel network card, 2.6.29 kernel.

Initiator: 1.7GHz Xeon, x86_32, 1GB of memory limited to 256MB by kernel command line to have less test data footprint, dual port 1Gbps E1000 Intel network card, 2.6.27 kernel, open-iscsi 2.0-870-rc3.

Quoted 6 lines

The target exported a 5GB file on XFS for FILEIO and 5GB partition for BLOCKIO.

All the tests were ran 3 times and average written. All the values are in MB/s. The tests were ran with CFQ and deadline IO schedulers on the target. All other parameters on both target and initiator were default.

These are indeed interesting results. There are some aspects of the test setup I do not understand however:
* All tests have been run with buffered I/O instead of direct I/O (iflag=direct / oflag=direct). My experience is that the results of tests with direct I/O are easier to reproduce (less variation between runs). So I have been wondering why the tests have been run with buffered I/O instead ?

* It is well known that having more memory in the target system improves performance because of read and write caching. What did you want to demonstrate by limiting the memory of the target system ?

* Which SCST options were enabled on the target ? Was e.g. the NV_CACHE option enabled ?

Quoted 7 lines · Bart Van Assche, on 03/30/2009 10:06 PM wrote:

These are indeed interesting results. There are some aspects of the test setup I do not understand however:
* All tests have been run with buffered I/O instead of direct I/O (iflag=direct / oflag=direct). My experience is that the results of tests with direct I/O are easier to reproduce (less variation between runs). So I have been wondering why the tests have been run with buffered I/O instead ?

Real applications use buffered I/O, hence it should be used in tests. It evaluates all the storage stack on both initiator and target as a whole. The results are very reproducible, variation is about 10%.

Quoted 8 lines · Vladislav Bolkhovitin

Quoted 7 lines · Bart Van Assche, on 03/30/2009 10:06 PM wrote:

These are indeed interesting results. There are some aspects of the test setup I do not understand however:
* All tests have been run with buffered I/O instead of direct I/O (iflag=direct / oflag=direct). My experience is that the results of tests with direct I/O are easier to reproduce (less variation between runs). So I have been wondering why the tests have been run with buffered I/O instead ?

Quoted 3 lines

Real applications use buffered I/O, hence it should be used in tests. It
evaluates all the storage stack on both initiator and target as a whole.
The results are very reproducible, variation is about 10%.

Most applications do indeed use buffered I/O. Database software however often uses direct I/O. It might be interesting to publish performance results for both buffered I/O and direct I/O.

A quote from
the paper "Asynchronous I/O Support in Linux 2.5" by Bhattacharya e.a. (Linux Symposium, Ottawa, 2003):

Direct I/O (raw and O_DIRECT) transfers data between a user buffer and a device without copying the data through the kernel’s buffer cache. This mechanism can boost performance if the data is unlikely to be used again in the short term (during a disk backup, for example), or for applications such as large database management systems that perform their own caching.

Bart.

------------------------------------------------------------------------------
_______________________________________________
Scst-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scst-devel

Quoted 4 lines · Bart Van Assche, on 03/30/2009 10:53 PM wrote:

Most applications do indeed use buffered I/O. Database software however often uses direct I/O. It might be interesting to publish performance results for both buffered I/O and direct I/O.

Yes, sure

Quoted 10 lines

A quote from
the paper "Asynchronous I/O Support in Linux 2.5" by Bhattacharya e.a. (Linux Symposium, Ottawa, 2003):

Direct I/O (raw and O_DIRECT) transfers data between a user buffer and a device without copying the data through the kernel’s buffer cache. This mechanism can boost performance if the data is unlikely to be used again in the short term (during a disk backup, for example), or for applications such as large database management systems that perform their own caching.

Please don't misread phrase "unlikely to be used again in the short term". If you have read-ahead, all your cached data is *likely* to be used "again" in the near future after they were read from storage, although only once in the first read by application. The same is true for write-back caching, where data written to the cache once for each command. Both read-ahead and write back are very important for good performance and O_DIRECT throws them away. All the modern HDDs have a memory buffer (cache) at least 2MB big on the cheapest ones. This cache is essential for performance, although how can it make any difference if the host computer has, say, 1000 times more memory?

Thus, to work effectively with O_DIRECT an application has to be very smart to workaround the lack of read-ahead and write back.

I personally consider O_DIRECT (as well as BLOCKIO) as nothing more than a workaround for possible flaws in the storage subsystem. If O_DIRECT works better, then in 99+% cases there is something in the storage subsystem, which should be fixed to perform better.

To be complete, there is one case where O_DIRECT and BLOCKIO have an
advantage: both of them transfer data zero-copy. So they are good if your memory is too slow comparing to storage (InfiniBand case, for instance) and additional data copy hurts performance noticeably.

IET just needs to fix how it does it workload with CFQ which somehow SCST has overcome. Of course SCST tweaks the Linux kernel to gain some extra speed.

Quoted 3 lines · Ross S. W. Walker

IET just needs to fix how it does it workload with CFQ which somehow SCST has overcome. Of course SCST tweaks the Linux kernel to gain some extra speed.

I'm not familiar with the implementation details of CFQ, but I know that one of the changes between SCST 1.0.0 and SCST 1.0.1 is that the default number of kernel threads of the scst_vdisk kernel module has been increased to 5. Could this explain the performance difference between SCST and IET for FILEIO and BLOCKIO ?

I. SEQUENTIAL ACCESS OVER SINGLE LINE

1. # dd if=/dev/sdX of=/dev/null bs=512K count=2000

                       ISCSI-SCST      IET             STGT
NULLIO:                 106             105             103 FILEIO/CFQ:             82              57              55 FILEIO/deadline         69              69              67 BLOCKIO/CFQ             81              28              - BLOCKIO/deadline        80              66              -

Quoted 11 lines · Bart Van Assche

Quoted 3 lines · Ross S. W. Walker

IET just needs to fix how it does it workload with CFQ which somehow SCST has overcome. Of course SCST tweaks the Linux kernel to gain some extra speed.

I'm not familiar with the implementation details of CFQ, but I know that one of the changes between SCST 1.0.0 and SCST 1.0.1 is that the default number of kernel threads of the scst_vdisk kernel module has been increased to 5. Could this explain the performance difference between SCST and IET for FILEIO and BLOCKIO ?

Thank for the update. IET has used 8 threads per target for ages now, I don't think it is that.

It may be how the I/O threads are forked in SCST that causes them to be in the same I/O context with each other.

I'm pretty sure implementing a version of the patch that was used for the dump command (found on the LKML) will fix this.

But thanks goes to Vlad for pointing this dificiency out so we can fix it to help make IET even better.

Quoted 16 lines · Ross Walker

Quoted 10 lines · Bart Van Assche

Quoted 3 lines · Ross S. W. Walker

IET just needs to fix how it does it workload with CFQ which somehow SCST has overcome. Of course SCST tweaks the Linux kernel to gain some extra speed.

I'm not familiar with the implementation details of CFQ, but I know that one of the changes between SCST 1.0.0 and SCST 1.0.1 is that the default number of kernel threads of the scst_vdisk kernel module has been increased to 5. Could this explain the performance difference between SCST and IET for FILEIO and BLOCKIO ?

Thank for the update. IET has used 8 threads per target for ages now, I don't think it is that.

It may be how the I/O threads are forked in SCST that causes them to be in the same I/O context with each other.

Quoted 3 lines

I'm pretty sure implementing a version of the patch that was used for the dump command (found on the LKML) will fix this.

But thanks goes to Vlad for pointing this dificiency out so we can fix it to help make IET even better.

SCST explicitly fiddles with the io context to get this to happen. It has a hack to block to export alloc_io_context:

http://marc.info/?t=122893564800003

Quoted 20 lines · James Bottomley, on 04/02/2009 12:23 AM wrote:

Quoted 19 lines · Ross Walker

Quoted 10 lines · Bart Van Assche

Quoted 3 lines · Ross S. W. Walker

IET just needs to fix how it does it workload with CFQ which somehow SCST has overcome. Of course SCST tweaks the Linux kernel to gain some extra speed.

I'm not familiar with the implementation details of CFQ, but I know that one of the changes between SCST 1.0.0 and SCST 1.0.1 is that the default number of kernel threads of the scst_vdisk kernel module has been increased to 5. Could this explain the performance difference between SCST and IET for FILEIO and BLOCKIO ?

Thank for the update. IET has used 8 threads per target for ages now, I don't think it is that.

It may be how the I/O threads are forked in SCST that causes them to be in the same I/O context with each other.

I'm pretty sure implementing a version of the patch that was used for the dump command (found on the LKML) will fix this.

But thanks goes to Vlad for pointing this dificiency out so we can fix it to help make IET even better.

Quoted 4 lines

SCST explicitly fiddles with the io context to get this to happen. It has a hack to block to export alloc_io_context:

http://marc.info/?t=122893564800003

Correct, although I wouldn't call it "fiddle", rather "grouping" ;)

But that's not the only reason for good performance. Particularly, it can't explain Bart's tmpfs results from the previous message, where the majority of I/O done to/from RAM without any I/O scheduler involved. (Or does I/O scheduler also involved with tmpfs?) Bart has 4GB RAM, if I remember correctly, i.e. the test data set was 25% of RAM.

Quoted 8 lines · Vladislav Bolkhovitin, on 04/02/2009 11:38 AM wrote:

Quoted 5 lines · James Bottomley, on 04/02/2009 12:23 AM wrote:

SCST explicitly fiddles with the io context to get this to happen. It has a hack to block to export alloc_io_context:

http://marc.info/?t=122893564800003

Correct, although I wouldn't call it "fiddle", rather "grouping" ;)

Ross S. W. Walker, on 03/30/2009 10:33 PM wrote:

I would be interested in knowing how your code defeats CFQ's extremely high latency? Does your code reach into the io scheduler too? If not, some code hints would be great.

Quoted 5 lines

But that's not the only reason for good performance. Particularly, it can't explain Bart's tmpfs results from the previous message, where the majority of I/O done to/from RAM without any I/O scheduler involved. (Or does I/O scheduler also involved with tmpfs?) Bart has 4GB RAM, if I remember correctly, i.e. the test data set was 25% of RAM.

To remove any suspicions that I'm playing dirty games here I should note

Quoted 6 lines · Vladislav Bolkhovitin

Quoted 5 lines · Vladislav Bolkhovitin, on 04/02/2009 11:38 AM wrote:

Quoted 4 lines · James Bottomley, on 04/02/2009 12:23 AM wrote:

SCST explicitly fiddles with the io context to get this to happen. It has a hack to block to export alloc_io_context:

Quoted 3 lines

Quoted 3 lines

Quoted 1 line

http://marc.info/?t=122893564800003

Correct, although I wouldn't call it "fiddle", rather "grouping" ;)

Call it what you like,

Quoted 5 lines · Vladislav Bolkhovitin

Ross S. W. Walker, on 03/30/2009 10:33 PM wrote:

I would be interested in knowing how your code defeats CFQ's extremely high latency? Does your code reach into the io scheduler too? If not, some code hints would be great.

Quoted 4 lines · Ross S. W. Walker, on 03/30/2009 10:33 PM wrote:

I would be interested in knowing how your code defeats CFQ's extremely high latency? Does your code reach into the io scheduler too? If not, some code hints would be great.

Hmm, CFQ doesn't have any extra processing latency, especially "extremely", hence there is nothing to defeat. If it had, how could it been chosen as the default?

Quoted 7 lines · Vladislav Bolkhovitin

Quoted 6 lines · Vladislav Bolkhovitin, on 04/02/2009 11:38 AM wrote:

Quoted 4 lines · James Bottomley, on 04/02/2009 12:23 AM wrote:

SCST explicitly fiddles with the io context to get this to happen. It has a hack to block to export alloc_io_context:

http://marc.info/?t=122893564800003

Correct, although I wouldn't call it "fiddle", rather "grouping" ;)

Call it what you like,

Quoted 5 lines · Vladislav Bolkhovitin

Ross S. W. Walker, on 03/30/2009 10:33 PM wrote:

I would be interested in knowing how your code defeats CFQ's extremely high latency? Does your code reach into the io scheduler too? If not, some code hints would be great.

Hmm, CFQ doesn't have any extra processing latency, especially "extremely", hence there is nothing to defeat. If it had, how could it been chosen as the default?

----------

List:       linux-scsi
Subject:    [PATCH][RFC 13/23]: Export of alloc_io_context() function
From:       Vladislav Bolkhovitin <vst () vlnb ! net>
Date:       2008-12-10 18:49:19
Message-ID: 49400F2F.4050603 () vlnb ! net

This patch exports alloc_io_context() function. For performance reasons SCST queues commands using a pool of IO threads. It is considerably better for performance (>30% increase on sequential reads) if threads in
a pool have the same IO context. Since SCST can be built as a module,
it needs alloc_io_context() function exported.

<snip>
----------

I call that lying.

Quoted 2 lines

Quoted 1 line

But that's not the only reason for good performance. Particularly, it can't explain Bart's tmpfs results from the previous message, where the majority of I/O done to/from RAM without any I/O scheduler involved. (Or does I/O scheduler also involved with tmpfs?) Bart has 4GB RAM, if I remember correctly, i.e. the test data set was 25% of RAM.

To remove any suspicions that I'm playing dirty games here I should note

<snip>

I don't know what games your playing at, but do me a favor, if your too stupid enough to realize when your caught in a lie and to just shut up then please do me the favor and leave me out of any further correspondence from you.

Thank you,

-Ross

On Mon, Mar 30, 2009 at 7:33 PM, Vladislav Bolkhovitin <[email protected]> wrote:
==================================================================

Quoted 10 lines

I. SEQUENTIAL ACCESS OVER SINGLE LINE

1. # dd if=/dev/sdX of=/dev/null bs=512K count=2000

ISCSI-SCST IET STGT

NULLIO:                 106             105             103
FILEIO/CFQ:             82              57              55
FILEIO/deadline         69              69              67
BLOCKIO/CFQ             81              28              -
BLOCKIO/deadline        80              66              -

I have repeated some of these performance tests for iSCSI over IPoIB (two DDR PCIe 1.0 ConnectX HCA's connected back to back). The results for the buffered I/O test with a block size of 512K (initiator) against a file of 1GB residing on a tmpfs filesystem on the target are as follows:

write-test: iSCSI-SCST 243 MB/s; IET 192 MB/s.
read-test: iSCSI-SCST 291 MB/s; IET 223 MB/s.

And for a block size of 4 KB:

write-test: iSCSI-SCST 43 MB/s; IET 42 MB/s.
read-test: iSCSI-SCST 288 MB/s; IET 221 MB/s.

Or: depending on the test scenario, SCST transfers data between 2% and 30% faster via the iSCSI protocol over this network.

Something that is not relevant for this comparison, but interesting to
know: with the SRP implementation in SCST the maximal read throughput is 1290 MB/s on the same setup.

Bart.

------------------------------------------------------------------------------
_______________________________________________
Scst-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scst-devel

Think what you want and do what you want. You can even filter out all e-mails from me, that's your right. But:

1. As I wrote grouping threads into a single IO context doesn't explain all the performance difference and finding out reasons for other's performance problems isn't something I can afford at the moment.

2. CFQ doesn't have any processing latency and has never had. Learn to understand what are your writing about and how to correctly express yourself at first. You asked about that latency and I replied that there is nothing to defeat.

3. SCST doesn't have any hooks into CFQ and not going to have in the considerable future.

Quoted 9 lines · Bart Van Assche, on 04/02/2009 12:14 AM wrote:

I have repeated some of these performance tests for iSCSI over IPoIB (two DDR PCIe 1.0 ConnectX HCA's connected back to back). The results for the buffered I/O test with a block size of 512K (initiator) against a file of 1GB residing on a tmpfs filesystem on the target are as follows:

write-test: iSCSI-SCST 243 MB/s; IET 192 MB/s.
read-test: iSCSI-SCST 291 MB/s; IET 223 MB/s.

Quoted 4 lines

And for a block size of 4 KB:

write-test: iSCSI-SCST 43 MB/s; IET 42 MB/s.
read-test: iSCSI-SCST 288 MB/s; IET 221 MB/s.

Do you have any thoughts why writes are so bad? It shouldn't be so..

-----Original Message-----
From: Bart Van Assche [mailto:[email protected]]
Sent: Friday, April 03, 2009 10:09 AM
To: Vladislav Bolkhovitin
Cc: scst-devel; [email protected]; [email protected]
Subject: Re: [Scst-devel] ISCSI-SCST performance (with also IET and STGTdata)

Quoted 2 lines · Vladislav Bolkhovitin

Quoted 1 line · Bart Van Assche, on 04/02/2009 12:14 AM wrote:

I have repeated some of these performance tests for iSCSI

over IPoIB

Quoted 1 line

Quoted 1 line

(two DDR PCIe 1.0 ConnectX HCA's connected back to back).

The results

Quoted 7 lines

Quoted 7 lines

for the buffered I/O test with a block size of 512K (initiator) against a file of 1GB residing on a tmpfs filesystem on the target are as follows:

write-test: iSCSI-SCST 243 MB/s; IET 192 MB/s.
read-test: iSCSI-SCST 291 MB/s; IET 223 MB/s.

Quoted 6 lines

Quoted 4 lines

And for a block size of 4 KB:

write-test: iSCSI-SCST 43 MB/s; IET 42 MB/s.
read-test: iSCSI-SCST 288 MB/s; IET 221 MB/s.

Do you have any thoughts why writes are so bad? It shouldn't be so..

It's not impossible that with the 4 KB write test I hit the limits of the initiator system (Intel E6750 CPU, 2.66 GHz, two cores). Some statistics I gathered during the 4 KB write test:
Target: CPU load 0.5, 16500 mlx4-comp-0 interrupts per second, same number of interrupts processed by each core (8250/s).
Initiator: CPU load 1.0, 32850 mlx4-comp-0 interrupts per second, all interrupts occurred on the same core.

Bart.

--------------------------------------------------------------
----------------
_______________________________________________
Scst-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scst-devel

Quoted 21 lines

Quoted 12 lines · Vladislav Bolkhovitin

Quoted 11 lines · Bart Van Assche, on 04/02/2009 12:14 AM wrote:

I have repeated some of these performance tests for iSCSI over IPoIB
(two DDR PCIe 1.0 ConnectX HCA's connected back to back). The results
for the buffered I/O test with a block size of 512K (initiator) against a file of 1GB residing on a tmpfs filesystem on the target are as follows:

write-test: iSCSI-SCST 243 MB/s; IET 192 MB/s.
read-test: iSCSI-SCST 291 MB/s; IET 223 MB/s.

Quoted 6 lines

Quoted 4 lines

And for a block size of 4 KB:

write-test: iSCSI-SCST 43 MB/s; IET 42 MB/s.
read-test: iSCSI-SCST 288 MB/s; IET 221 MB/s.

Do you have any thoughts why writes are so bad? It shouldn't be so..

Quoted 7 lines

It's not impossible that with the 4 KB write test I hit the limits of the initiator system (Intel E6750 CPU, 2.66 GHz, two cores). Some statistics I gathered during the 4 KB write test:
Target: CPU load 0.5, 16500 mlx4-comp-0 interrupts per second, same number of interrupts processed by each core (8250/s).
Initiator: CPU load 1.0, 32850 mlx4-comp-0 interrupts per second, all interrupts occurred on the same core.

Are you using connected mode IPoIB and setting the MTU to 4KB? Would fragmentation of IPoIB drive up the interrupt rates?

Quoted 9 lines · Bart Van Assche, on 04/02/2009 12:14 AM wrote:

I have repeated some of these performance tests for iSCSI over IPoIB (two DDR PCIe 1.0 ConnectX HCA's connected back to back). The results for the buffered I/O test with a block size of 512K (initiator) against a file of 1GB residing on a tmpfs filesystem on the target are as follows:

write-test: iSCSI-SCST 243 MB/s; IET 192 MB/s.
read-test: iSCSI-SCST 291 MB/s; IET 223 MB/s.

Quoted 4 lines

And for a block size of 4 KB:

write-test: iSCSI-SCST 43 MB/s; IET 42 MB/s.
read-test: iSCSI-SCST 288 MB/s; IET 221 MB/s.

Do you have any thoughts why writes are so bad? It shouldn't be so..

Quoted 7 lines · Vladislav Bolkhovitin

Quoted 6 lines · Bart Van Assche, on 04/02/2009 12:14 AM wrote:

I have repeated some of these performance tests for iSCSI over IPoIB (two DDR PCIe 1.0 ConnectX HCA's connected back to back). The results for the buffered I/O test with a block size of 512K (initiator) against a file of 1GB residing on a tmpfs filesystem on the target are as follows:

Quoted 8 lines

Quoted 7 lines

write-test: iSCSI-SCST 243 MB/s; IET 192 MB/s.
read-test: iSCSI-SCST 291 MB/s; IET 223 MB/s.

And for a block size of 4 KB:

write-test: iSCSI-SCST 43 MB/s; IET 42 MB/s.
read-test: iSCSI-SCST 288 MB/s; IET 221 MB/s.

Do you have any thoughts why writes are so bad? It shouldn't be so..

By this time I have run the following variation of the 4 KB write test:
* Target: iSCSI-SCST was exporting a 1 GB file residing on a tmpfs filesystem.
* Initiator: two processes were writing 4 KB blocks as follows:
dd if=/dev/zero of=/dev/sdb bs=4K seek=0 count=131072 oflag=sync & dd if=/dev/zero of=/dev/sdb bs=4K seek=131072 count=131072 oflag=sync &

Results:
* Each dd process on the initiator was writing at a speed of 37.8 MB/s, or a combined writing speed of 75.6 MB/s.
* CPU load on the initiator system during the test: 2.0.
* According to /proc/interrupts, about 38000 mlx4-comp-0 interrupts were triggered per second.

These results confirm that the initiator system was the bottleneck during the 4 KB write test, not the target system.

Bart.

------------------------------------------------------------------------------
_______________________________________________
Scst-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scst-devel