Skip to contentSkip to conversation

tiobench read 50% regression with 2.6.30-rc1

DISCUSSION Zhang, Yanmin 3 participants 12 messages Apr 9, 2009 – Apr 15, 2009 Block layer

lore
Share & tools
lore
#0 lore raw

Comparing with 2.6.29's result, tiobench (read) has about 50% regression with 2.6.30-rc1 on all my machines. Bisect down to below patch.

b029195dda0129b427c6e579a3bb3ae752da3a93 is first bad commit commit b029195dda0129b427c6e579a3bb3ae752da3a93

Author: Jens Axboe <[email protected]>
Date:   Tue Apr 7 11:38:31 2009 +0200

cfq-iosched: don't let idling interfere with plugging

When CFQ is waiting for a new request from a process, currently it'll immediately restart queuing when it sees such a request. This doesn't work very well with streamed IO, since we then end up splitting IO that would otherwise have been merged nicely. For a simple dd test, this causes 10x as many requests to be issued as we should have. Normally this goes unnoticed due to the low overhead of requests at the device side, but some hardware is very sensitive to request sizes and there it can cause big slow downs.

Command to start the testing:
#tiotest -k0 -k1 -k3 -f 80 -t 32

It's a multi-threaded program and starts 32 threads. Every thread does I/O on its own 80MB file.

yanmin

#1 lore raw
Quoted 21 lines · Yanmin

Comparing with 2.6.29's result, tiobench (read) has about 50% regression with 2.6.30-rc1 on all my machines. Bisect down to below patch.

b029195dda0129b427c6e579a3bb3ae752da3a93 is first bad commit commit b029195dda0129b427c6e579a3bb3ae752da3a93

Author: Jens Axboe <[email protected]>
Date:   Tue Apr 7 11:38:31 2009 +0200

cfq-iosched: don't let idling interfere with plugging

When CFQ is waiting for a new request from a process, currently it'll immediately restart queuing when it sees such a request. This doesn't work very well with streamed IO, since we then end up splitting IO that would otherwise have been merged nicely. For a simple dd test, this causes 10x as many requests to be issued as we should have. Normally this goes unnoticed due to the low overhead of requests at the device side, but some hardware is very sensitive to request sizes and there it can cause big slow downs.

Quoted 5 lines

Command to start the testing:
#tiotest -k0 -k1 -k3 -f 80 -t 32

It's a multi-threaded program and starts 32 threads. Every thread does I/O on its own 80MB file.

It's not a huge surprise that we regressed there. I'll get this fixed up next week. Can you I talk you into trying to change the 'quantum' sysfs variable for the drive? It's in /sys/block/xxx/queue/iosched where xxx is your drive(s). It's set to 4, if you could try progressively larger settings and retest, that would help get things started.

Thanks!

Signature · 1 line
Jens Axboe
#2 lore raw
Quoted 22 lines · Jens Axboe
Quoted 21 lines · Yanmin

Comparing with 2.6.29's result, tiobench (read) has about 50% regression with 2.6.30-rc1 on all my machines. Bisect down to below patch.

b029195dda0129b427c6e579a3bb3ae752da3a93 is first bad commit commit b029195dda0129b427c6e579a3bb3ae752da3a93

Author: Jens Axboe <[email protected]>
Date:   Tue Apr 7 11:38:31 2009 +0200

cfq-iosched: don't let idling interfere with plugging

When CFQ is waiting for a new request from a process, currently it'll immediately restart queuing when it sees such a request. This doesn't work very well with streamed IO, since we then end up splitting IO that would otherwise have been merged nicely. For a simple dd test, this causes 10x as many requests to be issued as we should have. Normally this goes unnoticed due to the low overhead of requests at the device side, but some hardware is very sensitive to request sizes and there it can cause big slow downs.

Quoted 5 lines
Quoted 5 lines

Command to start the testing:
#tiotest -k0 -k1 -k3 -f 80 -t 32

It's a multi-threaded program and starts 32 threads. Every thread does I/O on its own 80MB file.

The files should be created before the testing and pls. drop page caches by "echo 3 >/proc/sys/vm/drop_caches" before testing.

Quoted 6 lines

It's not a huge surprise that we regressed there. I'll get this fixed up next week. Can you I talk you into trying to change the 'quantum' sysfs variable for the drive? It's in /sys/block/xxx/queue/iosched where xxx is your drive(s). It's set to 4, if you could try progressively larger settings and retest, that would help get things started.

I tried 4,8,16,64,128 and didn't find result difference.

#3 lore raw
Quoted 32 lines · Yanmin
Quoted 22 lines · Jens Axboe
Quoted 21 lines · Yanmin

Comparing with 2.6.29's result, tiobench (read) has about 50% regression with 2.6.30-rc1 on all my machines. Bisect down to below patch.

b029195dda0129b427c6e579a3bb3ae752da3a93 is first bad commit commit b029195dda0129b427c6e579a3bb3ae752da3a93

Author: Jens Axboe <[email protected]>
Date:   Tue Apr 7 11:38:31 2009 +0200

cfq-iosched: don't let idling interfere with plugging

When CFQ is waiting for a new request from a process, currently it'll immediately restart queuing when it sees such a request. This doesn't work very well with streamed IO, since we then end up splitting IO that would otherwise have been merged nicely. For a simple dd test, this causes 10x as many requests to be issued as we should have. Normally this goes unnoticed due to the low overhead of requests at the device side, but some hardware is very sensitive to request sizes and there it can cause big slow downs.

Quoted 5 lines
Quoted 5 lines

Command to start the testing:
#tiotest -k0 -k1 -k3 -f 80 -t 32

It's a multi-threaded program and starts 32 threads. Every thread does I/O on its own 80MB file.

The files should be created before the testing and pls. drop page caches by "echo 3 >/proc/sys/vm/drop_caches" before testing.

Quoted 1 line
Quoted 6 lines
Quoted 5 lines

It's not a huge surprise that we regressed there. I'll get this fixed up next week. Can you I talk you into trying to change the 'quantum' sysfs variable for the drive? It's in /sys/block/xxx/queue/iosched where xxx is your drive(s). It's set to 4, if you could try progressively larger settings and retest, that would help get things started.

I tried 4,8,16,64,128 and didn't find result difference.

OK, that's good. I'm away from the source atm, but I think I know what is going on. We need to kick the queue for anything but the first queued request. I'll get it fixed up next week.

Signature · 1 line
Jens Axboe
#4 lore raw
In reply to Zhang, Yanmin
Quoted 32 lines · Yanmin
Quoted 22 lines · Jens Axboe
Quoted 21 lines · Yanmin

Comparing with 2.6.29's result, tiobench (read) has about 50% regression with 2.6.30-rc1 on all my machines. Bisect down to below patch.

b029195dda0129b427c6e579a3bb3ae752da3a93 is first bad commit commit b029195dda0129b427c6e579a3bb3ae752da3a93

Author: Jens Axboe <[email protected]>
Date:   Tue Apr 7 11:38:31 2009 +0200

cfq-iosched: don't let idling interfere with plugging

When CFQ is waiting for a new request from a process, currently it'll immediately restart queuing when it sees such a request. This doesn't work very well with streamed IO, since we then end up splitting IO that would otherwise have been merged nicely. For a simple dd test, this causes 10x as many requests to be issued as we should have. Normally this goes unnoticed due to the low overhead of requests at the device side, but some hardware is very sensitive to request sizes and there it can cause big slow downs.

Quoted 5 lines
Quoted 5 lines

Command to start the testing:
#tiotest -k0 -k1 -k3 -f 80 -t 32

It's a multi-threaded program and starts 32 threads. Every thread does I/O on its own 80MB file.

The files should be created before the testing and pls. drop page caches by "echo 3 >/proc/sys/vm/drop_caches" before testing.

Quoted 1 line
Quoted 6 lines
Quoted 5 lines

It's not a huge surprise that we regressed there. I'll get this fixed up next week. Can you I talk you into trying to change the 'quantum' sysfs variable for the drive? It's in /sys/block/xxx/queue/iosched where xxx is your drive(s). It's set to 4, if you could try progressively larger settings and retest, that would help get things started.

I tried 4,8,16,64,128 and didn't find result difference.

Can you try with this patch?

block/cfq-iosched.c +9 -2
index a4809de..66f00e5 100644
@@ -1905,10 +1905,17 @@ cfq_rq_enqueued(struct cfq_data *cfqd, struct cfq_queue *cfqq,
 		 * Remember that we saw a request from this process, but
 		 * don't start queuing just yet. Otherwise we risk seeing lots
 		 * of tiny requests, because we disrupt the normal plugging
-		 * and merging.
+		 * and merging. If the request is already larger than a single
+		 * page, let it rip immediately. For that case we assume that
+		 * merging is already done.
 		 */
-		if (cfq_cfqq_wait_request(cfqq))
+		if (cfq_cfqq_wait_request(cfqq)) {
+			if (blk_rq_bytes(rq) > PAGE_CACHE_SIZE) {
+				del_timer(&cfqd->idle_slice_timer);
+				blk_start_queueing(cfqd->queue);
+			}
 			cfq_mark_cfqq_must_dispatch(cfqq);
+		}
 	} else if (cfq_should_preempt(cfqd, cfqq, rq)) {
 		/*
 		 * not the active queue - expire current slice if it is
Signature · 1 line
Jens Axboe
#5 lore raw
Quoted 12 lines · Jens Axboe
Quoted 11 lines · Yanmin
Quoted 10 lines · Jens Axboe
Quoted 7 lines · Yanmin

Comparing with 2.6.29's result, tiobench (read) has about 50% regression with 2.6.30-rc1 on all my machines. Bisect down to below patch.

b029195dda0129b427c6e579a3bb3ae752da3a93 is first bad commit commit b029195dda0129b427c6e579a3bb3ae752da3a93

Author: Jens Axboe <[email protected]>
Date:   Tue Apr 7 11:38:31 2009 +0200
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines

It's not a huge surprise that we regressed there. I'll get this fixed up next week. Can you I talk you into trying to change the 'quantum' sysfs variable for the drive? It's in /sys/block/xxx/queue/iosched where xxx is your drive(s). It's set to 4, if you could try progressively larger settings and retest, that would help get things started.

I tried 4,8,16,64,128 and didn't find result difference.

Can you try with this patch?

The new patch does fix it.

Quoted 26 lines
block/cfq-iosched.c +9 -2
index a4809de..66f00e5 100644
@@ -1905,10 +1905,17 @@ cfq_rq_enqueued(struct cfq_data *cfqd, struct cfq_queue *cfqq,
 		 * Remember that we saw a request from this process, but
 		 * don't start queuing just yet. Otherwise we risk seeing lots
 		 * of tiny requests, because we disrupt the normal plugging
-		 * and merging.
+		 * and merging. If the request is already larger than a single
+		 * page, let it rip immediately. For that case we assume that
+		 * merging is already done.
 		 */
-		if (cfq_cfqq_wait_request(cfqq))
+		if (cfq_cfqq_wait_request(cfqq)) {
+			if (blk_rq_bytes(rq) > PAGE_CACHE_SIZE) {
+				del_timer(&cfqd->idle_slice_timer);
+				blk_start_queueing(cfqd->queue);
+			}
 			cfq_mark_cfqq_must_dispatch(cfqq);
+		}
 	} else if (cfq_should_preempt(cfqd, cfqq, rq)) {
 		/*
 		 * not the active queue - expire current slice if it is
#6 lore raw
Quoted 18 lines · Yanmin
Quoted 17 lines · Jens Axboe
Quoted 16 lines · Yanmin
Quoted 15 lines · Jens Axboe
Quoted 7 lines · Yanmin

Comparing with 2.6.29's result, tiobench (read) has about 50% regression with 2.6.30-rc1 on all my machines. Bisect down to below patch.

b029195dda0129b427c6e579a3bb3ae752da3a93 is first bad commit commit b029195dda0129b427c6e579a3bb3ae752da3a93

Author: Jens Axboe <[email protected]>
Date:   Tue Apr 7 11:38:31 2009 +0200

It's not a huge surprise that we regressed there. I'll get this fixed up next week. Can you I talk you into trying to change the 'quantum' sysfs variable for the drive? It's in /sys/block/xxx/queue/iosched where xxx is your drive(s). It's set to 4, if you could try progressively larger settings and retest, that would help get things started.

Quoted 4 lines
Quoted 3 lines
Quoted 1 line

I tried 4,8,16,64,128 and didn't find result difference.

Can you try with this patch?

The new patch does fix it.

Great, thanks for retesting.

Signature · 1 line
Jens Axboe
#7 lore raw
In reply to Jens Axboe
Quoted 56 lines · Jens Axboe
Quoted 32 lines · Yanmin
Quoted 22 lines · Jens Axboe
Quoted 21 lines · Yanmin

Comparing with 2.6.29's result, tiobench (read) has about 50% regression with 2.6.30-rc1 on all my machines. Bisect down to below patch.

b029195dda0129b427c6e579a3bb3ae752da3a93 is first bad commit commit b029195dda0129b427c6e579a3bb3ae752da3a93

Author: Jens Axboe <[email protected]>
Date:   Tue Apr 7 11:38:31 2009 +0200

cfq-iosched: don't let idling interfere with plugging

When CFQ is waiting for a new request from a process, currently it'll immediately restart queuing when it sees such a request. This doesn't work very well with streamed IO, since we then end up splitting IO that would otherwise have been merged nicely. For a simple dd test, this causes 10x as many requests to be issued as we should have. Normally this goes unnoticed due to the low overhead of requests at the device side, but some hardware is very sensitive to request sizes and there it can cause big slow downs.

Quoted 5 lines
Quoted 5 lines

Command to start the testing:
#tiotest -k0 -k1 -k3 -f 80 -t 32

It's a multi-threaded program and starts 32 threads. Every thread does I/O on its own 80MB file.

The files should be created before the testing and pls. drop page caches by "echo 3 >/proc/sys/vm/drop_caches" before testing.

Quoted 1 line
Quoted 6 lines
Quoted 5 lines

It's not a huge surprise that we regressed there. I'll get this fixed up next week. Can you I talk you into trying to change the 'quantum' sysfs variable for the drive? It's in /sys/block/xxx/queue/iosched where xxx is your drive(s). It's set to 4, if you could try progressively larger settings and retest, that would help get things started.

I tried 4,8,16,64,128 and didn't find result difference.

Can you try with this patch?

block/cfq-iosched.c +3 -2
index a4809de..66f00e5 100644
@@ -1905,10 +1905,17 @@ cfq_rq_enqueued(struct cfq_data *cfqd, struct cfq_queue *cfqq,
 		 * Remember that we saw a request from this process, but
 		 * don't start queuing just yet. Otherwise we risk seeing lots
 		 * of tiny requests, because we disrupt the normal plugging
-		 * and merging.
+		 * and merging. If the request is already larger than a single
+		 * page, let it rip immediately. For that case we assume that
+		 * merging is already done.
 		 */
-		if (cfq_cfqq_wait_request(cfqq))

block/cfq-iosched.c cfq_rq_enqueued(struct cfq_data *cfqd, struct cfq_queue *cfqq, · line 1921

Quoted 10 lines
block/cfq-iosched.c +6 -0
+		if (cfq_cfqq_wait_request(cfqq)) {
+			if (blk_rq_bytes(rq) > PAGE_CACHE_SIZE) {
+				del_timer(&cfqd->idle_slice_timer);
+				blk_start_queueing(cfqd->queue);
+			}
 			cfq_mark_cfqq_must_dispatch(cfqq);
+		}
 	} else if (cfq_should_preempt(cfqd, cfqq, rq)) {
 		/*
 		 * not the active queue - expire current slice if it is

I tested this using iozone to read a file from an NFS client. The iozone command line was:
iozone -s 2000000 -r 64 -f /mnt/test/testfile -i 1 -w

The numbers in the nfsd's row represent the number of nfsd threads. I included numbers for the deadline scheduler as well for comparison.

               v2.6.29

nfsd's | 1 | 2 | 4 | 8
--------+---------------+-------+------

cfq     | 91356 | 66391 | 61942 | 51674
deadline| 43207 | 67436 | 96289 | 107784
              2.6.30-rc1

nfsd's | 1 | 2 | 4 | 8
--------+---------------+-------+------

cfq     | 43127 | 22354 | 20858 | 21179
deadline| 43732 | 68059 | 76659 | 83231
          2.6.30-rc1 + cfq fix

nfsd's | 1 | 2 | 4 | 8
--------+-----------------+-------+------
cfq | 114602 | 102280 | 43479 | 43160

As you can see, for 1 and 2 threads, the patch *really* helps out. We still don't get back the performance for 4 and 8 nfsd threads, though. It's interesting to note that the deadline scheduler regresses for 4 and 8 threads, as well. I think we've still got some digging to do.

I'll try the cfq close cooperator patches next.

Cheers,
Jeff

#8 lore raw
Quoted 92 lines · Jeff Moyer
Quoted 56 lines · Jens Axboe
Quoted 32 lines · Yanmin
Quoted 22 lines · Jens Axboe
Quoted 21 lines · Yanmin

Comparing with 2.6.29's result, tiobench (read) has about 50% regression with 2.6.30-rc1 on all my machines. Bisect down to below patch.

b029195dda0129b427c6e579a3bb3ae752da3a93 is first bad commit commit b029195dda0129b427c6e579a3bb3ae752da3a93

Author: Jens Axboe <[email protected]>
Date:   Tue Apr 7 11:38:31 2009 +0200

cfq-iosched: don't let idling interfere with plugging

When CFQ is waiting for a new request from a process, currently it'll immediately restart queuing when it sees such a request. This doesn't work very well with streamed IO, since we then end up splitting IO that would otherwise have been merged nicely. For a simple dd test, this causes 10x as many requests to be issued as we should have. Normally this goes unnoticed due to the low overhead of requests at the device side, but some hardware is very sensitive to request sizes and there it can cause big slow downs.

Quoted 5 lines
Quoted 5 lines

Command to start the testing:
#tiotest -k0 -k1 -k3 -f 80 -t 32

It's a multi-threaded program and starts 32 threads. Every thread does I/O on its own 80MB file.

The files should be created before the testing and pls. drop page caches by "echo 3 >/proc/sys/vm/drop_caches" before testing.

Quoted 1 line
Quoted 6 lines
Quoted 5 lines

It's not a huge surprise that we regressed there. I'll get this fixed up next week. Can you I talk you into trying to change the 'quantum' sysfs variable for the drive? It's in /sys/block/xxx/queue/iosched where xxx is your drive(s). It's set to 4, if you could try progressively larger settings and retest, that would help get things started.

I tried 4,8,16,64,128 and didn't find result difference.

Can you try with this patch?

block/cfq-iosched.c +3 -2
index a4809de..66f00e5 100644
@@ -1905,10 +1905,17 @@ cfq_rq_enqueued(struct cfq_data *cfqd, struct cfq_queue *cfqq,
 		 * Remember that we saw a request from this process, but
 		 * don't start queuing just yet. Otherwise we risk seeing lots
 		 * of tiny requests, because we disrupt the normal plugging
-		 * and merging.
+		 * and merging. If the request is already larger than a single
+		 * page, let it rip immediately. For that case we assume that
+		 * merging is already done.
 		 */
-		if (cfq_cfqq_wait_request(cfqq))

block/cfq-iosched.c cfq_rq_enqueued(struct cfq_data *cfqd, struct cfq_queue *cfqq, · line 1921

Quoted 10 lines
block/cfq-iosched.c +6 -0
+		if (cfq_cfqq_wait_request(cfqq)) {
+			if (blk_rq_bytes(rq) > PAGE_CACHE_SIZE) {
+				del_timer(&cfqd->idle_slice_timer);
+				blk_start_queueing(cfqd->queue);
+			}
 			cfq_mark_cfqq_must_dispatch(cfqq);
+		}
 	} else if (cfq_should_preempt(cfqd, cfqq, rq)) {
 		/*
 		 * not the active queue - expire current slice if it is

I tested this using iozone to read a file from an NFS client. The iozone command line was:
iozone -s 2000000 -r 64 -f /mnt/test/testfile -i 1 -w

The numbers in the nfsd's row represent the number of nfsd threads. I included numbers for the deadline scheduler as well for comparison.

               v2.6.29

nfsd's | 1 | 2 | 4 | 8
--------+---------------+-------+------

cfq     | 91356 | 66391 | 61942 | 51674
deadline| 43207 | 67436 | 96289 | 107784
              2.6.30-rc1

nfsd's | 1 | 2 | 4 | 8
--------+---------------+-------+------

cfq     | 43127 | 22354 | 20858 | 21179
deadline| 43732 | 68059 | 76659 | 83231
          2.6.30-rc1 + cfq fix
Quoted 8 lines

nfsd's | 1 | 2 | 4 | 8
--------+-----------------+-------+------
cfq | 114602 | 102280 | 43479 | 43160

As you can see, for 1 and 2 threads, the patch *really* helps out. We still don't get back the performance for 4 and 8 nfsd threads, though. It's interesting to note that the deadline scheduler regresses for 4 and 8 threads, as well. I think we've still got some digging to do.

Wow, that does indeed look pretty good!

Quoted 1 line

I'll try the cfq close cooperator patches next.

I have a pending update on the coop patch that isn't pushed out yet, I hope to have it finalized and tested later today. Hopefully, with that, we should be able to maintain > 100Mb/sec for 4 and 8 threads.

Signature · 1 line
Jens Axboe
#9 lore raw
Quoted 28 lines · Jens Axboe
Quoted 27 lines · Jeff Moyer

I tested this using iozone to read a file from an NFS client. The iozone command line was:
iozone -s 2000000 -r 64 -f /mnt/test/testfile -i 1 -w

The numbers in the nfsd's row represent the number of nfsd threads. I included numbers for the deadline scheduler as well for comparison.

               v2.6.29

nfsd's | 1 | 2 | 4 | 8
--------+---------------+-------+------

cfq     | 91356 | 66391 | 61942 | 51674
deadline| 43207 | 67436 | 96289 | 107784
              2.6.30-rc1

nfsd's | 1 | 2 | 4 | 8
--------+---------------+-------+------

cfq     | 43127 | 22354 | 20858 | 21179
deadline| 43732 | 68059 | 76659 | 83231
          2.6.30-rc1 + cfq fix

nfsd's | 1 | 2 | 4 | 8
--------+-----------------+-------+------
cfq | 114602 | 102280 | 43479 | 43160

Quoted 6 lines
Quoted 4 lines

As you can see, for 1 and 2 threads, the patch *really* helps out. We still don't get back the performance for 4 and 8 nfsd threads, though. It's interesting to note that the deadline scheduler regresses for 4 and 8 threads, as well. I think we've still got some digging to do.

Wow, that does indeed look pretty good!

Please don't lose the later part of the message, which is that both CFQ and deadline regress for 4 and 8 threads when moving from 2.6.29 to 2.6.30-rc1! There have been way too many changes to keep track of in this area recently, and I'd like to get some confidence that we understand the performance implications of them.

Quoted 5 lines
Quoted 1 line

I'll try the cfq close cooperator patches next.

I have a pending update on the coop patch that isn't pushed out yet, I hope to have it finalized and tested later today. Hopefully, with that, we should be able to maintain > 100Mb/sec for 4 and 8 threads.

OK, please let me know when and where this is available and I'll give it a try.

Cheers,
Jeff

#10 lore raw
Quoted 35 lines · Jeff Moyer
Quoted 33 lines · Jens Axboe
Quoted 31 lines · Jeff Moyer

I tested this using iozone to read a file from an NFS client. The iozone command line was:
iozone -s 2000000 -r 64 -f /mnt/test/testfile -i 1 -w

The numbers in the nfsd's row represent the number of nfsd threads. I included numbers for the deadline scheduler as well for comparison.

               v2.6.29

nfsd's | 1 | 2 | 4 | 8
--------+---------------+-------+------

cfq     | 91356 | 66391 | 61942 | 51674
deadline| 43207 | 67436 | 96289 | 107784
              2.6.30-rc1

nfsd's | 1 | 2 | 4 | 8
--------+---------------+-------+------

cfq     | 43127 | 22354 | 20858 | 21179
deadline| 43732 | 68059 | 76659 | 83231
          2.6.30-rc1 + cfq fix

nfsd's | 1 | 2 | 4 | 8
--------+-----------------+-------+------
cfq | 114602 | 102280 | 43479 | 43160

As you can see, for 1 and 2 threads, the patch *really* helps out. We still don't get back the performance for 4 and 8 nfsd threads, though. It's interesting to note that the deadline scheduler regresses for 4 and 8 threads, as well. I think we've still got some digging to do.

Quoted 7 lines
Quoted 1 line

Wow, that does indeed look pretty good!

Please don't lose the later part of the message, which is that both CFQ and deadline regress for 4 and 8 threads when moving from 2.6.29 to 2.6.30-rc1! There have been way too many changes to keep track of in this area recently, and I'd like to get some confidence that we understand the performance implications of them.

I'm definitely not, in fact pending patches should make it better! For the nfs case, I'm still a big proponent of explicitly getting the io context sharing in place, instead of adding code at the other end to try and fix things up.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

I'll try the cfq close cooperator patches next.

I have a pending update on the coop patch that isn't pushed out yet, I hope to have it finalized and tested later today. Hopefully, with that, we should be able to maintain > 100Mb/sec for 4 and 8 threads.

OK, please let me know when and where this is available and I'll give it a try.

I sent you email about ~10 minutes ago or so about that very patch.

Signature · 1 line
Jens Axboe
#11 lore raw
Quoted 38 lines · Jens Axboe
Quoted 37 lines · Jeff Moyer
Quoted 34 lines · Jens Axboe
Quoted 23 lines · Jeff Moyer

I tested this using iozone to read a file from an NFS client. The iozone command line was:
iozone -s 2000000 -r 64 -f /mnt/test/testfile -i 1 -w

The numbers in the nfsd's row represent the number of nfsd threads. I included numbers for the deadline scheduler as well for comparison.

               v2.6.29

nfsd's | 1 | 2 | 4 | 8
--------+---------------+-------+------

cfq     | 91356 | 66391 | 61942 | 51674
deadline| 43207 | 67436 | 96289 | 107784
              2.6.30-rc1

nfsd's | 1 | 2 | 4 | 8
--------+---------------+-------+------

cfq     | 43127 | 22354 | 20858 | 21179
deadline| 43732 | 68059 | 76659 | 83231
          2.6.30-rc1 + cfq fix
Quoted 8 lines

nfsd's | 1 | 2 | 4 | 8
--------+-----------------+-------+------
cfq | 114602 | 102280 | 43479 | 43160

As you can see, for 1 and 2 threads, the patch *really* helps out. We still don't get back the performance for 4 and 8 nfsd threads, though. It's interesting to note that the deadline scheduler regresses for 4 and 8 threads, as well. I think we've still got some digging to do.

Wow, that does indeed look pretty good!

Quoted 7 lines
Quoted 5 lines

Please don't lose the later part of the message, which is that both CFQ and deadline regress for 4 and 8 threads when moving from 2.6.29 to 2.6.30-rc1! There have been way too many changes to keep track of in this area recently, and I'd like to get some confidence that we understand the performance implications of them.

I'm definitely not, in fact pending patches should make it better! For

OK, I'll look through the changelog of your for-linus branch.

Quoted 3 lines

the nfs case, I'm still a big proponent of explicitly getting the io context sharing in place, instead of adding code at the other end to try and fix things up.

OK, I have an ugly patch that wires that up. I guess it's time to clean that up and test it.

Quoted 10 lines
Quoted 8 lines
Quoted 5 lines
Quoted 1 line

I'll try the cfq close cooperator patches next.

I have a pending update on the coop patch that isn't pushed out yet, I hope to have it finalized and tested later today. Hopefully, with that, we should be able to maintain > 100Mb/sec for 4 and 8 threads.

OK, please let me know when and where this is available and I'll give it a try.

I sent you email about ~10 minutes ago or so about that very patch.

Yep, got it. Thanks!

-Jeff