Skip to contentSkip to conversation

I/O regression after 2.6.0-test5

BENCHMARK [email protected] 4 participants 10 messages Oct 21, 2003 – Oct 23, 2003 Block layer

lore
Share & tools
lore

[email protected]

author
#0 lore raw
Quoted 2 lines

If you have time, would you please try testing as-iosched.c from test5 in a later kernel (it won't go into test8-mm1 though).

copying drivers/block/as-iosched.c from 2.6.0-test5 to test8 looks like it fixes the regression. Here are the results so far.

In the AIM7 database benchmark, jobs/minute using test5 as-iosched.c doubled. At 32 tasks, jobs/min went from 300 to 601.
At 256 tasks, jobs/min went from 552 to 1010.

test5 as-iosched.c:
cat suite7-dbase-2.6.0-test8-as
Benchmark Version Machine Run Date
AIM Multiuser Benchmark - Suite VII "1.1" dev4-003

Tasks   Jobs/Min        Real    CPU     
32      601.0           316.3   134.0   
64      782.6           485.8   259.2   
96      869.1           656.1   387.6   
128     919.3           827.0   522.8   
160     955.0           995.2   650.3   
192     981.9           1161.5  790.9   
224     1002.3          1327.5  910.2   
256     1009.9          1505.7  1027.7

Unmodififed test8.
cat suite7-dbase-2.6.0-test8
Benchmark Version Machine Run Date
AIM Multiuser Benchmark - Suite VII "1.1" dev4-003

Tasks   Jobs/Min        Real    CPU     
32      299.8           634.0   128.6   
64      450.5           843.9   246.5   
96      549.7           1037.4  366.0   
128     558.7           1360.8  497.6   
160     522.1           1820.2  612.2   
192     515.5           2212.4  742.7   
224     529.9           2510.8  859.5   
256     552.4           2753.0  980.4

The AIM7 fileserver benchmark is back up to previous levels when test8 uses the test5 as-iosched.c:

test5 as-iosched.c with test8.
cat suite7-fserver-2.6.0-test8-as
Benchmark Version Machine Run Date
AIM Multiuser Benchmark - Suite VII "1.1" dev4-003

Tasks   Jobs/Min        Real    CPU     
4       80.5            301.0   37.0    
8       133.0           364.6   64.0    
12      166.7           436.3   93.4    
16      175.9           551.2   111.5   
20      186.3           650.7   133.8   
24      202.8           717.3   161.2   
28      207.1           819.3   200.0   
32      210.6           920.7   226.5

Unmodified test8.
cat suite7-fserver-2.6.0-test8
Benchmark Version Machine Run Date
AIM Multiuser Benchmark - Suite VII "1.1" dev4-003

Tasks   Jobs/Min        Real    CPU     
4       42.3            573.6   40.9    
8       63.3            765.7   71.6    
12      83.2            874.2   95.0    
16      87.5            1108.4  102.5   
20      105.0           1154.6  136.7   
24      111.2           1308.1  162.2   
28      130.7           1297.9  185.8   
32      134.2           1444.5  212.9

My tiobench test hasn't completed yet, but test8 with test5's as-iosched.c is making good progress. I think Nick nailed the changeset.

Signature · 2 lines
Randy Hron
http://home.earthlink.net/~rwhron/kernel/bigbox.html
#1 lore raw
Quoted 11 lines · [email protected] wrote:
Quoted 3 lines

If you have time, would you please try testing as-iosched.c from test5 in a later kernel (it won't go into test8-mm1 though).

copying drivers/block/as-iosched.c from 2.6.0-test5 to test8 looks like it fixes the regression. Here are the results so far.

In the AIM7 database benchmark, jobs/minute using test5 as-iosched.c doubled. At 32 tasks, jobs/min went from 300 to 601.
At 256 tasks, jobs/min went from 552 to 1010.

Yep, looks right. Thanks for your thoroughness. There are a few important direct IO bug fixes gone in there as well (hopefully they aren't causing the regressions... unlikely), so for stability you'll want to continue to use the test8 version.

I'll get a patch out soon.

#2 lore raw
In reply to [email protected]

sql bench on mysql, una master, three slave, with old as-iosched.c runs mutch better, thanx.

Quoted 96 lines · On Tue, 21 Oct 2003 [email protected] wrote:

Date: Tue, 21 Oct 2003 09:05:01 -0400
From: [email protected]
To: [email protected], [email protected]
Cc: [email protected], [email protected], [email protected]
Subject: Re: [BENCHMARK] I/O regression after 2.6.0-test5

Quoted 2 lines

If you have time, would you please try testing as-iosched.c from test5 in a later kernel (it won't go into test8-mm1 though).

copying drivers/block/as-iosched.c from 2.6.0-test5 to test8 looks like it fixes the regression. Here are the results so far.

In the AIM7 database benchmark, jobs/minute using test5 as-iosched.c doubled. At 32 tasks, jobs/min went from 300 to 601.
At 256 tasks, jobs/min went from 552 to 1010.

test5 as-iosched.c:
cat suite7-dbase-2.6.0-test8-as
Benchmark Version Machine Run Date
AIM Multiuser Benchmark - Suite VII "1.1" dev4-003

Tasks   Jobs/Min        Real    CPU
32      601.0           316.3   134.0
64      782.6           485.8   259.2
96      869.1           656.1   387.6
128     919.3           827.0   522.8
160     955.0           995.2   650.3
192     981.9           1161.5  790.9
224     1002.3          1327.5  910.2
256     1009.9          1505.7  1027.7

Unmodififed test8.
cat suite7-dbase-2.6.0-test8
Benchmark Version Machine Run Date
AIM Multiuser Benchmark - Suite VII "1.1" dev4-003

Tasks   Jobs/Min        Real    CPU
32      299.8           634.0   128.6
64      450.5           843.9   246.5
96      549.7           1037.4  366.0
128     558.7           1360.8  497.6
160     522.1           1820.2  612.2
192     515.5           2212.4  742.7
224     529.9           2510.8  859.5
256     552.4           2753.0  980.4

The AIM7 fileserver benchmark is back up to previous levels when test8 uses the test5 as-iosched.c:

test5 as-iosched.c with test8.
cat suite7-fserver-2.6.0-test8-as
Benchmark Version Machine Run Date
AIM Multiuser Benchmark - Suite VII "1.1" dev4-003

Tasks   Jobs/Min        Real    CPU
4       80.5            301.0   37.0
8       133.0           364.6   64.0
12      166.7           436.3   93.4
16      175.9           551.2   111.5
20      186.3           650.7   133.8
24      202.8           717.3   161.2
28      207.1           819.3   200.0
32      210.6           920.7   226.5

Unmodified test8.
cat suite7-fserver-2.6.0-test8
Benchmark Version Machine Run Date
AIM Multiuser Benchmark - Suite VII "1.1" dev4-003

Tasks   Jobs/Min        Real    CPU
4       42.3            573.6   40.9
8       63.3            765.7   71.6
12      83.2            874.2   95.0
16      87.5            1108.4  102.5
20      105.0           1154.6  136.7
24      111.2           1308.1  162.2
28      130.7           1297.9  185.8
32      134.2           1444.5  212.9

My tiobench test hasn't completed yet, but test8 with test5's as-iosched.c is making good progress. I think Nick nailed the changeset.

--
Randy Hron
http://home.earthlink.net/~rwhron/kernel/bigbox.html

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#3 lore raw
In reply to [email protected]
Quoted 7 lines · [email protected] wrote:
Quoted 3 lines

If you have time, would you please try testing as-iosched.c from test5 in a later kernel (it won't go into test8-mm1 though).

copying drivers/block/as-iosched.c from 2.6.0-test5 to test8 looks like it fixes the regression. Here are the results so far.

Could you please try the following patch against 2.6.0-test8. Thanks. I'll have to come up with something more comprehensive because this destroys find | xargs grep during other IO. It looks like this might be the cause of a lot of AS's database problems though.

I'm working on something...

 linux-2.6-npiggin/drivers/block/as-iosched.c |   23 +++--------------------
 1 files changed, 3 insertions(+), 20 deletions(-)
drivers/block/as-iosched.c renamed from drivers/block/as-iosched.c~as-fix +3 -20
diff -puN drivers/block/as-iosched.c~as-fix drivers/block/as-iosched.c
@@ -99,7 +99,6 @@ struct as_data {
 	sector_t last_sector[2];	/* last REQ_SYNC & REQ_ASYNC sectors */
 	struct list_head *dispatch;	/* driver dispatch queue */
 	struct list_head *hash;		/* request hash */
-	unsigned long new_success; /* anticipation success on new proc */
 	unsigned long current_batch_expires;
 	unsigned long last_check_fifo[2];
 	int changed_batch;		/* 1: waiting for old batch to end */

@@ -585,18 +584,11 @@ static void as_antic_stop(struct as_data
 	int status = ad->antic_status;
 
 	if (status == ANTIC_WAIT_REQ || status == ANTIC_WAIT_NEXT) {
-		struct as_io_context *aic;
-
 		if (status == ANTIC_WAIT_NEXT)
 			del_timer(&ad->antic_timer);
 		ad->antic_status = ANTIC_FINISHED;
 		/* see as_work_handler */
 		kblockd_schedule_work(&ad->antic_work);
-
-		aic = ad->io_context->aic;
-		if (aic->seek_samples == 0)
-			/* new process */
-			ad->new_success = (ad->new_success * 3) / 4 + 256;
 	}
 }
 

@@ -612,14 +604,8 @@ static void as_antic_timeout(unsigned lo
 	spin_lock_irqsave(q->queue_lock, flags);
 	if (ad->antic_status == ANTIC_WAIT_REQ
 			|| ad->antic_status == ANTIC_WAIT_NEXT) {
-		struct as_io_context *aic;
 		ad->antic_status = ANTIC_FINISHED;
 		kblockd_schedule_work(&ad->antic_work);
-
-		aic = ad->io_context->aic;
-		if (aic->seek_samples == 0)
-			/* new process */
-			ad->new_success = (ad->new_success * 3) / 4;
 	}
 	spin_unlock_irqrestore(q->queue_lock, flags);
 }

@@ -708,11 +694,10 @@ static int as_can_break_anticipation(str
 		return 1;
 	}
 
-	if (ad->new_success < 256 &&
-			(aic->seek_samples == 0 || aic->ttime_samples == 0)) {
+	if (aic->seek_samples == 0 && aic->ttime_samples == 0) {
 		/*
-		 * Process has just started IO and we have a bad history of
-		 * success anticipating on new processes!
+		 * Process has just started IO. Don't anticipate.
+		 * TODO! Must fix this up.
 		 */
 		return 1;
 	}

@@ -1822,8 +1807,6 @@ static int as_init(request_queue_t *q, e
 	if (ad->write_batch_count < 2)
 		ad->write_batch_count = 2;
 
-	ad->new_success = 512;
-
 	return 0;
 }

_

#4 lore raw

Sorry, this patch didn't fix our performance problems. Mary just finished running dbt2 on test8 with your patch:

NOTPM   kernel          scheduler
965     2.6.0-test8-np  AS
1632    2.6.-test6-mm4  deadline

This is an 8-way system with DAC960 and 12 LUNs, using raw devices. That's still quite a sizeable drop.

Quoted 9 lines · Nick Piggin

Could you please try the following patch against 2.6.0-test8. Thanks. I'll have to come up with something more comprehensive because this destroys find | xargs grep during other IO. It looks like this might be the cause of a lot of AS's database problems though.

I'm working on something...

#5 lore raw

I don't know how useful this might but. I've been trying to track down which patches to as-iosched resulted in the performance drop. The results so far are a little confusing. I have been running reaim on an 8-way system, comparing as-iosched performance with deadline, on each kernel version. The numbers are "number of jobs per minute". So, larger is better.

Deadline As-iosched
2.6.0-test5 8542 8589
2.6.0-test5-mm1 8303 8401
2.6.0-test5-mm2 8309 8224
2.6.0-test5-mm3 8222 8417
2.6.0-test6 8302 6934 ****
2.6.0-test6-mm1 8375 8163
2.6.0-test6-mm2 ??? 8309

I'm still getting data on test6-mm*. It's curious that performance dropped for test6, but came back for test6-mm1.

Quoted 11 lines · Dave Olien

Sorry, this patch didn't fix our performance problems. Mary just finished running dbt2 on test8 with your patch:

NOTPM   kernel          scheduler
965     2.6.0-test8-np  AS
1632    2.6.-test6-mm4  deadline

This is an 8-way system with DAC960 and 12 LUNs, using raw devices. That's still quite a sizeable drop.

#6 lore raw
In reply to Dave Olien
Quoted 7 lines · Dave Olien

Sorry, this patch didn't fix our performance problems. Mary just finished running dbt2 on test8 with your patch:

NOTPM   kernel          scheduler
965     2.6.0-test8-np  AS
1632    2.6.-test6-mm4  deadline

Thanks. hmm.
And NOTPM was better with AS in test5? Does using as-iosched.c from test5 in a test8 kernel help?

#7 lore raw
Quoted 7 lines · Nick Piggin
Quoted 3 lines · Dave Olien

Sorry, this patch didn't fix our performance problems. Mary just finished running dbt2 on test8 with your patch:

Quoted 9 lines
Quoted 4 lines
NOTPM   kernel          scheduler
965     2.6.0-test8-np  AS
1632    2.6.-test6-mm4  deadline

Thanks. hmm.
And NOTPM was better with AS in test5? Does using as-iosched.c from test5 in a test8 kernel help?

If that helps, can you try this patch.

 linux-2.6-npiggin/drivers/block/as-iosched.c |   23 +++--------------------
 1 files changed, 3 insertions(+), 20 deletions(-)
drivers/block/as-iosched.c renamed from drivers/block/as-iosched.c~as-fix +3 -20
diff -puN drivers/block/as-iosched.c~as-fix drivers/block/as-iosched.c
@@ -99,7 +99,6 @@ struct as_data {
 	sector_t last_sector[2];	/* last REQ_SYNC & REQ_ASYNC sectors */
 	struct list_head *dispatch;	/* driver dispatch queue */
 	struct list_head *hash;		/* request hash */
-	unsigned long new_success; /* anticipation success on new proc */
 	unsigned long current_batch_expires;
 	unsigned long last_check_fifo[2];
 	int changed_batch;		/* 1: waiting for old batch to end */

@@ -585,18 +584,11 @@ static void as_antic_stop(struct as_data
 	int status = ad->antic_status;
 
 	if (status == ANTIC_WAIT_REQ || status == ANTIC_WAIT_NEXT) {
-		struct as_io_context *aic;
-
 		if (status == ANTIC_WAIT_NEXT)
 			del_timer(&ad->antic_timer);
 		ad->antic_status = ANTIC_FINISHED;
 		/* see as_work_handler */
 		kblockd_schedule_work(&ad->antic_work);
-
-		aic = ad->io_context->aic;
-		if (aic->seek_samples == 0)
-			/* new process */
-			ad->new_success = (ad->new_success * 3) / 4 + 256;
 	}
 }
 

@@ -612,14 +604,8 @@ static void as_antic_timeout(unsigned lo
 	spin_lock_irqsave(q->queue_lock, flags);
 	if (ad->antic_status == ANTIC_WAIT_REQ
 			|| ad->antic_status == ANTIC_WAIT_NEXT) {
-		struct as_io_context *aic;
 		ad->antic_status = ANTIC_FINISHED;
 		kblockd_schedule_work(&ad->antic_work);
-
-		aic = ad->io_context->aic;
-		if (aic->seek_samples == 0)
-			/* new process */
-			ad->new_success = (ad->new_success * 3) / 4;
 	}
 	spin_unlock_irqrestore(q->queue_lock, flags);
 }

@@ -708,11 +694,10 @@ static int as_can_break_anticipation(str
 		return 1;
 	}
 
-	if (ad->new_success < 256 &&
-			(aic->seek_samples == 0 || aic->ttime_samples == 0)) {
+	if (aic->seek_samples == 0 || aic->ttime_samples == 0) {
 		/*
-		 * Process has just started IO and we have a bad history of
-		 * success anticipating on new processes!
+		 * Process has just started IO. Don't anticipate.
+		 * TODO! Must fix this up.
 		 */
 		return 1;
 	}

@@ -1822,8 +1807,6 @@ static int as_init(request_queue_t *q, e
 	if (ad->write_batch_count < 2)
 		ad->write_batch_count = 2;
 
-	ad->new_success = 512;
-
 	return 0;
 }

_

#8 lore raw
In reply to Nick Piggin

test5 didn't work with dbt2. The database run kept hanging. I've been running reaim, which doesn't have those problems with older kernels, but has shown similar performance problems.

I need to also check again that reaim and dbt2 still show similar performance issues with newer kernel versions.

I've lost my test machine for today. But I'll try test5 as-iosched in a test8 kernel with reaim tomorrow. If it works well, I'll try your following patch. Probably get it in the mail tomorrow afternoon.

Dave

Quoted 16 lines · Nick Piggin
Quoted 7 lines · Dave Olien

Sorry, this patch didn't fix our performance problems. Mary just finished running dbt2 on test8 with your patch:

NOTPM   kernel          scheduler
965     2.6.0-test8-np  AS
1632    2.6.-test6-mm4  deadline

Thanks. hmm.
And NOTPM was better with AS in test5? Does using as-iosched.c from test5 in a test8 kernel help?

#9 lore raw
Quoted 11 lines · Dave Olien

test5 didn't work with dbt2. The database run kept hanging. I've been running reaim, which doesn't have those problems with older kernels, but has shown similar performance problems.

I need to also check again that reaim and dbt2 still show similar performance issues with newer kernel versions.

I've lost my test machine for today. But I'll try test5 as-iosched in a test8 kernel with reaim tomorrow. If it works well, I'll try your following patch. Probably get it in the mail tomorrow afternoon.

Hi Dave,
Yeah thats right - I forgot. It was hanging because of as-iosched.c too, so don't bother trying to run it :(

Anyway, Randy can reproduce regressions, so I'll work on his first.