Skip to contentSkip to conversation

aio-stress throughput regressions from 2.6.11 to 2.6.12

DISCUSSION Suparna Bhattacharya 4 participants 9 messages Jul 1, 2005 – Jul 11, 2005

Subsystems (2)
VFS SCSI
lore
Share & tools
lore

Suparna Bhattacharya

author
#0 lore raw

Has anyone else noticed major throughput regressions for random reads/writes with aio-stress in 2.6.12 ?
Or have there been any other FS/IO regressions lately ?

On one test system I see a degradation from around 17+ MB/s to 11MB/s for random O_DIRECT AIO (aio-stress -o3 testext3/rwfile5) from 2.6.11 to 2.6.12. It doesn't seem filesystem specific. Not good :(

BTW, Chris/Ben, it doesn't look like the changes to aio.c have had an impact (I copied those back to my 2.6.11 tree and tried the runs with no effect) So it is something else ...

Ideas/thoughts/observations ?

Regards
Suparna

Signature · 3 lines
Suparna Bhattacharya ([email protected])
Linux Technology Center
IBM Software Lab, India
#1 lore raw
Quoted 3 lines · Suparna Bhattacharya

On one test system I see a degradation from around 17+ MB/s to 11MB/s for random O_DIRECT AIO (aio-stress -o3 testext3/rwfile5) from 2.6.11 to 2.6.12. It doesn't seem filesystem specific. Not good :(

What sort of io subsystem does it have? Also, does changing the elevator make any difference? I'm away for the long weekend, but will help look into this next week.

-ben

author
#2 lore raw
Quoted 6 lines · Benjamin LaHaise
Quoted 3 lines · Suparna Bhattacharya

On one test system I see a degradation from around 17+ MB/s to 11MB/s for random O_DIRECT AIO (aio-stress -o3 testext3/rwfile5) from 2.6.11 to 2.6.12. It doesn't seem filesystem specific. Not good :(

What sort of io subsystem does it have? Also, does changing the

aic7xxx.

Quoted 1 line

elevator make any difference? I'm away for the long weekend, but will

I tried switching to the noop elevator - the regression was still there.

Quoted 1 line

help look into this next week.

Do you see the regression as well, or is it just me ?

Regards
Suparna

Quoted 2 lines

-ben

Signature · 3 lines
Suparna Bhattacharya ([email protected])
Linux Technology Center
IBM Software Lab, India
#3 lore raw
Quoted 1 line · Suparna Bhattacharya

Do you see the regression as well, or is it just me ?

Throughput for me on an ICH6 SATA system using O_DIRECT seems to remain pretty constant across 2.6.11-something-FC3 to 2.6.13-rc2 around 31MB/s. 2.6.11 proper hangs for me, though.

-ben

Signature · 1 line
"Time is what keeps everything from happening all at once." -- John Wheeler
#4 lore raw
In reply to Suparna Bhattacharya
Quoted 8 lines · Suparna Bhattacharya

Has anyone else noticed major throughput regressions for random reads/writes with aio-stress in 2.6.12 ?
Or have there been any other FS/IO regressions lately ?

On one test system I see a degradation from around 17+ MB/s to 11MB/s for random O_DIRECT AIO (aio-stress -o3 testext3/rwfile5) from 2.6.11 to 2.6.12. It doesn't seem filesystem specific. Not good :(

Quoted 5 lines

BTW, Chris/Ben, it doesn't look like the changes to aio.c have had an impact (I copied those back to my 2.6.11 tree and tried the runs with no effect) So it is something else ...

Ideas/thoughts/observations ?

Lets try to narrow it down a bit:

aio-stress -o 3 -d 1 will set the depth to 1, (io_submit then wait one request at a time). This doesn't take the aio subsystem out of the picture, but it does make the block layer interaction more or less the same as non-aio benchmarks. If this is slow, I would suspect something in the block layer, and iozone -I -i 2 -w -f testext3/rwfile5 should also show the regression.

If it doesn't regress, I would suspect something in the aio core. My first attempts at the context switch reduction patches caused this kind of regression. There was too much latency in sending the events up to userland.

Other options:

Try different elevators
Try O_SYNC aio random writes
Try aio random reads
Try buffers random reads

-chris
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>

#5 lore raw
Quoted 3 lines · Chris Mason

If it doesn't regress, I would suspect something in the aio core. My first attempts at the context switch reduction patches caused this kind of regression. There was too much latency in sending the events up to userland.

AIO will by its very nature have a higher rate of context switches unless the subsystem in question does its completion from the interrupt handler. There are a few optimizations in this area that we can do to improve things for some of the common usage models: when sleeping in read_events() we can do iocb runs in the sleeping task instead of switching to the aio helper daemon. That might help for Oracle's usage model of aio, and it should also help aio-stress.

There are also other ways of reducing the overhead of the context switches. O_DIRECT operations could take note of the mm they are operating on and do its get_user_pages() on the mm without the tlb being flushed.

-ben

Signature · 6 lines
"Time is what keeps everything from happening all at once." -- John Wheeler
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to [email protected].  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
author
#6 lore raw
In reply to Chris Mason
Quoted 13 lines · Chris Mason
Quoted 12 lines · Suparna Bhattacharya

Has anyone else noticed major throughput regressions for random reads/writes with aio-stress in 2.6.12 ?
Or have there been any other FS/IO regressions lately ?

On one test system I see a degradation from around 17+ MB/s to 11MB/s for random O_DIRECT AIO (aio-stress -o3 testext3/rwfile5) from 2.6.11 to 2.6.12. It doesn't seem filesystem specific. Not good :(

BTW, Chris/Ben, it doesn't look like the changes to aio.c have had an impact (I copied those back to my 2.6.11 tree and tried the runs with no effect) So it is something else ...

Quoted 5 lines
Quoted 1 line

Ideas/thoughts/observations ?

Lets try to narrow it down a bit:

aio-stress -o 3 -d 1 will set the depth to 1, (io_submit then wait one request

This doesn't regress - the problem really happens when we don't wait one at a time.

Quoted 5 lines

at a time). This doesn't take the aio subsystem out of the picture, but it does make the block layer interaction more or less the same as non-aio benchmarks. If this is slow, I would suspect something in the block layer, and iozone -I -i 2 -w -f testext3/rwfile5 should also show the regression.

Quoted 7 lines

If it doesn't regress, I would suspect something in the aio core. My first attempts at the context switch reduction patches caused this kind of regression. There was too much latency in sending the events up to userland.

Other options:

Try different elevators

Still regresses (I tried noop instead of as)

Quoted 3 lines

Try O_SYNC aio random writes
Try aio random reads
Try buffers random reads

Again all these end up waiting one at a time with mainline because it forces buffered AIO to be synchronous, so we the regression doesn't show up. But, when I apply my patches to make buffered fsAIO async, so we aren't waiting one at a time -- there is a similar regression. In fact it was this regression that made me go back and check if it was happening with AIO-DIO as well.

Regards
Suparna

Quoted 2 lines

-chris

Signature · 9 lines
Suparna Bhattacharya ([email protected])
Linux Technology Center
IBM Software Lab, India

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to [email protected].  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
author
#7 lore raw

Re: aio-stress regressions in 2.6.12 narrowed down to AIC7xxx

OK, I think I have narrowed it down to aic7xxx driver, because when I specify CONFIG_SCSI_AIC7XXX_OLD, it does not regress.

The regression occurs from 2.6.11 to 2.6.12 (from 17MB/s it goes down to 11MB/s)

The regression is still there in 2.6.13-rc1 + the "speed fix" patch(discussed in the recent aic7xxx regression thread on linux-scsi)

Recreate by running: aio-stress -O -o3 <1 GB testfile>

Config options:
CONFIG_SCSI_AIC7XXX=y
CONFIG_AIC7XXX_CMDS_PER_DEVICE=32
CONFIG_AIC7XXX_RESET_DELAY_MS=15000
CONFIG_AIC7XXX_DEBUG_ENABLE=y
CONFIG_AIC7XXX_DEBUG_MASK=0
CONFIG_AIC7XXX_REG_PRETTY_PRINT=y

Comparing dmesg outputs
On 2.6.11
scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.36
<Adaptec aic7896/97 Ultra2 SCSI adapter> aic7896/97: Ultra2 Wide Channel B, SCSI Id=7, 32/253 SCBs

(scsi0:A:0): 80.000MB/s transfers (40.000MHz, offset 63, 16bit)
Vendor: IBM-ESXS Model: ST318305LC !# Rev: B245
Type: Direct-Access ANSI SCSI revision: 03
scsi0:A:0:0: Tagged Queuing enabled. Depth 32 (scsi0:A:1): 80.000MB/s transfers (40.000MHz, offset 63, 16bit)
Vendor: IBM-ESXS Model: ST318305LC !# Rev: B245
...

On 2.6.12
scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.36
<Adaptec aic7896/97 Ultra2 SCSI adapter> aic7896/97: Ultra2 Wide Channel B, SCSI Id=7, 32/253 SCBs

Vendor: IBM-ESXS Model: ST318305LC !# Rev: B245
Type: Direct-Access ANSI SCSI revision: 03
scsi0:A:0:0: Tagged Queuing enabled. Depth 32
target0:0:0: Beginning Domain Validation
WIDTH IS 1
(scsi0:A:0): 6.600MB/s transfers (16bit)
(scsi0:A:0): 80.000MB/s transfers (40.000MHz, offset 63, 16bit)
target0:0:0: Ending Domain Validation
Vendor: IBM-ESXS Model: ST318305LC !# Rev: B245
...

On 2.6.13-rc1 + "speed fix"
scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.36
<Adaptec aic7896/97 Ultra2 SCSI adapter> aic7896/97: Ultra2 Wide Channel B, SCSI Id=7, 32/253 SCBs

target0:0:0: asynchronous.
Vendor: IBM-ESXS Model: ST318305LC !# Rev: B245
Type: Direct-Access ANSI SCSI revision: 03
scsi0:A:0:0: Tagged Queuing enabled. Depth 32
target0:0:0: Beginning Domain Validation target0:0:0: wide asynchronous.
target0:0:0: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 63) target0:0:0: Ending Domain Validation
target0:0:1: asynchronous.
Vendor: IBM-ESXS Model: ST318305LC !# Rev: B245
...

Regards
Suparna

Quoted 64 lines · Suparna Bhattacharya
Quoted 13 lines · Chris Mason
Quoted 12 lines · Suparna Bhattacharya

Has anyone else noticed major throughput regressions for random reads/writes with aio-stress in 2.6.12 ?
Or have there been any other FS/IO regressions lately ?

On one test system I see a degradation from around 17+ MB/s to 11MB/s for random O_DIRECT AIO (aio-stress -o3 testext3/rwfile5) from 2.6.11 to 2.6.12. It doesn't seem filesystem specific. Not good :(

BTW, Chris/Ben, it doesn't look like the changes to aio.c have had an impact (I copied those back to my 2.6.11 tree and tried the runs with no effect) So it is something else ...

Quoted 5 lines
Quoted 1 line

Ideas/thoughts/observations ?

Lets try to narrow it down a bit:

aio-stress -o 3 -d 1 will set the depth to 1, (io_submit then wait one request

This doesn't regress - the problem really happens when we don't wait one at a time.

Quoted 5 lines

at a time). This doesn't take the aio subsystem out of the picture, but it does make the block layer interaction more or less the same as non-aio benchmarks. If this is slow, I would suspect something in the block layer, and iozone -I -i 2 -w -f testext3/rwfile5 should also show the regression.

Quoted 7 lines

If it doesn't regress, I would suspect something in the aio core. My first attempts at the context switch reduction patches caused this kind of regression. There was too much latency in sending the events up to userland.

Other options:

Try different elevators

Still regresses (I tried noop instead of as)

Quoted 3 lines

Try O_SYNC aio random writes
Try aio random reads
Try buffers random reads

Again all these end up waiting one at a time with mainline because it forces buffered AIO to be synchronous, so we the regression doesn't show up. But, when I apply my patches to make buffered fsAIO async, so we aren't waiting one at a time -- there is a similar regression. In fact it was this regression that made me go back and check if it was happening with AIO-DIO as well.

Regards
Suparna

Quoted 2 lines

-chris

Signature · 9 lines
Suparna Bhattacharya ([email protected])
Linux Technology Center
IBM Software Lab, India

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to [email protected].  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Signature · 3 lines
Suparna Bhattacharya ([email protected])
Linux Technology Center
IBM Software Lab, India
#8 lore raw
In reply to Suparna Bhattacharya
Quoted 8 lines · Suparna Bhattacharya

Has anyone else noticed major throughput regressions for random reads/writes with aio-stress in 2.6.12 ?
Or have there been any other FS/IO regressions lately ?

On one test system I see a degradation from around 17+ MB/s to 11MB/s for random O_DIRECT AIO (aio-stress -o3 testext3/rwfile5) from 2.6.11 to 2.6.12. It doesn't seem filesystem specific. Not good :(

Quoted 9 lines

BTW, Chris/Ben, it doesn't look like the changes to aio.c have had an impact (I copied those back to my 2.6.11 tree and tried the runs with no effect) So it is something else ...

Ideas/thoughts/observations ?

Regards
Suparna

I'm too seeing a regression, but between 2.6.10 and 2.6.12 and using
sysbench + MySQL (with POSIX AIO). The difference is roughly 8% slower for 2.6.12. I'm currently trying to trace it.

aio-stress shows no difference so it probably does not come from
kernel AIO.

Sébastien.

Signature · 16 lines
------------------------------------------------------

  Sébastien Dugué                BULL/FREC:B1-247
  phone: (+33) 476 29 77 70      Bullcom: 229-7770

  mailto:[email protected]

  Linux POSIX AIO: http://www.bullopensource.org/posix
  
------------------------------------------------------

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to [email protected].  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>