Skip to contentSkip to conversation

CFQ iosched praise: good perfomance and better latency

DISCUSSION Pedro Larroy 4 participants 8 messages Apr 19, 2004

lore
Share & tools
lore
#0 lore raw

Hi

I've been trying CFQ ioscheduler in my software raid5 with nice results, I've observed that a latency pattern still exists, just as in the anticipatory ioscheduler, but those spikes are now much lower (from 6ms with AS to 2ms with CFQ as seen in the bottom of
http://pedro.larroy.com/devel/iolat/analisys/),
plus apps seems to get a fair amount of io so they don't get starved.

Seems a good choice for io loaded boxes. Thanks Jens Axboe.

Regards.

Signature · 4 lines
Pedro Larroy Tovar | Linux & Network consultant |  piotr%member.fsf.org 

Software patents are a threat to innovation in Europe please check: 
	http://www.eurolinux.org/     
#1 lore raw
Quoted 11 lines · Pedro Larroy

Hi

I've been trying CFQ ioscheduler in my software raid5 with nice results, I've observed that a latency pattern still exists, just as in the anticipatory ioscheduler, but those spikes are now much lower (from 6ms with AS to 2ms with CFQ as seen in the bottom of
http://pedro.larroy.com/devel/iolat/analisys/),
plus apps seems to get a fair amount of io so they don't get starved.

Seems a good choice for io loaded boxes. Thanks Jens Axboe.

Although AS isn't at its best when behind raid devices (it should probably be in front of them), you could be seeing some problem with the raid code.

I'd be interested to see what the graph looks like with elevator=noop

#2 lore raw
Quoted 11 lines · Nick Piggin
Quoted 9 lines · Pedro Larroy

Hi

I've been trying CFQ ioscheduler in my software raid5 with nice results, I've observed that a latency pattern still exists, just as in the anticipatory ioscheduler, but those spikes are now much lower (from 6ms with AS to 2ms with CFQ as seen in the bottom of
http://pedro.larroy.com/devel/iolat/analisys/),
plus apps seems to get a fair amount of io so they don't get starved.

Quoted 8 lines
Quoted 2 lines

Seems a good choice for io loaded boxes. Thanks Jens Axboe.

Although AS isn't at its best when behind raid devices (it should probably be in front of them), you could be seeing some problem with the raid code.

I'd be interested to see what the graph looks like with elevator=noop

This isn't a very surprising result, is it? AS throws away latency to gain throughput. Pedro is measuring latency...

#3 lore raw
Quoted 20 lines · Andrew Morton
Quoted 18 lines · Nick Piggin
Quoted 11 lines · Pedro Larroy

Hi

I've been trying CFQ ioscheduler in my software raid5 with nice results, I've observed that a latency pattern still exists, just as in the anticipatory ioscheduler, but those spikes are now much lower (from 6ms with AS to 2ms with CFQ as seen in the bottom of
http://pedro.larroy.com/devel/iolat/analisys/),
plus apps seems to get a fair amount of io so they don't get starved.

Seems a good choice for io loaded boxes. Thanks Jens Axboe.

Although AS isn't at its best when behind raid devices (it should probably be in front of them), you could be seeing some problem with the raid code.

Quoted 6 lines
Quoted 1 line

I'd be interested to see what the graph looks like with elevator=noop

This isn't a very surprising result, is it? AS throws away latency to gain throughput. Pedro is measuring latency...

Well I think Pedro actually means *seconds*, not ms. The URL shows AS peaks at nearly 10 seconds latency, and CFQ over 2s.

It really seems like a raid problem though, because latency measured at the individual devices is under 250ms for AS.

#4 lore raw
Quoted 24 lines · Nick Piggin
Quoted 23 lines · Andrew Morton
Quoted 19 lines · Nick Piggin
Quoted 11 lines · Pedro Larroy

Hi

I've been trying CFQ ioscheduler in my software raid5 with nice results, I've observed that a latency pattern still exists, just as in the anticipatory ioscheduler, but those spikes are now much lower (from 6ms with AS to 2ms with CFQ as seen in the bottom of
http://pedro.larroy.com/devel/iolat/analisys/),
plus apps seems to get a fair amount of io so they don't get starved.

Seems a good choice for io loaded boxes. Thanks Jens Axboe.

Although AS isn't at its best when behind raid devices (it should probably be in front of them), you could be seeing some problem with the raid code.

I'd be interested to see what the graph looks like with elevator=noop

Quoted 6 lines
Quoted 3 lines

This isn't a very surprising result, is it? AS throws away latency to gain throughput. Pedro is measuring latency...

Well I think Pedro actually means *seconds*, not ms. The URL shows AS peaks at nearly 10 seconds latency, and CFQ over 2s.

Yes, I meant seconds, my mistake. I will be testing elevator=noop this evening.

Quoted 3 lines

It really seems like a raid problem though, because latency measured at the individual devices is under 250ms for AS.

Probably. But I was surprised to find that bonnie gave similar results with CFQ and with AS when benchmarking the swraid5.

Regards.

Signature · 4 lines
Pedro Larroy Tovar | Linux & Network consultant |  piotr%member.fsf.org 

Software patents are a threat to innovation in Europe please check: 
	http://www.eurolinux.org/     
#5 lore raw
Quoted 9 lines · Pedro Larroy
Quoted 2 lines · Nick Piggin

Well I think Pedro actually means *seconds*, not ms. The URL shows AS peaks at nearly 10 seconds latency, and CFQ over 2s.

Yes, I meant seconds, my mistake. I will be testing elevator=noop this evening.

That would be interesting.

Quoted 7 lines
Quoted 3 lines

It really seems like a raid problem though, because latency measured at the individual devices is under 250ms for AS.

Probably. But I was surprised to find that bonnie gave similar results with CFQ and with AS when benchmarking the swraid5.

I haven't used bonnie, but I think it is single threaded, isn't it? If that is the case, then the IO scheduler will make little or no difference, so your result is not surprising.

#6 lore raw
Quoted 18 lines · Nick Piggin
Quoted 9 lines · Pedro Larroy
Quoted 2 lines · Nick Piggin

Well I think Pedro actually means *seconds*, not ms. The URL shows AS peaks at nearly 10 seconds latency, and CFQ over 2s.

Yes, I meant seconds, my mistake. I will be testing elevator=noop this evening.

That would be interesting.

Quoted 5 lines
Quoted 3 lines

It really seems like a raid problem though, because latency measured at the individual devices is under 250ms for AS.

Quoted 6 lines
Quoted 2 lines

Probably. But I was surprised to find that bonnie gave similar results with CFQ and with AS when benchmarking the swraid5.

I haven't used bonnie, but I think it is single threaded, isn't it? If that is the case, then the IO scheduler will make little or no difference, so your result is not surprising.

Seems your suspicions were correct, the delay patterns are pretty similar with all the schedulers, and the big delays aren't caused by the ioscheduler aparently. I've updated the graphs. In 2.6.5-mm3
at least, all the ioschedulers give alike latencies. I wonder now how did I get previous measures around 6000ms. I think I blamed a previous misbehaving kernel version on the ioscheduler. My apologies.

Is there any interest to hack in md code? IIRC the plans are to use dm in the near future.

Regards.

Signature · 4 lines
Pedro Larroy Tovar | Linux & Network consultant |  piotr%member.fsf.org 

Software patents are a threat to innovation in Europe please check: 
	http://www.eurolinux.org/     
#7 lore raw
In reply to Nick Piggin
Quoted 11 lines · Nick Piggin
Quoted 9 lines · Pedro Larroy

Hi
I've been trying CFQ ioscheduler in my software raid5 with nice results,
I've observed that a latency pattern still exists, just as in the anticipatory ioscheduler, but those spikes are now much lower (from 6ms with AS to 2ms with CFQ as seen in the bottom of
http://pedro.larroy.com/devel/iolat/analisys/),
plus apps seems to get a fair amount of io so they don't get starved. Seems a good choice for io loaded boxes. Thanks Jens Axboe.

Quoted 3 lines

Although AS isn't at its best when behind raid devices (it should probably be in front of them), you could be seeing some problem with the raid code.

Hrm. So, if AS isn't a good to have behind RAID devices, but is reasonable before them, is there any easy way to configure a system like that?

I don't see an easy way to change the IO scheduler on a per-device basis anywhere...

Daniel

Signature · 3 lines
A wonderful discovery, psychoanalysis.  
Makes quite simple people feel they're complex.
        -- S. N. Behrman