Skip to contentSkip to conversation

async I/O seems to be blocking on 2.6.15

DISCUSSION Brent Baccala 3 participants 3 messages Nov 7, 2006

lore
Share & tools
lore
#0 lore raw
Quoted 9 lines · Kenneth W

I've tried that myself too and see similar result. One thing to note is that I/O being submitted are pretty big at 1MB, so the vector list inside bio is going to be pretty long and it will take a while to construct that. Drop the size for each I/O to something like 4KB will significantly reduce the time. I haven't done the measurement whether the time to submit I/O grows linearly with respect to I/O size. Most likely it will. If it is not, then we might have a scaling problem (though I don't believe we have this problem).

Quoted 3 lines

- Ken

I'm basically an end user here (as far as the kernel is concerned), so let me ask the basic "dumb user" question here:

How should I do my async I/O if I just want to read or write sequentially through a file, using O_DIRECT, and letting the CPU get some work done in the meantime? What about more random access?

I've already concluded that I should try to keep my read and write files on seperate disks and hopefully on seperate controllers, but I still seem to be fighting this thing to keep it from blocking.

-bwb

Brent Baccala [email protected]

#1 lore raw

Brent Baccala wrote on Monday, November 06, 2006 4:04 PM

Quoted 13 lines · Kenneth W
Quoted 8 lines

I've tried that myself too and see similar result. One thing to note is that I/O being submitted are pretty big at 1MB, so the vector list inside bio is going to be pretty long and it will take a while to construct that. Drop the size for each I/O to something like 4KB will significantly reduce the time. I haven't done the measurement whether the time to submit I/O grows linearly with respect to I/O size. Most likely it will. If it is not, then we might have a scaling problem (though I don't believe we have this problem).

I'm basically an end user here (as far as the kernel is concerned), so let me ask the basic "dumb user" question here:

Quoted 7 lines

How should I do my async I/O if I just want to read or write sequentially through a file, using O_DIRECT, and letting the CPU get some work done in the meantime? What about more random access?

I've already concluded that I should try to keep my read and write files on seperate disks and hopefully on seperate controllers, but I still seem to be fighting this thing to keep it from blocking.

You bark rightfully at the fact that kernel blocks in the async path when block queue is full. It would be nice that -EAGAIN is returned so app can do other stuff.

My earlier post was commenting on the measurement of I/O submit when nothing blocks (which happens to be extremely important to me). Your measurement of 7ms seems excessive. While mine and Jens came at around 90 micro-second which isn't all that unreasonable given the size of the I/O request.

- Ken

#2 lore raw
In reply to Brent Baccala
Quoted 15 lines · Brent Baccala
Quoted 9 lines · Kenneth W

I've tried that myself too and see similar result. One thing to note is that I/O being submitted are pretty big at 1MB, so the vector list inside bio is going to be pretty long and it will take a while to construct that. Drop the size for each I/O to something like 4KB will significantly reduce the time. I haven't done the measurement whether the time to submit I/O grows linearly with respect to I/O size. Most likely it will. If it is not, then we might have a scaling problem (though I don't believe we have this problem).

Quoted 3 lines

- Ken

Quoted 6 lines

I'm basically an end user here (as far as the kernel is concerned), so let me ask the basic "dumb user" question here:

How should I do my async I/O if I just want to read or write sequentially through a file, using O_DIRECT, and letting the CPU get some work done in the meantime? What about more random access?

For sequential io, you'll pretty quickly hit the transfer size sweet spot where growing the io larger wont yield any benefits. In this case you don't want a huge queue depth, 100 ios is an insane amount for sequential io. If you have a properly sized io unit and a depth of 4 or so, I doubt you'll see any improvement beyond that on most hardware.

For random io, you want a bigger queue depth. If the hardware can do queueing, you want to make sure that you can fill that queue. 100 ios is still a lot in this case, though.

Quoted 3 lines

I've already concluded that I should try to keep my read and write files on seperate disks and hopefully on seperate controllers, but I still seem to be fighting this thing to keep it from blocking.

Shrink your queue size and/or io size :-)

Signature · 1 line
Jens Axboe