async I/O seems to be blocking on 2.6.15
Quoted 9 lines · Kenneth W
I've tried that myself too and see similar result. One thing to note is that I/O being submitted are pretty big at 1MB, so the vector list inside bio is going to be pretty long and it will take a while to construct that. Drop the size for each I/O to something like 4KB will significantly reduce the time. I haven't done the measurement whether the time to submit I/O grows linearly with respect to I/O size. Most likely it will. If it is not, then we might have a scaling problem (though I don't believe we have this problem).
Quoted 3 lines
- Ken
I'm basically an end user here (as far as the kernel is concerned), so let me ask the basic "dumb user" question here:
How should I do my async I/O if I just want to read or write sequentially through a file, using O_DIRECT, and letting the CPU get some work done in the meantime? What about more random access?
I've already concluded that I should try to keep my read and write files on seperate disks and hopefully on seperate controllers, but I still seem to be fighting this thing to keep it from blocking.
-bwb
Brent Baccala [email protected]
Brent Baccala wrote on Monday, November 06, 2006 4:04 PM
Quoted 13 lines · Kenneth W
Quoted 8 lines
I've tried that myself too and see similar result. One thing to note is that I/O being submitted are pretty big at 1MB, so the vector list inside bio is going to be pretty long and it will take a while to construct that. Drop the size for each I/O to something like 4KB will significantly reduce the time. I haven't done the measurement whether the time to submit I/O grows linearly with respect to I/O size. Most likely it will. If it is not, then we might have a scaling problem (though I don't believe we have this problem).
I'm basically an end user here (as far as the kernel is concerned), so let me ask the basic "dumb user" question here:
Quoted 7 lines
How should I do my async I/O if I just want to read or write sequentially through a file, using
O_DIRECT, and letting the CPU get some work done in the meantime? What about more random access?I've already concluded that I should try to keep my read and write files on seperate disks and hopefully on seperate controllers, but I still seem to be fighting this thing to keep it from blocking.
You bark rightfully at the fact that kernel blocks in the async path when block queue is full. It would be nice that -EAGAIN is returned so app can do other stuff.
My earlier post was commenting on the measurement of I/O submit when nothing blocks (which happens to be extremely important to me). Your measurement of 7ms seems excessive. While mine and Jens came at around 90 micro-second which isn't all that unreasonable given the size of the I/O request.
- Ken
Quoted 15 lines · Brent Baccala
Quoted 9 lines · Kenneth W
I've tried that myself too and see similar result. One thing to note is that I/O being submitted are pretty big at 1MB, so the vector list inside bio is going to be pretty long and it will take a while to construct that. Drop the size for each I/O to something like 4KB will significantly reduce the time. I haven't done the measurement whether the time to submit I/O grows linearly with respect to I/O size. Most likely it will. If it is not, then we might have a scaling problem (though I don't believe we have this problem).
Quoted 3 lines
- Ken
Quoted 6 lines
I'm basically an end user here (as far as the kernel is concerned), so let me ask the basic "dumb user" question here:
How should I do my async I/O if I just want to read or write sequentially through a file, using
O_DIRECT, and letting the CPU get some work done in the meantime? What about more random access?
For sequential io, you'll pretty quickly hit the transfer size sweet spot where growing the io larger wont yield any benefits. In this case you don't want a huge queue depth, 100 ios is an insane amount for sequential io. If you have a properly sized io unit and a depth of 4 or so, I doubt you'll see any improvement beyond that on most hardware.
For random io, you want a bigger queue depth. If the hardware can do queueing, you want to make sure that you can fill that queue. 100 ios is still a lot in this case, though.
Quoted 3 lines
I've already concluded that I should try to keep my read and write files on seperate disks and hopefully on seperate controllers, but I still seem to be fighting this thing to keep it from blocking.
Shrink your queue size and/or io size :-)
Signature · 1 line
Jens Axboe