adding aio_readv/writev
Ben,
Currently there is no way to initiate an aio readv/writev in 2.5. There were no aio_readv/writev calls in 2.4 either - I'm wondering if there was any particular reason for excluding readv/writev operations from aio ?
The read/readv paths have anyway been merged for raw/O_DIRECT and regular file read/writes. So why not expose the vector read/write to the user by adding the IOCB_CMD_PREADV/IOCB_CMD_READV and IOCB_CMD_PWRITEV/IOCB_CMD_WRITEV commands to the aio set. Without that, raw/O_DIRECT readv users would need to unnecessarily cycle through their iovecs at a library level submitting them individually. For larger iovecs, user/library code would needlessly deal with multiple completions. While I'm not sure of the performance impact of the absence of aio_readv/writev, it seems easy enough to provide.
Most of the functions are already in place. We would only
need a way to pass the iovec through the iocb.
I was thinking of something like this:
struct iocb {
+union {
__u64 aio_buf
+ __u64 aio_iovp
+}
+union {
__u64 aio_nbytes+ __u64 aio_nsegs
+}
allowing the iovec * & nsegs to be passed into sys_io_submit. Some code would be added (within case handling of IOCB_CMD_READV within io_submit_one) to copy & verify the iovec pointers and then call aio_readv/aio_writev (if its defined for the fs).
What do you think ? I wanted to get some feedback before trying to code this up.
While we are on the topic of expanding aio operations, what about providing IOCB_CMD_READ/WRITE, distinct from their pread/pwrite counterparts ? Do you think thats needed ?
- Shailabh
A message that is not in the archive (sent off-list or not yet received): 1032555981.2082.10.camel@dell_ss3.pdx.osdl.net
Quoted 3 lines · Stephen Hemminger
Why not batch up multiple requests with one
io_submit? It has the same effect, except there would be multiple responses.
Even though the multiple iocb's enter the kernel together, they still get processed individually so a fair amount of unnecessary data transmission and function invocation are still occurring in the submit code path.
Depending on how long it takes for io_submit_one to return, there might be a reduced probability for merging of io requests at the i/o scheduler.
Finally, the multiple responses need to be handled as you mentioned. I suppose the application could wait for the last request (in the io_submit list) and that would most probably ensure that the preceding ones were complete as well but its not a guarantee offered by the aio API, right ? Besides, the application needs the data (represented by multiple requests) at one go so partial completion isn't likely to be useful and will only be an overhead.
While a quantitative assessment of the above tradeoffs is possible, it will be difficult to make a good comparison before "true" aio functionality is in place for 2.5. Such an assessment is unlikely to happen before the feature freeze takes effect. So I'm making a case for putting in async vector I/O interfaces in for the following three reasons:
- the synchronous API does provide separate entry points for vector I/O. Extending the same to the async interfaces, especially when it doesn't even involve creating new syscalls, seems natural for completeness.
- underlying in-kernel infrastructure already supports it, so no major changes are needed.
- there exists atleast one major application class (databases) that uses vectored I/O heavily and benefits from async I/O. Hence async vectored I/O is also likely to be useful. Can anyone else with experience on other OS's comment on this ?
Comments, reasons for not doing async readv/writev directly welcome.
- Shailabh
Quoted 34 lines · Shailabh Nagar
Quoted 31 lines
Ben,
Currently there is no way to initiate an aio readv/writev in 2.5. There were no aio_readv/writev calls in 2.4 either - I'm wondering if there was any particular reason for excluding readv/writev operations from aio ?
The read/readv paths have anyway been merged for raw/O_DIRECT and regular file read/writes. So why not expose the vector read/write to the user by adding the IOCB_CMD_PREADV/IOCB_CMD_READV and IOCB_CMD_PWRITEV/IOCB_CMD_WRITEV commands to the aio set. Without that, raw/O_DIRECT readv users would need to unnecessarily cycle through their iovecs at a library level submitting them individually. For larger iovecs, user/library code would needlessly deal with multiple completions. While I'm not sure of the performance impact of the absence of aio_readv/writev, it seems easy enough to provide.
Most of the functions are already in place. We would only
need a way to pass the iovec through the iocb.I was thinking of something like this:
struct iocb {
(quoted diff) +6
-0+union { __u64 aio_buf + __u64 aio_iovp +} +union { __u64 aio_nbytes + __u64 aio_nsegs +}allowing the iovec * & nsegs to be passed into
sys_io_submit. Some code would be added (within case handling ofIOCB_CMD_READVwithinio_submit_one) to copy & verify the iovec pointers and then call aio_readv/aio_writev (if its defined for the fs).What do you think ? I wanted to get some feedback before trying to code this up.
While we are on the topic of expanding aio operations, what about providing IOCB_CMD_READ/WRITE, distinct from their pread/pwrite counterparts ? Do you think thats needed ?
- Shailabh
Quoted 1 line
Quoted 1 line
Comments, reasons for not doing async readv/writev directly welcome.
How about the case for it... See Pages 404-406 [Section 12.7] of Richard Steven's ``Advanced Programming in the Unix Environment'' [aka APUE]. Richard measures almost a factor of 2 difference in system time between using vectored I/O and not using it on a Sun and on a x86.
Quoted 4 lines
Quoted 4 lines
- there exists at least one major application class (databases) that uses vectored I/O heavily and benefits from async I/O. Hence async vectored I/O is also likely to be useful. Can anyone else with experience on other OS's comment on this ?
.... a number of other comments/arguments from other .... responces removed to get the meat of the discussion.
Becareful out there.....
Large commercial applications such as Oracle DB, IBM's DB2 or Netscape Enterprise server for that matter - are very modular in their interface to OS because they have to be and have been, ported and tuned to run on a number of different OS's and HW architectures. When I see some one say something about ``Oracle'' doing X or Y - I get a little worried.
Which version, which port etc... e.g. Oracle's DB running on VMS has a different I/O system interface that is different from any of it's Unix implementations...... oh yes - was the clustered or not... did it have the X package etc...
The point is that the UNIX implementations of Oracle DB vary widely..... This is also true of every >>major<< application package I have worked/consulted seen some of the insides (SAP, Informix, Netscape, etc...).
Solaris and Tru64 [and I would expect AIX, HP-UX
etc... but I only know these two personally] each offer a highly parallel I/O, asynchronous (but proprietary) interface. Oracle's Sun group (or the old DEC group) exploit the >>private<< interfaces -- to make the code work better - they do.
That may or may not be what you have seen on some ``simple Un*x'' port - which is a starting point for them - that's not the code they ship on the high end revenue systems.
Oracle/IBM/Netscape etc... do this cause the want to grab customers from their competetors (DB2, Informix, etc.)... they invest in using the best interfaces available.... if they are available AND if they can help them sell more copies of their product.
So... let's get back to the basic issue....
We know that vectored/scatter gather I/O can help a number of real applications ... Richard demonstrated that. We have some examples [like DB2] that have use vectored I/O successfully. We also know asynchronous I/O has been demonstrated to be useful and know that some commerical folks have used that.
I'm gather from some of the comments, adding async/vectored will make an already complex subsystem, even more so [i.e. not a resounding endorsement for sure this is easy].
So the question is can async vectored I/O be implemented to have a positive gain, such as it did within the traditonal one. If the complexity is too high and it does not help much...then
maybe this is a Chimera to leave alone. But.... if it can be
done with some level of elegance... well.... the past history is that the commerical folks have used those features.
I know this this sounds a little bit like:
``if you build it - they will come.''
But I would say it's more to this point:
``if you build it and this new feature shows some real value
AND the application can exploit it ... in time, they will because if they don't their competetors will.''
Clem Cole
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 3 lines · Benjamin LaHaise
Only db2 uses vectored io heavily. Oracle does not, and none of the open source databases do. Vectored io is pretty useless for most people.
writev is extremely important for networking as it avoids small packets.
Why do people have such tunnel vision around aio to disk? Aio to network is far more important, as networks are much slower than disks.
Hi,
Quoted 7 lines · John Gardiner Myers
Quoted 4 lines · Benjamin LaHaise
Only db2 uses vectored io heavily. Oracle does not, and none of the open source databases do. Vectored io is pretty useless for most people.
writev is extremely important for networking as it avoids small packets.
No, all you can infer from that is that "some method for avoiding small packets is important for networking." TCP_CORK already does that in Linux, for tcp at least, without requiring writev. (Of course, normal nonblocking writev is still there if you want it.)
--Stephen
Quoted 5 lines · Stephen C. Tweedie
No, all you can infer from that is that "some method for avoiding small packets is important for networking."
TCP_CORKalready does that in Linux, for tcp at least, without requiring writev. (Of course, normal nonblocking writev is still there if you want it.)
TCP_CORK is indeed effective for avoiding small packets. Be that as it may, the source data for network writes are frequently in discontiguous buffers and writev is nonetheless still important for networking. The alternative in the aio model is to waste a lot of resources delivering io completions the application doesn't care about.