Extensible AIO interface

So, I and other people keep running into things where we really need to add an interface to pass some auxiliary... stuff along with a pread() or pwrite().

A few examples:

* IO scheduler hints...
* Cache hints...

* Passing checksums out to userspace. We've got bio integrity, which is a (somewhat) generic interface for passing data checksums between the filesystem and the hardware.

Hence, AIO attributes.

An iocb_attr has an id field, and a size field - and some amount of data specific to that attribute.

Quoted 3 lines · Kent Overstreet

So, I and other people keep running into things where we really need to add an interface to pass some auxiliary... stuff along with a pread() or pwrite().

Sure. Martin (cc:ed) will sympathize.

Quoted 5 lines

A few examples:

* IO scheduler hints...
* Cache hints...

Quoted 3 lines

* Passing checksums out to userspace. We've got bio integrity, which is a (somewhat) generic interface for passing data checksums between the filesystem and the hardware.

Hmm, careful here. I think that in DIF/DIX the checksums are per-sector, not per IO, right? That'd mean that the PAGE_SIZE attr limit in this patch would be magically creating different max IO size limits on different architectures. That doesn't seem great.

Quoted 2 lines

Hence, AIO attributes.

I have to be honest: I really don't like tying the interface to AIO, but I guess it's the only per-io facility we have today. It'd be nice to include sync O_DIRECT when designing the interface to make sure that it is possible to use generic syscalls in the future without running up against unexpected problems.

Quoted 2 lines

An iocb_attr has an id field, and a size field - and some amount of data specific to that attribute.

I'd hope that we can come up with a less fragile interface. The kernel would have to scan the attributes to make sure that there aren't malicious sizes. I only quickly glanced at the loops, but it seemed like you could have a 0 size attribute in there and _next() would spin forever.

Not just per sector, Per hardware sector. For passing around checksums userspace would have to find out the hardware sector size and checksum type/size via a different interface, and then the attribute would contain a pointer to a buffer that can hold the appropriate number of checksums.

I don't think there's anything fragile about the basic idea though. Or do you have some way of improving upon it in mind?

The idea with the size field is that it's just sizeof(the particular attribute struct), so when userspace is appending attributes it just sets size = sizeof() and attr_list->size += attr->size.

Quoted 5 lines

Not just per sector, Per hardware sector. For passing around checksums userspace would have to find out the hardware sector size and checksum type/size via a different interface, and then the attribute would contain a pointer to a buffer that can hold the appropriate number of checksums.

All problems fall to another layer of indirection? :)

But yes, that's
fair. I was obviously just assuming that the checksums would be in the attribute.

But now we're talking about layers of user pointers. Would the attribute parser need to verify/copy pointers before downstream kernel code tries to work with it? Would it be up to the attribute consumers to verify the pointers that the core doesn't really touch?

Are these
second pointers native (enter compat goo) or u64s?

Quoted 2 lines

I don't think there's anything fragile about the basic idea though. Or do you have some way of improving upon it in mind?

Nothing super great is springing to mind, no.

Quoted 3 lines

The idea with the size field is that it's just sizeof(the particular attribute struct), so when userspace is appending attributes it just sets size = sizeof() and attr_list->size += attr->size.

I suppose. But this also raises the spectre of aligning the packed attributes to match their struct definitions. It's the netlink(3) macros all over again, right? I guess unaligned accesses aren't *that* big a deal. But still.

And what about duplicate instances of a given attribute id? Use the first? The last? Error? Depends on the id?

It just seems like there are a lot of corner cases that can go wrong with an API that is so free form. I'd like something a lot harder to make mistakes with.

- z
(being That Guy today, apparently :/)

The generic code wouldn't know about any user pointers inside attributes, so it'd have to be downstream consumers. Hopefully there won't be many attributes with user pointers in them (I don't expect there to be), so we won't have too much of this messyness.

Quoted 4 lines

The generic code wouldn't know about any user pointers inside attributes, so it'd have to be downstream consumers. Hopefully there won't be many attributes with user pointers in them (I don't expect there to be), so we won't have too much of this messyness.

I really don't like this. We should have learned this lesson with ioctl structs that are nested pointers.

What about security bits that are trying to determine if attributes are OK?

What about contexts that can't easily deal with userspace pointers? We tend to copy into relatively more accessible kernel memory as early as possible.

What about fuse trying to extend this interface out to their fs clients? Look at the horrible mess it implements to bounce nested ioctl data parsing between the kernel's user pointer copying and the fuse client's parsing of that copied data.

Let's not do this again, please. I think it's a fallacy to claim that the interface can be opaque to high levels and only parsed by lower levels. The interface should be explicit and fully specified on entry so that all levels have trivial access to it.

Seems to me it'd be no different from security considerations when introducing new ioctls. I.e., messy, ad hoc, easy to get wrong, but sometimes no way around it.

It really has to be ad hoc if it's extensible, unfortunately.

The only way of getting around _that_ would be with some kind of reflective type system, so that generic code could parse (in some fashion) the types of the various attributes, and for pointers copy the user data into the kernel and do whatever access controls in generic code.

Hello, Kent.

Quoted 11 lines · Kent Overstreet

Seems to me it'd be no different from security considerations when introducing new ioctls. I.e., messy, ad hoc, easy to get wrong, but sometimes no way around it.

It really has to be ad hoc if it's extensible, unfortunately.

The only way of getting around _that_ would be with some kind of reflective type system, so that generic code could parse (in some fashion) the types of the various attributes, and for pointers copy the user data into the kernel and do whatever access controls in generic code.

I'm not userland API expert by any stretch of imagination but the basic mechanism to pass data around seems sane to me and aio as stinky as it is seems to be the only logical stuff for IOs w/ extra attributes although alignment is always painful with any form of concatenated opaque structures.

However, I don't think it's a good idea to try to implement something which is a neutral transport of opaque data between userland and lower layers. Things like that sound attractive with unlimited possibilities but reality seems to have the tendancy to make a big mess out of setups like that.

So, if we're gonna do this, let's define what attributes we want to have and let them be processed at the interface layer and fed to lower layers afterwards - e.g. for cgroup context association, associate the resulting bios with the target cgroup from the aio layer.

I'm quite skeptical of general usefulness of having opaque knobs to lower IO stack which don't have proper generic abstraction. Nobody can make proper use of things like that. Well, not nobody, maybe if the lower stack, the interface and the application are implemented by a single organization over relatively short span of time, maybe it would be useful for them, but that isn't something which generic interface design should focus on.

It's okay to allow some side channel thing for specific hacky uses but I really hope the general design were focused around properly abstracted attributes which can be understood and handled by the upper layer.

Quoted 5 lines

However, I don't think it's a good idea to try to implement something which is a neutral transport of opaque data between userland and lower layers. Things like that sound attractive with unlimited possibilities but reality seems to have the tendancy to make a big mess out of setups like that.

I don't see how the "neutral transport of opaque data" itself is a bad thing. We want something simple and sane to build actual interfaces on top of - once we've got that, we can either build clean generic well defined interfaces or we can make a mess like with ioctls :P

It's like any other mechanism. There's good syscalls and bad syscalls...

Say we implement an attr to control a block layer cache. That attr could be parsed/validated in high level code (if there's any to do) - that I don't object to. But the high level code isn't going to /know/ whether there was any block cache in the stack that handled the attr. If the attr is passed down to the block cache, that block cache can return that it was handled.

Quoted 4 lines

It's okay to allow some side channel thing for specific hacky uses but I really hope the general design were focused around properly abstracted attributes which can be understood and handled by the upper layer.

Completely agreed. I want to leave that side channel open for experimentation, and so we have a way of implementing one off hacky stuff when we need to - but normal mainline stuff should be sane and well designed.

Hello, Kent.

Quoted 11 lines · Kent Overstreet

Quoted 5 lines

However, I don't think it's a good idea to try to implement something which is a neutral transport of opaque data between userland and lower layers. Things like that sound attractive with unlimited possibilities but reality seems to have the tendancy to make a big mess out of setups like that.

I don't see how the "neutral transport of opaque data" itself is a bad thing. We want something simple and sane to build actual interfaces on top of - once we've got that, we can either build clean generic well defined interfaces or we can make a mess like with ioctls :P

Quoted 1 line

It's like any other mechanism. There's good syscalls and bad syscalls...

Depending on what a feature aims for, the design and implementation vary greatly. If you go for completely generic extensible stuff which can be used to warp space-time continuum, it's easy to end up with a monstrosity with generic and programmable parser, verifier, accessor and so on.

Quoted 6 lines

Say we implement an attr to control a block layer cache. That attr could be parsed/validated in high level code (if there's any to do) - that I don't object to. But the high level code isn't going to /know/ whether there was any block cache in the stack that handled the attr. If the attr is passed down to the block cache, that block cache can return that it was handled.

My point is that if it doesn't fit the generic abstract model as in fadvise(2), it probably isn't worth supporting in any generic manner.

Quoted 9 lines

Quoted 4 lines

It's okay to allow some side channel thing for specific hacky uses but I really hope the general design were focused around properly abstracted attributes which can be understood and handled by the upper layer.

Completely agreed. I want to leave that side channel open for experimentation, and so we have a way of implementing one off hacky stuff when we need to - but normal mainline stuff should be sane and well designed.

So, I think we can aim for something simple and modest (the only thing I can think of at the moment is task association) and provide simple framework which can be used for specific custom usages. Let's please not go overboard with generic parser / verifier which supports pointer indirection or whatnot.

And what about duplicate instances of a given attribute id? Use the first? The last? Error? Depends on the id?

Quoted 6 lines

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Kent" == Kent Overstreet <[email protected]> writes:

Hmm, careful here. I think that in DIF/DIX the checksums are per-sector, not per IO, right? That'd mean that the PAGE_SIZE attr limit in this patch would be magically creating different max IO size limits on different architectures. That doesn't seem great.

So, I and other people keep running into things where we really need to add an interface to pass some auxiliary... stuff along with a pread() or pwrite().

A few examples:

* IO scheduler hints. Some userspace program wants to, per IO, specify either priorities or a cgroup - by specifying a cgroup you can have a fileserver in userspace that makes use of cfq's per cgroup bandwidth quotas.

* Cache hints. For bcache and other things, userspace may want to specify "this data should be cached", "this data should bypass the cache", etc.

* Passing checksums out to userspace. We've got bio integrity, which is a (somewhat) generic interface for passing data checksums between the filesystem and the hardware. There are various circumstances under which you may want to pass these checksums out to userspace, and if so we ought to have a generic way of doing it.

Hence, AIO attributes.

* FUTURE STUFF:

Return values:

Some attributes are probably going to want to return something to userspace.

If nothing else, we want this so that userspace can tell if anything handled the attributes it specified - as dynamic as the io stack can be, with something extensible like this there really isn't any generic way of knowing ahead of time if something is going to interpret any attribute - we want to return at least an error code.

One could imagine sticking the return in the attribute itself, but I don't want to do this. For some things (checksums), the attribute will contain a pointer to a buffer - that's fine. But I don't want the attributes themselves to be writeable.

Quoted 10 lines · Kent Overstreet

So, I and other people keep running into things where we really need to add an interface to pass some auxiliary... stuff along with a pread() or pwrite().

A few examples:

* IO scheduler hints. Some userspace program wants to, per IO, specify either priorities or a cgroup - by specifying a cgroup you can have a fileserver in userspace that makes use of cfq's per cgroup bandwidth quotas.

You can do this today by splitting I/O between processes and placing those processes in different cgroups. For io priority, there is ioprio_set, which incurs an extra system call, but can be used. Not elegant, but possible.

Quoted 2 lines

* Cache hints. For bcache and other things, userspace may want to specify "this data should be cached", "this data should bypass the cache", etc.

Please explain how you will differentiate this from posix_fadvise.

Quoted 5 lines

* Passing checksums out to userspace. We've got bio integrity, which is a (somewhat) generic interface for passing data checksums between the filesystem and the hardware. There are various circumstances under which you may want to pass these checksums out to userspace, and if so we ought to have a generic way of doing it.

Yes, that needs a new interface.

Quoted 1 line

Hence, AIO attributes.

*No.* Start with the non-AIO case first.

Quoted 7 lines

* FUTURE STUFF:

Return values:

Some attributes are probably going to want to return something to userspace.

Quoted 5 lines

If nothing else, we want this so that userspace can tell if anything handled the attributes it specified - as dynamic as the io stack can be, with something extensible like this there really isn't any generic way of knowing ahead of time if something is going to interpret any attribute - we want to return at least an error code.

Seems odd to me. Why not expose supported attributes via some other call? fcntl?

Quoted 4 lines

One could imagine sticking the return in the attribute itself, but I don't want to do this. For some things (checksums), the attribute will contain a pointer to a buffer - that's fine. But I don't want the attributes themselves to be writeable.

One could imagine that attributes don't return anything, because, well, they're properties of something else, and properties don't return anything.

Quoted 13 lines · Jeff Moyer

Quoted 10 lines · Kent Overstreet

So, I and other people keep running into things where we really need to add an interface to pass some auxiliary... stuff along with a pread() or pwrite().

A few examples:

* IO scheduler hints. Some userspace program wants to, per IO, specify either priorities or a cgroup - by specifying a cgroup you can have a fileserver in userspace that makes use of cfq's per cgroup bandwidth quotas.

Quoted 4 lines

You can do this today by splitting I/O between processes and placing those processes in different cgroups. For io priority, there is ioprio_set, which incurs an extra system call, but can be used. Not elegant, but possible.

Yes - those are things I'm trying to replace. Doing it that way is a real pain, both as it's a lousy interface for this and it does impact performance (ioprio_set doesn't really work too well with aio, too).

Quoted 3 lines

Quoted 2 lines

* Cache hints. For bcache and other things, userspace may want to specify "this data should be cached", "this data should bypass the cache", etc.

Quoted 1 line

Please explain how you will differentiate this from posix_fadvise.

Oh sorry, I think about SSD caching so much I forget to say that's what I'm talking about. posix_fadvise is for the page cache, we want something different for an SSD cache (IMO it'd be really ugly to use it for both, and posix_fadvise() can't really specifify everything we'd want to for an SSD cache).

Quoted 26 lines · Kent Overstreet

Quoted 13 lines · Jeff Moyer

Quoted 10 lines · Kent Overstreet

So, I and other people keep running into things where we really need to add an interface to pass some auxiliary... stuff along with a pread() or pwrite().

A few examples:

* IO scheduler hints. Some userspace program wants to, per IO, specify either priorities or a cgroup - by specifying a cgroup you can have a fileserver in userspace that makes use of cfq's per cgroup bandwidth quotas.

Quoted 4 lines

You can do this today by splitting I/O between processes and placing those processes in different cgroups. For io priority, there is ioprio_set, which incurs an extra system call, but can be used. Not elegant, but possible.

Yes - those are things I'm trying to replace. Doing it that way is a real pain, both as it's a lousy interface for this and it does impact performance (ioprio_set doesn't really work too well with aio, too).

Quoted 3 lines

Quoted 2 lines

* Cache hints. For bcache and other things, userspace may want to specify "this data should be cached", "this data should bypass the cache", etc.

Quoted 7 lines

Quoted 1 line

Please explain how you will differentiate this from posix_fadvise.

Oh sorry, I think about SSD caching so much I forget to say that's what I'm talking about. posix_fadvise is for the page cache, we want something different for an SSD cache (IMO it'd be really ugly to use it for both, and posix_fadvise() can't really specifify everything we'd want to for an SSD cache).

Similar discussions about posix_fadvise() are being had for marking ranges of files as volatile (i.e. useful for determining what can be evicted from a cache when space reclaim is required).

https://lkml.org/lkml/2012/10/2/501

If you have requirements for specific cache management, then it might be worth seeing if you can steer an existing interface proposal for some form of cache management in the direction you need.

Quoted 39 lines · Dave Chinner

Quoted 26 lines · Kent Overstreet

Quoted 13 lines · Jeff Moyer

Quoted 10 lines · Kent Overstreet

So, I and other people keep running into things where we really need to add an interface to pass some auxiliary... stuff along with a pread() or pwrite().

A few examples:

* IO scheduler hints. Some userspace program wants to, per IO, specify either priorities or a cgroup - by specifying a cgroup you can have a fileserver in userspace that makes use of cfq's per cgroup bandwidth quotas.

Quoted 4 lines

You can do this today by splitting I/O between processes and placing those processes in different cgroups. For io priority, there is ioprio_set, which incurs an extra system call, but can be used. Not elegant, but possible.

Yes - those are things I'm trying to replace. Doing it that way is a real pain, both as it's a lousy interface for this and it does impact performance (ioprio_set doesn't really work too well with aio, too).

Quoted 3 lines

Quoted 2 lines

* Cache hints. For bcache and other things, userspace may want to specify "this data should be cached", "this data should bypass the cache", etc.

Quoted 7 lines

Quoted 1 line

Please explain how you will differentiate this from posix_fadvise.

Oh sorry, I think about SSD caching so much I forget to say that's what I'm talking about. posix_fadvise is for the page cache, we want something different for an SSD cache (IMO it'd be really ugly to use it for both, and posix_fadvise() can't really specifify everything we'd want to for an SSD cache).

Similar discussions about posix_fadvise() are being had for marking ranges of files as volatile (i.e. useful for determining what can be evicted from a cache when space reclaim is required).

Quoted 1 line

https://lkml.org/lkml/2012/10/2/501

Hmm, interesting

Speaking as an implementor though, hints that aren't associated with any specific IO are harder to make use of - stuff is in the cache. What you really want is to know, for a given IO, whether to cache it or not, and possibly where in the LRU to stick it.

Well, it's quite possible that different implementations would have no trouble making use of those kinds of hints, I'm no doubt biased by having implemented bcache. With bcache though, cache replacement is done in terms of physical address space, not logical (i.e. the address space of the device being cached).

So to handle posix_fadvise, we'd have to walk the btree and chase pointers to buckets, and modify the bucket priorities up or down... but what about the other data in those buckets? It's not clear what should happen, but there isn't any good way to take that into account.

(The exception is dropping data from the cache entirely, we can just invalidate that part of the keyspace and garbage collection will reclaim the buckets they pointed to. Bcache does that for discard requests, currently).

Quoted 13 lines · Jeff Moyer

Quoted 10 lines · Kent Overstreet

So, I and other people keep running into things where we really need to add an interface to pass some auxiliary... stuff along with a pread() or pwrite().

A few examples:

* IO scheduler hints. Some userspace program wants to, per IO, specify either priorities or a cgroup - by specifying a cgroup you can have a fileserver in userspace that makes use of cfq's per cgroup bandwidth quotas.

Quoted 4 lines

You can do this today by splitting I/O between processes and placing those processes in different cgroups. For io priority, there is ioprio_set, which incurs an extra system call, but can be used. Not elegant, but possible.

Yes - those are things I'm trying to replace. Doing it that way is a real pain, both as it's a lousy interface for this and it does impact performance (ioprio_set doesn't really work too well with aio, too).

Quoted 4 lines

Quoted 2 lines

* Cache hints. For bcache and other things, userspace may want to specify "this data should be cached", "this data should bypass the cache", etc.

Please explain how you will differentiate this from posix_fadvise.

Oh sorry, I think about SSD caching so much I forget to say that's what I'm talking about. posix_fadvise is for the page cache, we want something different for an SSD cache (IMO it'd be really ugly to use it for both, and posix_fadvise() can't really specifify everything we'd want to for an SSD cache).

Quoted 3 lines

Quoted 1 line

Hence, AIO attributes.

*No.* Start with the non-AIO case first.

Why? It is orthogonal to AIO (and I should make that clearer), but to do it for sync IO we'd need new syscalls that take an extra argument so IMO it's a bit easier to start with AIO.

Might be worth implementing the sync interface sooner rather than later just to discover any potential issues, I suppose.

Quoted 13 lines

Quoted 12 lines

* FUTURE STUFF:

Return values:

Some attributes are probably going to want to return something to userspace.

If nothing else, we want this so that userspace can tell if anything handled the attributes it specified - as dynamic as the io stack can be, with something extensible like this there really isn't any generic way of knowing ahead of time if something is going to interpret any attribute - we want to return at least an error code.

Quoted 2 lines

Seems odd to me. Why not expose supported attributes via some other call? fcntl?

It's not possible in general - consider stacking block devices, and attrs that are supported only by specific block drivers. I.e. if you've got lvm on top of bcache or bcache on top of md, we can pass the attr down with the IO but we can't determine ahead of time, in general, where the IO is going to go.

But that probably isn't true for most attrs so it probably would be a good idea to have an interface for querying what's supported, and even for device specific ones you could query what a device supports.

Quoted 8 lines

Quoted 4 lines

One could imagine sticking the return in the attribute itself, but I don't want to do this. For some things (checksums), the attribute will contain a pointer to a buffer - that's fine. But I don't want the attributes themselves to be writeable.

One could imagine that attributes don't return anything, because, well, they're properties of something else, and properties don't return anything.

With a strict definition of attribute, yeah. One of the real uses cases we have for this is per IO timings, for aio - right now we've got an interface for the kernel to tell userspace how long a syscall took (don't think it's upstream yet - Paul's been behind that stuff), but it only really makes sense with synchronous syscalls.

These AIO attributes would be useful for that too, but I'd _much_ prefer if the timing information was explicitly returned instead of using a pointer to a buffer.

Quoted 19 lines · Kent Overstreet

Quoted 17 lines · Jeff Moyer

Quoted 10 lines · Kent Overstreet

So, I and other people keep running into things where we really need to add an interface to pass some auxiliary... stuff along with a pread() or pwrite().

A few examples:

* IO scheduler hints. Some userspace program wants to, per IO, specify either priorities or a cgroup - by specifying a cgroup you can have a fileserver in userspace that makes use of cfq's per cgroup bandwidth quotas.

You can do this today by splitting I/O between processes and placing those processes in different cgroups. For io priority, there is ioprio_set, which incurs an extra system call, but can be used. Not elegant, but possible.

Quoted 3 lines

Yes - those are things I'm trying to replace. Doing it that way is a real pain, both as it's a lousy interface for this and it does impact performance (ioprio_set doesn't really work too well with aio, too).

ioprio_set works fine with aio, since the I/O is issued in the caller's context. Perhaps you're thinking of writeback I/O?

Quoted 10 lines

Quoted 4 lines

Quoted 2 lines

* Cache hints. For bcache and other things, userspace may want to specify "this data should be cached", "this data should bypass the cache", etc.

Please explain how you will differentiate this from posix_fadvise.

Oh sorry, I think about SSD caching so much I forget to say that's what I'm talking about. posix_fadvise is for the page cache, we want something different for an SSD cache (IMO it'd be really ugly to use it for both, and posix_fadvise() can't really specifify everything we'd want to for an SSD cache).

DESCRIPTION
Programs can use posix_fadvise() to announce an intention to access file data in a specific pattern in the future, thus allowing the kernel to perform appropriate optimizations.

That description seems broad enough to include disk caches as well. You haven't exactly stated what's missing.

Quoted 8 lines

Quoted 3 lines

Quoted 1 line

Hence, AIO attributes.

*No.* Start with the non-AIO case first.

Why? It is orthogonal to AIO (and I should make that clearer), but to do it for sync IO we'd need new syscalls that take an extra argument so IMO it's a bit easier to start with AIO.

Quoted 2 lines

Might be worth implementing the sync interface sooner rather than later just to discover any potential issues, I suppose.

Looking back to preadv and pwritev, it was wrong to only add them to libaio (and that later got corrected). I'd just like to see things start out with the sync interfaces, since you'll get more eyes on the code (not everyone cares about aio) and that helps to work out any interface issues.

Quoted 5 lines

It's not possible in general - consider stacking block devices, and attrs that are supported only by specific block drivers. I.e. if you've got lvm on top of bcache or bcache on top of md, we can pass the attr down with the IO but we can't determine ahead of time, in general, where the IO is going to go.

If the io stack is static (meaning you setup a device once, then open it and do io to it, and it doesn't change while you're doing io), how would you not know where the IO is going to go?

Quoted 3 lines

But that probably isn't true for most attrs so it probably would be a good idea to have an interface for querying what's supported, and even for device specific ones you could query what a device supports.

OK.

Quoted 9 lines

Quoted 8 lines

Quoted 4 lines

One could imagine sticking the return in the attribute itself, but I don't want to do this. For some things (checksums), the attribute will contain a pointer to a buffer - that's fine. But I don't want the attributes themselves to be writeable.

One could imagine that attributes don't return anything, because, well, they're properties of something else, and properties don't return anything.

Quoted 5 lines

With a strict definition of attribute, yeah. One of the real uses cases we have for this is per IO timings, for aio - right now we've got an interface for the kernel to tell userspace how long a syscall took (don't think it's upstream yet - Paul's been behind that stuff), but it only really makes sense with synchronous syscalls.

Something beyond recording the time spent in the kernel? Paul who? I agree the per io timing for aio may be coarse-grained today (you can time the difference between io_submit returning and the event being returned by io_getevents, but that says nothing of when the io was issued to the block layer). I'm curious to know exactly what granularity you want here, and what an application would do with that information. You can currently access a whole lot of detail of the io path through blktrace, but that is not easily done from within an application.

So, I and other people keep running into things where we really need to add an interface to pass some auxiliary... stuff along with a pread() or pwrite().