aio is unlikely
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 29 lines · Linux Kernel Mailing List
Gitweb: http://git.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b8522ead3534c6cd06752b47a3bc380956191a2a Commit: b8522ead3534c6cd06752b47a3bc380956191a2a Parent: b41eeef14d7c73af6d16c7d02b7a939082a137ff Author: Andrew Morton <[email protected]> AuthorDate: Wed May 9 02:34:58 2007 -0700 Committer: Linus Torvalds <[email protected]> CommitDate: Wed May 9 12:30:54 2007 -0700aio is unlikely
Stick an
unlikely()aroundis_aio(): I assert that most IO is synchronous.Cc Suparna Bhattacharya <[email protected]>Cc Ingo Molnar <[email protected]>Cc Benjamin LaHaise <[email protected]>Cc Zach Brown <[email protected]>Cc Ulrich Drepper <[email protected]>Cc Christoph Hellwig <[email protected]>Signed-off-by Andrew Morton <[email protected]>Signed-off-by Linus Torvalds <[email protected]>--- include/linux/aio.h | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-)include/linux/aio.h +0
-0
include/linux/aio.h int FASTCALL(io_submit_one(struct kioctx *ctx, struct iocb __user *user_iocb, · line 229
Quoted 5 lines
include/linux/aio.h +1
-1@@ -226,7 +226,8 @@ int FASTCALL(io_submit_one(struct kioctx *ctx, struct iocb __user *user_iocb,__put_ioctx (kioctx); \ }while (0 ) -#define in_aio () !is_sync_wait(current->io_wait) +#define in_aio() (unlikely(!is_sync_wait(current->io_wait)))
Please revert. Workload-dependent "likelihood" should not cause programmers to add such markers.
This is a common misunderstanding about unlikely() and likely(). The branch prediction used for each assumes 99% unlikely or 99% likely, which is not true at all for workload-dependent code.
Even if only 1% of Linux users use AIO, for that 1%, the 'unlikely' marker causes repeated branch mispredictions.
likely() and unlikely() should be used for cases where code is likely/unlikely for EVERYBODY.
Jeff
Quoted 10 lines · On Wed, 09 May 2007 18:06:58 -0400 Jeff Garzik <[email protected]> wrote:
Review
Quoted 2 lines
(quoted diff) +1
-1-#define in_aio() !is_sync_wait(current->io_wait) +#define in_aio() (unlikely(!is_sync_wait(current->io_wait)))Please revert. Workload-dependent "likelihood" should not cause programmers to add such markers.
This is a common misunderstanding about
unlikely()andlikely(). The branch prediction used for each assumes 99% unlikely or 99% likely, which is not true at all for workload-dependent code.
Quoted 5 lines
Even if only 1% of Linux users use AIO, for that 1%, the 'unlikely' marker causes repeated branch mispredictions.
likely()andunlikely()should be used for cases where code is likely/unlikely for EVERYBODY.
a) disagree with the above
b) if in_aio() ever returns true we do
printk(KERN_ERR "%s(%s:%d) called in async context!\n", __FUNCTION__, __FILE__, __LINE__);
so I sure hope it's unlikely for all workloads.
Quoted 8 lines · Andrew Morton
a) disagree with the above
b) if
in_aio()ever returns true we doprintk(KERN_ERR "%s(%s:%d) called in async context!\n", __FUNCTION__, __FILE__, __LINE__);so I sure hope it's unlikely for all workloads.
hrm, indeed. Ignore me.
Jeff
Quoted 2 lines · Andrew Morton
aio is unlikely
Stick anunlikely()aroundis_aio(): I assert that most IO is
synchronous.
Quoted 11 lines
(quoted diff) +1
-1-#define in_aio() !is_sync_wait(current->io_wait) +#define in_aio() (unlikely(!is_sync_wait(current->io_wait)))Quoted 5 lines · Jeff Garzik
Review
Quoted 2 lines
(quoted diff) +1
-1-#define in_aio() !is_sync_wait(current->io_wait) +#define in_aio() (unlikely(!is_sync_wait(current->io_wait)))Please revert. Workload-dependent "likelihood" should not cause programmers to add such markers.
Quoted 8 lines
a) disagree with the above
b) if
in_aio()ever returns true we doprintk(KERN_ERR "%s(%s:%d) called in async context!\n", __FUNCTION__, __FILE__, __LINE__);so I sure hope it's unlikely for all workloads.
Shouldn't unlikely() go where in_aio() is actually used, if we printk(error) there?
Isn't putting likely/unlikely into a boolean function-like macro itself asking for later trouble?
--Alex.
Quoted 23 lines · On Fri, 18 May 2007 16:49:49 -0400 "Alex Volkov" <[email protected]> wrote:
Quoted 2 lines · Andrew Morton
aio is unlikely
Stick anunlikely()aroundis_aio(): I assert that most IO issynchronous.
Quoted 18 lines
(quoted diff) +1
-1-#define in_aio() !is_sync_wait(current->io_wait) +#define in_aio() (unlikely(!is_sync_wait(current->io_wait)))Quoted 5 lines · Jeff Garzik
Review
Quoted 2 lines
(quoted diff) +1
-1-#define in_aio() !is_sync_wait(current->io_wait) +#define in_aio() (unlikely(!is_sync_wait(current->io_wait)))Please revert. Workload-dependent "likelihood" should not cause programmers to add such markers.
a) disagree with the above
b) if
in_aio()ever returns true we doprintk(KERN_ERR "%s(%s:%d) called in async context!\n", __FUNCTION__, __FILE__, __LINE__);
Quoted 7 lines
Quoted 1 line
so I sure hope it's unlikely for all workloads.
Shouldn't
unlikely()go wherein_aio()is actually used, if we printk(error) there?
Isn't putting likely/unlikely into a boolean function-like macro itself asking for later trouble?
Yes, if you agree with Jeff's original point.
But I don't, actually. Sure, on some machines+workloads, AIO is more common than sync IO. But I expect that when we sum across all the machines+workloads in the world, sync IO is more common and is hence the case we should optimise for.
That's assuming that the unlikely() actually does something.
Quoted 8 lines · Andrew Morton
Yes, if you agree with Jeff's original point.
But I don't, actually. Sure, on some machines+workloads, AIO is more common than sync IO. But I expect that when we sum across all the machines+workloads in the world, sync IO is more common and is hence the case we should optimise for.
That's assuming that the
unlikely()actually does something.
For the record, I agreed with your counter-point, and retract[ed?] my disagreement...
Jeff
Quoted 8 lines · Andrew Morton
Yes, if you agree with Jeff's original point.
But I don't, actually. Sure, on some machines+workloads, AIO is more common than sync IO. But I expect that when we sum across all the machines+workloads in the world, sync IO is more common and is hence the case we should optimise for.
That's assuming that the
unlikely()actually does something.
But as Jeff said, that's not what unlikely is for. It should only be used when it is unlikely for everybody, all the time, because when it is right, it helps rather little, but when it is wrong, it hurts a lot.
Quoted 8 lines · On Fri, 18 May 2007 17:54:32 -0400 Phillip Susi <[email protected]> wrote:
Quoted 7 lines · Andrew Morton
Yes, if you agree with Jeff's original point.
But I don't, actually. Sure, on some machines+workloads, AIO is more common than sync IO. But I expect that when we sum across all the machines+workloads in the world, sync IO is more common and is hence the case we should optimise for.
Quoted 5 lines
Quoted 1 line
That's assuming that the
unlikely()actually does something.But as Jeff said, that's not what unlikely is for. It should only be used when it is unlikely for everybody, all the time, because when it is right, it helps rather little, but when it is wrong, it hurts a lot.
It does? Tell us more.
Quoted 11 lines · Andrew Morton
Quoted 8 lines · On Fri, 18 May 2007 17:54:32 -0400 Phillip Susi <[email protected]> wrote:
Quoted 7 lines · Andrew Morton
Yes, if you agree with Jeff's original point.
But I don't, actually. Sure, on some machines+workloads, AIO is more common than sync IO. But I expect that when we sum across all the machines+workloads in the world, sync IO is more common and is hence the case we should optimise for.
Quoted 4 lines
Quoted 2 lines
Quoted 1 line
That's assuming that the
unlikely()actually does something.But as Jeff said, that's not what unlikely is for. It should only be used when it is unlikely for everybody, all the time, because when it is right, it helps rather little, but when it is wrong, it hurts a lot.
It does? Tell us more.
It is difficult to quantify either way. The details are both CPU-specific and compiler-specific. The best information can be culled from the gcc list archives, which is where I obtained my knowledge on the subject (which is now ~2 years old).
Under the hood, likely() and unlikely() are implemented as percentage predictions. likely() is implemented in the kernel as a 99-100% chance of success, and unlikely() is implemented as a 0-1% chance of success.
As such, for our purposes, likely() and unlikely() should only be used when a situation is [likely | unlikely] across all runtime configurations. So if you mark a branch unlikely() when it is hit often by 1% of your users, that is an incorrect usage.
The effects are probably most dramatic on older CPUs. Repeatedly hitting an unlikely() can cause a pipeline stall on every single access.
Branch delay slots are filled improperly, with obvious implications.
But on modern hardware, I would /guess/ that the effect of repeatedly hitting an unlikely() would be mitigated by smarter branch prediction.
We really need a GCC expert to answer this question in any more detail.
Jeff
Quoted 15 lines · Jeff Garzik
Quoted 7 lines · Andrew Morton
Quoted 1 line · On Fri, 18 May 2007 17:54:32 -0400 Phillip Susi <[email protected]> wrote:
But as Jeff said, that's not what unlikely is for. It should only be used when it is unlikely for everybody, all the time, because when it is right, it helps rather little, but when it is wrong, it hurts a lot.
It does? Tell us more.
It is difficult to quantify either way. The details are both CPU-specific and compiler-specific. The best information can be culled from the gcc list archives, which is where I obtained my knowledge on the subject (which is now ~2 years old).
Under the hood,
likely()andunlikely()are implemented as percentage predictions.likely()is implemented in the kernel as a 99-100% chance of success, andunlikely()is implemented as a 0-1% chance of success.
Quoted 8 lines
As such, for our purposes,
likely()andunlikely()should only be used when a situation is [likely | unlikely] across all runtime configurations. So if you mark a branchunlikely()when it is hit often by 1% of your users, that is an incorrect usage.The effects are probably most dramatic on older CPUs. Repeatedly hitting an
unlikely()can cause a pipeline stall on every single access.
Branch delay slots are filled improperly, with obvious implications.But on modern hardware, I would /guess/ that the effect of repeatedly hitting an
unlikely()would be mitigated by smarter branch prediction.We really need a GCC expert to answer this question in any more detail.
Aside from using branch constructs or hints that help the predictor guess the right way... I think gcc will move unlikely paths right past the end of the "likely" fastpath, so it can increase code size and be somewhat suboptimal in terms of icache usage.
I don't know particularly why it would hurt a lot more when it goes wrong than it helps when it goes right, though.
Also, I don't think I agree that it should be used where it is correct for all users. We make rt_task unlikely in the scheduler, and I measured that a very long time ago was IIRC good for nearly 5% pipe based context switching peformance. Systems running a lot of rt tasks aren't going to like it, but bugger them :)
Signature · 1 line
SUSE Labs, Novell Inc.
Quoted 4 lines · Nick Piggin
Aside from using branch constructs or hints that help the predictor guess the right way... I think gcc will move unlikely paths right past the end of the "likely" fastpath, so it can increase code size and be somewhat suboptimal in terms of icache usage.
Thanks for the reminder. GCC definitely does code movement.
ISTR the code movement might even be "extreme", once the unit-at-a-time support arrived, placing "cold" code at the end of the compiled module.
Jeff
Quoted 5 lines · In article <058f01c7998e$1406e370$650df7cd@MUMBA> you wrote:
Review
Quoted 2 lines
(quoted diff) +1
-1-#define in_aio() !is_sync_wait(current->io_wait) +#define in_aio() (unlikely(!is_sync_wait(current->io_wait)))Shouldn't
unlikely()go wherein_aio()is actually used, if we printk(error) there?
Actually I would just remove that define and use the method directly, if this is the only place where it is used. If it is used in multiple places, the unlikely is most likely wrong .)
Gruss
Bernd