Fix Unlikely(x) == y

The patch below was not yet tested. If it's correct as it is, please comment. ---
Fix Unlikely(x) == y

Quoted 4 lines · On Sat, 16 Feb 2008 17:08:01 +0100 Roel Kluin <[email protected]> wrote:

The patch below was not yet tested. If it's correct as it is, please comment. ---
Fix Unlikely(x) == y

you found a great set of bugs..
but to be honest... I suspect it's just best to remove unlikely altogether for these cases; unlikely() is almost a go-faster-stripes thing, and if you don't know how to use it you shouldn't be using it... so just removing it for all wrong cases is actually the best thing to do imo.

Quoted 8 lines · Arjan van de Ven

Quoted 4 lines · On Sat, 16 Feb 2008 17:08:01 +0100 Roel Kluin <[email protected]> wrote:

The patch below was not yet tested. If it's correct as it is, please comment. ---
Fix Unlikely(x) == y

Quoted 6 lines

you found a great set of bugs..
but to be honest... I suspect it's just best to remove unlikely altogether for these cases; unlikely() is almost a go-faster-stripes thing, and if you don't know how to use it you shouldn't be using it... so just removing it for all wrong cases is actually the best thing to do imo.

Well, eventhough the author may not know how to use it, "unlikely" at least indicates the intention of the author, or his knowledge of what should happen here. I'd suggest leaving it where it is because the authot of this code is in best position to know that this branch is unlikely to happen, eventhough he does not correctly use the macro.

Quoted 16 lines · On Sat, 16 Feb 2008 18:33:16 +0100 Willy Tarreau <[email protected]> wrote:

Quoted 8 lines · Arjan van de Ven

Quoted 4 lines · On Sat, 16 Feb 2008 17:08:01 +0100 Roel Kluin <[email protected]> wrote:

The patch below was not yet tested. If it's correct as it is, please comment. ---
Fix Unlikely(x) == y

Quoted 6 lines

you found a great set of bugs..
but to be honest... I suspect it's just best to remove unlikely altogether for these cases; unlikely() is almost a go-faster-stripes thing, and if you don't know how to use it you shouldn't be using it... so just removing it for all wrong cases is actually the best thing to do imo.

Quoted 6 lines

Well, eventhough the author may not know how to use it, "unlikely" at least indicates the intention of the author, or his knowledge of what should happen here. I'd suggest leaving it where it is because the authot of this code is in best position to know that this branch is unlikely to happen, eventhough he does not correctly use the macro.

you have more faith in the authors knowledge of how his code actually behaves than I think is warranted :) Or faith in that he knows what "unlikely" means.
I should write docs about this; but unlikely() means:
1) It happens less than 0.01% of the cases.
2) The compiler couldn't have figured this out by itself
(NULL pointer checks are compiler done already, same for some other conditions)
3) It's a hot codepath where shaving 0.5 cycles (less even on x86) matters
(and the author is ok with taking a 500 cycles hit if he's wrong)

If you think unlikely() means something else, we should fix what it maps to towards gcc ;) (to.. be empty ;)

Quoted 2 lines

If you think unlikely() means something else, we should fix what it maps to towards gcc ;) (to.. be empty ;)

eventhough the gcc docs say it's just a hint to help the compiler optimize the branch it takes by default, I too have noticed that it more often does bad than good. Code gets completely reordered and even sometimes partially duplicated (especially when the branch is a return).

Last but not least, gcc 4 tends to emit stupid checks, to the point that I have replaced unlikely(x) with (x) in my code when gcc >= 4 is detected. What I observe is that the following code :

    if (unlikely(p == NULL)) ...

often gets coded like this :

    reg1 = (p == NULL)
    if (reg1 != 0) ...

... which clobbers reg1 for nothing and performs a double test.

But yes, I assumed that the author considered its use to be legitimate (I've not looked at the code). Maybe you're right and it should be removed, but in this case we would need a large audit of the abuses of unlikely()...

Quoted 16 lines · On Sat, 16 Feb 2008 18:33:16 +0100 Willy Tarreau <[email protected]> wrote:

Quoted 8 lines · Arjan van de Ven

Quoted 4 lines · On Sat, 16 Feb 2008 17:08:01 +0100 Roel Kluin <[email protected]> wrote:

The patch below was not yet tested. If it's correct as it is, please comment. ---
Fix Unlikely(x) == y

Quoted 6 lines

you found a great set of bugs..
but to be honest... I suspect it's just best to remove unlikely altogether for these cases; unlikely() is almost a go-faster-stripes thing, and if you don't know how to use it you shouldn't be using it... so just removing it for all wrong cases is actually the best thing to do imo.

Quoted 6 lines

Well, eventhough the author may not know how to use it, "unlikely" at least indicates the intention of the author, or his knowledge of what should happen here. I'd suggest leaving it where it is because the authot of this code is in best position to know that this branch is unlikely to happen, eventhough he does not correctly use the macro.

you have more faith in the authors knowledge of how his code actually behaves than I think is warranted :) Or faith in that he knows what "unlikely" means.
I should write docs about this; but unlikely() means:
1) It happens less than 0.01% of the cases.
2) The compiler couldn't have figured this out by itself
(NULL pointer checks are compiler done already, same for some other conditions)
3) It's a hot codepath where shaving 0.5 cycles (less even on x86) matters
(and the author is ok with taking a 500 cycles hit if he's wrong)

If you think unlikely() means something else, we should fix what it maps to towards gcc ;) (to.. be empty ;)

Quoted 37 lines · Arjan van de Ven

Quoted 16 lines · On Sat, 16 Feb 2008 18:33:16 +0100 Willy Tarreau <[email protected]> wrote:

Quoted 8 lines · Arjan van de Ven

Quoted 4 lines · On Sat, 16 Feb 2008 17:08:01 +0100 Roel Kluin <[email protected]> wrote:

The patch below was not yet tested. If it's correct as it is, please comment. ---
Fix Unlikely(x) == y

Quoted 6 lines

you found a great set of bugs..
but to be honest... I suspect it's just best to remove unlikely altogether for these cases; unlikely() is almost a go-faster-stripes thing, and if you don't know how to use it you shouldn't be using it... so just removing it for all wrong cases is actually the best thing to do imo.

Quoted 7 lines

Well, eventhough the author may not know how to use it, "unlikely" at least indicates the intention of the author, or his knowledge of what should happen here. I'd suggest leaving it where it is because the authot of this code is in best position to know that this branch is unlikely to happen, eventhough he does not correctly use the macro.

you have more faith in the authors knowledge of how his code actually behaves than I think is warranted :) Or faith in that he knows what "unlikely" means. I should write docs about this; but unlikely() means: 1) It happens less than 0.01% of the cases.
2) The compiler couldn't have figured this out by itself (NULL pointer checks are compiler done already, same for some
other conditions) 3) It's a hot codepath where shaving 0.5 cycles (less even on x86) matters (and the author is ok with taking a 500 cycles hit if he's wrong)

Quoted 2 lines

If you think unlikely() means something else, we should fix what it maps to towards gcc ;) (to.. be empty ;)

Well, I didn't consider what today's compiler does, but used it as a general indicator, because I think that code will be around a long time. If you show me some test results that prove it causes harm I might consider removing it.

Last but not least, gcc 4 tends to emit stupid checks, to the point that I have replaced unlikely(x) with (x) in my code when gcc >= 4 is detected. What I observe is that the following code :

    if (unlikely(p == NULL)) ...

often gets coded like this :

    reg1 = (p == NULL)
    if (reg1 != 0) ...

... which clobbers reg1 for nothing and performs a double test.

Quoted 8 lines · Willy Tarreau

Last but not least, gcc 4 tends to emit stupid checks, to the point that I have replaced unlikely(x) with (x) in my code when gcc >= 4 is detected. What I observe is that the following code :

    if (unlikely(p == NULL)) ...

often gets coded like this :

Quoted 4 lines

    reg1 = (p == NULL)
    if (reg1 != 0) ...

... which clobbers reg1 for nothing and performs a double test.

This really only can happen in GCC 4.0.x and 4.1.x and cannot happen for 4.2 or 4.3 really because of the way __builtin_expect is handled for those two.

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

you found a great set of bugs..
but to be honest... I suspect it's just best to remove unlikely altogether for these cases; unlikely() is almost a go-faster-stripes thing, and if you don't know how to use it you shouldn't be using it... so just removing it for all wrong cases is actually the best thing to do imo.

for mordern (last 10 years) x86 cpus... the cpu branchpredictor is better than the coder in general. Same for most other architectures.

unlikely() creates bigger code as well.

Now... we're talking about your super duper hotpath function here right? One where you care about 0.5 cycle speed improvement? (less on modern systems ;)

On Sat, 2008-02-16 at 10:39 -0800, Arjan van de Ven wrote:
...

Quoted 8 lines

for mordern (last 10 years) x86 cpus... the cpu branchpredictor is better than the coder in general. Same for most other architectures.

unlikely() creates bigger code as well.

Now... we're talking about your super duper hotpath function here right? One where you care about 0.5 cycle speed improvement? (less on modern systems ;)

The first patch was to platforms/ps3 code, which runs on the Cell, in particular the PPE ... which is not an x86 :)

eg:

[michael@schoenaich ~]$ cat branch.c
#include <stdio.h>
int main(void)

{
        int i, j;

        for (i = 0, j = 0; i < 1000000000; i++)
                if (i % 4 == 0)
                        j++;

        printf("j = %d\n", j);
        return 0;
}

[michael@schoenaich ~]$ ppu-gcc -Wall -O3 -o branch branch.c [michael@schoenaich ~]$ time ./branch
real 0m5.172s

[michael@schoenaich ~]$ cat branch.c
..

        for (i = 0, j = 0; i < 1000000000; i++)
                if (__builtin_expect(i % 4 == 0, 0))
                        j++;

..
[michael@schoenaich ~]$ ppu-gcc -Wall -O3 -o branch branch.c [michael@schoenaich ~]$ time ./branch
real 0m3.762s

Which looks as though unlikely() is helping. Admittedly we don't have a lot of kernel code that looks like that, but at least unlikely is doing what we want it to.

cheers

Quoted 44 lines · Michael Ellerman

On Sat, 2008-02-16 at 10:39 -0800, Arjan van de Ven wrote:
...

Quoted 8 lines

for mordern (last 10 years) x86 cpus... the cpu branchpredictor is better than the coder in general. Same for most other architectures.

unlikely() creates bigger code as well.

Now... we're talking about your super duper hotpath function here right? One where you care about 0.5 cycle speed improvement? (less on modern systems ;)

The first patch was to platforms/ps3 code, which runs on the Cell, in particular the PPE ... which is not an x86 :)

eg:

[michael@schoenaich ~]$ cat branch.c
#include <stdio.h>
int main(void)

{
        int i, j;

        for (i = 0, j = 0; i < 1000000000; i++)
                if (i % 4 == 0)
                        j++;

        printf("j = %d\n", j);
        return 0;
}

[michael@schoenaich ~]$ ppu-gcc -Wall -O3 -o branch branch.c [michael@schoenaich ~]$ time ./branch
real 0m5.172s

[michael@schoenaich ~]$ cat branch.c
..

        for (i = 0, j = 0; i < 1000000000; i++)
                if (__builtin_expect(i % 4 == 0, 0))
                        j++;

..
[michael@schoenaich ~]$ ppu-gcc -Wall -O3 -o branch branch.c [michael@schoenaich ~]$ time ./branch
real 0m3.762s

Quoted 3 lines

Which looks as though unlikely() is helping. Admittedly we don't have a lot of kernel code that looks like that, but at least unlikely is doing what we want it to.

This means it generates faster code with a current gcc for your platform.

But a future gcc might e.g. replace the whole loop with a division (gcc SVN head (that will soon become gcc 4.3) already does transformations like replacing loops with divisions [1]).

Quoted 6 lines · Adrian Bunk

This means it generates faster code with a current gcc for your platform.

But a future gcc might e.g. replace the whole loop with a division (gcc SVN head (that will soon become gcc 4.3) already does transformations like replacing loops with divisions [1]).

Hence shouldn't we ask the gcc people what's the purpose of __builtin_expect(), if it doesn't live up to its promise?

With kind regards,

Geert Uytterhoeven

Hence shouldn't we ask the gcc people what's the purpose of __builtin_expect(), if it doesn't live up to its promise?

you have more faith in the authors knowledge of how his code actually behaves than I think is warranted :)

Or faith in that he knows what "unlikely" means.
I should write docs about this; but unlikely() means:
1) It happens less than 0.01% of the cases.
2) The compiler couldn't have figured this out by itself
(NULL pointer checks are compiler done already, same for some other conditions)
3) It's a hot codepath where shaving 0.5 cycles (less even on x86) matters (and the author is ok with taking a 500 cycles hit if he's wrong)

Quoted 2 lines · Geert Uytterhoeven

Hence shouldn't we ask the gcc people what's the purpose of __builtin_expect(), if it doesn't live up to its promise?

__builtin_expect() is useful on FRV where you _have_ to give each branch and conditional branch instruction a measure of probability whether the branch will be taken.

David

__builtin_expect() is useful on FRV where you _have_ to give each branch and conditional branch instruction a measure of probability whether the branch will be taken.

Quoted 3 lines · David Howells

__builtin_expect() is useful on FRV where you _have_ to give each branch and conditional branch instruction a measure of probability whether the branch will be taken.

What does gcc do the 99.998% of the time we don't have likely/unlikely coded?

Quoted 5 lines

This means it generates faster code with a current gcc for your platform.

But a future gcc might e.g. replace the whole loop with a division (gcc SVN head (that will soon become gcc 4.3) already does transformations like replacing loops with divisions [1]).

Quoted 8 lines · Geert Uytterhoeven

Quoted 6 lines · Adrian Bunk

This means it generates faster code with a current gcc for your platform.

But a future gcc might e.g. replace the whole loop with a division (gcc SVN head (that will soon become gcc 4.3) already does transformations like replacing loops with divisions [1]).

Quoted 2 lines

Hence shouldn't we ask the gcc people what's the purpose of __builtin_expect(), if it doesn't live up to its promise?

That's a different issue.

My point here is that we do not know how the latest gcc available in the year 2010 might transform this code, and how a likely/unlikely placed there might influence gcc's optimizations then.

Quoted 2 lines · Arjan van de Ven

you have more faith in the authors knowledge of how his code actually behaves than I think is warranted :)

iirc there was a mm patch some time ago to keep track of the actual unlikely values at runtime and it showed indeed some wrong ones. But the far majority of them are probably correct.

Quoted 8 lines

Or faith in that he knows what "unlikely" means.
I should write docs about this; but unlikely() means:
1) It happens less than 0.01% of the cases.
2) The compiler couldn't have figured this out by itself (NULL pointer checks are compiler done already, same for some other
conditions) 3) It's a hot codepath where shaving 0.5 cycles (less even on x86) matters (and the author is ok with taking a 500 cycles hit if he's wrong)

One more thing unlikely() does is to move the unlikely code out of line. So it should conserve some icache in critical functions, which might well be worth some more cycles (don't have numbers though).

But overall I agree with you that unlikely is in most cases a bad idea (and I submitted the original patch introducing it originally @). That is because it is often used in situations where gcc's default branch prediction heuristics do would make exactly the same decision

           if (unlikely(x == NULL))

is simply totally useless because gcc already assumes all x == NULL tests are unlikely. I appended some of the builtin heuristics from a recent gcc source so people can see them.

Note in particular the last predictors; assuming branch ending with goto, including call, causing early function return or returning negative constant are not taken. Just these alone are likely 95+% of the unlikelies in the kernel.

-Andi

Actually one thing I don't like about gcc is that I think it still emits cmovs for likely/unlikely branches, which is silly (the gcc developers seem to be in love with that instruction). If that goes away, then branch hints may be even better.

On Tue, 19 Feb 2008 13:33:53 +1100

Quoted 4 lines · Nick Piggin

Actually one thing I don't like about gcc is that I think it still emits cmovs for likely/unlikely branches, which is silly (the gcc developers seem to be in love with that instruction). If that goes away, then branch hints may be even better.

only for -Os and only if the result is smaller afaik.

(cmov tends to be a performance loss most of the time so for -O2 and such it isn't used as far as I know.. it does make for nice small code however ;-)

Quoted 4 lines

Note in particular the last predictors; assuming branch ending with goto, including call, causing early function return or returning negative constant are not taken. Just these alone are likely 95+% of the unlikelies in the kernel.

Yes, gcc should be able to do pretty good heuristics, considering the quite good numbers that cold CPU predictors can attain. However for really performance critical code (or really "never" executed code), then I think it is OK to have the hints and not have to rely on gcc heuristics.

Quoted 10 lines · Nick Piggin

Quoted 4 lines

Note in particular the last predictors; assuming branch ending with goto, including call, causing early function return or returning negative constant are not taken. Just these alone are likely 95+% of the unlikelies in the kernel.

Yes, gcc should be able to do pretty good heuristics, considering the quite good numbers that cold CPU predictors can attain. However for really performance critical code (or really "never" executed code), then I think it is OK to have the hints and not have to rely on gcc heuristics.

in my experience, the real problem is that gcc does what *it* wants and not what *you* want. I've been annoyed a lot by the way it coded some loops that could really be blazingly fast, but which resulted in a ton of branches due to its predictors. And using unlikely() there was a real mess, because instead of just hinting the compiler with probabilities to write some linear code for the *most* common case, it ended up with awful branches everywhere with code sent far away and even duplicated for some branches.

Sometimes, for performance critical paths, I would like gcc to be dumb and follow *my* code and not its hard-coded probabilities. For instance, in a tree traversal, you really know how you want to build your loop. And these days, it seems like the single method of getting it your way is doing asm, which obviously is not portable :-(

Maybe one thing we would need would be the ability to assign probabilities to each branch based on what we expect, so that gcc could build a better tree keeping most frequently used code tight.

Quoted 14 lines · Adrian Bunk

Quoted 10 lines · Geert Uytterhoeven

Quoted 6 lines · Adrian Bunk

This means it generates faster code with a current gcc for your platform.

But a future gcc might e.g. replace the whole loop with a division (gcc SVN head (that will soon become gcc 4.3) already does transformations like replacing loops with divisions [1]).

Hence shouldn't we ask the gcc people what's the purpose of __builtin_expect(), if it doesn't live up to its promise?

That's a different issue.

Quoted 3 lines

My point here is that we do not know how the latest gcc available in the year 2010 might transform this code, and how a likely/unlikely placed there might influence gcc's optimizations then.

You're right, we don't know. But if giving the compiler _more_ information causes it to produce vastly inferior code then we should be filing gcc bugs. After all the unlikely/likely is just a hint, if gcc knows better it can always ignore it.

cheers

Quoted 17 lines · Andi Kleen

Quoted 2 lines · Arjan van de Ven

you have more faith in the authors knowledge of how his code actually behaves than I think is warranted :)

iirc there was a mm patch some time ago to keep track of the actual unlikely values at runtime and it showed indeed some wrong ones. But the far majority of them are probably correct.

Quoted 8 lines

Or faith in that he knows what "unlikely" means.
I should write docs about this; but unlikely() means:
1) It happens less than 0.01% of the cases.
2) The compiler couldn't have figured this out by itself (NULL pointer checks are compiler done already, same for some other
conditions) 3) It's a hot codepath where shaving 0.5 cycles (less even on x86) matters (and the author is ok with taking a 500 cycles hit if he's wrong)

Quoted 3 lines

One more thing unlikely() does is to move the unlikely code out of line. So it should conserve some icache in critical functions, which might well be worth some more cycles (don't have numbers though).

I actually once measured context switching performance in the scheduler, and removing the unlikely hint for testing RT tasks IIRC gave about 5% performance drop.

This was on a P4 which is very different from more modern CPUs both in terms of branch performance characteristics,

and icache characteristics.

However, the P4's branch predictor is pretty good, and it should easily

Actually one thing I don't like about gcc is that I think it still emits cmovs for likely/unlikely branches,

which is silly (the gcc developers

the quite good numbers that cold CPU predictors can attain. However for really performance critical code (or really "never" executed code), then I think it is OK to have the hints and not have to rely on gcc heuristics.

Sometimes, for performance critical paths, I would like gcc to be dumb and follow *my* code and not its hard-coded probabilities.

Maybe one thing we would need would be the ability to assign probabilities to each branch based on what we expect, so that gcc could build a better tree keeping most frequently used code tight.

Hmm I've just noticed -fno-guess-branch-probability in the man, I never tried it.

Quoted 3 lines · Nick Piggin

I actually once measured context switching performance in the scheduler, and removing the unlikely hint for testing RT tasks IIRC gave about 5% performance drop.

OT: what benchmarks did you use for that? I had a change some time ago to the CFS scheduler to avoid unpredicted indirect calls for the common case, but I wasn't able to benchmark a difference with the usual suspect benchmark (lmbench). Since it increased code size by
a few bytes it was rejected then.

Quoted 4 lines

This was on a P4 which is very different from more modern CPUs both in terms of branch performance characteristics,

and icache characteristics.

Hmm, the P4 the trace cache actually should not care about inline code that is not executed.

Quoted 1 line

However, the P4's branch predictor is pretty good, and it should easily

I think it depends on the generation. Prescott class branch prediction should be much better than the earlier ones.

Quoted 2 lines

Actually one thing I don't like about gcc is that I think it still emits cmovs for likely/unlikely branches,

That's -Os.

Quoted 1 line

which is silly (the gcc developers

It depends on the CPU. e.g. on K8 and P6 using CMOV if possible makes sense. P4 doesn't like it though.

Quoted 4 lines

the quite good numbers that cold CPU predictors can attain. However for really performance critical code (or really "never" executed code), then I think it is OK to have the hints and not have to rely on gcc heuristics.

But only when the explicit hints are different from what the implicit branch predictors would predict anyways. And if you look at the heuristics that is not often the case...

Quoted 6 lines · Andi Kleen

Quoted 3 lines · Nick Piggin

I actually once measured context switching performance in the scheduler, and removing the unlikely hint for testing RT tasks IIRC gave about 5% performance drop.

Quoted 5 lines

OT: what benchmarks did you use for that? I had a change some time ago to the CFS scheduler to avoid unpredicted indirect calls for the common case, but I wasn't able to benchmark a difference with the usual suspect benchmark (lmbench). Since it increased code size by
a few bytes it was rejected then.

I think it was just a simple context switch benchmark, but not lmbench (which I found to be a bit too variable). But it was a long time ago...

Quoted 5 lines

Quoted 2 lines

However, the P4's branch predictor is pretty good, and it should easily

I think it depends on the generation. Prescott class branch prediction should be much better than the earlier ones.

I was using a Nocona Xeon, which I think is a Prescott class?

And don't they have much higher mispredict penalty (than older P4s)?

Quoted 6 lines

Quoted 4 lines

Actually one thing I don't like about gcc is that I think it still emits cmovs for likely/unlikely branches,

That's -Os.

And -O2 and -O3, on the gccs that I'm using, AFAIKS.

Quoted 4 lines

Quoted 3 lines

which is silly (the gcc developers

Quoted 2 lines

It depends on the CPU. e.g. on K8 and P6 using CMOV if possible makes sense. P4 doesn't like it though.

If the branch is completely predictable (eg. annotated), then I think branches should be used anyway. Even on well predicted branches, cmov is similar speed on microbenchmarks, but it will increase data hazards I think, so it will probably be worse for some real world situations.

Quoted 7 lines

Quoted 6 lines

the quite good numbers that cold CPU predictors can attain. However for really performance critical code (or really "never" executed code), then I think it is OK to have the hints and not have to rely on gcc heuristics.

Quoted 3 lines

But only when the explicit hints are different from what the implicit branch predictors would predict anyways. And if you look at the heuristics that is not often the case...

But a likely branch will be _strongly_ predicted to be taken, wheras a lot of the gcc heuristics simply have slightly more or slightly less probability. So it's not just a question of which way is more likely, but also _how_ likely it is to go that way.

Quoted 2 lines · Nick Piggin

I think it was just a simple context switch benchmark, but not lmbench (which I found to be a bit too variable). But it was a long time ago...

Do you still have it?

I thought about writing my own but ended up being too lazy for that @)

Quoted 6 lines

Quoted 4 lines

Quoted 2 lines

Actually one thing I don't like about gcc is that I think it still emits cmovs for likely/unlikely branches,

That's -Os.

And -O2 and -O3, on the gccs that I'm using, AFAIKS.

Well if it still happens on gcc 4.2 with P4 tuning you should perhaps open a gcc PR. They tend to ignore these bugs mostly in my experience, but sometimes they act on them.

Quoted 10 lines

Quoted 4 lines

Quoted 1 line

which is silly (the gcc developers

It depends on the CPU. e.g. on K8 and P6 using CMOV if possible makes sense. P4 doesn't like it though.

If the branch is completely predictable (eg. annotated), then I think branches should be used anyway. Even on well predicted branches, cmov is similar speed on microbenchmarks, but it will increase data hazards I think, so it will probably be worse for some real world situations.

At least the respective optimization manuals say they should be used. I presume they only made this recommendation after some extensive benchmarking.

Quoted 4 lines

But a likely branch will be _strongly_ predicted to be taken, wheras a lot of the gcc heuristics simply have slightly more or slightly less probability. So it's not just a question of which way is more likely, but also _how_ likely it is to go that way.

Yes, but a lot of the heuristics are pretty strong (>80%) and gcc will act on them unless it has a very strong contra cue. And that should normally not be the case.

Quoted 2 lines

Sometimes, for performance critical paths, I would like gcc to be dumb and follow *my* code and not its hard-coded probabilities.

If you really want that, simple: just disable optimization @)

Quoted 3 lines

Maybe one thing we would need would be the ability to assign probabilities to each branch based on what we expect, so that gcc could build a better tree keeping most frequently used code tight.

Just use profile feedback then for user space. I don't think it's a good idea for kernel code though because it leads to unreproducible binaries which would wreck the development model.

Quoted 2 lines

Hmm I've just noticed -fno-guess-branch-probability in the man, I never tried it.

Or -fno-reorder-blocks