Efficient x86 and x86_64 NOP microbenchmarks
A message that is not in the archive (sent off-list or not yet received): 20080813191926.GB15547@Krystal
[
Thanks to Mathieu Desnoyers who forward this to me. Currently my ISP
for goodmis.org is having issues:
https://help.domaindirect.com/index.php?_m=news&_a=viewnews&newsid=104
]
Quoted 16 lines
----- Forwarded message from Andi Kleen <[email protected]> -----
Quoted 2 lines
So microbenchmarking this way will probably make some things look unrealistically good.
Must be careful to miss the big picture here.
We have two assumptions here in this thread:
- Normal
alternative()nops are relatively infrequent, typically in points with enough pipeline bubbles anyways, and it likely doesn't matter how they are encode. And also they don't have an issue with mult part instructions anyways because they're not patched at runtime, so always the best known can be used.
Quoted 4 lines
- The one case where nops are very frequent and matter and multipart
is a problem is with ftrace noping out the call to mcount at runtime because that happens on every function entry. Even there the overhead is not that big, but at least measurable in kernel builds.
The problem is not ftrace noping out the call at runtime. The problem is ftrace changing the nops back to calls to mcount.
The nop part is simple, straight forward and not an issue that we are talking here. The issue is which kind of nop to use. The bug with the multi-part nop happens when we _enable_ tracing. That is, when someone runs the tracer. The issue with the multi-part nop is that a task could have been preempted after it executed the first nop and before the second part. Then we enable tracing, and when the task is scheduled back in, it now will execute half the call to the mcount function.
I want this point very clear. If you never run tracing, this bug will not happen. And the bug only happens on enabling the tracer, not on the disabling part. Not to mention that the bug itself will only happen 1 in a billion.
Quoted 4 lines
Now the numbers have shown that just by not using frame pointer ( -pg right now implies frame pointer) you can get more benefit than what you lose from using non optimal nops.
No, I can easily make a patch that does not use frame pointers but still uses -pg. We just can not print the parent function in the trace. This can easily be added to a config, as well as easily implemented.
Quoted 10 lines
So for me the best strategy would be to get rid of the frame pointer
and ignore the nops. This unfortunately would require going away
from -pg and instead post process gcc output to insert "call mcount"
manually. But the nice advantage of that is that you could actually set up a custom table of callers built in a ELF section and with that you don't actually need the runtime patching (which is only
done currently because there's no global table of mcount calls),
but could do everything instop_machine(). Without
runtime patching you also don't need single part nops.
I'm totally confused here. How do you enable function tracing? How do we make a call to the code that will trace a function was hit?
Quoted 3 lines
I think that would be the best option. I especially like it because it would prevent forcing frame pointer which seems to be costlier than any kinds of nosp.
As I stated, the frame pointer part is only to record the parent function in tracing. ie:
ls-4866 [00] 177596.041275: _spin_unlock <-journal_stop
Here we see that the function _spin_unlock was called by the function journal_stop. We can easily turn off parent tracing now, with:
# echo noprint-parent > /debug/tracing/iter_ctrl
which gives us just:
ls-4866 [00] 177596.041275: _spin_unlock
If we disable frame pointers, the noprint-parent option would be forced. Not that devastating, but it gives the option to still have function tracing to the user without the requirement of having frame pointers.
I would still require that the irqsoff tracer add frame pointers, just because knowing that the long latency of interrupts disabled happened at local_irq_save doesn't cut it ;-)
Anyway, who would want to run with frame pointers disabled? If you ever get a bug crash, the stack trace is pretty much useless.
-- Steve
Quoted 1 line · Steven Rostedt
No, I can easily make a patch that does not use frame pointers but still uses -pg. We just can not print the parent function in the trace. This can easily be added to a config, as well as easily implemented.
Why? You can always get the calling function, because its return address is on the stack (assuming mcount is called before the function puts its own frame on the stack). But without a frame pointer, you can't necessarily get the caller's caller.
But I think Andi's point is that gcc forces frame pointers on when you enable mcount, so there's no choice in the matter.
J
Just a curious run of Mathieu's micro benchmark:
NR_TESTS 10000000
test empty cycles : 182500444
test 2-bytes jump cycles : 195969127
test 5-bytes jump cycles : 197000202
test 3/2 nops cycles : 201333408
test 5-bytes nop with long prefix cycles : 205000067
test 5-bytes P6 nop cycles : 205000227
test Generic 1/4 5-bytes nops cycles : 200000077
test K7 1/4 5-bytes nops cycles : 197549045And this was on a Pentium III 847.461 MHz box (my old toshiba laptop)
The jumps here played the best, but that could just be cache issues. But interesting to see that of the nops, the K7 1/4 faired the best.
-- Steve
Quoted 6 lines · Steven Rostedt
Quoted 4 lines
Now the numbers have shown that just by not using frame pointer ( -pg right now implies frame pointer) you can get more benefit than what you lose from using non optimal nops.
No, I can easily make a patch that does not use frame pointers but still
Not without patching gcc. Try it. The patch is not very difficult and i did it here, but it needs a patch.
Quoted 1 line
If we disable frame pointers, the noprint-parent option would be forced.
Actually you can get the parent without frame pointer if you just force gcc to emit mcount before touching the stack frame (and manual insertion pass would do that). Then parent is at 4(%esp)/8(%rsp) Again teaching gcc that is not very difficult, but it needs a patch.
Quoted 3 lines
I would still require that the irqsoff tracer add frame pointers, just because knowing that the long latency of interrupts disabled happened at
local_irq_savedoesn't cut it ;-)
Nope.
Quoted 3 lines
Anyway, who would want to run with frame pointers disabled? If you ever get a bug crash, the stack trace is pretty much useless.
First that's not true (remember most production kernels run without frame pointers, also e.g. crash or systemtap know how to do proper unwinding without slow frame pointers) and if you want it runtime also you can always add the dwarf2 unwinder (like the opensuse kernel does) and get better backtraces than you could ever get with frame pointers (that is because e.g. most assembler code doesn't even bother to set up frame pointers, but it is all dwarf2 annotated)
Also I must say the whole ftrace noping exercise is pretty pointless without avoiding frame pointers because it does save less than what you lose unconditionally from the "select FRAME_POINTER"
-Andi
Quoted 4 lines · Andi Kleen
Also I must say the whole ftrace noping exercise is pretty pointless without avoiding frame pointers because it does save less than what you lose unconditionally from the "select
FRAME_POINTER"
Andi, you seem to have missed the whole point. This is a _correctness_ issue as long as the nop is not a single instruction. And the workaround for that is uglier than just making a single-instruction nop.
So the question now is to find a good nop that _is_ a single atomic instruction. Your blathering about frame pointers is missing the whole point!
Linus
Quoted 7 lines · Andi Kleen
Quoted 5 lines · Steven Rostedt
Quoted 5 lines
Now the numbers have shown that just by not using frame pointer ( -pg right now implies frame pointer) you can get more benefit than what you lose from using non optimal nops.
Quoted 6 lines
Quoted 2 lines
No, I can easily make a patch that does not use frame pointers but still
Not without patching gcc. Try it. The patch is not very difficult and i did it here, but it needs a patch.
OK, I admit you are right ;-)
I got the error message:
gcc: -pg and -fomit-frame-pointer are incompatible
-- Steve