Skip to contentSkip to conversation

Efficient x86 and x86_64 NOP microbenchmarks

DISCUSSION Steven Rostedt 4 participants 6 messages Aug 13, 2008 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): 20080813191926.GB15547@Krystal

#1 lore raw

[
Thanks to Mathieu Desnoyers who forward this to me. Currently my ISP
for goodmis.org is having issues:
https://help.domaindirect.com/index.php?_m=news&_a=viewnews&newsid=104
]

Quoted 16 lines

----- Forwarded message from Andi Kleen <[email protected]> -----

Quoted 2 lines

So microbenchmarking this way will probably make some things look unrealistically good.

Must be careful to miss the big picture here.

We have two assumptions here in this thread:

- Normal alternative() nops are relatively infrequent, typically in points with enough pipeline bubbles anyways, and it likely doesn't matter how they are encode. And also they don't have an issue with mult part instructions anyways because they're not patched at runtime, so always the best known can be used.

Quoted 4 lines

- The one case where nops are very frequent and matter and multipart
is a problem is with ftrace noping out the call to mcount at runtime because that happens on every function entry. Even there the overhead is not that big, but at least measurable in kernel builds.

The problem is not ftrace noping out the call at runtime. The problem is ftrace changing the nops back to calls to mcount.

The nop part is simple, straight forward and not an issue that we are talking here. The issue is which kind of nop to use. The bug with the multi-part nop happens when we _enable_ tracing. That is, when someone runs the tracer. The issue with the multi-part nop is that a task could have been preempted after it executed the first nop and before the second part. Then we enable tracing, and when the task is scheduled back in, it now will execute half the call to the mcount function.

I want this point very clear. If you never run tracing, this bug will not happen. And the bug only happens on enabling the tracer, not on the disabling part. Not to mention that the bug itself will only happen 1 in a billion.

Quoted 4 lines

Now the numbers have shown that just by not using frame pointer ( -pg right now implies frame pointer) you can get more benefit than what you lose from using non optimal nops.

No, I can easily make a patch that does not use frame pointers but still uses -pg. We just can not print the parent function in the trace. This can easily be added to a config, as well as easily implemented.

Quoted 10 lines

So for me the best strategy would be to get rid of the frame pointer
and ignore the nops. This unfortunately would require going away
from -pg and instead post process gcc output to insert "call mcount"
manually. But the nice advantage of that is that you could actually set up a custom table of callers built in a ELF section and with that you don't actually need the runtime patching (which is only
done currently because there's no global table of mcount calls),
but could do everything in stop_machine(). Without
runtime patching you also don't need single part nops.

I'm totally confused here. How do you enable function tracing? How do we make a call to the code that will trace a function was hit?

Quoted 3 lines

I think that would be the best option. I especially like it because it would prevent forcing frame pointer which seems to be costlier than any kinds of nosp.

As I stated, the frame pointer part is only to record the parent function in tracing. ie:

            ls-4866  [00] 177596.041275: _spin_unlock <-journal_stop

Here we see that the function _spin_unlock was called by the function journal_stop. We can easily turn off parent tracing now, with:

# echo noprint-parent > /debug/tracing/iter_ctrl

which gives us just:

            ls-4866  [00] 177596.041275: _spin_unlock

If we disable frame pointers, the noprint-parent option would be forced. Not that devastating, but it gives the option to still have function tracing to the user without the requirement of having frame pointers.

I would still require that the irqsoff tracer add frame pointers, just because knowing that the long latency of interrupts disabled happened at local_irq_save doesn't cut it ;-)

Anyway, who would want to run with frame pointers disabled? If you ever get a bug crash, the stack trace is pretty much useless.

-- Steve

#2 lore raw
Quoted 1 line · Steven Rostedt

No, I can easily make a patch that does not use frame pointers but still uses -pg. We just can not print the parent function in the trace. This can easily be added to a config, as well as easily implemented.

Why? You can always get the calling function, because its return address is on the stack (assuming mcount is called before the function puts its own frame on the stack). But without a frame pointer, you can't necessarily get the caller's caller.

But I think Andi's point is that gcc forces frame pointers on when you enable mcount, so there's no choice in the matter.

J

#3 lore raw

Just a curious run of Mathieu's micro benchmark:

NR_TESTS 10000000

                      test empty cycles : 182500444
               test 2-bytes jump cycles : 195969127
               test 5-bytes jump cycles : 197000202
                   test 3/2 nops cycles : 201333408
test 5-bytes nop with long prefix cycles : 205000067
             test 5-bytes P6 nop cycles : 205000227
   test Generic 1/4 5-bytes nops cycles : 200000077
        test K7 1/4 5-bytes nops cycles : 197549045

And this was on a Pentium III 847.461 MHz box (my old toshiba laptop)

The jumps here played the best, but that could just be cache issues. But interesting to see that of the nops, the K7 1/4 faired the best.

-- Steve

#4 lore raw
In reply to Steven Rostedt
Quoted 6 lines · Steven Rostedt
Quoted 4 lines

Now the numbers have shown that just by not using frame pointer ( -pg right now implies frame pointer) you can get more benefit than what you lose from using non optimal nops.

No, I can easily make a patch that does not use frame pointers but still

Not without patching gcc. Try it. The patch is not very difficult and i did it here, but it needs a patch.

Quoted 1 line

If we disable frame pointers, the noprint-parent option would be forced.

Actually you can get the parent without frame pointer if you just force gcc to emit mcount before touching the stack frame (and manual insertion pass would do that). Then parent is at 4(%esp)/8(%rsp) Again teaching gcc that is not very difficult, but it needs a patch.

Quoted 3 lines

I would still require that the irqsoff tracer add frame pointers, just because knowing that the long latency of interrupts disabled happened at local_irq_save doesn't cut it ;-)

Nope.

Quoted 3 lines

Anyway, who would want to run with frame pointers disabled? If you ever get a bug crash, the stack trace is pretty much useless.

First that's not true (remember most production kernels run without frame pointers, also e.g. crash or systemtap know how to do proper unwinding without slow frame pointers) and if you want it runtime also you can always add the dwarf2 unwinder (like the opensuse kernel does) and get better backtraces than you could ever get with frame pointers (that is because e.g. most assembler code doesn't even bother to set up frame pointers, but it is all dwarf2 annotated)

Also I must say the whole ftrace noping exercise is pretty pointless without avoiding frame pointers because it does save less than what you lose unconditionally from the "select FRAME_POINTER"

-Andi

#5 lore raw
Quoted 4 lines · Andi Kleen

Also I must say the whole ftrace noping exercise is pretty pointless without avoiding frame pointers because it does save less than what you lose unconditionally from the "select FRAME_POINTER"

Andi, you seem to have missed the whole point. This is a _correctness_ issue as long as the nop is not a single instruction. And the workaround for that is uglier than just making a single-instruction nop.

So the question now is to find a good nop that _is_ a single atomic instruction. Your blathering about frame pointers is missing the whole point!

Linus

#6 lore raw
In reply to Andi Kleen
Quoted 7 lines · Andi Kleen
Quoted 5 lines · Steven Rostedt
Quoted 5 lines

Now the numbers have shown that just by not using frame pointer ( -pg right now implies frame pointer) you can get more benefit than what you lose from using non optimal nops.

Quoted 6 lines
Quoted 2 lines

No, I can easily make a patch that does not use frame pointers but still

Not without patching gcc. Try it. The patch is not very difficult and i did it here, but it needs a patch.

OK, I admit you are right ;-)

I got the error message:

gcc: -pg and -fomit-frame-pointer are incompatible

-- Steve