Multiple handlers per marker
Quoted 35 lines · * Mathieu Desnoyers ([email protected]) wrote:
Quoted 34 lines · * Mathieu Desnoyers ([email protected]) wrote:
Quoted 6 lines · * Mike Mason ([email protected]) wrote:
Hi Mathieu,
Are you aware of any working being done to allow multiple handlers to be attached to a marker? Something like what kprobes allows. I've started to look into this and don't want to duplicate efforts.
Nope, but I know we will have to address this.
Something along the lines of walking an RCU list of function pointers, calling them.
The only downside I see is that we will have to pass a
va_list* instead of real va args. The could make the marker site a little bit bigger and will change the probe callback arguments.What do you think about these ideas ?
If we can find a way to make the common case (only one probe connected) _ultra_ fast, and yet architecture independent, that would be awesome. A simple call is kind of hard to beat though.. So we may have to think about a design with :
- One call at the marker site
- if 1 probe is installed :
- If the format string is empty, connect a probe without va args.
- If the format string is not empty, connect a "stage 1" probe that takes the va args, starts/ends theva_listand calls _one_ function (let's call it "stage 2" probe), that takesva_listas parameter.
- if more than 1 probe is installed :
- The stage 1 probe creates theva_listand passes it to each function connected, iterated with an RCU list.
Quoted 7 lines
Quoted 4 lines
What do you think ?
Mathieu
I'm working on an implementation.
It's ready for testing. Please grab
http://ltt.polymtl.ca/lttng/patch-2.6.24-rc1-git13-lttng-0.10-pre18.tar.bz2
patch name :
markers-support-multiple-probes.patch
It still need to go through patchcheck.pl and some polishing, but it seems to work fine for me with multiple probes (the sample marker, sample probe and multiple instances of my lttng probes can connect/disconnect without problem).
Currently, the "connect/disconnect" and "arm/disarm" operations are separate. However, they could be merged. Any comment/preference on this? Being separate, a probe provider can wait until the very last moment before it activates its markers, with a minimalistic impact on the system, but it is not such a strong argument.
Mathieu
Signature · 3 lines
Mathieu Desnoyers Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Quoted 37 lines · Mathieu Desnoyers
Quoted 36 lines · * Mathieu Desnoyers ([email protected]) wrote:
Quoted 35 lines · * Mathieu Desnoyers ([email protected]) wrote:
Quoted 4 lines · * Mike Mason ([email protected]) wrote:
Hi Mathieu,
Are you aware of any working being done to allow multiple handlers to be attached to a marker? Something like what kprobes allows. I've started to look into this and don't want to duplicate efforts.
Nope, but I know we will have to address this.
Something along the lines of walking an RCU list of function pointers, calling them.
The only downside I see is that we will have to pass a
va_list* instead of real va args. The could make the marker site a little bit bigger and will change the probe callback arguments.What do you think about these ideas ?
If we can find a way to make the common case (only one probe connected) _ultra_ fast, and yet architecture independent, that would be awesome. A simple call is kind of hard to beat though.. So we may have to think about a design with :
- One call at the marker site
- if 1 probe is installed :
- If the format string is empty, connect a probe without va args.
- If the format string is not empty, connect a "stage 1" probe that takes the va args, starts/ends theva_listand calls _one_ function (let's call it "stage 2" probe), that takesva_listas parameter.
- if more than 1 probe is installed :
- The stage 1 probe creates theva_listand passes it to each function connected, iterated with an RCU list.What do you think ?
Mathieu
Quoted 8 lines
Quoted 2 lines
I'm working on an implementation.
It's ready for testing. Please grab
http://ltt.polymtl.ca/lttng/patch-2.6.24-rc1-git13-lttng-0.10-pre18.tar.bz2
patch name :markers-support-multiple-probes.patch
This patch alone doesn't apply cleanly at all on 2.6.24-rc1-git14. Are there other patches in this series I should apply first?
Mike
Quoted 14 lines
It still need to go through patchcheck.pl and some polishing, but it seems to work fine for me with multiple probes (the sample marker, sample probe and multiple instances of my lttng probes can connect/disconnect without problem).
Currently, the "connect/disconnect" and "arm/disarm" operations are separate. However, they could be merged. Any comment/preference on this? Being separate, a probe provider can wait until the very last moment before it activates its markers, with a minimalistic impact on the system, but it is not such a strong argument.
Mathieu
Quoted 43 lines · * Mike Mason ([email protected]) wrote:
Quoted 42 lines · Mathieu Desnoyers
Quoted 41 lines · * Mathieu Desnoyers ([email protected]) wrote:
Quoted 34 lines · * Mathieu Desnoyers ([email protected]) wrote:
Quoted 6 lines · * Mike Mason ([email protected]) wrote:
Hi Mathieu,
Are you aware of any working being done to allow multiple handlers to be attached to a marker? Something like what kprobes allows. I've started to look into this and don't want to duplicate efforts.
Nope, but I know we will have to address this.
Something along the lines of walking an RCU list of function pointers, calling them.
The only downside I see is that we will have to pass a
va_list* instead of real va args. The could make the marker site a little bit bigger and will change the probe callback arguments.What do you think about these ideas ?
If we can find a way to make the common case (only one probe connected) _ultra_ fast, and yet architecture independent, that would be awesome. A simple call is kind of hard to beat though.. So we may have to think about a design with :
- One call at the marker site
- if 1 probe is installed :
- If the format string is empty, connect a probe without va args.
- If the format string is not empty, connect a "stage 1" probe that
takes
the va args, starts/ends theva_listand calls _one_ function (let's call it "stage 2" probe), that takesva_listas parameter.
- if more than 1 probe is installed :
- The stage 1 probe creates theva_listand passes it to each function connected, iterated with an RCU list.Quoted 4 lines
What do you think ?
Mathieu
I'm working on an implementation.
Quoted 8 lines
Quoted 4 lines
It's ready for testing. Please grab
http://ltt.polymtl.ca/lttng/patch-2.6.24-rc1-git13-lttng-0.10-pre18.tar.bz2
patch name :
markers-support-multiple-probes.patchThis patch alone doesn't apply cleanly at all on 2.6.24-rc1-git14. Are there other patches in this series I should apply first?
Yes, the following ones should suffice :
# instrumentation menu removal add-kconfig-to-arch.patch add-arch-supports-oprofile.patch add-arch-supports-kprobes.patch move-kconfig-instrumentation-to-arch.patch # kprobes-use-mutex-for-insn-pages.patch kprobes-dont-use-kprobes-mutex-in-arch-code.patch kprobes-declare-kprobes-mutex-static.patch declare-array.patch text-edit-lock-architecture-independent-code.patch text-edit-lock-alternative-i386-and-x86_64.patch text-edit-lock-kprobes-architecture-independent.patch text-edit-lock-kprobes-i386.patch text-edit-lock-kprobes-x86_64.patch text-edit-lock-i386-standardize-debug-rodata.patch text-edit-lock-x86_64-standardize-debug-rodata.patch # immediate-values-architecture-independent-code.patch immediate-values-kconfig-embedded.patch immediate-values-move-kprobes-i386-restore-interrupt-to-kdebug-h.patch add-asm-compat-to-x86.patch immediate-values-i386-optimization.patch immediate-values-powerpc-optimization.patch immediate-values-documentation.patch # linux-kernel-markers-immediate-values.patch # markers-support-multiple-probes.patch
Tell me if you still have rejects.
Mathieu
Quoted 13 lines
Mike
Quoted 10 lines
It still need to go through patchcheck.pl and some polishing, but it seems to work fine for me with multiple probes (the sample marker, sample probe and multiple instances of my lttng probes can connect/disconnect without problem).
Currently, the "connect/disconnect" and "arm/disarm" operations are separate. However, they could be merged. Any comment/preference on this? Being separate, a probe provider can wait until the very last moment before it activates its markers, with a minimalistic impact on the system, but it is not such a strong argument.
Mathieu
Signature · 3 lines
Mathieu Desnoyers Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Quoted 81 lines · Mathieu Desnoyers
Quoted 39 lines · * Mike Mason ([email protected]) wrote:
Quoted 38 lines · Mathieu Desnoyers
Quoted 37 lines · * Mathieu Desnoyers ([email protected]) wrote:
Quoted 36 lines · * Mathieu Desnoyers ([email protected]) wrote:
Quoted 4 lines · * Mike Mason ([email protected]) wrote:
Hi Mathieu,
Are you aware of any working being done to allow multiple handlers to be attached to a marker? Something like what kprobes allows. I've started to look into this and don't want to duplicate efforts.
Nope, but I know we will have to address this.
Something along the lines of walking an RCU list of function pointers, calling them.
The only downside I see is that we will have to pass a
va_list* instead of real va args. The could make the marker site a little bit bigger and will change the probe callback arguments.What do you think about these ideas ?
If we can find a way to make the common case (only one probe connected) _ultra_ fast, and yet architecture independent, that would be awesome. A simple call is kind of hard to beat though.. So we may have to think about a design with :
- One call at the marker site
- if 1 probe is installed :
- If the format string is empty, connect a probe without va args.
- If the format string is not empty, connect a "stage 1" probe that
takes
the va args, starts/ends theva_listand calls _one_ function (let's call it "stage 2" probe), that takesva_listas parameter.
- if more than 1 probe is installed :
- The stage 1 probe creates theva_listand passes it to each function connected, iterated with an RCU list.What do you think ?
Mathieu
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
I'm working on an implementation.
It's ready for testing. Please grab
http://ltt.polymtl.ca/lttng/patch-2.6.24-rc1-git13-lttng-0.10-pre18.tar.bz2
patch name :
markers-support-multiple-probes.patchThis patch alone doesn't apply cleanly at all on 2.6.24-rc1-git14. Are there other patches in this series I should apply first?
Yes, the following ones should suffice :
# instrumentation menu removal add-kconfig-to-arch.patch add-arch-supports-oprofile.patch add-arch-supports-kprobes.patch move-kconfig-instrumentation-to-arch.patch # kprobes-use-mutex-for-insn-pages.patch kprobes-dont-use-kprobes-mutex-in-arch-code.patch kprobes-declare-kprobes-mutex-static.patch declare-array.patch text-edit-lock-architecture-independent-code.patch text-edit-lock-alternative-i386-and-x86_64.patch text-edit-lock-kprobes-architecture-independent.patch text-edit-lock-kprobes-i386.patch text-edit-lock-kprobes-x86_64.patch text-edit-lock-i386-standardize-debug-rodata.patch text-edit-lock-x86_64-standardize-debug-rodata.patch # immediate-values-architecture-independent-code.patch immediate-values-kconfig-embedded.patch immediate-values-move-kprobes-i386-restore-interrupt-to-kdebug-h.patch add-asm-compat-to-x86.patch immediate-values-i386-optimization.patch immediate-values-powerpc-optimization.patch immediate-values-documentation.patch # linux-kernel-markers-immediate-values.patch # markers-support-multiple-probes.patch
Quoted 1 line
Tell me if you still have rejects.
I applied the above patches to 2.6.24-rc1-git14. They applied fine with just a few offsets until the last patch, which yielded this result:
patching file include/linux/marker.h
Hunk #5 succeeded at 162 with fuzz 2.
patching file kernel/marker.c
Hunk #14 FAILED at 534.
Hunk #15 FAILED at 587.
Hunk #16 FAILED at 621.
Hunk #17 FAILED at 732.
Hunk #18 FAILED at 769.
Hunk #19 succeeded at 791 (offset 12 lines).
5 out of 19 hunks FAILED -- saving rejects to file kernel/marker.c.rej patching file kernel/module.c
Hunk #1 succeeded at 1998 (offset -3 lines).
Hunk #2 succeeded at 2608 (offset -37 lines).
Hunk #3 succeeded at 2651 with fuzz 1 (offset -3 lines). patching file include/linux/module.h
Hunk #1 FAILED at 468.
Hunk #2 succeeded at 572 (offset -2 lines).
1 out of 2 hunks FAILED -- saving rejects to file include/linux/module.h.rej patching file samples/markers/probe-example.c
Mike
Quoted 17 lines
Mathieu
Quoted 12 lines
Mike
Quoted 10 lines
It still need to go through patchcheck.pl and some polishing, but it seems to work fine for me with multiple probes (the sample marker, sample probe and multiple instances of my lttng probes can connect/disconnect without problem).
Currently, the "connect/disconnect" and "arm/disarm" operations are separate. However, they could be merged. Any comment/preference on this? Being separate, a probe provider can wait until the very last moment before it activates its markers, with a minimalistic impact on the system, but it is not such a strong argument.
Mathieu
Quoted 114 lines · * Mike Mason ([email protected]) wrote:
Quoted 80 lines · Mathieu Desnoyers
Quoted 47 lines · * Mike Mason ([email protected]) wrote:
Quoted 46 lines · Mathieu Desnoyers
Quoted 45 lines · * Mathieu Desnoyers ([email protected]) wrote:
Quoted 44 lines · * Mathieu Desnoyers ([email protected]) wrote:
Quoted 6 lines · * Mike Mason ([email protected]) wrote:
Hi Mathieu,
Are you aware of any working being done to allow multiple handlers to be attached to a marker? Something like what kprobes allows. I've started to look into this and don't want to duplicate efforts.
Nope, but I know we will have to address this.
Something along the lines of walking an RCU list of function pointers, calling them.
The only downside I see is that we will have to pass a
va_list* instead
of real va args. The could make the marker site a little bit bigger and
will change the probe callback arguments.What do you think about these ideas ?
If we can find a way to make the common case (only one probe connected)
_ultra_ fast, and yet architecture independent, that would be awesome. A
simple call is kind of hard to beat though.. So we may have to think about a design with :- One call at the marker site
- if 1 probe is installed :
- If the format string is empty, connect a probe without va args.
- If the format string is not empty, connect a "stage 1" probe that
takes
the va args, starts/ends theva_listand calls _one_ function
(let's
call it "stage 2" probe), that takesva_listas parameter.
- if more than 1 probe is installed :
- The stage 1 probe creates theva_listand passes it to each
function
connected, iterated with an RCU list.What do you think ?
Mathieu
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
I'm working on an implementation.
It's ready for testing. Please grab
http://ltt.polymtl.ca/lttng/patch-2.6.24-rc1-git13-lttng-0.10-pre18.tar.bz2
patch name :
markers-support-multiple-probes.patchThis patch alone doesn't apply cleanly at all on 2.6.24-rc1-git14. Are there other patches in this series I should apply first?
Yes, the following ones should suffice :
# instrumentation menu removal add-kconfig-to-arch.patch add-arch-supports-oprofile.patch add-arch-supports-kprobes.patch move-kconfig-instrumentation-to-arch.patch # kprobes-use-mutex-for-insn-pages.patch kprobes-dont-use-kprobes-mutex-in-arch-code.patch kprobes-declare-kprobes-mutex-static.patch declare-array.patch text-edit-lock-architecture-independent-code.patch text-edit-lock-alternative-i386-and-x86_64.patch text-edit-lock-kprobes-architecture-independent.patch text-edit-lock-kprobes-i386.patch text-edit-lock-kprobes-x86_64.patch text-edit-lock-i386-standardize-debug-rodata.patch text-edit-lock-x86_64-standardize-debug-rodata.patch # immediate-values-architecture-independent-code.patch immediate-values-kconfig-embedded.patch immediate-values-move-kprobes-i386-restore-interrupt-to-kdebug-h.patch add-asm-compat-to-x86.patchQuoted 8 lines
immediate-values-i386-optimization.patch immediate-values-powerpc-optimization.patch immediate-values-documentation.patch # linux-kernel-markers-immediate-values.patch # markers-support-multiple-probes.patch Tell me if you still have rejects.I applied the above patches to 2.6.24-rc1-git14. They applied fine with just a few offsets until the last patch, which yielded this result:
patching file
include/linux/marker.h
Hunk #5 succeeded at 162 with fuzz 2.
patching filekernel/marker.c
Hunk #14 FAILED at 534.
Hunk #15 FAILED at 587.
Hunk #16 FAILED at 621.
Hunk #17 FAILED at 732.
Hunk #18 FAILED at 769.
Hunk #19 succeeded at 791 (offset 12 lines).
5 out of 19 hunks FAILED -- saving rejects to filekernel/marker.c.rejpatching filekernel/module.c
Hunk #1 succeeded at 1998 (offset -3 lines).
Hunk #2 succeeded at 2608 (offset -37 lines).
Hunk #3 succeeded at 2651 with fuzz 1 (offset -3 lines). patching fileinclude/linux/module.h
Hunk #1 FAILED at 468.
Hunk #2 succeeded at 572 (offset -2 lines).
1 out of 2 hunks FAILED -- saving rejects to fileinclude/linux/module.h.rej
patching filesamples/markers/probe-example.c
Quoted 2 lines
Mike
Ok, I released a new patchset, which should fix your problem :
http://ltt.polymtl.ca/lttng/patch-2.6.24-rc2-lttng-0.10-pre20.tar.bz2
You simply have to apply all patches up to
markers-support-multiple-probes.patch
I have moved the patch earlier in the patchset so you don't have to apply lttng. I also fixed the coding style and bugs I encountered during my testing. You may also want to try out markers-multi-probes-test.patch, which is a test module that I used to make sure the probes were correct upon multiple connect/disconnect. It is useful when you activate the "marker_debug" integer in kernel/marker.c.
For those interested in lttng, this version should be used with :
http://ltt.polymtl.ca/lttng/ltt-control-0.46-06112007.tar.gz
http://ltt.polymtl.ca/packages/lttv-0.10.0-pre2-07112007.tar.gz
Mathieu
Quoted 14 lines
Quoted 13 lines
Mathieu
Quoted 12 lines
Mike
Quoted 10 lines
It still need to go through patchcheck.pl and some polishing, but it seems to work fine for me with multiple probes (the sample marker, sample probe and multiple instances of my lttng probes can connect/disconnect without problem).
Currently, the "connect/disconnect" and "arm/disarm" operations are separate. However, they could be merged. Any comment/preference on this? Being separate, a probe provider can wait until the very last moment before it activates its markers, with a minimalistic impact on the system, but it is not such a strong argument.
Mathieu
Signature · 3 lines
Mathieu Desnoyers Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68