kprobes: Kprobes jump optimization support
Subsystems (3)
cover kprobes: Kprobes jump optimization support
Hi,
Here are the RFC patchset of the kprobes jump optimization (a.k.a. Djprobe, which I had developed two years ago). This version supports all features of Kprobes (probe disabling, module .init probing, debugfs interface).
These patches can be applied on -tip tree + x86 instruction decoder which I re-sent right now (see below). This is another example of x86 instruction decoder.
This patch depends on:
kprobes: cleanup fix_riprel() using insn decoder on x86
kprobes: checks probe address is instruction boudary on x86
x86: x86 instruction decoder build-time selftest
x86: instruction decoder API
Jump Optimized Kprobes
======================
o Concept
Kprobes uses the int3 breakpoint instruction on x86 for instrumenting
probes into running kernel. Jump optimization allows kprobes to replace breakpoint with a jump instruction for reducing probing overhead drastically.
o Performance
An optimized kprobe 5 times faster than a kprobe.
Optimizing probes gains its performance. Usually, a kprobe hit takes
0.5 to 1.0 microseconds to process. On the other hand, a jump optimized probe hit takes less than 0.1 microseconds (actual number depends on the processor). Here is a sample overheads.
Intel(R) Xeon(R) CPU E5410 @ 2.33GHz (without debugging options)
x86-32 x86-64 kprobe: 0.68us 0.91us kprobe+booster: 0.27us 0.40us kprobe+optimized: 0.06us 0.06us
kretprobe : 0.95us 1.21us kretprobe+booster: 0.53us 0.71us kretprobe+optimized: 0.30us 0.35us
(booster skips single-stepping)
Note that jump optimization also consumes more memory, but not so much.
It just uses ~200 bytes, so, even if you use ~10,000 probes, it just consumes a few MB.
o Usage
Set CONFIG_OPTPROBES=y when building a kernel, then all *probes will be
optimized if possible.
Kprobes decodes probed function and checks whether the target instructions
can be optimized(replaced with a jump) safely. If it can't be, Kprobes just doesn't optimize it.
o Optimization
Before preparing optimization, Kprobes inserts original(user-defined)
kprobe on the specified address. So, even if the kprobe is not possible to be optimized, it just uses a normal kprobe.
- Safety check
First, Kprobes gets the address of probed function and checks whether the
optimized region, which will be replaced by a jump instruction, does NOT straddle the function boundary, because if the optimized region reaches the next function, its caller causes unexpected results.
Next, Kprobes decodes whole body of probed function and checks there is
NO indirect jump, and near jump which jumps into the optimized region (except the 1st byte of jump), because if some jump instruction jumps into the middle of another instruction, it causes unexpected results too.
Kprobes also measures the length of instructions which will be replaced
by a jump instruction, because a jump instruction is longer than 1 byte, it may replaces multiple instructions, and it checks whether those instructions can be executed out-of-line.
- Preparing detour code
Then, Kprobes prepares "detour" buffer, which contains exception emulating
code (push/pop registers, call handler), copied instructions(Kprobes copies instructions which will be replaced by a jump, to the detour buffer), and a jump which jumps back to the original execution path.
- Pre-optimization
After preparing detour code, Kprobes enqueues the kprobe to optimizing list
and kicks kprobe-optimizer workqueue to optimize it. To wait other optimized probes, kprobe-optimizer will delay to work.
When the optimized-kprobe is hit before optimization, its handler
changes IP(instruction pointer) to copied code and exits. So, the instructions which were copied to detour buffer are executed on the detour buffer.
- Optimization
Kprobe-optimizer doesn't start instruction-replacing soon, it waits
synchronize_sched for safety, because some processors are possible to be interrupted on the instructions which will be replaced by a jump instruction. As you know, synchronize_sched() can ensure that all interruptions which were executed when synchronize_sched() was called are done, only if CONFIG_PREEMPT=n. So, this version supports only the kernel with CONFIG_PREEMPT=n.(*)
After that, kprobe-optimizer replaces the 4 bytes right after int3 breakpoint
with relative-jump destination, and synchronize caches on all processors. Next, it replaces int3 with relative-jump opcode, and synchronize caches again.
- Unoptimization
When unregistering, disabling kprobe or being blocked by other kprobe,
an optimized-kprobe will be unoptimized. Before kprobe-optimizer runs, the kprobe just be dequeued from the optimized list. When the optimization has been done, it replaces a jump with int3 breakpoint and original code.
First it puts int3 at the first byte of the jump, synchronize caches
on all processors, and replaces the 4 bytes right after int3 with the original code.
(*)This optimization-safety checking may be replaced with stop-machine method
which ksplice is done for supporting CONFIG_PREEMPT=y kernel.
Thank you,
---
Masami Hiramatsu (6):
kprobes: add documents of jump optimization
kprobes: x86: support kprobes jump optimization on x86
kprobes: x86: cleanup save/restore registers
kprobes: kprobes jump optimization core
kprobes: introducing generic insn_slot framework
kprobes: use list instead of hlist for insn_pages
Documentation/kprobes.txt | 172 ++++++++++++- arch/Kconfig | 11 + arch/x86/Kconfig | 1 arch/x86/include/asm/kprobes.h | 31 ++ arch/x86/kernel/kprobes.c | 528 ++++++++++++++++++++++++++++++++++------ include/linux/kprobes.h | 38 +++ kernel/kprobes.c | 508 +++++++++++++++++++++++++++++++------- 7 files changed, 1096 insertions(+), 193 deletions(-)
Signature · 7 lines
Masami Hiramatsu Software Engineer Hitachi Computer Products (America), Inc. Software Solutions Division e-mail: [email protected]
RFC PATCH 1/6 kprobes: use list instead of hlist for insn_pages
Use struct list instead of struct hlist for managing insn_pages, because insn_pages doesn't use hash table.
---
kernel/kprobes.c | 30 +++++++++++------------------- 1 files changed, 11 insertions(+), 19 deletions(-)
kernel/kprobes.c +11 -19
@@ -103,7 +103,7 @@ static struct kprobe_blackpoint kprobe_blacklist[] = {#define INSNS_PER_PAGE (PAGE_SIZE/(MAX_INSN_SIZE * sizeof(kprobe_opcode_t)))struct kprobe_insn_page { -struct hlist_node hlist; +struct list_head list;kprobe_opcode_t *insns;/* Page of instruction slots */ char slot_used[INSNS_PER_PAGE ];int nused;
@@ -117,7 +117,7 @@ enum kprobe_slot_state { };static DEFINE_MUTEX (kprobe_insn_mutex);/* Protects kprobe_insn_pages */ -static struct hlist_head kprobe_insn_pages; +static LIST_HEAD (kprobe_insn_pages);static int kprobe_garbage_slots;static int collect_garbage_slots (void );
@@ -152,10 +152,9 @@ loop_end:static kprobe_opcode_t __kprobes *__get_insn_slot (void ) {struct kprobe_insn_page *kip; -struct hlist_node *pos; retry: -hlist_for_each_entry (kip, pos, &kprobe_insn_pages, hlist) { +list_for_each_entry (kip, &kprobe_insn_pages, list) {if (kip->nused <INSNS_PER_PAGE ) {int i;for (i =0 ; i <INSNS_PER_PAGE ; i++) {
@@ -189,8 +188,8 @@ static kprobe_opcode_t __kprobes *__get_insn_slot(void)kfree (kip);return NULL ; } -INIT_HLIST_NODE (&kip->hlist); -hlist_add_head (&kip->hlist, &kprobe_insn_pages); +INIT_LIST_HEAD (&kip->list); +list_add (&kip->list, &kprobe_insn_pages);memset (kip->slot_used,SLOT_CLEAN ,INSNS_PER_PAGE ); kip->slot_used[0 ] =SLOT_USED ; kip->nused =1 ;
@@ -219,12 +218,8 @@ static int __kprobes collect_one_slot(struct kprobe_insn_page *kip, int idx)* so as not to have to set it up again the * next time somebody inserts a probe. */ -hlist_del (&kip->hlist); -if (hlist_empty (&kprobe_insn_pages)) { -INIT_HLIST_NODE (&kip->hlist); -hlist_add_head (&kip->hlist, - &kprobe_insn_pages); - }else { +if (!list_is_singular (&kprobe_insn_pages)) { +list_del (&kip->list);module_free (NULL , kip->insns);kfree (kip); }
@@ -235,8 +230,7 @@ static int __kprobes collect_one_slot(struct kprobe_insn_page *kip, int idx)static int __kprobes collect_garbage_slots (void ) { -struct kprobe_insn_page *kip; -struct hlist_node *pos, *next; +struct kprobe_insn_page *kip, *next;int safety;/* Ensure no-one is preepmted on the garbages */
@@ -246,7 +240,7 @@ static int __kprobes collect_garbage_slots(void)if (safety !=0 )return -EAGAIN ; -hlist_for_each_entry_safe (kip, pos, next, &kprobe_insn_pages, hlist) { +list_for_each_entry_safe (kip, next, &kprobe_insn_pages, list) {int i;if (kip->ngarbage ==0 )continue ;
@@ -264,19 +258,17 @@ static int __kprobes collect_garbage_slots(void)void __kprobes free_insn_slot (kprobe_opcode_t * slot,int dirty) {struct kprobe_insn_page *kip; -struct hlist_node *pos;mutex_lock (&kprobe_insn_mutex); -hlist_for_each_entry (kip, pos, &kprobe_insn_pages, hlist) { +list_for_each_entry (kip, &kprobe_insn_pages, list) {if (kip->insns <= slot && slot < kip->insns + (INSNS_PER_PAGE *MAX_INSN_SIZE )) {int i = (slot - kip->insns) /MAX_INSN_SIZE ;if (dirty) { kip->slot_used[i] =SLOT_DIRTY ; kip->ngarbage++; - }else { + }else collect_one_slot (kip, i); - }break ; } }
Signature · 7 lines
Masami Hiramatsu Software Engineer Hitachi Computer Products (America), Inc. Software Solutions Division e-mail: [email protected]
RFC PATCH 2/6 kprobes: introducing generic insn_slot framework
In reply to Masami HiramatsuMake insn_slot framework support various size slots.
Current insn_slot just supports one-size instruction buffer slot. However, kprobes jump optimization needs larger size buffers.
---
kernel/kprobes.c | 96 +++++++++++++++++++++++++++++++++--------------------- 1 files changed, 58 insertions(+), 38 deletions(-)
kernel/kprobes.c +58 -38
@@ -100,26 +100,38 @@ static struct kprobe_blackpoint kprobe_blacklist[] = {* stepping on the instruction on a vmalloced/kmalloced/data page * is a recipe for disaster */ -#define INSNS_PER_PAGE (PAGE_SIZE/(MAX_INSN_SIZE * sizeof(kprobe_opcode_t))) -struct kprobe_insn_page {struct list_head list;kprobe_opcode_t *insns;/* Page of instruction slots */ -char slot_used[INSNS_PER_PAGE ];int nused;int ngarbage; +char slot_used[1 ]; +}; + +struct kprobe_insn_cache { +struct list_head pages;/* list of kprobe_insn_page */ +size_t insn_size;/* size of instruction slot */ +int nr_garbage; }; +static int slots_per_page (struct kprobe_insn_cache *c) +{ +return PAGE_SIZE /(c->insn_size *sizeof (kprobe_opcode_t)); +} +enum kprobe_slot_state {SLOT_CLEAN =0 ,SLOT_DIRTY =1 ,SLOT_USED =2 , }; -static DEFINE_MUTEX (kprobe_insn_mutex);/* Protects kprobe_insn_pages */ -static LIST_HEAD (kprobe_insn_pages); -static int kprobe_garbage_slots; -static int collect_garbage_slots (void ); +static DEFINE_MUTEX (kprobe_insn_mutex);/* Protects kprobe_insn_slots */ +static struct kprobe_insn_cache kprobe_insn_slots = { + .pages =LIST_HEAD_INIT (kprobe_insn_slots.pages), + .insn_size =MAX_INSN_SIZE , + .nr_garbage =0 , +}; +static int __kprobes collect_garbage_slots (struct kprobe_insn_cache *c);static int __kprobes check_safety (void ) {
@@ -149,32 +161,33 @@ loop_end:* __get_insn_slot() - Find a slot on an executable page for an instruction. * We allocate an executable page if there's no room on existing ones. */ -static kprobe_opcode_t __kprobes *__get_insn_slot (void ) +static kprobe_opcode_t __kprobes *__get_insn_slot (struct kprobe_insn_cache *c) {struct kprobe_insn_page *kip; retry: -list_for_each_entry (kip, &kprobe_insn_pages, list) { -if (kip->nused <INSNS_PER_PAGE ) { +list_for_each_entry (kip, &c->pages, list) { +if (kip->nused <slots_per_page (c)) {int i; -for (i =0 ; i <INSNS_PER_PAGE ; i++) { +for (i =0 ; i <slots_per_page (c); i++) {if (kip->slot_used[i] ==SLOT_CLEAN ) { kip->slot_used[i] =SLOT_USED ; kip->nused++; -return kip->insns + (i *MAX_INSN_SIZE ); +return kip->insns + (i * c->insn_size); } } -/* Surprise! No unused slots. Fix kip->nused. */ - kip->nused =INSNS_PER_PAGE ; +/* kip->nused is broken. */ +BUG (); } }/* If there are any garbage slots, collect it and try again. */ -if (kprobe_garbage_slots &&collect_garbage_slots () ==0 ) { +if (c->nr_garbage &&collect_garbage_slots (c) ==0 ) goto retry; - } +/* All out of space. Need to allocate a new page. Use slot 0. */ - kip =kmalloc (sizeof (struct kprobe_insn_page ),GFP_KERNEL ); + kip =kmalloc (sizeof (struct kprobe_insn_page ) +slots_per_page (c) -1 , +GFP_KERNEL );if (!kip)return NULL ;
@@ -189,19 +202,20 @@ static kprobe_opcode_t __kprobes *__get_insn_slot(void)return NULL ; }INIT_LIST_HEAD (&kip->list); -list_add (&kip->list, &kprobe_insn_pages); -memset (kip->slot_used,SLOT_CLEAN ,INSNS_PER_PAGE ); +memset (kip->slot_used,SLOT_CLEAN ,slots_per_page (c)); kip->slot_used[0 ] =SLOT_USED ; kip->nused =1 ; kip->ngarbage =0 ; +list_add (&kip->list, &c->pages);return kip->insns; } +kprobe_opcode_t __kprobes *get_insn_slot (void ) { -kprobe_opcode_t *ret; +kprobe_opcode_t *ret =NULL ;mutex_lock (&kprobe_insn_mutex); - ret =__get_insn_slot (); + ret =__get_insn_slot (&kprobe_insn_slots);mutex_unlock (&kprobe_insn_mutex);return ret; }
@@ -218,7 +232,7 @@ static int __kprobes collect_one_slot(struct kprobe_insn_page *kip, int idx)* so as not to have to set it up again the * next time somebody inserts a probe. */ -if (!list_is_singular (&kprobe_insn_pages)) { +if (!list_is_singular (&kip->list)) {list_del (&kip->list);module_free (NULL , kip->insns);kfree (kip);
@@ -228,7 +242,7 @@ static int __kprobes collect_one_slot(struct kprobe_insn_page *kip, int idx)return 0 ; } -static int __kprobes collect_garbage_slots (void ) +static int __kprobes collect_garbage_slots (struct kprobe_insn_cache *c) {struct kprobe_insn_page *kip, *next;int safety;
@@ -240,42 +254,48 @@ static int __kprobes collect_garbage_slots(void)if (safety !=0 )return -EAGAIN ; -list_for_each_entry_safe (kip, next, &kprobe_insn_pages, list) { +list_for_each_entry_safe (kip, next, &c->pages, list) {int i;if (kip->ngarbage ==0 )continue ; kip->ngarbage =0 ;/* we will collect all garbages */ -for (i =0 ; i <INSNS_PER_PAGE ; i++) { +for (i =0 ; i <slots_per_page (c); i++) {if (kip->slot_used[i] ==SLOT_DIRTY &&collect_one_slot (kip, i))break ; } } - kprobe_garbage_slots =0 ; + c->nr_garbage =0 ;return 0 ; } -void __kprobes free_insn_slot (kprobe_opcode_t * slot,int dirty) +static void __kprobes __free_insn_slot (struct kprobe_insn_cache *c, +kprobe_opcode_t *slot,int dirty) {struct kprobe_insn_page *kip; -mutex_lock (&kprobe_insn_mutex); -list_for_each_entry (kip, &kprobe_insn_pages, list) { -if (kip->insns <= slot && - slot < kip->insns + (INSNS_PER_PAGE *MAX_INSN_SIZE )) { -int i = (slot - kip->insns) /MAX_INSN_SIZE ; +list_for_each_entry (kip, &c->pages, list) { +long idx = ((long )slot - (long )kip->insns) / c->insn_size; +if (idx >=0 && idx <slots_per_page (c)) { +WARN_ON (kip->slot_used[idx] !=SLOT_USED );if (dirty) { - kip->slot_used[i] =SLOT_DIRTY ; + kip->slot_used[idx] =SLOT_DIRTY ; kip->ngarbage++; +if (++c->nr_garbage >slots_per_page (c)) +collect_garbage_slots (c); }else -collect_one_slot (kip, i); -break ; +collect_one_slot (kip, idx); +return ; } } +/* Could not free this slot. */ +WARN_ON (1 ); +} -if (dirty && ++kprobe_garbage_slots >INSNS_PER_PAGE ) -collect_garbage_slots (); - +void __kprobes free_insn_slot (kprobe_opcode_t * slot,int dirty) +{ +mutex_lock (&kprobe_insn_mutex); +__free_insn_slot (&kprobe_insn_slots, slot, dirty);mutex_unlock (&kprobe_insn_mutex); }#endif
Signature · 7 lines
Masami Hiramatsu Software Engineer Hitachi Computer Products (America), Inc. Software Solutions Division e-mail: [email protected]
RFC PATCH 4/6 kprobes: x86: cleanup save/restore registers
In reply to Masami HiramatsuIntroduce SAVE/RESOTRE_REGS_STRING for cleanup kretprobe-trampoline asm code. These macros will be used for emulating interruption.
---
arch/x86/kernel/kprobes.c | 128 ++++++++++++++++++++++++--------------------- 1 files changed, 67 insertions(+), 61 deletions(-)
arch/x86/kernel/kprobes.c +67 -61
@@ -568,6 +568,69 @@ static int __kprobes kprobe_handler(struct pt_regs *regs)return 0 ; } +#ifdef CONFIG_X86_64 +#define SAVE_REGS_STRING \ +/* Skip cs, ip, orig_ax. */ \ + " subq $24, %rsp\n" \ + " pushq %rdi\n" \ + " pushq %rsi\n" \ + " pushq %rdx\n" \ + " pushq %rcx\n" \ + " pushq %rax\n" \ + " pushq %r8\n" \ + " pushq %r9\n" \ + " pushq %r10\n" \ + " pushq %r11\n" \ + " pushq %rbx\n" \ + " pushq %rbp\n" \ + " pushq %r12\n" \ + " pushq %r13\n" \ + " pushq %r14\n" \ + " pushq %r15\n" +#define RESTORE_REGS_STRING \ + " popq %r15\n" \ + " popq %r14\n" \ + " popq %r13\n" \ + " popq %r12\n" \ + " popq %rbp\n" \ + " popq %rbx\n" \ + " popq %r11\n" \ + " popq %r10\n" \ + " popq %r9\n" \ + " popq %r8\n" \ + " popq %rax\n" \ + " popq %rcx\n" \ + " popq %rdx\n" \ + " popq %rsi\n" \ + " popq %rdi\n" \ +/* Skip orig_ax, ip, cs */ \ + " addq $24, %rsp\n" +#else +#define SAVE_REGS_STRING \ +/* Skip cs, ip, orig_ax and gs. */ \ + " subl $16, %esp\n" \ + " pushl %fs\n" \ + " pushl %ds\n" \ + " pushl %es\n" \ + " pushl %eax\n" \ + " pushl %ebp\n" \ + " pushl %edi\n" \ + " pushl %esi\n" \ + " pushl %edx\n" \ + " pushl %ecx\n" \ + " pushl %ebx\n" +#define RESTORE_REGS_STRING \ + " popl %ebx\n" \ + " popl %ecx\n" \ + " popl %edx\n" \ + " popl %esi\n" \ + " popl %edi\n" \ + " popl %ebp\n" \ + " popl %eax\n" \ +/* Skip ds, es, fs, gs, orig_ax, and ip. Note: don't pop cs here*/ \ + " addl $24, %esp\n" +#endif +/* * When a retprobed function returns, this code saves registers and * calls trampoline_handler() runs, which calls the kretprobe's handler.
@@ -581,65 +644,16 @@ static void __used __kprobes kretprobe_trampoline_holder(void)/* We don't bother saving the ss register */ " pushq %rsp\n" " pushfq\n" -/* -* Skip cs, ip, orig_ax. -* trampoline_handler() will plug in these values -*/ -" subq $24, %rsp\n" -" pushq %rdi\n" -" pushq %rsi\n" -" pushq %rdx\n" -" pushq %rcx\n" -" pushq %rax\n" -" pushq %r8\n" -" pushq %r9\n" -" pushq %r10\n" -" pushq %r11\n" -" pushq %rbx\n" -" pushq %rbp\n" -" pushq %r12\n" -" pushq %r13\n" -" pushq %r14\n" -" pushq %r15\n" +SAVE_REGS_STRING " movq %rsp, %rdi\n" " call trampoline_handler\n" /* Replace saved sp with true return address. */ " movq %rax, 152(%rsp)\n" -" popq %r15\n" -" popq %r14\n" -" popq %r13\n" -" popq %r12\n" -" popq %rbp\n" -" popq %rbx\n" -" popq %r11\n" -" popq %r10\n" -" popq %r9\n" -" popq %r8\n" -" popq %rax\n" -" popq %rcx\n" -" popq %rdx\n" -" popq %rsi\n" -" popq %rdi\n" -/* Skip orig_ax, ip, cs */ -" addq $24, %rsp\n" +RESTORE_REGS_STRING " popfq\n"#else " pushf\n" -/* -* Skip cs, ip, orig_ax and gs. -* trampoline_handler() will plug in these values -*/ -" subl $16, %esp\n" -" pushl %fs\n" -" pushl %es\n" -" pushl %ds\n" -" pushl %eax\n" -" pushl %ebp\n" -" pushl %edi\n" -" pushl %esi\n" -" pushl %edx\n" -" pushl %ecx\n" -" pushl %ebx\n" +SAVE_REGS_STRING " movl %esp, %eax\n" " call trampoline_handler\n" /* Move flags to cs */
@@ -647,15 +661,7 @@ static void __used __kprobes kretprobe_trampoline_holder(void)" movl %edx, 52(%esp)\n" /* Replace saved flags with true return address. */ " movl %eax, 56(%esp)\n" -" popl %ebx\n" -" popl %ecx\n" -" popl %edx\n" -" popl %esi\n" -" popl %edi\n" -" popl %ebp\n" -" popl %eax\n" -/* Skip ds, es, fs, gs, orig_ax and ip */ -" addl $24, %esp\n" +RESTORE_REGS_STRING " popf\n"#endif " ret\n");
Signature · 7 lines
Masami Hiramatsu Software Engineer Hitachi Computer Products (America), Inc. Software Solutions Division e-mail: [email protected]
RFC PATCH 3/6 kprobes: kprobes jump optimization core
In reply to Masami HiramatsuIntroduce kprobes jump optimization arch-independent parts. Kprobes uses breakpoint instruction for interrupting execution flow, on some architectures, it can be replaced by a jump instruction and interruption emulation code. This gains kprobs' performance drastically.
---
arch/Kconfig | 11 + include/linux/kprobes.h | 38 +++++ kernel/kprobes.c | 394 ++++++++++++++++++++++++++++++++++++++++++----- 3 files changed, 398 insertions(+), 45 deletions(-)
arch/Kconfig +11 -0
@@ -44,6 +44,15 @@ config KPROBES for kernel debugging, non-intrusive instrumentation and testing. If in doubt, say "N". +config OPTPROBES + bool "Kprobes jump optimization support (EXPERIMENTAL)" + depends on KPROBES + depends on !PREEMPT + depends on HAVE_OPTPROBES + help + This option will allow kprobes to optimize breakpoint to + a jump for reducing its overhead. + config HAVE_EFFICIENT_UNALIGNED_ACCESS bool help
@@ -79,6 +88,8 @@ config HAVE_KPROBES config HAVE_KRETPROBES bool +config HAVE_OPTPROBES + bool # # An arch should select this if it provides all these things: #
include/linux/kprobes.h +38 -0
@@ -122,6 +122,11 @@ struct kprobe {/* Kprobe status flags */ #define KPROBE_FLAG_GONE 1/* breakpoint has already gone */ #define KPROBE_FLAG_DISABLED 2/* probe is temporarily disabled */ +#define KPROBE_FLAG_OPTIMIZED 4/* +* probe is really optimized. +* NOTE: +* this flag is only for optimized_kprobe. +*/ /* Has this kprobe gone ? */ static inline int kprobe_gone (struct kprobe *p)
@@ -134,6 +139,12 @@ static inline int kprobe_disabled(struct kprobe *p) {return p->flags & (KPROBE_FLAG_DISABLED |KPROBE_FLAG_GONE ); } + +/* Is this kprobe really running optimized path ? */ +static inline int kprobe_optimized (struct kprobe *p) +{ +return p->flags &KPROBE_FLAG_OPTIMIZED ; +}/* * Special probe type that uses setjmp-longjmp type tricks to resume * execution at a specified entry with a matching prototype corresponding
@@ -249,6 +260,33 @@ extern kprobe_opcode_t *get_insn_slot(void);extern void free_insn_slot (kprobe_opcode_t *slot,int dirty);extern void kprobes_inc_nmissed_count (struct kprobe *p); +#ifdef CONFIG_OPTPROBES +/* +* Internal structure for direct jump optimized probe +*/ +struct optimized_kprobe { +struct kprobe kp; +struct list_head list;/* list for optimizing queue */ +struct arch_optimized_insn optinsn; +}; + +/* Architecture dependent functions for direct jump optimization */ +extern int arch_prepared_optinsn (struct arch_optimized_insn *optinsn); +extern int arch_check_optimized_kprobe (struct optimized_kprobe *op); +extern int arch_prepare_optimized_kprobe (struct optimized_kprobe *op); +extern void arch_remove_optimized_kprobe (struct optimized_kprobe *op); +extern int arch_optimize_kprobe (struct optimized_kprobe *op); +extern void arch_unoptimize_kprobe (struct optimized_kprobe *op); +extern int arch_detour_optimized_kprobe (struct optimized_kprobe *op, +struct pt_regs *regs); +extern kprobe_opcode_t *get_optinsn_slot (void ); +extern void free_optinsn_slot (kprobe_opcode_t *slot,int dirty); +extern int arch_within_optimized_kprobe (struct optimized_kprobe *op, +unsigned long addr); + +extern void opt_pre_handler (struct kprobe *p,struct pt_regs *regs); +#endif /* CONFIG_OPTPROBES */ +/* Get the kprobe at this addr (if any) - called with preemption disabled */ struct kprobe *get_kprobe (void *addr); void kretprobe_hash_lock(struct task_struct *tsk,
kernel/kprobes.c +349 -45
576 lines not on this page · raw message
Signature · 7 lines
Masami Hiramatsu Software Engineer Hitachi Computer Products (America), Inc. Software Solutions Division e-mail: [email protected]
RFC PATCH 6/6 kprobes: add documents of jump optimization
In reply to Masami HiramatsuAdd documentations about kprobe jump optimization to Documentation/kprobes.txt.
---
Documentation/kprobes.txt | 172 ++++++++++++++++++++++++++++++++++++++++++--- 1 files changed, 159 insertions(+), 13 deletions(-)
Documentation/kprobes.txt +159 -13
@@ -1,6 +1,7 @@ Title : Kernel Probes (Kprobes) Authors : Jim Keniston <[email protected]> : Prasanna S Panchamukhi <[email protected]> + : Masami Hiramatsu <[email protected]> CONTENTS
@@ -14,6 +15,7 @@ CONTENTS 8. Kprobes Example 9. Jprobes Example 10. Kretprobes Example +11. Optimization Example Appendix A: The kprobes debugfs interface 1. Concepts: Kprobes, Jprobes, Return Probes
@@ -42,13 +44,13 @@ registration/unregistration of a group of *probes. These functions can speed up unregistration process when you have to unregister a lot of probes at once. -The next three subsections explain how the different types of -probes work. They explain certain things that you'll need to -know in order to make the best use of Kprobes -- e.g., the -difference between a pre_handler and a post_handler, and how -to use the maxactive and nmissed fields of a kretprobe. But -if you're in a hurry to start using Kprobes, you can skip ahead -to section 2. +The next four subsections explain how the different types of +probes work and how the optimization works. They explain certain +things that you'll need to know in order to make the best use of +Kprobes -- e.g., the difference between a pre_handler and +a post_handler, and how to use the maxactive and nmissed fields of +a kretprobe. But if you're in a hurry to start using Kprobes, you +can skip ahead to section 2. 1.1 How Does a Kprobe Work?
@@ -161,13 +163,105 @@ In case probed function is entered but there is no kretprobe_instance object available, then in addition to incrementing the nmissed count, the user entry_handler invocation is also skipped. +1.4 How Does the Optimization Work? + + If you configured kernel with CONFIG_OPTPROBES=y (currently this option is +supported on x86/x86-64, non-preemptive kernel), kprobes tries to use a +jump instruction instead of breakpoint instruction automatically. + +1.4.1 Init a Kprobe + + Before preparing optimization, Kprobes inserts original(user-defined) +kprobe on the specified address. So, even if the kprobe is not +possible to be optimized, it just uses a normal kprobe. + +1.4.2 Safety check + + First, Kprobes gets the address of probed function and checks whether the +optimized region, which will be replaced by a jump instruction, does NOT +straddle the function boundary, because if the optimized region reaches the +next function, its caller causes unexpected results. + Next, Kprobes decodes whole body of probed function and checks there is +NO indirect jump, and near jump which jumps into the optimized region (except +the 1st byte of jump), because if some jump instruction jumps into the middle +of another instruction, it causes unexpected results too. + Kprobes also measures the length of instructions which will be replaced +by a jump instruction, because a jump instruction is longer than 1 byte, +it may replaces multiple instructions, and it checks whether those +instructions can be executed out-of-line. + +1.4.3 Preparing detour buffer + + Then, Kprobes prepares "detour" buffer, which contains exception emulating +code (push/pop registers, call handler), copied instructions(Kprobes copies +instructions which will be replaced by a jump, to the detour buffer), and +a jump which jumps back to the original execution path. + +1.4.4 Pre-optimization + + After preparing detour buffer, Kprobes checks that the probe is *NOT* in +the below cases; + - The probe has either break_handler or post_handler. + - Other probes are probing the instructions which will be replaced by + a jump instruction. + - The probe is disabled. +In above cases, Kprobes just doesn't start optimizating the probe. + + If the kprobe can be optimized, Kprobes enqueues the kprobe to optimizing +list and kicks kprobe-optimizer workqueue to optimize it. To wait other +optimized probes, kprobe-optimizer will delay to work. + When the optimized-kprobe is hit before optimization, its handler changes +IP(instruction pointer) to copied code and exits. So, the instructions which +were copied to detour buffer are executed on the detour buffer. + +1.4.5 Optimization + + Kprobe-optimizer doesn't start instruction-replacing soon, it waits +synchronize_sched for safety, because some processors are possible to be +interrupted on the instructions which will be replaced by a jump instruction. +As you know, synchronize_sched() can ensure that all interruptions which were +executed when synchronize_sched() was called are done, only if +CONFIG_PREEMPT=n. So, this version supports only the kernel with +CONFIG_PREEMPT=n.(*) + After that, kprobe-optimizer replaces the 4 bytes right after int3 +breakpoint with relative-jump destination, and synchronize caches on all +processors. And then, it replaces int3 with relative-jump opcode, and +synchronize caches again. + + After optimizing the probe, a CPU hits the jump instruction and jumps to +the out-of-line buffer directly. Thus the breakpoint exception is skipped. + +1.4.6 Unoptimization + + When unregistering, disabling kprobe or being blocked by other kprobe, +an optimized-kprobe will be unoptimized. Before kprobe-optimizer runs, +the kprobe just be dequeued from the optimized list. When the optimization +has been done, it replaces a jump with int3 breakpoint and original code. + First it puts int3 at the first byte of the jump, synchronize caches +on all processors, replaces the 4 bytes right after int3 with the original +code and synchronize caches again. + +(*)This optimization-safety checking may be replaced with stop-machine method + which ksplice is done for supporting CONFIG_PREEMPT=y kernel. + +NOTE for geeks: +The jump optimization changes the kprobe's pre_handler behavior. +Without optimization, pre_handler can change kernel execution path by +changing regs->ip and return 1. However, after optimizing the probe, +that modification is ignored. Thus, if you'd like to tweak kernel +execution path, you need to avoid optimization. In that case, you can +choose either, + - Set empty function to post_handler or break_handler. + or + - Config CONFIG_OPTPROBES=n. + 2. Architectures Supported Kprobes, jprobes, and return probes are implemented on the following architectures: -- i386 -- x86_64 (AMD-64, EM64T) +- i386 (Supports jump optimization) +- x86_64 (AMD-64, EM64T) (Supports jump optimization) - ppc64 - ia64 (Does not support probes on instruction slot1.) - sparc64 (Return probes not yet implemented.)
@@ -193,6 +287,10 @@ it useful to "Compile the kernel with debug info" (CONFIG_DEBUG_INFO), so you can use "objdump -d -l vmlinux" to see the source-to-object code mapping. +If you want to reduce probing overhead, set "Kprobes jump optimization +support" (CONFIG_OPTPROBES) to "y". You can find this option under +"Kprobes" line. + 4. API Reference The Kprobes API includes a "register" function and an "unregister"
@@ -387,9 +485,12 @@ the probe which has been registered. 5. Kprobes Features and Limitations -Kprobes allows multiple probes at the same address. Currently, -however, there cannot be multiple jprobes on the same function at -the same time. +Kprobes allows multiple probes at the same address even if it is optimized. +Currently, however, there cannot be multiple jprobes on the same function +at the same time. And also, optimized kprobes can not invoke the +post_handler and the break_handler. So if you attempt to install the probe +which has the the post_handler or the break_handler at the same address of +an optimized kprobe, the probe will be unoptimized automatically. In general, you can install a probe anywhere in the kernel. In particular, you can probe interrupt handlers. Known exceptions
@@ -453,6 +554,37 @@ reason, Kprobes doesn't support return probes (or kprobes or jprobes) on the x86_64 version of __switch_to(); the registration functions return -EINVAL. +On x86/x86-64, since the Jump Optimization of Kprobes modifies instructions +widely, there are some limitations for optimization. To explain it, +we introduce some terminology. Image certain binary line which is +constructed by 2 byte instruction, 2byte instruction and 3byte instruction. + + IA + | +[-2][-1][0][1][2][3][4][5][6][7] + [ins1][ins2][ ins3 ] + [<- DCR ->] + [<- JTPR ->] + +ins1: 1st Instruction +ins2: 2nd Instruction +ins3: 3rd Instruction +IA: Insertion Address +JTPR: Jump Target Prohibition Region +DCR: Detoured Code Region + +The instructions in DCR are copied to the out-of-line buffer +of the djprobe instance, because the bytes in JTPR are replaced by +a jump instruction. So, there are several limitations. + +a) The instructions in DCR must be relocatable. +b) The instructions in DCR must not include call instruction. +c) JTPR must not be targeted by any jump or call instruction. +d) DCR must not straddle the border betweeen functions. + +Anyway, these limitations are checked by in-kernel instruction decoder, +so you don't need to care about that. + 6. Probe Overhead On a typical CPU in use in 2005, a kprobe hit takes 0.5 to 1.0
@@ -476,6 +608,19 @@ k = 0.49 usec; j = 0.76; r = 0.80; kr = 0.82; jr = 1.07 ppc64: POWER5 (gr), 1656 MHz (SMT disabled, 1 virtual CPU per physical CPU) k = 0.77 usec; j = 1.31; r = 1.26; kr = 1.45; jr = 1.99 +6.1 Optimized Probe Overhead + +Typically, an optimized kprobe hit takes 0.07 to 0.1 microseconds to +process. Here are sample overhead figures (in usec) for x86-64 architectures. +k = unoptimized kprobe, b = boosted(single-step skipped), o = optimized kprobe, +r = unoptimized kretprobe, rb = boosted kretprobe, ro = optimized kretprobe. + +i386: Intel(R) Xeon(R) E5410, 2.33GHz, 4656.90 bogomips +k = 0.68 usec; b = 0.27; o = 0.06; r = 0.95; rb = 0.53; ro = 0.30 + +x86-64: Intel(R) Xeon(R) E5410, 2.33GHz, 4656.90 bogomips +k = 0.91 usec; b = 0.40; o = 0.06; r = 1.21; rb = 0.71; ro = 0.35 + 7. TODO a. SystemTap (http://sourceware.org/systemtap): Provides a simplified
@@ -523,7 +668,8 @@ is also specified. Following columns show probe status. If the probe is on a virtual address that is no longer valid (module init sections, module virtual addresses that correspond to modules that've been unloaded), such probes are marked with [GONE]. If the probe is temporarily disabled, -such probes are marked with [DISABLED]. +such probes are marked with [DISABLED]. If the probe is optimized, it is +marked with [OPTIMIZED]. /debug/kprobes/enabled: Turn kprobes ON/OFF forcibly.
Signature · 7 lines
Masami Hiramatsu Software Engineer Hitachi Computer Products (America), Inc. Software Solutions Division e-mail: [email protected]
RFC PATCH 5/6 kprobes: x86: support kprobes jump optimization on x86
In reply to Masami HiramatsuIntroduce x86 arch-specific optimization code, which supports both of x86-32 and x86-64.
This code also supports safety checking, which decodes whole of a function in which probe is inserted, and checks following conditions before optimization:
- The optimized instructions which will be replaced by a jump instruction don't straddle the function boundary.
- There is no indirect jump instruction, because it will jumps into the address range which is replaced by jump operand.
- There is no jump/loop instruction which jumps into the address range which is replaced by jump operand.
---
arch/x86/Kconfig | 1 arch/x86/include/asm/kprobes.h | 31 +++ arch/x86/kernel/kprobes.c | 400 ++++++++++++++++++++++++++++++++++++++-- 3 files changed, 409 insertions(+), 23 deletions(-)
arch/x86/Kconfig +1 -0
@@ -29,6 +29,7 @@ config X86 select ARCH_WANT_OPTIONAL_GPIOLIB select ARCH_WANT_FRAME_POINTERS select HAVE_KRETPROBES + select HAVE_OPTPROBES select HAVE_FTRACE_MCOUNT_RECORD select HAVE_DYNAMIC_FTRACE select HAVE_FUNCTION_TRACER
arch/x86/include/asm/kprobes.h +30 -1
@@ -32,7 +32,10 @@ struct kprobe;typedef u8 kprobe_opcode_t ;#define BREAKPOINT_INSTRUCTION 0xcc -#define RELATIVEJUMP_INSTRUCTION 0xe9 +#define RELATIVEJUMP_OPCODE 0xe9 +#define RELATIVECALL_OPCODE 0xe8 +#define RELATIVE_ADDR_SIZE 4 +#define RELATIVE_JUMP_SIZE (sizeof(kprobe_opcode_t) + RELATIVE_ADDR_SIZE)#define MAX_INSN_SIZE 16#define MAX_STACK_SIZE 64#define MIN_STACK_SIZE (ADDR ) \
@@ -44,6 +47,17 @@ typedef u8 kprobe_opcode_t;#define flush_insn_slot (p) do { } while (0) +/* optinsn template addresses */ +extern kprobe_opcode_t optprobe_template_entry; +extern kprobe_opcode_t optprobe_template_val; +extern kprobe_opcode_t optprobe_template_call; +extern kprobe_opcode_t optprobe_template_end; +#define MAX_OPTIMIZED_LENGTH (MAX_INSN_SIZE + RELATIVE_ADDR_SIZE) +#define MAX_OPTINSN_SIZE \ + (((unsigned long)&optprobe_template_end - \ + (unsigned long)&optprobe_template_entry) + \ + MAX_OPTIMIZED_LENGTH + RELATIVE_JUMP_SIZE) +extern const int kretprobe_blacklist_size;void arch_remove_kprobe (struct kprobe *p);
@@ -64,6 +78,21 @@ struct arch_specific_insn {int boostable; }; +struct arch_optimized_insn { +/* copy of the original instructions */ +kprobe_opcode_t copied_insn[RELATIVE_ADDR_SIZE ]; +/* detour code buffer */ +kprobe_opcode_t *insn; +/* the size of instructions copied to detour code buffer */ +size_t size; +}; + +/* Return true (!0) if optinsn is prepared for optimization. */ +static inline int arch_prepared_optinsn (struct arch_optimized_insn *optinsn) +{ +return optinsn->size; +} +struct prev_kprobe {struct kprobe *kp;unsigned long status;
arch/x86/kernel/kprobes.c +378 -22
456 lines not on this page · raw message
Signature · 7 lines
Masami Hiramatsu Software Engineer Hitachi Computer Products (America), Inc. Software Solutions Division e-mail: [email protected]
Quoted 1 line · Masami Hiramatsu
- Safety check
[...]
Quoted 4 lines
Next, Kprobes decodes whole body of probed function and checks there is
NO indirect jump, and near jump which jumps into the optimized region (except the 1st byte of jump), because if some jump instruction jumps into the middle of another instruction, it causes unexpected results too.
Hi Masami,
I think your safety check algorithm is wrong.
There are several ways in which the kernel might jump into the optimized region that cannot be detected by examining just the probed function:
(1) The compiler is allowed to do cross-function optimization within a compilation unit where code in one function jumps into the middle of another function.
(2) If you have a switch statement that looks like:
switch (foo) {
case 1:
printk("a1");
break;
case 2:
printk("a2");
break;
case 3:
printk("a3");
break;
case 4:
printk("a4");
break;
case 5:
printk("a5");
break;
}(i.e. a large number of cases indexed by a small range of integers; depending on your compiler you may need more cases), gcc will implement it using a jump table in the .rodata section. On x86_32, the generated assembly for the switch will read from the jump table at an offset of 4 * (foo - 1) and then jump to that address to reach the code for the appropriate case. These jump tables can result in jumps from within the probed function into the optimized region in a way that your algorithm would not detect.
(3) If the code that you're overwriting can throw an exception, it might be that that there is a jump from the .fixup section back into the probed function that overlaps the optimized region.
The other comment I have about this approach is that it seems you've written a completely new x86 disassembler in order to do binary code analysis in the kernel.
Ksplice has been using the udis86 disassembler for this purpose:
<http://udis86.sourceforge.net/>. udis86 is intended for binary code analysis and is designed to be embedded into kernels and other applications.
udis86 generates all its instruction table data from an XML opcode file, which is I think what H. Peter Anvin was suggesting you should do in this previous thread on your instruction decoder:
<http://lkml.indiana.edu/hypermail/linux/kernel/0904.0/01929.html>
Compared to e.g. libopcodes it is still quite small -- there's a total of about 3000 lines of C, plus some instruction tables that are automatically generated from an XML description of the instructions.
The upstream developer has merged patches from Anders Kaseorg and myself that make it build as part of the core Linux kernel without any changes other than adding a kernel Makefile. I think if we're going to putting an x86 disassembler into the kernel, it might be better to use something like udis86 that provides a little more information about the instructions being disassembled and is more data-driven.
-Tim Abbott
Hi Tim,
Thank you for your comment!
Quoted 11 lines · Tim Abbott
Quoted 1 line · Masami Hiramatsu
- Safety check
[...]
Quoted 4 lines
Next, Kprobes decodes whole body of probed function and checks there is
NO indirect jump, and near jump which jumps into the optimized region (except the 1st byte of jump), because if some jump instruction jumps into the middle of another instruction, it causes unexpected results too.Hi Masami,
Quoted 8 lines
I think your safety check algorithm is wrong.
There are several ways in which the kernel might jump into the optimized region that cannot be detected by examining just the probed function:
(1) The compiler is allowed to do cross-function optimization within a compilation unit where code in one function jumps into the middle of another function.
Hmm, right. This is a real problem. I think I have 2 options for this issue. I counted the cross-function jumps on my kernel roughly, and I found there were about 856 jumps jump into 158 functions (it might be incorrect, I just used 'objdump -d', and it also disassembles data section...)
- Making a blacklist of target functions or addresses which cross-function jumps jump into. This will be done by disassembling kernel when starting up kernel and loading modules.(or, at build-time)
- Just disables cross-function optimization by adding --param min-crossjump-insns=XXXX where XXXX is enough big number, when CONFIG_OPTPROBES=y
Quoted 20 lines
(2) If you have a switch statement that looks like:
switch (foo) { case 1: printk("a1"); break; case 2: printk("a2"); break; case 3: printk("a3"); break; case 4: printk("a4"); break; case 5: printk("a5"); break; }
Quoted 8 lines
(i.e. a large number of cases indexed by a small range of integers; depending on your compiler you may need more cases), gcc will implement it using a jump table in the .rodata section. On
x86_32, the generated assembly for the switch will read from the jump table at an offset of 4 * (foo - 1) and then jump to that address to reach the code for the appropriate case. These jump tables can result in jumps from within the probed function into the optimized region in a way that your algorithm would not detect.
These jumps are uses indirect jumps. As I wrote in the document, Kprobes doesn't optimize a probe which probes the function including indirect jump instructions.
Quoted 3 lines
(3) If the code that you're overwriting can throw an exception, it might be that that there is a jump from the .fixup section back into the probed function that overlaps the optimized region.
Sorry, I forgot to describe this case on the document. When the addresses of optimized instructions are listed on .fixup section, kprobes also doesn't optimize it.
Quoted 8 lines
The other comment I have about this approach is that it seems you've written a completely new x86 disassembler in order to do binary code analysis in the kernel.
Ksplice has been using the udis86 disassembler for this purpose:
<http://udis86.sourceforge.net/>. udis86 is intended for binary code analysis and is designed to be embedded into kernels and other applications.
I know, when I asked about udis86, I heard that ksplice just uses it for supporting old kernels and new kernel implementation doesn't need it. So, I made a new decoder.
Quoted 7 lines
udis86 generates all its instruction table data from an XML opcode file, which is I think what H. Peter Anvin was suggesting you should do in this previous thread on your instruction decoder:
<http://lkml.indiana.edu/hypermail/linux/kernel/0904.0/01929.html>
Compared to e.g. libopcodes it is still quite small -- there's a total of about 3000 lines of C, plus some instruction tables that are automatically generated from an XML description of the instructions.
I'm not so sure about udis86. Can I use it in exception path (and kprobes)? Is that XML things enough easy to be maintained?
If so, and the maintainer of udis86 is eager to push it in the kernel and he will maintain his code etc., it might be acceptable, aah, after rewriting it according to linux c style:)
Quoted 7 lines
The upstream developer has merged patches from Anders Kaseorg and myself that make it build as part of the core Linux kernel without any changes other than adding a kernel
Makefile. I think if we're going to putting an x86 disassembler into the kernel, it might be better to use something like udis86 that provides a little more information about the instructions being disassembled and is more data-driven.
Thank you,
Quoted 2 lines
-Tim Abbott
Signature · 7 lines
Masami Hiramatsu Software Engineer Hitachi Computer Products (America), Inc. Software Solutions Division e-mail: [email protected]
Quoted 10 lines · Masami Hiramatsu
Quoted 7 lines
udis86 generates all its instruction table data from an XML opcode file, which is I think what H. Peter Anvin was suggesting you should do in this previous thread on your instruction decoder:
<http://lkml.indiana.edu/hypermail/linux/kernel/0904.0/01929.html>
Compared to e.g. libopcodes it is still quite small -- there's a total of about 3000 lines of C, plus some instruction tables that are automatically generated from an XML description of the instructions.I'm not so sure about udis86. Can I use it in exception path (and kprobes)? Is that XML things enough easy to be maintained?
And one big question is that who needs full featured disassembler in kernel. It seems that kprobes and other potential user only need an instruction decoder (and an instruction emulator too.)
Thank you,
Signature · 7 lines
Masami Hiramatsu Software Engineer Hitachi Computer Products (America), Inc. Software Solutions Division e-mail: [email protected]
Quoted 9 lines · Masami Hiramatsu
Quoted 3 lines
(1) The compiler is allowed to do cross-function optimization within a compilation unit where code in one function jumps into the middle of another function.
Hmm, right. This is a real problem. I think I have 2 options for this issue. I counted the cross-function jumps on my kernel roughly, and I found there were about 856 jumps jump into 158 functions (it might be incorrect, I just used 'objdump -d', and it also disassembles data section...)
Hmm, almost all of these functions are jumped from .fixup section. Obviously, these should be mentioned.
Quoted 3 lines
- Making a blacklist of target functions or addresses which cross-function jumps jump into. This will be done by disassembling kernel when starting up kernel and loading modules.(or, at build-time)
If we can make this blacklist, it will include above .fixup section.
Quoted 4 lines
- Just disables cross-function optimization by adding --param min-crossjump-insns=XXXX where XXXX is enough big number, when
CONFIG_OPTPROBES=y
BTW, I've tried to compile with min-crossjump-insns=32768. It actually increased vmlinux size (about 1%),
text data bss dec hex filename
5767574 2022487 1521264 9311325 8e145d vmlinux-crossjump 5809551 2022487 1521264 9353302 8eb856 vmlinux-noncrossjump
However, with crossjumps opt, it seems not to add any jumps which jump into the middle of other functions...
Thanks,
Signature · 7 lines
Masami Hiramatsu Software Engineer Hitachi Computer Products (America), Inc. Software Solutions Division e-mail: [email protected]