Full syscall argument decode in "perf trace"
Subsystems (11)
Em Tue, Sep 17, 2013 at 05:10:55PM +0200, Denys Vlasenko escreveu:
Quoted 7 lines
I'm trying to figure out how to extend "perf trace".
Currently, it shows syscall names and arguments, and only them. Meaning that syscalls such as open(2) are shown as:
open(filename: 140736118412184, flags: 0, mode: 140736118403776) = 3
Quoted 7 lines
The problem is, of course, that user wants to see the filename per se, not the address of its first byte.
To improve that, we need to fetch the pointed-to data.
There are two approaches to this: extending "raw_syscalls:sys_{enter,exit}" tracepoint so that it returns this data, or selectively stopping the traced process when it reaches the thacepoint.
We don't want to stop the process at all, this is one of the major advantages of 'perf trace' over 'strace'.
Look at the tmp.perf/trace2 branch in my git repo, tglx and Ingo added a tracepoint to vfs_getname to use that.
Quoted 5 lines
First solution is attractive performance-wise, but requires a lot of new code: *ALL* syscalls will need to know which arguments are pointers, how large their pointed-to data structures are, and (remember readv and friends!) some of pointed-to structures themselves contain pointers which reference even more data.
Well, we can look at DWARF to get the function signatures, types, librarize 'perf probe' and insert probes in the syscalls we want decoding.
That for the cases where we don't have a tracepoint or when adding a new tracepoint is not an option.
And this all with what we have in the kernel right now.
Also for 'perf trace' look at my perf/core branch, where we have more syscall arg beautifiers and the machinery that is getting in place to allow that.
Longer term we could have something like dtrace's CTF to have a more compact type only ELF section that always go with the kernel, like we have CFI in binaries these days.
- Arnaldo
----- End forwarded message -----
Quoted 11 lines · Arnaldo Carvalho de Melo
Em Tue, Sep 17, 2013 at 05:10:55PM +0200, Denys Vlasenko escreveu:
Quoted 9 lines
I'm trying to figure out how to extend "perf trace".
Currently, it shows syscall names and arguments, and only them. Meaning that syscalls such as open(2) are shown as:
open(filename: 140736118412184, flags: 0, mode: 140736118403776) = 3The problem is, of course, that user wants to see the filename per se, not the address of its first byte.
Quoted 7 lines
Quoted 4 lines
To improve that, we need to fetch the pointed-to data.
There are two approaches to this: extending "raw_syscalls:sys_{enter,exit}" tracepoint so that it returns this data, or selectively stopping the traced process when it reaches the thacepoint.We don't want to stop the process at all, this is one of the major advantages of 'perf trace' over 'strace'.
This is a worthy goal. strace is so slow exactly because it stops traced process so often. strace developers do want to avoid as many of these stops as possible.
I'm not sure that "not stopping ever" is achievable, though. There are cases where stopping is necessary.
For example, after clone() call, depending on the tracer needs, there may be operations which must be done on the new child before it is allowed to run.
strace used to use hideous, unsafe workarounds to catch children, until ptrace was augmented with features which made children stop immediately.
Do you think you can work around that? I just don't see how.
Quoted 2 lines
Look at the tmp.perf/trace2 branch in my git repo, tglx and Ingo added a tracepoint to
vfs_getnameto use that.
I know that this is the way how to fetch syscall args without stopping, yes.
The problem: ~100 more tracepoints need to be added merely to get to the point where strace already is, wrt quality of syscall decoding. strace has nearly 300 separate custom syscall formatting functions, some of them quite complex.
If we need to add syscall stopping feature (which, as I said above, will be necessary anyway IMO), then syscall decoding can be as good as strace *already*. Then, gradually more tracepoints are added to make it faster.
I am thinking about going into this direction.
Therefore my question should be restated as:
Would perf developers accept the "syscall pausing" feature, or it won't be accepted?
Signature · 1 line
vda
Quoted 4 lines · Denys Vlasenko
Therefore my question should be restated as:
Would perf developers accept the "syscall pausing" feature, or it won't be accepted?
I have been using perf-trace a lot lately specifically because it is effectively a 'passive' observer of the task (e.g., time-sensitive tasks can be traced with perf but not with strace).
Also, your solution would not work if the raw_syscall events are written to a file for later analysis where as using tracepoints to collect this information would.
David
Quoted 9 lines · David Ahern
Quoted 4 lines · Denys Vlasenko
Therefore my question should be restated as:
Would perf developers accept the "syscall pausing" feature, or it won't be accepted?
I have been using perf-trace a lot lately specifically because it is effectively a 'passive' observer of the task (e.g., time-sensitive tasks can be traced with perf but not with strace).
Yes, this is not just an important but a primary design goal for all perf
utilities: if the tracing buffers used are large enough then it should be a nearly zero-overhead observer that preserves all previous patterns of behavior as much as possible.
Thanks,
Ingo
Em Wed, Sep 18, 2013 at 01:35:13PM +0200, Denys Vlasenko escreveu:
Quoted 20 lines · Arnaldo Carvalho de Melo
Quoted 11 lines
Em Tue, Sep 17, 2013 at 05:10:55PM +0200, Denys Vlasenko escreveu:
Quoted 9 lines
I'm trying to figure out how to extend "perf trace".
Currently, it shows syscall names and arguments, and only them. Meaning that syscalls such as open(2) are shown as:
open(filename: 140736118412184, flags: 0, mode: 140736118403776) = 3The problem is, of course, that user wants to see the filename per se, not the address of its first byte.
Quoted 7 lines
Quoted 4 lines
To improve that, we need to fetch the pointed-to data.
There are two approaches to this: extending "raw_syscalls:sys_{enter,exit}" tracepoint so that it returns this data, or selectively stopping the traced process when it reaches the thacepoint.We don't want to stop the process at all, this is one of the major advantages of 'perf trace' over 'strace'.
Quoted 6 lines
This is a worthy goal. strace is so slow exactly because it stops traced process so often. strace developers do want to avoid as many of these stops as possible.
I'm not sure that "not stopping ever" is achievable, though. There are cases where stopping is necessary.
Can't we try first to achieve what is possible with existing infrastructure so that we can, out of the combo 'perf trace' and 'strace' have something that is better than plain 'strace'?
Quoted 9 lines
For example, after
clone()call, depending on the tracer needs, there may be operations which must be done on the new child before it is allowed to run.strace used to use hideous, unsafe workarounds to catch children, until ptrace was augmented with features which made children stop immediately.
Do you think you can work around that? I just don't see how.
I haven't even thought about it 8-)
Quoted 16 lines
Quoted 2 lines
Look at the tmp.perf/trace2 branch in my git repo, tglx and Ingo added a tracepoint to
vfs_getnameto use that.I know that this is the way how to fetch syscall args without stopping, yes.
The problem: ~100 more tracepoints need to be added merely to get to the point where strace already is, wrt quality of syscall decoding. strace has nearly 300 separate custom syscall formatting functions, some of them quite complex.
If we need to add syscall stopping feature (which, as I said above, will be necessary anyway IMO), then syscall decoding can be as good as strace *already*. Then, gradually more tracepoints are added to make it faster.
Quoted 6 lines
I am thinking about going into this direction.
Therefore my question should be restated as:
Would perf developers accept the "syscall pausing" feature, or it won't be accepted?
Do you have some patch for us to try?
- Arnaldo
Quoted 16 lines · Arnaldo Carvalho de Melo
Quoted 16 lines
Quoted 2 lines
Look at the tmp.perf/trace2 branch in my git repo, tglx and Ingo added a tracepoint to
vfs_getnameto use that.I know that this is the way how to fetch syscall args without stopping, yes.
The problem: ~100 more tracepoints need to be added merely to get to the point where strace already is, wrt quality of syscall decoding. strace has nearly 300 separate custom syscall formatting functions, some of them quite complex.
If we need to add syscall stopping feature (which, as I said above, will be necessary anyway IMO), then syscall decoding can be as good as strace *already*. Then, gradually more tracepoints are added to make it faster.
Quoted 8 lines
Quoted 6 lines
I am thinking about going into this direction.
Therefore my question should be restated as:
Would perf developers accept the "syscall pausing" feature, or it won't be accepted?
Do you have some patch for us to try?
I have a patch which is a bit strace specific: it sidesteps the question of the synchronization between traced process and its tracer by using ptrace's existing method of reporting stops.
This works for strace, and is very easy to implement. Naturally, other tracers (e.g. "perf trace" wouldn't want to start using ptrace! Synchronization needs to be done in some other way, not as a ptrace stop.
For one, the stopping flag needs to be a counter, so that more than one tracer can use this feature concurrently.
But anyway, I am attaching it.
It adds a new flag, attr.sysexit_stop, which makes process stop at next syscall exit when this tracepoint overflows.
Signature · 1 line
vda
include/uapi/linux/perf_event.h +3 -1
@@ -273,7 +273,9 @@ struct perf_event_attr { exclude_callchain_kernel : 1, /* exclude kernel callchains */ exclude_callchain_user : 1, /* exclude user callchains */ - __reserved_1 : 41; + sysexit_stop : 1, + + __reserved_1 : 40;union {__u32 wakeup_events;/* wakeup every n events */
kernel/events/core.c +4 -0
@@ -5026,6 +5026,10 @@ static int __perf_event_overflow(structirq_work_queue (&event->pending); } +if (!in_interrupt () && event->attr.sysexit_stop && current->ptrace) { +set_tsk_thread_flag (current,TIF_SYSCALL_TRACE ); + } +return ret; }
Quoted 30 lines · Denys Vlasenko
Quoted 10 lines · Arnaldo Carvalho de Melo
Quoted 10 lines
The problem: ~100 more tracepoints need to be added merely to get to the point where strace already is, wrt quality of syscall decoding. strace has nearly 300 separate custom syscall formatting functions, some of them quite complex.
If we need to add syscall stopping feature (which, as I said above, will be necessary anyway IMO), then syscall decoding can be as good as strace *already*. Then, gradually more tracepoints are added to make it faster.
Quoted 8 lines
Quoted 6 lines
I am thinking about going into this direction.
Therefore my question should be restated as:
Would perf developers accept the "syscall pausing" feature, or it won't be accepted?
Do you have some patch for us to try?
I have a patch which is a bit strace specific: it sidesteps the question of the synchronization between traced process and its tracer by using ptrace's existing method of reporting stops.
This works for strace, and is very easy to implement. Naturally, other tracers (e.g. "perf trace" wouldn't want to start using ptrace! Synchronization needs to be done in some other way, not as a ptrace stop.
Quoted 7 lines
For one, the stopping flag needs to be a counter, so that more than one tracer can use this feature concurrently.
But anyway, I am attaching it.
It adds a new flag, attr.sysexit_stop, which makes process stop at next syscall exit when this tracepoint overflows.
Here is the next iteration of the work in progress.
I added syscall masks.
This necessitated propagation of pointer to struct pt_regs which points to userspace registers from sys_{enter,exit} tracepoints to overflow handling functions, in order to get syscall#. (Yes, I discovered that pt_regs which was already there wasn't the *userspace* one).
The patch is tested: I have a modified version of strace which decodes all syscalls properly and which avoids stopping on all syscall entries and on a selected few syscall exits too.
As I see it, the next thing to tackle is the stopping method. (The current patch still uses my old ptrace-specific hack).
How about the following: add a per-task "pause counter". If it is <= 0, then task is not paused. If it is > 0, task is paused.
When an attached perf fd causes task to pause, the counter is incremented, a marker is written into the perf buffer, and task goes to sleep.
When tracer process sees the marker, it commands traced process to "unpause", which decrements the counter.
Why this way?
* this allows traced process to be paused by several tracers at once.
* this does not need heavy-weight notifications to be sent to tracers (unlike my current hack, which invokes the waitpid notification machinery, the source of much of strace's slowness).
* it might work even if counter increment is reordered relative to perf marker writing. if tracer sees the marker, it can "unpause" - decrement counter and cause it to go -1. The task is not paused (the rule is "<= 0", not "= 0"). Then kernel increments the counter, it's 0 now,
and task is still not paused. (I'm not sure whether such property is useful, but if it is, we have it - good :)
The downside is, we'd need one new field in task struct.
Does this look sensible to you?
arch/alpha/kernel/perf_event.c +1 -1
@@ -850,7 +850,7 @@ static void alpha_perf_event_irq_handlerperf_sample_data_init (&data,0 , hwc->last_period);if (alpha_perf_event_set_period (event, hwc, idx)) { -if (perf_event_overflow (event, &data, regs)) { +if (perf_event_overflow (event, &data, regs,NULL )) {/* Interrupts coming too quickly; "throttle" the * counter, i.e., disable it for a little while. */
arch/arm/kernel/perf_event_v6.c +1 -1
@@ -514,7 +514,7 @@ armv6pmu_handle_irq(int irq_num,if (!armpmu_event_set_period (event))continue ; -if (perf_event_overflow (event, &data, regs)) +if (perf_event_overflow (event, &data, regs,NULL )) cpu_pmu->disable (event); }
arch/arm/kernel/perf_event_v7.c +1 -1
@@ -1074,7 +1074,7 @@ static irqreturn_t armv7pmu_handle_irq(iif (!armpmu_event_set_period (event))continue ; -if (perf_event_overflow (event, &data, regs)) +if (perf_event_overflow (event, &data, regs,NULL )) cpu_pmu->disable (event); }
arch/arm/kernel/perf_event_xscale.c +2 -2
@@ -265,7 +265,7 @@ xscale1pmu_handle_irq(int irq_num, voidif (!armpmu_event_set_period (event))continue ; -if (perf_event_overflow (event, &data, regs)) +if (perf_event_overflow (event, &data, regs,NULL )) cpu_pmu->disable (event); }
@@ -606,7 +606,7 @@ xscale2pmu_handle_irq(int irq_num, voidif (!armpmu_event_set_period (event))continue ; -if (perf_event_overflow (event, &data, regs)) +if (perf_event_overflow (event, &data, regs,NULL )) cpu_pmu->disable (event); }
arch/arm64/kernel/perf_event.c +1 -1
@@ -1063,7 +1063,7 @@ static irqreturn_t armv8pmu_handle_irq(iif (!armpmu_event_set_period (event, hwc, idx))continue ; -if (perf_event_overflow (event, &data, regs)) +if (perf_event_overflow (event, &data, regs,NULL )) cpu_pmu->disable (hwc, idx); }
arch/metag/kernel/perf/perf_event.c +1 -1
@@ -789,7 +789,7 @@ static irqreturn_t metag_pmu_counter_ove* completed. Note the counter value may have been modified while it was * inactive to set it up ready for the next interrupt. */ -if (!perf_event_overflow (event, &sampledata, regs)) { +if (!perf_event_overflow (event, &sampledata, regs,NULL )) {__global_lock2 (flags); counter = (counter &0xff000000 ) | (metag_in32 (PERF_COUNT (idx)) &0x00ffffff );
arch/mips/kernel/perf_event_mipsxx.c +1 -1
@@ -746,7 +746,7 @@ static void handle_associated_event(struif (!mipspmu_event_set_period (event, hwc, idx))return ; -if (perf_event_overflow (event, data, regs)) +if (perf_event_overflow (event, data, regs,NULL ))mipsxx_pmu_disable_event (idx); }
arch/powerpc/perf/core-book3s.c +1 -1
@@ -1639,7 +1639,7 @@ static void record_and_restart(struct pe data.br_stack = &cpuhw->bhrb_stack; } -if (perf_event_overflow (event, &data, regs)) +if (perf_event_overflow (event, &data, regs,NULL ))power_pmu_stop (event,0 ); } }
arch/powerpc/perf/core-fsl-emb.c +1 -1
@@ -615,7 +615,7 @@ static void record_and_restart(struct peperf_sample_data_init (&data,0 , event->hw.last_period); -if (perf_event_overflow (event, &data, regs)) +if (perf_event_overflow (event, &data, regs,NULL ))fsl_emb_pmu_stop (event,0 ); } }
arch/sparc/kernel/perf_event.c +1 -1
@@ -1633,7 +1633,7 @@ static int __kprobes perf_event_nmi_handif (!sparc_perf_event_set_period (event, hwc, idx))continue ; -if (perf_event_overflow (event, &data, regs)) +if (perf_event_overflow (event, &data, regs,NULL ))sparc_pmu_stop (event,0 ); }
arch/x86/kernel/cpu/perf_event_amd_ibs.c +1 -1
@@ -580,7 +580,7 @@ static int perf_ibs_handle_irq(struct pe data.raw = &raw; } - throttle =perf_event_overflow (event, &data, ®s); + throttle =perf_event_overflow (event, &data, ®s,NULL ); out:if (throttle)perf_ibs_disable_event (perf_ibs, hwc, *config);
arch/x86/kernel/cpu/perf_event.c +1 -1
@@ -1225,7 +1225,7 @@ int x86_pmu_handle_irq(struct pt_regs *rif (!x86_perf_event_set_period (event))continue ; -if (perf_event_overflow (event, &data, regs)) +if (perf_event_overflow (event, &data, regs,NULL ))x86_pmu_stop (event,0 ); }
arch/x86/kernel/cpu/perf_event_intel.c +1 -1
@@ -1222,7 +1222,7 @@ again:if (has_branch_stack (event)) data.br_stack = &cpuc->lbr_stack; -if (perf_event_overflow (event, &data, regs)) +if (perf_event_overflow (event, &data, regs,NULL ))x86_pmu_stop (event,0 ); }
arch/x86/kernel/cpu/perf_event_intel_ds.c +1 -1
@@ -761,7 +761,7 @@ static void __intel_pmu_pebs_event(strucif (has_branch_stack (event)) data.br_stack = &cpuc->lbr_stack; -if (perf_event_overflow (event, &data, ®s)) +if (perf_event_overflow (event, &data, ®s,NULL ))x86_pmu_stop (event,0 ); }
arch/x86/kernel/cpu/perf_event_knc.c +1 -1
@@ -251,7 +251,7 @@ again:perf_sample_data_init (&data,0 , event->hw.last_period); -if (perf_event_overflow (event, &data, regs)) +if (perf_event_overflow (event, &data, regs,NULL ))x86_pmu_stop (event,0 ); }
arch/x86/kernel/cpu/perf_event_p4.c +1 -1
@@ -1037,7 +1037,7 @@ static int p4_pmu_handle_irq(struct pt_rcontinue ; -if (perf_event_overflow (event, &data, regs)) +if (perf_event_overflow (event, &data, regs,NULL ))x86_pmu_stop (event,0 ); }
include/linux/ftrace_event.h +3 -3
@@ -376,10 +376,10 @@ extern void *perf_trace_buf_prepare(intstatic inline void perf_trace_buf_submit (void *raw_data,int size,int rctx,u64 addr, -u64 count,struct pt_regs *regs,void *head, -struct task_struct *task) +u64 count,struct pt_regs *regs,struct pt_regs *user_regs, +void *head,struct task_struct *task) { -perf_tp_event (addr, count, raw_data, size, regs, head, rctx, task); +perf_tp_event (addr, count, raw_data, size, regs, user_regs, head, rctx, task); }#endif
include/linux/perf_event.h +3 -2
@@ -602,7 +602,8 @@ extern void perf_prepare_sample(struct pextern int perf_event_overflow (struct perf_event *event,struct perf_sample_data *data, -struct pt_regs *regs); +struct pt_regs *regs, +struct pt_regs *user_regs);static inline bool is_sampling_event (struct perf_event *event) {
@@ -717,7 +718,7 @@ static inline bool perf_paranoid_kernel(extern void perf_event_init (void );extern void perf_tp_event (u64 addr,u64 count,void *record, -int entry_size,struct pt_regs *regs, +int entry_size,struct pt_regs *regs,struct pt_regs *user_regs,struct hlist_head *head,int rctx,struct task_struct *task);extern void perf_bp_event (struct perf_event *event,void *data);
include/trace/events/syscalls.h +2 -0
@@ -30,6 +30,7 @@ TRACE_EVENT_FN(sys_enter,TP_fast_assign ( __entry->id = id;syscall_get_arguments (current, regs,0 ,6 , __entry->args); + user_regs = regs; ), TP_printk("NR %ld (%lx, %lx, %lx, %lx, %lx, %lx)",
@@ -56,6 +57,7 @@ TRACE_EVENT_FN(sys_exit,TP_fast_assign ( __entry->id = syscall_get_nr(current, regs); __entry->ret = ret; + user_regs = regs; ), TP_printk("NR %ld = %ld",
include/trace/ftrace.h +5 -1
@@ -519,6 +519,8 @@ ftrace_raw_event_##call(void *__data, pr struct ftrace_raw_##call *entry; \struct ring_buffer *buffer; \unsigned long irq_flags; \ +/* dummy. "assign" macro param might need it to exist: */ \ + struct pt_regs __maybe_unused *user_regs; \int __data_size; \int pc; \ \
@@ -652,6 +654,8 @@ perf_trace_##call(void *__data, proto) struct ftrace_data_offsets_##call __maybe_unused __data_offsets;\ struct ftrace_raw_##call *entry; \ struct pt_regs __regs; \ + /* "assign" macro parameter might overwrite it: */ \ +struct pt_regs *user_regs =NULL ; \u64 __addr =0 , __count =1 ; \struct task_struct *__task =NULL ; \struct hlist_head *head; \
@@ -681,7 +685,7 @@ perf_trace_##call(void *__data, proto) \ head =this_cpu_ptr (event_call->perf_events); \perf_trace_buf_submit (entry, __entry_size, rctx, __addr, \ - __count, &__regs, head, __task); \ + __count, &__regs, user_regs, head, __task); \ }/*
include/uapi/linux/perf_event.h +13 -1
@@ -273,7 +273,10 @@ struct perf_event_attr { exclude_callchain_kernel : 1, /* exclude kernel callchains */ exclude_callchain_user : 1, /* exclude user callchains */ - __reserved_1 : 41; + sysenter_stop : 1, + sysexit_stop : 1, + + __reserved_1 : 39;union {__u32 wakeup_events;/* wakeup every n events */
@@ -304,6 +307,15 @@ struct perf_event_attr {/* Align to u64. */ __u32 __reserved_2; + +/* +* If sys{enter,exit}_stop should ignore some syscalls, +* these bitmasks specify which to ignore. Otherwise set to 0/NULL. +*/ +unsigned sysenter_mask_len; +unsigned sysexit_mask_len; +unsigned long *sysenter_mask_ptr; +unsigned long *sysexit_mask_ptr; };#define perf_flags (attr) (*(&(attr)->read_format + 1))
kernel/events/core.c +71 -17
204 lines not on this page · raw message
kernel/trace/trace_event_perf.c +1 -1
@@ -282,7 +282,7 @@ perf_ftrace_function_call(unsigned long head =this_cpu_ptr (event_function.perf_events);perf_trace_buf_submit (entry,ENTRY_SIZE , rctx,0 , -1 , ®s, head,NULL ); +1 , ®s,NULL , head,NULL );#undef ENTRY_SIZE }
kernel/trace/trace_kprobe.c +2 -2
@@ -1193,7 +1193,7 @@ kprobe_perf_func(struct trace_probe *tp, head =this_cpu_ptr (call->perf_events);perf_trace_buf_submit (entry, size, rctx, - entry->ip,1 , regs, head,NULL ); + entry->ip,1 , regs,NULL , head,NULL ); }/* Kretprobe profile handler */
@@ -1225,7 +1225,7 @@ kretprobe_perf_func(struct trace_probe * head =this_cpu_ptr (call->perf_events);perf_trace_buf_submit (entry, size, rctx, - entry->ret_ip, 1, regs, head, NULL); + entry->ret_ip, 1, regs, NULL, head, NULL); }#endif /* CONFIG_PERF_EVENTS */
kernel/trace/trace_syscalls.c +2 -2
@@ -585,7 +585,7 @@ static void perf_syscall_enter(void *ign (unsigned long *)&rec->args); head =this_cpu_ptr (sys_data->enter_event->perf_events); -perf_trace_buf_submit (rec, size, rctx,0 ,1 , regs, head,NULL ); +perf_trace_buf_submit (rec, size, rctx,0 ,1 , regs, regs, head,NULL ); }static int perf_sysenter_enable (struct ftrace_event_call *call)
@@ -663,7 +663,7 @@ static void perf_syscall_exit(void *igno rec->ret =syscall_get_return_value (current, regs); head =this_cpu_ptr (sys_data->exit_event->perf_events); -perf_trace_buf_submit (rec, size, rctx,0 ,1 , regs, head,NULL ); +perf_trace_buf_submit (rec, size, rctx,0 ,1 , regs, regs, head,NULL ); }static int perf_sysexit_enable (struct ftrace_event_call *call)
kernel/trace/trace_uprobe.c +1 -1
@@ -862,7 +862,7 @@ static void uprobe_perf_print(struct trafor (i =0 ; i < tu->nr_args; i++)call_fetch (&tu->args[i].fetch, regs, data + tu->args[i].offset); -perf_trace_buf_submit (entry, size, rctx,0 ,1 , regs, head,NULL ); +perf_trace_buf_submit (entry, size, rctx,0 ,1 , regs,NULL , head,NULL ); out:preempt_enable (); }