perf/x86: Add perf text poke event
Subsystems (2)
Adrian Hunter
reviewercover perf/x86: Add perf text poke event
Hi
Here are patches to add a text poke event to record changes to kernel text (i.e. self-modifying code) in order to support tracers like Intel PT decoding through jump labels.
One question is whether this approach can be of use to other architectures, particularly ARM-CS.
The first patch makes the kernel change and the subsequent patches are tools changes.
The first 3 tools patches add support for updating perf tools' data cache with the changed bytes.
The last 2 patches are Intel PT specific tools changes.
The patches are based on Arnaldo's tree perf/core branch.
Example:
For jump labels, the kernel needs
CONFIG_JUMP_LABEL=y
and also an easy to flip jump label is in sysctl_schedstats() which needs
CONFIG_SCHEDSTATS=y
CONFIG_PROC_SYSCTL=y
Also note the 'sudo perf record' is put into the background which, as written, needs sudo credential caching (otherwise the background task will stop awaiting the sudo password), hence the 'sudo echo' to start.
Before:
$ sudo echo $ sudo perf record -o perf.data.before --kcore -a -e intel_pt//k -m,64M & [1] 1640 $ cat /proc/sys/kernel/sched_schedstats 0 $ sudo bash -c 'echo 1 > /proc/sys/kernel/sched_schedstats' $ cat /proc/sys/kernel/sched_schedstats 1 $ sudo bash -c 'echo 0 > /proc/sys/kernel/sched_schedstats' $ cat /proc/sys/kernel/sched_schedstats 0 $ sudo kill 1640 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 16.635 MB perf.data.before ] $ perf script -i perf.data.before --itrace=e >/dev/null Warning: 1946 instruction trace errors
After:
$ sudo echo $ sudo perf record -o perf.data.after --kcore -a -e intel_pt//k -m,64M & [1] 1882 $ cat /proc/sys/kernel/sched_schedstats 0 $ sudo bash -c 'echo 1 > /proc/sys/kernel/sched_schedstats' $ cat /proc/sys/kernel/sched_schedstats 1 $ sudo bash -c 'echo 0 > /proc/sys/kernel/sched_schedstats' $ cat /proc/sys/kernel/sched_schedstats 0 $ sudo kill 1882 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 10.893 MB perf.data.after ] $ perf script -i perf.data.after --itrace=e $
Adrian Hunter (6):
perf/x86: Add perf text poke event
perf dso: Refactor dso_cache__read() perf dso: Add dso__data_write_cache_addr() perf tools: Add support for PERF_RECORD_TEXT_POKE perf auxtrace: Add auxtrace_cache__remove() perf intel-pt: Add support for text poke events
arch/x86/include/asm/text-patching.h | 1 + arch/x86/kernel/alternative.c | 39 ++++++++- include/linux/perf_event.h | 6 ++ include/uapi/linux/perf_event.h | 28 ++++++- kernel/events/core.c | 90 +++++++++++++++++++- tools/include/uapi/linux/perf_event.h | 28 ++++++- tools/perf/arch/x86/util/intel-pt.c | 5 ++ tools/perf/builtin-record.c | 45 ++++++++++ tools/perf/lib/include/perf/event.h | 9 ++ tools/perf/util/auxtrace.c | 28 +++++++ tools/perf/util/auxtrace.h | 1 + tools/perf/util/dso.c | 135 +++++++++++++++++++++--------- tools/perf/util/dso.h | 7 ++ tools/perf/util/event.c | 39 +++++++++ tools/perf/util/event.h | 5 ++ tools/perf/util/evlist.h | 1 + tools/perf/util/intel-pt.c | 71 ++++++++++++++++ tools/perf/util/machine.c | 38 +++++++++ tools/perf/util/machine.h | 3 + tools/perf/util/perf_event_attr_fprintf.c | 1 + tools/perf/util/record.c | 10 +++ tools/perf/util/record.h | 1 + tools/perf/util/session.c | 22 +++++ tools/perf/util/tool.h | 3 +- 24 files changed, 570 insertions(+), 46 deletions(-)
Regards
Adrian
Adrian Hunter
reviewerRFC PATCH 1/6 perf/x86: Add perf text poke event
Record changes to kernel text (i.e. self-modifying code) in order to support tracers like Intel PT decoding through jump labels.
A copy of the running kernel code is needed as a reference point (e.g. from /proc/kcore). The text poke event records the old bytes and the new bytes so that the event can be processed forwards or backwards.
The text poke event has 'flags' to describe when the event is sent. At present the only flag is PERF_TEXT_POKE_UPDATE which indicates the point at which tools should update their reference kernel executable to change the old bytes to the new bytes.
Other architectures may wish to emit a pair of text poke events. One before and one after a text poke. In that case, PERF_TEXT_POKE_UPDATE flag would be set on only one of the pair.
In the case of Intel PT tracing, the executable code must be walked to reconstruct the control flow. For x86 a jump label text poke begins:
- write INT3 byte
- IPI-SYNC
At this point the actual control flow will be through the INT3 and handler and not hit the old or new instruction. Intel PT outputs FUP/TIP packets for the INT3, so the flow can still be decoded. Subsequently:
- emit RECORD_TEXT_POKE with the new instruction
- write instruction tail
- IPI-SYNC
- write first byte
- IPI-SYNC
So before the text poke event timestamp, the decoder will see either the old instruction flow or FUP/TIP of INT3. After the text poke event timestamp, the decoder will see either the new instruction flow or FUP/TIP of INT3. Thus decoders can use the timestamp as the point at which to modify the executable code.
--- arch/x86/include/asm/text-patching.h | 1 + arch/x86/kernel/alternative.c | 39 +++++++++++- include/linux/perf_event.h | 6 ++ include/uapi/linux/perf_event.h | 28 ++++++++- kernel/events/core.c | 90 +++++++++++++++++++++++++++- 5 files changed, 161 insertions(+), 3 deletions(-)
arch/x86/include/asm/text-patching.h +1 -0
@@ -30,6 +30,7 @@ struct text_poke_loc {void *addr;size_t len;const char opcode[POKE_MAX_OPCODE_SIZE ]; +u8 old; };extern void text_poke_early (void *addr,const void *opcode,size_t len);
arch/x86/kernel/alternative.c +38 -1
@@ -3,6 +3,7 @@#include <linux/module.h> #include <linux/sched.h> +#include <linux/perf_event.h> #include <linux/mutex.h> #include <linux/list.h> #include <linux/stringify.h>
@@ -1045,8 +1046,10 @@ void text_poke_bp_batch(struct text_poke_loc *tp, unsigned int nr_entries)/* * First step: add a int3 trap to the address that will be patched. */ -for (i =0 ; i < nr_entries; i++) +for (i =0 ; i < nr_entries; i++) { + tp[i].old = *(u8 *)tp[i].addr;text_poke (tp[i].addr, &int3,sizeof (int3)); + }on_each_cpu (do_sync_core,NULL ,1 );
@@ -1054,12 +1057,46 @@ void text_poke_bp_batch(struct text_poke_loc *tp, unsigned int nr_entries)* Second step: update all but the first byte of the patched range. */ for (i =0 ; i < nr_entries; i++) { +u8 old[POKE_MAX_OPCODE_SIZE ] = { tp[i].old, }; + +memcpy (old +sizeof (int3), + tp[i].addr +sizeof (int3), + tp[i].len -sizeof (int3)); +if (tp[i].len -sizeof (int3) >0 ) {text_poke ((char *)tp[i].addr +sizeof (int3), (const char *)tp[i].opcode +sizeof (int3), tp[i].len -sizeof (int3)); patched_all_but_first++; } + +/* +* Emit a perf event to record the text poke, primarily to +* support Intel PT decoding which must walk the executable code +* to reconstruct the trace. The flow up to here is: +* - write INT3 byte +* - IPI-SYNC +* At this point the actual control flow will be through the +* INT3 and handler and not hit the old or new instruction. +* Intel PT outputs FUP/TIP packets for the INT3, so the flow +* can still be decoded. Subsequently: +* - emit RECORD_TEXT_POKE with the new instruction +* - write instruction tail +* - IPI-SYNC +* - write first byte +* - IPI-SYNC +* So before the text poke event timestamp, the decoder will see +* either the old instruction flow or FUP/TIP of INT3. After the +* text poke event timestamp, the decoder will see either the +* new instruction flow or FUP/TIP of INT3. Thus decoders can +* use the timestamp as the point at which to modify the +* executable code. +* The old instruction is recorded so that the event can be +* processed forwards or backwards. +*/ +perf_event_text_poke ((unsigned long )tp[i].addr, +PERF_TEXT_POKE_UPDATE , old, tp[i].opcode, + tp[i].len); }if (patched_all_but_first) {
include/linux/perf_event.h +6 -0
@@ -1183,6 +1183,8 @@ extern void perf_event_exec(void);extern void perf_event_comm (struct task_struct *tsk,bool exec);extern void perf_event_namespaces (struct task_struct *tsk);extern void perf_event_fork (struct task_struct *tsk); +extern void perf_event_text_poke (unsigned long addr,u16 flags,const void *old_bytes, +const void *new_bytes,size_t len);/* Callchains */ DECLARE_PER_CPU(struct perf_callchain_entry, perf_callchain_entry);
@@ -1406,6 +1408,10 @@ static inline void perf_event_exec(void) { }static inline void perf_event_comm (struct task_struct *tsk,bool exec) { }static inline void perf_event_namespaces (struct task_struct *tsk) { }static inline void perf_event_fork (struct task_struct *tsk) { } +static inline void perf_event_text_poke (unsigned long addr,u16 flags, +const void *old_bytes, +const void *new_bytes, +size_t len) { }static inline void perf_event_init (void ) { }static inline int perf_swevent_get_recursion_context (void ) {return -1 ; }static inline void perf_swevent_put_recursion_context (int rctx) { }
include/uapi/linux/perf_event.h +27 -1
@@ -375,7 +375,8 @@ struct perf_event_attr { ksymbol : 1, /* include ksymbol events */ bpf_event : 1, /* include bpf events */ aux_output : 1, /* generate AUX records instead of events */ - __reserved_1 : 32; + text_poke : 1, /* include text poke events */ + __reserved_1 : 31;union {__u32 wakeup_events;/* wakeup every n events */
@@ -1000,6 +1001,26 @@ enum perf_event_type {*/ PERF_RECORD_BPF_EVENT =18 , +/* +* Records changes to kernel text i.e. self-modified code. +* 'flags' has PERF_TEXT_POKE_UPDATE (i.e. bit 0) set to +* indicate to tools to update old bytes to new bytes in the +* executable image. +* 'len' is the number of old bytes, which is the same as the number +* of new bytes. 'bytes' contains the old bytes followed immediately +* by the new bytes. +* +* struct { +* struct perf_event_header header; +* u64 addr; +* u16 flags; +* u16 len; +* u8 bytes[]; +* struct sample_id sample_id; +* }; +*/ +PERF_RECORD_TEXT_POKE =19 , + PERF_RECORD_MAX, /* non-ABI */ };
@@ -1041,6 +1062,11 @@ enum perf_callchain_context {#define PERF_AUX_FLAG_PARTIAL 0x04/* record contains gaps */ #define PERF_AUX_FLAG_COLLISION 0x08/* sample collided with another */ +/** +* PERF_RECORD_TEXT_POKE::flags bits +*/ +#define PERF_TEXT_POKE_UPDATE 0x01/* update old bytes to new bytes */ +#define PERF_FLAG_FD_NO_GROUP (1UL << 0)#define PERF_FLAG_FD_OUTPUT (1UL << 1)#define PERF_FLAG_PID_CGROUP (1UL << 2)/* pid=cgroup id, per-cpu mode only */
kernel/events/core.c +89 -1
120 lines not on this page · raw message
Signature · 1 line
2.17.1
Hi Adrian,
Quoted 7 lines · Adrian Hunter
Record changes to kernel text (i.e. self-modifying code) in order to support tracers like Intel PT decoding through jump labels.
A copy of the running kernel code is needed as a reference point (e.g. from
/proc/kcore). The text poke event records the old bytes and the new bytes so that the event can be processed forwards or backwards.
Quoted 8 lines
The text poke event has 'flags' to describe when the event is sent. At present the only flag is
PERF_TEXT_POKE_UPDATEwhich indicates the point at which tools should update their reference kernel executable to change the old bytes to the new bytes.Other architectures may wish to emit a pair of text poke events. One before and one after a text poke. In that case,
PERF_TEXT_POKE_UPDATEflag would be set on only one of the pair.
Thanks a lot for the patch set.
Below is my understanding for implementation 'emit a pair of text poke events' as you mentioned:
Since Arm64 instruction is fixed size, it doesn't need to rely on INT3 liked mechanism to replace instructions and directly uses two operations to alter instruction (modify instruction and flush icache line). So Arm64 has no chance to send perf event in the middle of altering instruction.
Thus we can send pair of text poke events in the kernel:
perf_event_text_poke(PERF_TEXT_POKE_UPDATE_PREV)
Change instruction
Flush icache
perf_event_text_poke(PERF_TEXT_POKE_UPDATE_POST)
In the userspace, if perf tool detects the instruction is changed from nop to branch, we need to update dso cache for the 'PERF_TEXT_POKE_UPDATE_PREV' event; if detect the instruction is changed from branch to nop, we need to update dso cache for 'PERF_TEXT_POKE_UPDATE_POST' event. The main idea is to ensure the branch instructions can be safely contained in the dso file and any branch samples can read out correct branch instruction.
Could you confirm this is the same with your understanding? Or I miss anything? I personally even think the pair events can be used for different arches (e.g. the solution can be reused on Arm64/x86, etc).
Quoted 8 lines
In the case of Intel PT tracing, the executable code must be walked to reconstruct the control flow. For x86 a jump label text poke begins:
- write INT3 byte
- IPI-SYNC
At this point the actual control flow will be through the INT3 and handler and not hit the old or new instruction. Intel PT outputs FUP/TIP packets for the INT3, so the flow can still be decoded. Subsequently:
- emitRECORD_TEXT_POKEwith the new instruction
s/RECORD_TEXT_POKE/PERF_TEXT_POKE_UPDATE
And I think emit the perf event is after the item 'write instruction tail' but not before it. So the commit log is not consistent with the change in the patch.
Quoted 7 lines
- write instruction tail
- IPI-SYNC
- write first byte
- IPI-SYNC
So before the text poke event timestamp, the decoder will see either the old instruction flow or FUP/TIP of INT3. After the text poke event timestamp, the decoder will see either the new instruction flow or FUP/TIP
I will check internally for timestamp on Arm/Arm64 ...
Thanks,
Leo Yan
Quoted 304 lines
of INT3. Thus decoders can use the timestamp as the point at which to modify the executable code.
Signed-off-by Adrian Hunter <[email protected]>--- arch/x86/include/asm/text-patching.h | 1 + arch/x86/kernel/alternative.c | 39 +++++++++++- include/linux/perf_event.h | 6 ++ include/uapi/linux/perf_event.h | 28 ++++++++- kernel/events/core.c | 90 +++++++++++++++++++++++++++- 5 files changed, 161 insertions(+), 3 deletions(-)arch/x86/include/asm/text-patching.h +1
-0@@ -30,6 +30,7 @@ struct text_poke_loc {void *addr;size_t len;const char opcode[POKE_MAX_OPCODE_SIZE ]; +u8 old; };extern void text_poke_early (void *addr,const void *opcode,size_t len);arch/x86/kernel/alternative.c +38
-1@@ -3,6 +3,7 @@#include <linux/module.h> #include <linux/sched.h> +#include <linux/perf_event.h> #include <linux/mutex.h> #include <linux/list.h> #include <linux/stringify.h> @@ -1045,8 +1046,10 @@ void text_poke_bp_batch(struct text_poke_loc *tp, unsigned int nr_entries)/* * First step: add a int3 trap to the address that will be patched. */ -for (i =0 ; i < nr_entries; i++) +for (i =0 ; i < nr_entries; i++) { + tp[i].old = *(u8 *)tp[i].addr;text_poke (tp[i].addr, &int3,sizeof (int3)); + }on_each_cpu (do_sync_core,NULL ,1 );@@ -1054,12 +1057,46 @@ void text_poke_bp_batch(struct text_poke_loc *tp, unsigned int nr_entries)* Second step: update all but the first byte of the patched range. */ for (i =0 ; i < nr_entries; i++) { +u8 old[POKE_MAX_OPCODE_SIZE ] = { tp[i].old, }; + +memcpy (old +sizeof (int3), + tp[i].addr +sizeof (int3), + tp[i].len -sizeof (int3)); +if (tp[i].len -sizeof (int3) >0 ) {text_poke ((char *)tp[i].addr +sizeof (int3), (const char *)tp[i].opcode +sizeof (int3), tp[i].len -sizeof (int3)); patched_all_but_first++; } + +/* +* Emit a perf event to record the text poke, primarily to +* support Intel PT decoding which must walk the executable code +* to reconstruct the trace. The flow up to here is: +* - write INT3 byte +* - IPI-SYNC +* At this point the actual control flow will be through the +* INT3 and handler and not hit the old or new instruction. +* Intel PT outputs FUP/TIP packets for the INT3, so the flow +* can still be decoded. Subsequently: +* - emit RECORD_TEXT_POKE with the new instruction +* - write instruction tail +* - IPI-SYNC +* - write first byte +* - IPI-SYNC +* So before the text poke event timestamp, the decoder will see +* either the old instruction flow or FUP/TIP of INT3. After the +* text poke event timestamp, the decoder will see either the +* new instruction flow or FUP/TIP of INT3. Thus decoders can +* use the timestamp as the point at which to modify the +* executable code. +* The old instruction is recorded so that the event can be +* processed forwards or backwards. +*/ +perf_event_text_poke ((unsigned long )tp[i].addr, +PERF_TEXT_POKE_UPDATE , old, tp[i].opcode, + tp[i].len); }if (patched_all_but_first) {include/linux/perf_event.h +6
-0@@ -1183,6 +1183,8 @@ extern void perf_event_exec(void);extern void perf_event_comm (struct task_struct *tsk,bool exec);extern void perf_event_namespaces (struct task_struct *tsk);extern void perf_event_fork (struct task_struct *tsk); +extern void perf_event_text_poke (unsigned long addr,u16 flags,const void *old_bytes, +const void *new_bytes,size_t len);/* Callchains */ DECLARE_PER_CPU(struct perf_callchain_entry, perf_callchain_entry);@@ -1406,6 +1408,10 @@ static inline void perf_event_exec(void) { }static inline void perf_event_comm (struct task_struct *tsk,bool exec) { }static inline void perf_event_namespaces (struct task_struct *tsk) { }static inline void perf_event_fork (struct task_struct *tsk) { } +static inline void perf_event_text_poke (unsigned long addr,u16 flags, +const void *old_bytes, +const void *new_bytes, +size_t len) { }static inline void perf_event_init (void ) { }static inline int perf_swevent_get_recursion_context (void ) {return -1 ; }static inline void perf_swevent_put_recursion_context (int rctx) { }include/uapi/linux/perf_event.h +27
-1@@ -375,7 +375,8 @@ struct perf_event_attr { ksymbol : 1, /* include ksymbol events */ bpf_event : 1, /* include bpf events */ aux_output : 1, /* generate AUX records instead of events */ - __reserved_1 : 32; + text_poke : 1, /* include text poke events */ + __reserved_1 : 31;union {__u32 wakeup_events;/* wakeup every n events */ @@ -1000,6 +1001,26 @@ enum perf_event_type {*/ PERF_RECORD_BPF_EVENT =18 , +/* +* Records changes to kernel text i.e. self-modified code. +* 'flags' has PERF_TEXT_POKE_UPDATE (i.e. bit 0) set to +* indicate to tools to update old bytes to new bytes in the +* executable image. +* 'len' is the number of old bytes, which is the same as the number +* of new bytes. 'bytes' contains the old bytes followed immediately +* by the new bytes. +* +* struct { +* struct perf_event_header header; +* u64 addr; +* u16 flags; +* u16 len; +* u8 bytes[]; +* struct sample_id sample_id; +* }; +*/ +PERF_RECORD_TEXT_POKE =19 , + PERF_RECORD_MAX, /* non-ABI */ };@@ -1041,6 +1062,11 @@ enum perf_callchain_context {#define PERF_AUX_FLAG_PARTIAL 0x04/* record contains gaps */ #define PERF_AUX_FLAG_COLLISION 0x08/* sample collided with another */ +/** +* PERF_RECORD_TEXT_POKE::flags bits +*/ +#define PERF_TEXT_POKE_UPDATE 0x01/* update old bytes to new bytes */ +#define PERF_FLAG_FD_NO_GROUP (1UL << 0)#define PERF_FLAG_FD_OUTPUT (1UL << 1)#define PERF_FLAG_PID_CGROUP (1UL << 2)/* pid=cgroup id, per-cpu mode only */ kernel/events/core.c +89
-1@@ -386,6 +386,7 @@ static atomic_t nr_freq_events __read_mostly;static atomic_t nr_switch_events__read_mostly ;static atomic_t nr_ksymbol_events__read_mostly ;static atomic_t nr_bpf_events__read_mostly ; +static atomic_t nr_text_poke_events__read_mostly ;static LIST_HEAD (pmus);static DEFINE_MUTEX (pmus_lock);@@ -4351,7 +4352,7 @@ static bool is_sb_event(struct perf_event *event)if (attr->mmap || attr->mmap_data || attr->mmap2 || attr->comm || attr->comm_exec || attr->task || attr->ksymbol || - attr->context_switch || + attr->context_switch || attr->text_poke || attr->bpf_event)return true;return false;@@ -4425,6 +4426,8 @@ static void unaccount_event(struct perf_event *event)atomic_dec (&nr_ksymbol_events);if (event->attr.bpf_event)atomic_dec (&nr_bpf_events); +if (event->attr.text_poke) +atomic_dec (&nr_text_poke_events);if (dec) {if (!atomic_add_unless (&perf_sched_count,-1 ,1 ))@@ -8070,6 +8073,89 @@ void perf_event_bpf_event(struct bpf_prog *prog,perf_iterate_sb (perf_event_bpf_output, &bpf_event,NULL ); } +struct perf_text_poke_event { +const void *old_bytes; +const void *new_bytes; +size_t pad; +u16 flags; +u16 len; + +struct { +struct perf_event_header header; + +u64 addr; + } event_id; +}; + +static int perf_event_text_poke_match (struct perf_event *event) +{ +return event->attr.text_poke; +} + +static void perf_event_text_poke_output (struct perf_event *event,void *data) +{ +struct perf_text_poke_event *text_poke_event = data; +struct perf_output_handle handle; +struct perf_sample_data sample; +u64 padding =0 ; +int ret; + +if (!perf_event_text_poke_match (event)) +return ; + +perf_event_header__init_id (&text_poke_event->event_id.header, &sample, event); + + ret =perf_output_begin (&handle, event, text_poke_event->event_id.header.size); +if (ret) +return ; + +perf_output_put (&handle, text_poke_event->event_id); +perf_output_put (&handle, text_poke_event->flags); +perf_output_put (&handle, text_poke_event->len); + +__output_copy (&handle, text_poke_event->old_bytes, text_poke_event->len); +__output_copy (&handle, text_poke_event->new_bytes, text_poke_event->len); + +if (text_poke_event->pad) +__output_copy (&handle, &padding, text_poke_event->pad); + +perf_event__output_id_sample (event, &handle, &sample); + +perf_output_end (&handle); +} + +void perf_event_text_poke (unsigned long addr,u16 flags,const void *old_bytes, +const void *new_bytes,size_t len) +{ +struct perf_text_poke_event text_poke_event; +size_t tot, pad; + +if (!atomic_read (&nr_text_poke_events)) +return ; + + tot =sizeof (text_poke_event.flags) + +sizeof (text_poke_event.len) + (len <<1 ); + pad =ALIGN (tot,sizeof (u64)) - tot; + + text_poke_event = (struct perf_text_poke_event ){ + .old_bytes = old_bytes, + .new_bytes = new_bytes, + .pad = pad, + .flags = flags, + .len = len, + .event_id = { + .header = { + .type =PERF_RECORD_TEXT_POKE , + .misc =PERF_RECORD_MISC_KERNEL , + .size =sizeof (text_poke_event.event_id) + tot + pad, + }, + .addr = addr, + }, + }; + +perf_iterate_sb (perf_event_text_poke_output, &text_poke_event,NULL ); +} +void perf_event_itrace_started (struct perf_event *event) { event->attach_state |=PERF_ATTACH_ITRACE ;@@ -10356,6 +10442,8 @@ static void account_event(struct perf_event *event)atomic_inc (&nr_ksymbol_events);if (event->attr.bpf_event)atomic_inc (&nr_bpf_events); +if (event->attr.text_poke) +atomic_inc (&nr_text_poke_events);if (inc) {/* Signature · 1 line
2.17.1
Peter Zijlstra
maintainerQuoted 21 lines · Leo Yan
Hi Adrian,
Quoted 7 lines · Adrian Hunter
Record changes to kernel text (i.e. self-modifying code) in order to support tracers like Intel PT decoding through jump labels.
A copy of the running kernel code is needed as a reference point (e.g. from
/proc/kcore). The text poke event records the old bytes and the new bytes so that the event can be processed forwards or backwards.Quoted 8 lines
The text poke event has 'flags' to describe when the event is sent. At present the only flag is
PERF_TEXT_POKE_UPDATEwhich indicates the point at which tools should update their reference kernel executable to change the old bytes to the new bytes.Other architectures may wish to emit a pair of text poke events. One before and one after a text poke. In that case,
PERF_TEXT_POKE_UPDATEflag would be set on only one of the pair.Thanks a lot for the patch set.
Quoted 8 lines
Below is my understanding for implementation 'emit a pair of text poke events' as you mentioned:
Since Arm64 instruction is fixed size, it doesn't need to rely on INT3 liked mechanism to replace instructions and directly uses two operations to alter instruction (modify instruction and flush icache line). So Arm64 has no chance to send perf event in the middle of altering instruction.
Right.
Quoted 11 lines
Thus we can send pair of text poke events in the kernel:
perf_event_text_poke(PERF_TEXT_POKE_UPDATE_PREV)Change instruction
Flush icache
perf_event_text_poke(PERF_TEXT_POKE_UPDATE_POST)In the userspace, if perf tool detects the instruction is changed from nop to branch,
There is _far_ more text poking than just jump_label's NOP/JMP transitions. Ftrace also does NOP/CALL, CALL/CALL, and the static_call infrastructure that I posted here:
https://lkml.kernel.org/r/[email protected]
Has: JMP/RET, JMP/JMP and if it has inline patching support: NOP/CALL, CALL/CALL, patching.
Anyway, the below argument doesn't care much, it works for NOP/JMP just fine.
Quoted 10 lines
we need to update dso cache for the '
PERF_TEXT_POKE_UPDATE_PREV' event; if detect the instruction is changed from branch to nop, we need to update dso cache for 'PERF_TEXT_POKE_UPDATE_POST' event. The main idea is to ensure the branch instructions can be safely contained in the dso file and any branch samples can read out correct branch instruction.Could you confirm this is the same with your understanding? Or I miss anything? I personally even think the pair events can be used for different arches (e.g. the solution can be reused on Arm64/x86, etc).
So the problem we have with PT is that it is a bit-stream of branch taken/not-taken decisions. In order to decode that we need to have an accurate view of the unconditional code flow.
Both NOP/JMP are unconditional and we need to exactly know which of the two was encountered.
With your scheme, I don't see how we can ever actually know that. When we get the PRE event, all we really know is that we're going to change a specific instruction into another. And at the POST event we know it has been done. But in between these two events, we have no clue which of the two instructions is live on which CPU (two CPUs might in fact have a different live instruction at the same time).
This means we _cannot_ unambiguously decode a taken/not-taken decision stream.
Does CS have this same problem, and how would the PRE/POST events help with that?
So our (x86) horrible (variable instruction length induced) complexity for text poking is actually in our advantage this one time. The exception and exception-return paths allow us to unambiguously know what happens around the time of poking.
Hi Peter,
[ + Mike Leach ]
Quoted 45 lines · Peter Zijlstra
Quoted 21 lines · Leo Yan
Hi Adrian,
Quoted 7 lines · Adrian Hunter
Record changes to kernel text (i.e. self-modifying code) in order to support tracers like Intel PT decoding through jump labels.
A copy of the running kernel code is needed as a reference point (e.g. from
/proc/kcore). The text poke event records the old bytes and the new bytes so that the event can be processed forwards or backwards.Quoted 8 lines
The text poke event has 'flags' to describe when the event is sent. At present the only flag is
PERF_TEXT_POKE_UPDATEwhich indicates the point at which tools should update their reference kernel executable to change the old bytes to the new bytes.Other architectures may wish to emit a pair of text poke events. One before and one after a text poke. In that case,
PERF_TEXT_POKE_UPDATEflag would be set on only one of the pair.Thanks a lot for the patch set.
Quoted 8 lines
Below is my understanding for implementation 'emit a pair of text poke events' as you mentioned:
Since Arm64 instruction is fixed size, it doesn't need to rely on INT3 liked mechanism to replace instructions and directly uses two operations to alter instruction (modify instruction and flush icache line). So Arm64 has no chance to send perf event in the middle of altering instruction.
Right.
Quoted 11 lines
Thus we can send pair of text poke events in the kernel:
perf_event_text_poke(PERF_TEXT_POKE_UPDATE_PREV)Change instruction
Flush icache
perf_event_text_poke(PERF_TEXT_POKE_UPDATE_POST)In the userspace, if perf tool detects the instruction is changed from nop to branch,
Quoted 8 lines
There is _far_ more text poking than just jump_label's NOP/JMP transitions. Ftrace also does NOP/CALL, CALL/CALL, and the
static_callinfrastructure that I posted here:https://lkml.kernel.org/r/[email protected]
Has: JMP/RET, JMP/JMP and if it has inline patching support: NOP/CALL, CALL/CALL, patching.
Thanks for the info. I took a bit time to look your patch set and Steven's patch set for dynamic function, though don't completely understand, but get more sense for the context.
Quoted 2 lines
Anyway, the below argument doesn't care much, it works for NOP/JMP just fine.
We can support NOP/JMP case as the first step, but later should can extend to support other transitions.
Quoted 11 lines
Quoted 10 lines
we need to update dso cache for the '
PERF_TEXT_POKE_UPDATE_PREV' event; if detect the instruction is changed from branch to nop, we need to update dso cache for 'PERF_TEXT_POKE_UPDATE_POST' event. The main idea is to ensure the branch instructions can be safely contained in the dso file and any branch samples can read out correct branch instruction.Could you confirm this is the same with your understanding? Or I miss anything? I personally even think the pair events can be used for different arches (e.g. the solution can be reused on Arm64/x86, etc).
Quoted 6 lines
So the problem we have with PT is that it is a bit-stream of branch taken/not-taken decisions. In order to decode that we need to have an accurate view of the unconditional code flow.
Both NOP/JMP are unconditional and we need to exactly know which of the two was encountered.
If I understand correctly, PT decoder needs to read out instructions from dso and decide the instruction type (NOP or JMP), and finally generate the accurate code flow.
So PT decoder relies on (cached) DSO for decoding. As I know, this might be different from Arm CS, since Arm CS decoder is merely generate packets and it doesn't need to rely on DSO for decoding.
I think my answer is not very convinced, in case I mislead, loop Mike to confirm for this.
Quoted 7 lines
With your scheme, I don't see how we can ever actually know that. When we get the PRE event, all we really know is that we're going to change a specific instruction into another. And at the POST event we know it has been done. But in between these two events, we have no clue which of the two instructions is live on which CPU (two CPUs might in fact have a different live instruction at the same time).
Quoted 5 lines
This means we _cannot_ unambiguously decode a taken/not-taken decision stream.
Does CS have this same problem, and how would the PRE/POST events help with that?
My purpose is to use PRE event and POST event to update cached DSO, thus perf tool can read out 'correct' instructions and fill them into instruction/branch samples. On the other hand, as mentioned, I don't aware the instruction read out from DSO will be used for Arm CS decoder; Arm CS also reads the instructions from DSO, usually it's used to decide instruction size or decide the sample flags (flags for CALL/RETURN/hw int, etc...), but it isn't used for decoder.
@Mike, please help confirm for this as well, from Arm CoreSight's decoder pespective.
Quoted 4 lines
So our (x86) horrible (variable instruction length induced) complexity for text poking is actually in our advantage this one time. The exception and exception-return paths allow us to unambiguously know what happens around the time of poking.
Thanks a lot for the info!
Leo.
Hi Leo,
Quoted 92 lines · Leo Yan
Hi Peter,
[ + Mike Leach ]
Quoted 45 lines · Peter Zijlstra
Quoted 21 lines · Leo Yan
Hi Adrian,
Quoted 7 lines · Adrian Hunter
Record changes to kernel text (i.e. self-modifying code) in order to support tracers like Intel PT decoding through jump labels.
A copy of the running kernel code is needed as a reference point (e.g. from
/proc/kcore). The text poke event records the old bytes and the new bytes so that the event can be processed forwards or backwards.Quoted 8 lines
The text poke event has 'flags' to describe when the event is sent. At present the only flag is
PERF_TEXT_POKE_UPDATEwhich indicates the point at which tools should update their reference kernel executable to change the old bytes to the new bytes.Other architectures may wish to emit a pair of text poke events. One before and one after a text poke. In that case,
PERF_TEXT_POKE_UPDATEflag would be set on only one of the pair.Thanks a lot for the patch set.
Quoted 8 lines
Below is my understanding for implementation 'emit a pair of text poke events' as you mentioned:
Since Arm64 instruction is fixed size, it doesn't need to rely on INT3 liked mechanism to replace instructions and directly uses two operations to alter instruction (modify instruction and flush icache line). So Arm64 has no chance to send perf event in the middle of altering instruction.
Right.
Quoted 11 lines
Thus we can send pair of text poke events in the kernel:
perf_event_text_poke(PERF_TEXT_POKE_UPDATE_PREV)Change instruction
Flush icache
perf_event_text_poke(PERF_TEXT_POKE_UPDATE_POST)In the userspace, if perf tool detects the instruction is changed from nop to branch,
Quoted 8 lines
There is _far_ more text poking than just jump_label's NOP/JMP transitions. Ftrace also does NOP/CALL, CALL/CALL, and the
static_callinfrastructure that I posted here:https://lkml.kernel.org/r/[email protected]
Has: JMP/RET, JMP/JMP and if it has inline patching support: NOP/CALL, CALL/CALL, patching.
Thanks for the info. I took a bit time to look your patch set and Steven's patch set for dynamic function, though don't completely understand, but get more sense for the context.
Quoted 2 lines
Anyway, the below argument doesn't care much, it works for NOP/JMP just fine.
We can support NOP/JMP case as the first step, but later should can extend to support other transitions.
Quoted 11 lines
Quoted 10 lines
we need to update dso cache for the '
PERF_TEXT_POKE_UPDATE_PREV' event; if detect the instruction is changed from branch to nop, we need to update dso cache for 'PERF_TEXT_POKE_UPDATE_POST' event. The main idea is to ensure the branch instructions can be safely contained in the dso file and any branch samples can read out correct branch instruction.Could you confirm this is the same with your understanding? Or I miss anything? I personally even think the pair events can be used for different arches (e.g. the solution can be reused on Arm64/x86, etc).
Quoted 6 lines
So the problem we have with PT is that it is a bit-stream of branch taken/not-taken decisions. In order to decode that we need to have an accurate view of the unconditional code flow.
Both NOP/JMP are unconditional and we need to exactly know which of the two was encountered.
If I understand correctly, PT decoder needs to read out instructions from dso and decide the instruction type (NOP or JMP), and finally generate the accurate code flow.
Quoted 7 lines
So PT decoder relies on (cached) DSO for decoding. As I know, this might be different from Arm CS, since Arm CS decoder is merely generate packets and it doesn't need to rely on DSO for decoding.
I think my answer is not very convinced, in case I mislead, loop Mike to confirm for this.
The CoreSight decoder needs the same information. When a branch / call instruction is traced an atom is generated in the trace to determine if that call as taken or not.
During the decode process the decoder walks the code image to match up atoms to call / branch instructions - which means for correct trace decode we must have an accurate code image as executed, as otherwise trace decode will go wrong.
Within perf, the OpenCSD decoder will call back into perf to ask for the memory image for a given address - which perf will supply from the list of .so / executable files it has in the .debug directory.
Self modifying code presents an issue in this respect.
Correlation of the modification information with the captured trace becomes the primary concern.
Regards
Mike
Quoted 31 lines
Quoted 7 lines
With your scheme, I don't see how we can ever actually know that. When we get the PRE event, all we really know is that we're going to change a specific instruction into another. And at the POST event we know it has been done. But in between these two events, we have no clue which of the two instructions is live on which CPU (two CPUs might in fact have a different live instruction at the same time).
Quoted 5 lines
This means we _cannot_ unambiguously decode a taken/not-taken decision stream.
Does CS have this same problem, and how would the PRE/POST events help with that?
My purpose is to use PRE event and POST event to update cached DSO, thus perf tool can read out 'correct' instructions and fill them into instruction/branch samples. On the other hand, as mentioned, I don't aware the instruction read out from DSO will be used for Arm CS decoder; Arm CS also reads the instructions from DSO, usually it's used to decide instruction size or decide the sample flags (flags for CALL/RETURN/hw int, etc...), but it isn't used for decoder.
@Mike, please help confirm for this as well, from Arm CoreSight's decoder pespective.
Quoted 4 lines
So our (x86) horrible (variable instruction length induced) complexity for text poking is actually in our advantage this one time. The exception and exception-return paths allow us to unambiguously know what happens around the time of poking.
Thanks a lot for the info!
Leo.
Signature · 3 lines
Mike Leach Principal Engineer, ARM Ltd. Manchester Design Centre. UK
Peter Zijlstra
maintainerQuoted 8 lines · Leo Yan
Quoted 4 lines · Peter Zijlstra
On Wed, Oct 30, 2019 at 06:47:47PM +0800, Leo Yan wrote:
Anyway, the below argument doesn't care much, it works for NOP/JMP just fine.
We can support NOP/JMP case as the first step, but later should can extend to support other transitions.
Since all instructions (with the possible exception of RET) are unconditional branch instructions: NOP, JMP, CALL. It makes no read difference to the argument below.
( I'm thinking RET might be special in that it reads the return address from the stack and therefore must emit the whole IP into the stream, as we cannot know the stack state )
Quoted 18 lines
Quoted 11 lines
Quoted 10 lines
we need to update dso cache for the '
PERF_TEXT_POKE_UPDATE_PREV' event; if detect the instruction is changed from branch to nop, we need to update dso cache for 'PERF_TEXT_POKE_UPDATE_POST' event. The main idea is to ensure the branch instructions can be safely contained in the dso file and any branch samples can read out correct branch instruction.Could you confirm this is the same with your understanding? Or I miss anything? I personally even think the pair events can be used for different arches (e.g. the solution can be reused on Arm64/x86, etc).
Quoted 6 lines
So the problem we have with PT is that it is a bit-stream of branch taken/not-taken decisions. In order to decode that we need to have an accurate view of the unconditional code flow.
Both NOP/JMP are unconditional and we need to exactly know which of the two was encountered.
Quoted 7 lines
If I understand correctly, PT decoder needs to read out instructions from dso and decide the instruction type (NOP or JMP), and finally generate the accurate code flow.
So PT decoder relies on (cached) DSO for decoding. As I know, this might be different from Arm CS, since Arm CS decoder is merely generate packets and it doesn't need to rely on DSO for decoding.
Given a start point (from a start or sync packet) we scan the instruction stream forward until the first conditional branch instruction. Then we consume the next available branch decision bit to know where to continue.
So yes, we need to have a correct text image available for this to work.
Quoted 10 lines
Quoted 10 lines
With your scheme, I don't see how we can ever actually know that. When we get the PRE event, all we really know is that we're going to change a specific instruction into another. And at the POST event we know it has been done. But in between these two events, we have no clue which of the two instructions is live on which CPU (two CPUs might in fact have a different live instruction at the same time).
This means we _cannot_ unambiguously decode a taken/not-taken decision stream.
Quoted 6 lines
Quoted 2 lines
Does CS have this same problem, and how would the PRE/POST events help with that?
My purpose is to use PRE event and POST event to update cached DSO, thus perf tool can read out 'correct' instructions and fill them into instruction/branch samples.
The thing is, as I argued, the instruction state between PRE and POST is ambiguous. This makes it impossible to decode the branch decision stream.
Suppose CPU0 emits the PRE event at T1 and the POST event at T5, but we have CPU1 covering the instruction at T3.
How do you decide where CPU1 goes and what the next conditional branch is?
[ + Will, Mark ]
Quoted 10 lines · Peter Zijlstra
Quoted 8 lines · Leo Yan
Quoted 4 lines · Peter Zijlstra
On Wed, Oct 30, 2019 at 06:47:47PM +0800, Leo Yan wrote:
Anyway, the below argument doesn't care much, it works for NOP/JMP just fine.
We can support NOP/JMP case as the first step, but later should can extend to support other transitions.
Quoted 7 lines
Since all instructions (with the possible exception of RET) are unconditional branch instructions: NOP, JMP, CALL. It makes no read difference to the argument below.
( I'm thinking RET might be special in that it reads the return address from the stack and therefore must emit the whole IP into the stream, as we cannot know the stack state )
To be honest, I don't have knowledge what's the exactly format for 'ret' in CoreSight trace; so would like to leave this to Mike.
Since Mike has confirmed that Arm CoreSight also needs the accurate code for decoding branch/call instructions, it's no doubt that we need to implement the same mechanism at here to update DSO for the accurate code flow. So the question is how to do this on Arm/Arm64 :)
Before move farward, I'd like to step back to describe clearly what's current problem on Arm64 and check one question for jump label:
I checked the kernel code, both kprobe and ftrace both uses stop_machine() to alter instructions, since all CPUs run into stop machine's synchronization, there have no race condition between instructions transition and CPUs execte the altered instruction; thus it's safe for kprobe and ftrace to use perf event PERF_TEXT_POKE_UPDATE to notify instruction transition and can allow us to read out 'correct' instruction for decoder.
But for jump label, it doesn't use the stop_machine() and perf event PERF_TEXT_POKE_UPDATE will introduce race condition as below (Let's see the example for transition from nop to branch):
CPU0 CPU1
NOP instruction
`-> static_key_enable()
`-> aarch64_insn_patch_text_nosync()
`-> perf event PERF_TEXT_POKE_UPDATE
-> Execute nop
instruction
`-> aarch64_insn_write()
`-> __flush_icache_range()Since x86 platform have INT3 as a mediate state, it can avoid the race condition between CPU0 (who is do transition) and other CPUs (who is possible to execute nop/branch).
Quoted 54 lines
Quoted 18 lines
Quoted 11 lines
Quoted 10 lines
we need to update dso cache for the '
PERF_TEXT_POKE_UPDATE_PREV' event; if detect the instruction is changed from branch to nop, we need to update dso cache for 'PERF_TEXT_POKE_UPDATE_POST' event. The main idea is to ensure the branch instructions can be safely contained in the dso file and any branch samples can read out correct branch instruction.Could you confirm this is the same with your understanding? Or I miss anything? I personally even think the pair events can be used for different arches (e.g. the solution can be reused on Arm64/x86, etc).
Quoted 6 lines
So the problem we have with PT is that it is a bit-stream of branch taken/not-taken decisions. In order to decode that we need to have an accurate view of the unconditional code flow.
Both NOP/JMP are unconditional and we need to exactly know which of the two was encountered.
Quoted 7 lines
If I understand correctly, PT decoder needs to read out instructions from dso and decide the instruction type (NOP or JMP), and finally generate the accurate code flow.
So PT decoder relies on (cached) DSO for decoding. As I know, this might be different from Arm CS, since Arm CS decoder is merely generate packets and it doesn't need to rely on DSO for decoding.
Given a start point (from a start or sync packet) we scan the instruction stream forward until the first conditional branch instruction. Then we consume the next available branch decision bit to know where to continue.
So yes, we need to have a correct text image available for this to work.
Quoted 10 lines
Quoted 10 lines
With your scheme, I don't see how we can ever actually know that. When we get the PRE event, all we really know is that we're going to change a specific instruction into another. And at the POST event we know it has been done. But in between these two events, we have no clue which of the two instructions is live on which CPU (two CPUs might in fact have a different live instruction at the same time).
This means we _cannot_ unambiguously decode a taken/not-taken decision stream.
Quoted 6 lines
Quoted 2 lines
Does CS have this same problem, and how would the PRE/POST events help with that?
My purpose is to use PRE event and POST event to update cached DSO, thus perf tool can read out 'correct' instructions and fill them into instruction/branch samples.
The thing is, as I argued, the instruction state between PRE and POST is ambiguous. This makes it impossible to decode the branch decision stream.
Quoted 5 lines
Suppose CPU0 emits the PRE event at T1 and the POST event at T5, but we have CPU1 covering the instruction at T3.
How do you decide where CPU1 goes and what the next conditional branch is?
Sorry for my not well thought.
I agree that T3 is an uncertain state with below flow:
CPU0 CPU1
perf event PERF_TEXT_POKE_UPDATE_PRE -> T1
Int3 / NOP -> T3
Int3 / branch -> T3' perf event PERF_TEXT_POKE_UPDATE_POST -> T5
Except if the trace has extra info and can use old/new instructions combination for analysis, otherwise PRE/POST pair events aren't helpful for resolve this issue (if trace decoder can do this, then the change in kernel will be much simpler).
Below are two potential options we can use on Arm64 platform:
- Change to use stop_machine() for jump label; this might introduce performance issue if jump label is altered frequently.
To mitigate the impaction, we can only use stop_machine() when detect the perf events are enabled, otherwise will rollback to use the old code path.
- We can use breakpoint to emulate the similiar flow with x86's int3, thus we can dismiss the race condition between one CPU alters instruction and other CPUs run into the alternative instruction.
@Will, @Mark, could you help review this? Appreciate any comments and suggestions. And please let me know if you want to consolidate related works with your side (or as you know if there have ongoing discussion or someone works on this).
Thanks,
Leo Yan
Peter Zijlstra
maintainerQuoted 5 lines · Leo Yan
Before move farward, I'd like to step back to describe clearly what's current problem on Arm64 and check one question for jump label:
I checked the kernel code, both kprobe and ftrace both uses
stop_machine()to alter instructions,
That's not currect for Aargh64, see aarch64_insn_patch_text_nosync(), which is used in both ftrace and jump_label.
Quoted 6 lines
since all CPUs run into stop
machine's synchronization, there have no race condition between instructions transition and CPUs execte the altered instruction; thus it's safe for kprobe and ftrace to use perf eventPERF_TEXT_POKE_UPDATEto notify instruction transition and can allow us to read out 'correct' instruction for decoder.
Agreed, IFF patching happens using stop_machine(), things are easy. ARM is (so far) exclusively using stop_machine() based text_poking, although the last time I spoke to Will about this, he said the _nosync stuff is possible on 32bit too, just nobody has bothered implementing it.
Quoted 14 lines
But for jump label, it doesn't use the
stop_machine()and perf eventPERF_TEXT_POKE_UPDATEwill introduce race condition as below (Let's see the example for transition from nop to branch):CPU0 CPU1
NOP instruction
`->static_key_enable()`-> aarch64_insn_patch_text_nosync() `-> perf event PERF_TEXT_POKE_UPDATE -> Execute nop instruction `-> aarch64_insn_write() `-> __flush_icache_range()
Quoted 3 lines
Since x86 platform have INT3 as a mediate state, it can avoid the race condition between CPU0 (who is do transition) and other CPUs (who is possible to execute nop/branch).
Ah, you found the _nosync thing in jump_label, here's the one in ftrace:
arch/arm64/kernel/ftrace.c: if (aarch64_insn_patch_text_nosync((void *)pc, new))
And yes, this is racy.
Quoted 37 lines
Quoted 9 lines
The thing is, as I argued, the instruction state between PRE and POST is ambiguous. This makes it impossible to decode the branch decision stream.
Suppose CPU0 emits the PRE event at T1 and the POST event at T5, but we have CPU1 covering the instruction at T3.
How do you decide where CPU1 goes and what the next conditional branch is?
Sorry for my not well thought.
I agree that T3 is an uncertain state with below flow:
CPU0 CPU1
perf eventPERF_TEXT_POKE_UPDATE_PRE-> T1Int3 / NOP -> T3 Int3 / branch -> T3'perf event
PERF_TEXT_POKE_UPDATE_POST-> T5Except if the trace has extra info and can use old/new instructions combination for analysis, otherwise PRE/POST pair events aren't helpful for resolve this issue (if trace decoder can do this, then the change in kernel will be much simpler).
Below are two potential options we can use on Arm64 platform:
- Change to use
stop_machine()for jump label; this might introduce performance issue if jump label is altered frequently.To mitigate the impaction, we can only use
stop_machine()when detect the perf events are enabled, otherwise will rollback to use the old code path.
Quoted 8 lines
- We can use breakpoint to emulate the similiar flow with x86's int3, thus we can dismiss the race condition between one CPU alters instruction and other CPUs run into the alternative instruction.
@Will, @Mark, could you help review this? Appreciate any comments and suggestions. And please let me know if you want to consolidate related works with your side (or as you know if there have ongoing discussion or someone works on this).
Given people are building larger Aargh64 machines (I've heard about 100+ CPUs already), I'm thinking the 3rd option is the most performant.
But yes, as you mention earlier, we can make this optional on the TEXT_POKE_UPDATE event being in use.
I'm thinking something along the lines of:
static uintptr_t nosync_addr;
static u32 nosync_insn;
int __kprobes aarch64_insn_patch_text_nosync(void *addr, u32 insn)
{
const u32 break = // some_breakpoint_insn;
uintptr_t tp = (uintptr_t)addr;
int ret;
lockdep_assert_held(&text_mutex);
/* A64 instructions must be word aligned */
if (tp & 0x3)
return -EINVAL;
if (perf_text_poke_update_enabled()) {
nosync_insn = insn;
smp_store_release(&nosync_addr, tp);
ret = aarch64_insn_write(addr, break);
if (ret == 0)
__flush_icache_range(tp, tp + AARCH64_INSN_SIZE);
perf_event_text_poke(....);
}
ret = aarch64_insn_write(addr, insn);
if (ret == 0)
__flush_icache_range(tp, tp + AARCH64_INSN_SIZE);
return ret;
}And have the 'break' handler do:
aarch64_insn_break_handler(struct pt_regs *regs)
{
unsigned long addr = smp_load_acquire(&nosync_addr);
u32 insn = nosync_insn;
if (regs->ip != addr)
return;
// emulate @insn
}I understood from Will the whole nosync scheme only works for a limited set of instructions, but you only have to implement emulation for the actual instructions used of course.
(which is what we do on x86)
Does this sound workable?
Peter Zijlstra
maintainerQuoted 24 lines · Peter Zijlstra
I'm thinking something along the lines of:
static
uintptr_tnosync_addr;
static u32nosync_insn;int
__kprobesaarch64_insn_patch_text_nosync(void *addr, u32 insn){ const u32 break = // some_breakpoint_insn; uintptr_t tp = (uintptr_t)addr; int ret; lockdep_assert_held(&text_mutex); /* A64 instructions must be word aligned */ if (tp & 0x3) return -EINVAL; if (perf_text_poke_update_enabled()) { nosync_insn = insn; smp_store_release(&nosync_addr, tp); ret = aarch64_insn_write(addr, break); if (ret == 0)
Quoted 8 lines
__flush_icache_range(tp, tp + AARCH64_INSN_SIZE); perf_event_text_poke(....); } ret = aarch64_insn_write(addr, insn); if (ret == 0) __flush_icache_range(tp, tp + AARCH64_INSN_SIZE);
// barrier that guarantees iflush completion ? dsb(osh) ? WRITE_ONCE(nosync_addr, NULL);
Quoted 23 lines
return ret; }And have the 'break' handler do:
aarch64_insn_break_handler(struct pt_regs *regs){ unsigned long addr = smp_load_acquire(&nosync_addr); u32 insn = nosync_insn; if (regs->ip != addr) return; // emulate @insn }I understood from Will the whole nosync scheme only works for a limited set of instructions, but you only have to implement emulation for the actual instructions used of course.
(which is what we do on x86)
Does this sound workable?
Hi Peter,
Quoted 10 lines · Peter Zijlstra
Quoted 5 lines · Leo Yan
Before move farward, I'd like to step back to describe clearly what's current problem on Arm64 and check one question for jump label:
I checked the kernel code, both kprobe and ftrace both uses
stop_machine()to alter instructions,That's not currect for Aargh64, see
aarch64_insn_patch_text_nosync(), which is used in both ftrace andjump_label.
Thanks for pointing out this.
Agree, for ftrace, it's complex and there have multiple pathes to use aarch64_insn_patch_text_nosync().
Below flow uses stop_machine() in ftrace:
ftrace_run_stop_machine()
stop_machine(__ftrace_modify_code, &command, NULL);
__ftrace_modify_code()
ftrace_modify_all_code()
ftrace_update_ftrace_func()
ftrace_modify_code()
aarch64_insn_patch_text_nosync()Below flow doesn't use stop_machine() in ftrace:
prepare_coming_module()
ftrace_module_enable()
process_cached_mods()
process_mod_list()
ftrace_hash_move_and_update_ops()
ftrace_ops_update_code()
ftrace_ops_update_code()
ftrace_run_modify_code()
ftrace_run_update_code()
arch_ftrace_update_code()
ftrace_modify_all_code()
ftrace_update_ftrace_func()
ftrace_modify_code()
aarch64_insn_patch_text_nosync()
Actually, there have other flows also will call into aarch64_insn_patch_text_nosync(), so at least we cannot say all ftrace flow uses stop_machine() to alter instructions.
Quoted 137 lines
Quoted 6 lines
since all CPUs run into stop
machine's synchronization, there have no race condition between instructions transition and CPUs execte the altered instruction; thus it's safe for kprobe and ftrace to use perf eventPERF_TEXT_POKE_UPDATEto notify instruction transition and can allow us to read out 'correct' instruction for decoder.Agreed, IFF patching happens using
stop_machine(), things are easy. ARM is (so far) exclusively usingstop_machine()basedtext_poking, although the last time I spoke to Will about this, he said the_nosyncstuff is possible on 32bit too, just nobody has bothered implementing it.Quoted 14 lines
But for jump label, it doesn't use the
stop_machine()and perf eventPERF_TEXT_POKE_UPDATEwill introduce race condition as below (Let's see the example for transition from nop to branch):CPU0 CPU1
NOP instruction
`->static_key_enable()`-> aarch64_insn_patch_text_nosync() `-> perf event PERF_TEXT_POKE_UPDATE -> Execute nop instruction `-> aarch64_insn_write() `-> __flush_icache_range()Quoted 3 lines
Since x86 platform have INT3 as a mediate state, it can avoid the race condition between CPU0 (who is do transition) and other CPUs (who is possible to execute nop/branch).
Ah, you found the
_nosyncthing injump_label, here's the one in ftrace:
arch/arm64/kernel/ftrace.c: if (aarch64_insn_patch_text_nosync((void *)pc, new))And yes, this is racy.
Quoted 37 lines
Quoted 9 lines
The thing is, as I argued, the instruction state between PRE and POST is ambiguous. This makes it impossible to decode the branch decision stream.
Suppose CPU0 emits the PRE event at T1 and the POST event at T5, but we have CPU1 covering the instruction at T3.
How do you decide where CPU1 goes and what the next conditional branch is?
Sorry for my not well thought.
I agree that T3 is an uncertain state with below flow:
CPU0 CPU1
perf eventPERF_TEXT_POKE_UPDATE_PRE-> T1Int3 / NOP -> T3 Int3 / branch -> T3'perf event
PERF_TEXT_POKE_UPDATE_POST-> T5Except if the trace has extra info and can use old/new instructions combination for analysis, otherwise PRE/POST pair events aren't helpful for resolve this issue (if trace decoder can do this, then the change in kernel will be much simpler).
Below are two potential options we can use on Arm64 platform:
- Change to use
stop_machine()for jump label; this might introduce performance issue if jump label is altered frequently.To mitigate the impaction, we can only use
stop_machine()when detect the perf events are enabled, otherwise will rollback to use the old code path.Quoted 8 lines
- We can use breakpoint to emulate the similiar flow with x86's int3, thus we can dismiss the race condition between one CPU alters instruction and other CPUs run into the alternative instruction.
@Will, @Mark, could you help review this? Appreciate any comments and suggestions. And please let me know if you want to consolidate related works with your side (or as you know if there have ongoing discussion or someone works on this).
Given people are building larger Aargh64 machines (I've heard about 100+ CPUs already), I'm thinking the 3rd option is the most performant.
But yes, as you mention earlier, we can make this optional on the
TEXT_POKE_UPDATEevent being in use.I'm thinking something along the lines of:
static
uintptr_tnosync_addr;
static u32nosync_insn;int
__kprobesaarch64_insn_patch_text_nosync(void *addr, u32 insn){ const u32 break = // some_breakpoint_insn; uintptr_t tp = (uintptr_t)addr; int ret; lockdep_assert_held(&text_mutex); /* A64 instructions must be word aligned */ if (tp & 0x3) return -EINVAL; if (perf_text_poke_update_enabled()) { nosync_insn = insn; smp_store_release(&nosync_addr, tp); ret = aarch64_insn_write(addr, break); if (ret == 0) __flush_icache_range(tp, tp + AARCH64_INSN_SIZE); perf_event_text_poke(....); } ret = aarch64_insn_write(addr, insn); if (ret == 0) __flush_icache_range(tp, tp + AARCH64_INSN_SIZE); return ret; }And have the 'break' handler do:
aarch64_insn_break_handler(struct pt_regs *regs){ unsigned long addr = smp_load_acquire(&nosync_addr); u32 insn = nosync_insn; if (regs->ip != addr) return; // emulate @insn }
Quoted 7 lines
I understood from Will the whole nosync scheme only works for a limited set of instructions, but you only have to implement emulation for the actual instructions used of course.
(which is what we do on x86)
Does this sound workable?
Very appreciate for the posted code (and another minor fixing in your next replying), the logic is quite clear.
Will do prototype for this, at the meantime, I'd like to give a bit more time for Will (or other Arm maintainers) to review this.
Thanks,
Leo Yan
On Mon, Nov 04, 2019 at 10:23:46AM +0800, Leo Yan wrote:
[...]
Quoted 125 lines
Quoted 125 lines
Quoted 14 lines
But for jump label, it doesn't use the
stop_machine()and perf eventPERF_TEXT_POKE_UPDATEwill introduce race condition as below (Let's see the example for transition from nop to branch):CPU0 CPU1
NOP instruction
`->static_key_enable()`-> aarch64_insn_patch_text_nosync() `-> perf event PERF_TEXT_POKE_UPDATE -> Execute nop instruction `-> aarch64_insn_write() `-> __flush_icache_range()Quoted 3 lines
Since x86 platform have INT3 as a mediate state, it can avoid the race condition between CPU0 (who is do transition) and other CPUs (who is possible to execute nop/branch).
Ah, you found the
_nosyncthing injump_label, here's the one in ftrace:
arch/arm64/kernel/ftrace.c: if (aarch64_insn_patch_text_nosync((void *)pc, new))And yes, this is racy.
Quoted 37 lines
Quoted 9 lines
The thing is, as I argued, the instruction state between PRE and POST is ambiguous. This makes it impossible to decode the branch decision stream.
Suppose CPU0 emits the PRE event at T1 and the POST event at T5, but we have CPU1 covering the instruction at T3.
How do you decide where CPU1 goes and what the next conditional branch is?
Sorry for my not well thought.
I agree that T3 is an uncertain state with below flow:
CPU0 CPU1
perf eventPERF_TEXT_POKE_UPDATE_PRE-> T1Int3 / NOP -> T3 Int3 / branch -> T3'perf event
PERF_TEXT_POKE_UPDATE_POST-> T5Except if the trace has extra info and can use old/new instructions combination for analysis, otherwise PRE/POST pair events aren't helpful for resolve this issue (if trace decoder can do this, then the change in kernel will be much simpler).
Below are two potential options we can use on Arm64 platform:
- Change to use
stop_machine()for jump label; this might introduce performance issue if jump label is altered frequently.To mitigate the impaction, we can only use
stop_machine()when detect the perf events are enabled, otherwise will rollback to use the old code path.Quoted 8 lines
- We can use breakpoint to emulate the similiar flow with x86's int3, thus we can dismiss the race condition between one CPU alters instruction and other CPUs run into the alternative instruction.
@Will, @Mark, could you help review this? Appreciate any comments and suggestions. And please let me know if you want to consolidate related works with your side (or as you know if there have ongoing discussion or someone works on this).
Given people are building larger Aargh64 machines (I've heard about 100+ CPUs already), I'm thinking the 3rd option is the most performant.
But yes, as you mention earlier, we can make this optional on the
TEXT_POKE_UPDATEevent being in use.I'm thinking something along the lines of:
static
uintptr_tnosync_addr;
static u32nosync_insn;int
__kprobesaarch64_insn_patch_text_nosync(void *addr, u32 insn){ const u32 break = // some_breakpoint_insn; uintptr_t tp = (uintptr_t)addr; int ret; lockdep_assert_held(&text_mutex); /* A64 instructions must be word aligned */ if (tp & 0x3) return -EINVAL; if (perf_text_poke_update_enabled()) { nosync_insn = insn; smp_store_release(&nosync_addr, tp); ret = aarch64_insn_write(addr, break); if (ret == 0) __flush_icache_range(tp, tp + AARCH64_INSN_SIZE); perf_event_text_poke(....); } ret = aarch64_insn_write(addr, insn); if (ret == 0) __flush_icache_range(tp, tp + AARCH64_INSN_SIZE); return ret; }And have the 'break' handler do:
aarch64_insn_break_handler(struct pt_regs *regs){ unsigned long addr = smp_load_acquire(&nosync_addr); u32 insn = nosync_insn; if (regs->ip != addr) return; // emulate @insn }
Quoted 7 lines
Quoted 7 lines
I understood from Will the whole nosync scheme only works for a limited set of instructions, but you only have to implement emulation for the actual instructions used of course.
(which is what we do on x86)
Does this sound workable?
I will update some status for prototype (the code has been uploaded into git server [1]) and found some issues for text poke perf event on arm64. These issues are mainly related with arch64 architecture.
- The first issue is for the nosync instruction emulation. On arm64, some instructions need to be single stepped and some instructions is emulated. Especially, after I read the code for kprobe implementation on Arm64. So the main idea for prototyping is to use the almos same method with kprobe for nosync instruction.
As result, there have duplicated code between kprobe and text poke handling. If this is the right way to move forward, we need to refactor the common code so that kprobe and text poke can share with each other.
- The second issue is race condition between the CPU alters instructions and other CPUs hit the altered instructions (and breakpointed).
Peter's suggestion uses global variables 'nosync_insn' and 'nosync_addr' to record the altered instruction. But from the testing I found for single static key, usually it will change for multiple address at once.
So this might cause the issue is: CPU(a) will loop to alter instructions for different address (sometimes the opcode also is different for different address), at the meantime, CPU(b) hits an altered instruction and handle exception for the breakpoint, if CPU(a) is continuing to alter instruction for the next address, thne CPU(a) might wrongly to use the value from 'nosync_insn' and 'nosync_addr'.
Simply to say, we cannot only support single nosync instruction but need to support multiple nosync instructions in the loop.
- The third issue is might be nested issue. E.g. for the injected break instruction, we have no method to pass perf event for this instruction; and if we connect with the first issue, if the instruction is single stepped with slot (the slot is allocated with get_insn_slot()), we cannot to allow the perf user space to know the instructions which are executed in the slots.
I am not sure if I think too complex here. But seems to me, we inject more instructions for poke text event, and if these injected instructions are executed but the userspace decoder has no way to pass the related info.
Just clarify, I am fine for merging this patch set, but we might consider more what's the best way on Arm64. Welcome any public comments and suggestions; I will sync internally for how to follow up this functionality.
Thanks,
Leo Yan
[1] https://git.linaro.org/people/leo.yan/linux-coresight.git/log/?h=arm64_perf_text_pork_prototype
Peter Zijlstra
maintainerQuoted 9 lines · Leo Yan
I will update some status for prototype (the code has been uploaded into git server [1]) and found some issues for text poke perf event on arm64. These issues are mainly related with arch64 architecture.
- The first issue is for the nosync instruction emulation. On arm64, some instructions need to be single stepped and some instructions is emulated. Especially, after I read the code for kprobe implementation on Arm64. So the main idea for prototyping is to use the almos same method with kprobe for nosync instruction.
This makes no sense to me what so ever. What actual instructions are patched with _nosync() ? ftrace/jump_label only use 'NOP/JMP/CALL'
For NOP you can advance regs->ip, for JMP you can adjust regs->ip, for CALL you adjust regs->ip and regs->r14 (IIUC you do something like:regs->r14 = regs->ip+4; regs->ip = func;)
(FWIW emulating CALL on x86 is fun because we get to PUSH something on the stack, let me know if you want to see the patches that were required to make that happen :-)
Quoted 17 lines
- The second issue is race condition between the CPU alters instructions and other CPUs hit the altered instructions (and breakpointed).
Peter's suggestion uses global variables '
nosync_insn' and 'nosync_addr' to record the altered instruction. But from the testing I found for single static key, usually it will change for multiple address at once.So this might cause the issue is: CPU(a) will loop to alter instructions for different address (sometimes the opcode also is different for different address), at the meantime, CPU(b) hits an altered instruction and handle exception for the breakpoint, if CPU(a) is continuing to alter instruction for the next address, thne CPU(a) might wrongly to use the value from '
nosync_insn' and 'nosync_addr'.
Quoted 2 lines
Simply to say, we cannot only support single nosync instruction but need to support multiple nosync instructions in the loop.
On x86 all actual text poking is serialized by text_mutex.
Quoted 11 lines
- The third issue is might be nested issue. E.g. for the injected break instruction, we have no method to pass perf event for this instruction; and if we connect with the first issue, if the instruction is single stepped with slot (the slot is allocated with
get_insn_slot()), we cannot to allow the perf user space to know the instructions which are executed in the slots.I am not sure if I think too complex here. But seems to me, we inject more instructions for poke text event, and if these injected instructions are executed but the userspace decoder has no way to pass the related info.
That's a misunderstanding, the text_poke event is a side-band event and as such delivered to all events that expressed interest in it. You don't need any exception to event mapping yourself.
Quoted 4 lines
Just clarify, I am fine for merging this patch set, but we might consider more what's the best way on Arm64. Welcome any public comments and suggestions; I will sync internally for how to follow up this functionality.
Thanks!
As a disclaimer: I'm having a hard time understanding what this thread is about.
Quoted 6 lines · Peter Zijlstra
Quoted 4 lines · Leo Yan
I will update some status for prototype (the code has been uploaded into git server [1]) and found some issues for text poke perf event on arm64. These issues are mainly related with arch64 architecture.
Quoted 8 lines
Quoted 5 lines
- The first issue is for the nosync instruction emulation. On arm64, some instructions need to be single stepped and some instructions is emulated. Especially, after I read the code for kprobe implementation on Arm64. So the main idea for prototyping is to use the almos same method with kprobe for nosync instruction.
This makes no sense to me what so ever. What actual instructions are patched with
_nosync()? ftrace/jump_label only use 'NOP/JMP/CALL'
'_nosync()' can be used to patch the following instructions:
B (JMP) BL (CALL) BRK (Breakpoint) SVC, HVC, SMC (System calls) NOP ISB (Pipe flush)
Our kprobes implementation prefers to single-step the instruction in an XOL buffer (we should do this by placing a BRK at the end of the buffer, but we currently use hardware step which is overkill imo). For instructions that are performing a PC-relative operation (e.g. an immediate branch), we can't run from the XOL buffer because we'd go to the wrong place; these instructions are therefore emulated in software...
Quoted 3 lines
For NOP you can advance
regs->ip, for JMP you can adjustregs->ip, for CALL you adjustregs->ipandregs->r14(IIUC you do something like:regs->r14= regs->ip+4;regs->ip= func;)
... in a manner similar to what you describe here. See simulate_b_bl(), although it's thankfully simpler than what x86 seems to have to do. This approach means we can avoid emulating the vast majority of the instruction set.
Quoted 3 lines
(FWIW emulating CALL on x86 is fun because we get to PUSH something on the stack, let me know if you want to see the patches that were required to make that happen :-)
No thanks ;)
Quoted 17 lines
Quoted 17 lines
- The second issue is race condition between the CPU alters instructions and other CPUs hit the altered instructions (and breakpointed).
Peter's suggestion uses global variables '
nosync_insn' and 'nosync_addr' to record the altered instruction. But from the testing I found for single static key, usually it will change for multiple address at once.So this might cause the issue is: CPU(a) will loop to alter instructions for different address (sometimes the opcode also is different for different address), at the meantime, CPU(b) hits an altered instruction and handle exception for the breakpoint, if CPU(a) is continuing to alter instruction for the next address, thne CPU(a) might wrongly to use the value from '
nosync_insn' and 'nosync_addr'.
Quoted 4 lines
Quoted 2 lines
Simply to say, we cannot only support single nosync instruction but need to support multiple nosync instructions in the loop.
On x86 all actual text poking is serialized by
text_mutex.
On arm64, we patch a *lot* and I think stop_machine() is the only safe choice for things like the alternatives, where we patch all sorts of things (cache maintenance, I/O accessors, atomics (until recently), user accessors. Even then, we have to provide our own synchronisation in __apply_alternatives_multi_stop() and use special cache-flushing (clean_dcache_range_nopatch()) to avoid the possibility of running the code that we're modifying. Outside of the alternatives, I'd still be wary about recursive debug exceptions if we were to patch jump labels with breakpoints.
Backing up though, I think I'm missing details about what this thread is trying to achieve. You're adding perf events so that coresight trace can take into account modifications of the kernel text, right? If so:
* Does this need to take into account more than just jump_label()?
* What consistency guarantees are needed by userspace? It's not clear to me how it correlates the new event with execution on other CPUs. Is this using timestamps or something else?
* What about module loading?
Finally, the whole point of the '_nosync()' stuff is that we don't have to synchronise. Therefore, there is a window where you really can't tell whether the instruction has been updated from the perspective of another CPU.
Will
Peter Zijlstra
maintainerQuoted 3 lines · Will Deacon
Backing up though, I think I'm missing details about what this thread is trying to achieve. You're adding perf events so that coresight trace can take into account modifications of the kernel text, right?
Yes, because ARM-CS / Intel-PT need to know the exact text at any one time in order to correctly decode their traces.
Quoted 3 lines
If so:
* Does this need to take into account more than just
jump_label()?
jump_label seems to be the initial target Adrian set, but yes, it needs to cover 'everything'.
That is, all single instruction patching crud around:
- optimized kprobes
(regular kprobes are exempt because they rely on exceptions)
- jump_labels
- static_call (still pending but still)
- ftrace fentry
We also need a solution for whole new chunks of text:
- modules
- ftrace trampolines
- optprobe trampolines
- JIT stuff
but that is so far not included; I had some ideas about a /dev/mem based interface that would report new ranges and wait for acks (from open file-desc) before freeing them.
Quoted 3 lines
* What consistency guarantees are needed by userspace? It's not clear to me how it correlates the new event with execution on other CPUs. Is this using timestamps or something else?
Something else :-)
Both CS/PT basically have a bit-stream encoding of taken / not-taken decisions. To decode they read the text until a conditional branch-point, then consume one bit from the stream and continue.
(note how unconditional branches -- jump_labels -- are expected to be correct in this scheme)
This means they need an exact representation of the text to be able to accurately decode.
This means timestamps or PRE/POST modify events are not acceptible. Because until the I-FLUSH has completed we do not know which CPU has which version of the instruction.
Instead we rely on exceptions; exceptions are differently encoded in the CS/PT data streams.
The scheme used is:
- overwrite target instruction with an exception (INT3 on x86, BRK on arm)
- sync (IPI broadcast CPUID or I-FLUSH completion)
at this point we know the instruction _will_ trap and CS/PT can observe this alternate flow. That is, the exception handler will emulate the instruction.
- emit the TEXT_POKE event with both the old and new instruction included
- overwrite the target instruction with the new instruction
- sync
at this point the new instruction should be valid.
Using this scheme we can at all times follow the instruction flow. Either it is an exception and the exception encoding helps us navigate, or, on either size, we'll know the old/new instruction.
Quoted 1 line
* What about module loading?
I mentioned that above; that is still pending.
Quoted 4 lines
Finally, the whole point of the '
_nosync()' stuff is that we don't have to synchronise. Therefore, there is a window where you really can't tell whether the instruction has been updated from the perspective of another CPU.
Right, and much of that is preserved with the above scheme, nowhere do you have to wait/disturb other CPUs, except for the I-FLUSH completion.
Does this help?
Quoted 9 lines · Peter Zijlstra
Quoted 3 lines · Will Deacon
Backing up though, I think I'm missing details about what this thread is trying to achieve. You're adding perf events so that coresight trace can take into account modifications of the kernel text, right?
Yes, because ARM-CS / Intel-PT need to know the exact text at any one time in order to correctly decode their traces.
Quoted 6 lines
Quoted 3 lines
If so:
* Does this need to take into account more than just
jump_label()?
jump_labelseems to be the initial target Adrian set, but yes, it needs to cover 'everything'.
Including alternatives, which are what get me worried since the potential for recursion is pretty high there (on arm64, at least).
Quoted 10 lines
That is, all single instruction patching crud around:
- optimized kprobes
(regular kprobes are exempt because they rely on exceptions)
-jump_labels
-static_call(still pending but still)
- ftrace fentryWe also need a solution for whole new chunks of text:
Quoted 8 lines
- modules
- ftrace trampolines
- optprobe trampolines
- JIT stuffbut that is so far not included; I had some ideas about a
/dev/membased interface that would report new ranges and wait for acks (from open file-desc) before freeing them.
I think that it would be nice not to end up with two very different interfaces for this. But I see this is still just an RFC, so maybe the full picture will emerge once we solve more of these use-cases.
Quoted 20 lines
Quoted 3 lines
* What consistency guarantees are needed by userspace? It's not clear to me how it correlates the new event with execution on other CPUs. Is this using timestamps or something else?
Something else :-)
Both CS/PT basically have a bit-stream encoding of taken / not-taken decisions. To decode they read the text until a conditional branch-point, then consume one bit from the stream and continue.
(note how unconditional branches --
jump_labels-- are expected to be correct in this scheme)This means they need an exact representation of the text to be able to accurately decode.
This means timestamps or PRE/POST modify events are not acceptible. Because until the I-FLUSH has completed we do not know which CPU has which version of the instruction.
Quoted 7 lines
Instead we rely on exceptions; exceptions are differently encoded in the CS/PT data streams.
The scheme used is:
- overwrite target instruction with an exception (INT3 on x86, BRK on arm)
- sync (IPI broadcast CPUID or I-FLUSH completion)
Hmm. Wouldn't this sync also need to drain the trace buffers for all other CPUs so that we make sure that the upcoming TEXT_POKE event occurs after all prior trace data, which could've been from before the breakpoint was installed?
Quoted 25 lines
at this point we know the instruction _will_ trap and CS/PT can observe this alternate flow. That is, the exception handler will emulate the instruction.
- emit the
TEXT_POKEevent with both the old and new instruction included- overwrite the target instruction with the new instruction
- syncat this point the new instruction should be valid.
Using this scheme we can at all times follow the instruction flow. Either it is an exception and the exception encoding helps us navigate, or, on either size, we'll know the old/new instruction.
Quoted 1 line
* What about module loading?
I mentioned that above; that is still pending.
Quoted 4 lines
Finally, the whole point of the '
_nosync()' stuff is that we don't have to synchronise. Therefore, there is a window where you really can't tell whether the instruction has been updated from the perspective of another CPU.
Quoted 4 lines
Right, and much of that is preserved with the above scheme, nowhere do you have to wait/disturb other CPUs, except for the I-FLUSH completion.
Does this help?
Yes, thank you for explaining it!
Will
Peter Zijlstra
maintainerQuoted 12 lines · Will Deacon
Quoted 11 lines · Peter Zijlstra
Quoted 3 lines · Will Deacon
Backing up though, I think I'm missing details about what this thread is trying to achieve. You're adding perf events so that coresight trace can take into account modifications of the kernel text, right?
Yes, because ARM-CS / Intel-PT need to know the exact text at any one time in order to correctly decode their traces.
Quoted 2 lines
If so:
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
* Does this need to take into account more than just
jump_label()?
jump_labelseems to be the initial target Adrian set, but yes, it needs to cover 'everything'.Including alternatives, which are what get me worried since the potential for recursion is pretty high there (on arm64, at least).
So I had not considered alternatives because they're typically ran once at boot (and module load) and never seen again. That would make them just part of loading new text.
But you mentioned wanting to run them at hotplug time... which is more 'interresting'.
Quoted 15 lines
Quoted 15 lines
That is, all single instruction patching crud around:
- optimized kprobes
(regular kprobes are exempt because they rely on exceptions)
-jump_labels
-static_call(still pending but still)
- ftrace fentryWe also need a solution for whole new chunks of text:
- modules
- ftrace trampolines
- optprobe trampolines
- JIT stuff
Quoted 7 lines
Quoted 3 lines
but that is so far not included; I had some ideas about a
/dev/membased interface that would report new ranges and wait for acks (from open file-desc) before freeing them.I think that it would be nice not to end up with two very different interfaces for this. But I see this is still just an RFC, so maybe the full picture will emerge once we solve more of these use-cases.
The general distinction is between new text mappings and changing them once they exist.
In general a text mapping is large and doesn't change (much). Once you get an event it exist you can copy it out at your convenience, all you really need to make sure of it that it doesn't dissapear before you've completed your copy.
OTOH dynamic text like jump_labels can happen quite frequently and we cannot wait for all observers to have observed/copied the new state before we allow changing it again -- ie. we need a buffered event.
So we don't want to stick whole new text things into a buffer (a module might be larger than the buffer) but we cannot be lazy with text updates.
That is, yes it sucks, but these are two different cases.
Quoted 5 lines
Quoted 5 lines
Instead we rely on exceptions; exceptions are differently encoded in the CS/PT data streams.
The scheme used is:
Quoted 7 lines
Quoted 2 lines
- overwrite target instruction with an exception (INT3 on x86, BRK on arm)
- sync (IPI broadcast CPUID or I-FLUSH completion)Hmm. Wouldn't this sync also need to drain the trace buffers for all other CPUs so that we make sure that the upcoming
TEXT_POKEevent occurs after all prior trace data, which could've been from before the breakpoint was installed?
All we need to ensure is that the breakpoint is visible before the event. That way we know that before the event we have the old instruction, after the event we have the new instruction, and any ambiguity must be resolved with exception packets.
That is, if there is concurrency, the trace buffer will be 'flushed' by the exception. If there is no concurrency, we don't care and can assume old/new depending on timestamps relative to the event.
Quoted 15 lines
Quoted 15 lines
at this point we know the instruction _will_ trap and CS/PT can observe this alternate flow. That is, the exception handler will emulate the instruction.
- emit the
TEXT_POKEevent with both the old and new instruction included- overwrite the target instruction with the new instruction
- syncat this point the new instruction should be valid.
Using this scheme we can at all times follow the instruction flow. Either it is an exception and the exception encoding helps us navigate, or, on either size, we'll know the old/new instruction.
A message that is not in the archive (sent off-list or not yet received): CAJ9a7VgZH7g=rFDpKf=FzEcyBVLS_WjqbrqtRnjOi7WOY4st+w@mail.gmail.com
Peter Zijlstra
maintainerQuoted 14 lines · Mike Leach
Quoted 11 lines · Leo Yan
Quoted 7 lines
Since all instructions (with the possible exception of RET) are unconditional branch instructions: NOP, JMP, CALL. It makes no read difference to the argument below.
( I'm thinking RET might be special in that it reads the return address from the stack and therefore must emit the whole IP into the stream, as we cannot know the stack state )
To be honest, I don't have knowledge what's the exactly format for 'ret' in CoreSight trace; so would like to leave this to Mike.
Quoted 8 lines
For ETM trace we do not have to output the entire address into he stream if:
- address compression allows us to emit only the changed ls bit from the last address.
- the address is identical to one of the last three addresses emitted ( we just emit a ‘same address encoding’
- we are using return stack compression and the address is top of the return stack (we emit nothing and the decoder gets the address from its own mode, of the return stack)
Cute. I don't actually know what PT does, but I figured there had to at least be the option to provide more information.
Peter Zijlstra
maintainerQuoted 43 lines · Adrian Hunter
Record changes to kernel text (i.e. self-modifying code) in order to support tracers like Intel PT decoding through jump labels.
A copy of the running kernel code is needed as a reference point (e.g. from
/proc/kcore). The text poke event records the old bytes and the new bytes so that the event can be processed forwards or backwards.The text poke event has 'flags' to describe when the event is sent. At present the only flag is
PERF_TEXT_POKE_UPDATEwhich indicates the point at which tools should update their reference kernel executable to change the old bytes to the new bytes.Other architectures may wish to emit a pair of text poke events. One before and one after a text poke. In that case,
PERF_TEXT_POKE_UPDATEflag would be set on only one of the pair.In the case of Intel PT tracing, the executable code must be walked to reconstruct the control flow. For x86 a jump label text poke begins:
- write INT3 byte
- IPI-SYNC
At this point the actual control flow will be through the INT3 and handler and not hit the old or new instruction. Intel PT outputs FUP/TIP packets for the INT3, so the flow can still be decoded. Subsequently:
- emitRECORD_TEXT_POKEwith the new instruction
- write instruction tail
- IPI-SYNC
- write first byte
- IPI-SYNC
So before the text poke event timestamp, the decoder will see either the old instruction flow or FUP/TIP of INT3. After the text poke event timestamp, the decoder will see either the new instruction flow or FUP/TIP of INT3. Thus decoders can use the timestamp as the point at which to modify the executable code.Signed-off-by Adrian Hunter <[email protected]>include/uapi/linux/perf_event.h +3
-0@@ -1000,6 +1001,26 @@ enum perf_event_type {*/ PERF_RECORD_BPF_EVENT = 18, +/* +* Records changes to kernel text i.e. self-modified code. +* 'flags' has PERF_TEXT_POKE_UPDATE (i.e. bit 0) set to
include/uapi/linux/perf_event.h enum perf_event_type { · line 1016
Quoted 10 lines
include/uapi/linux/perf_event.h +10
-0+* indicate to tools to update old bytes to new bytes in the +* executable image. +* 'len' is the number of old bytes, which is the same as the number +* of new bytes. 'bytes' contains the old bytes followed immediately +* by the new bytes. +* +* struct { +* struct perf_event_header header; +* u64 addr; +* u16 flags;
What's the purpose of the 'flags' field? We currently only have the 1 flag defined, but what possible other flags are you thinking of?
Quoted 25 lines
include/uapi/linux/perf_event.h +13
-0+* u16 len; +* u8 bytes[]; +* struct sample_id sample_id; +* }; +*/ +PERF_RECORD_TEXT_POKE =19 , + PERF_RECORD_MAX, /* non-ABI */ };@@ -1041,6 +1062,11 @@ enum perf_callchain_context {#define PERF_AUX_FLAG_PARTIAL 0x04/* record contains gaps */ #define PERF_AUX_FLAG_COLLISION 0x08/* sample collided with another */ +/** +* PERF_RECORD_TEXT_POKE::flags bits +*/ +#define PERF_TEXT_POKE_UPDATE 0x01/* update old bytes to new bytes */ +#define PERF_FLAG_FD_NO_GROUP (1UL << 0)#define PERF_FLAG_FD_OUTPUT (1UL << 1)#define PERF_FLAG_PID_CGROUP (1UL << 2)/* pid=cgroup id, per-cpu mode only */ +void perf_event_text_poke(unsigned long addr, u16 flags, const void *old_bytes,
include/uapi/linux/perf_event.h enum perf_callchain_context {
Quoted 10 lines
include/uapi/linux/perf_event.h +10
-0+ const void *new_bytes, size_t len) +{ +struct perf_text_poke_event text_poke_event; +size_t tot, pad; + +if (!atomic_read (&nr_text_poke_events)) +return ; + + tot =sizeof (text_poke_event.flags) + +sizeof (text_poke_event.len) + (len <<1 );
Maybe use: 'len * 2', compiler should be smart enough.
Quoted 10 lines
kernel/events/core.c +10
-0+ pad =ALIGN (tot,sizeof (u64)) - tot; + + text_poke_event = (struct perf_text_poke_event ){ + .old_bytes = old_bytes, + .new_bytes = new_bytes, + .pad = pad, + .flags = flags, + .len = len, + .event_id = { + .header = {
kernel/events/core.c void perf_event_bpf_event(struct bpf_prog *prog, · line 8157
Quoted 10 lines
kernel/events/core.c +10
-0+ .type = PERF_RECORD_TEXT_POKE, + .misc = PERF_RECORD_MISC_KERNEL, + .size = sizeof(text_poke_event.event_id) + tot + pad, + }, + .addr = addr, + }, + }; + +perf_iterate_sb (perf_event_text_poke_output, &text_poke_event,NULL ); +}
Also, I'm thinking this patch has a problem witharch_unoptimize_kprobe() as it stands today.
I've got a patch pending for that:
https://lkml.kernel.org/r/[email protected]
That rewrites that a little, but it will be slightly differently broken after that.
The problem is that while we can (and do) ignore regular kprobes (they use text_poke() which isn't instrumented, and they don't need to be, because they're always going through INT3), we cannot ignore optprobes.
Installing the optprobe works as expected, but unoptimizing is difficult and as it stands your patch will not report the old text (you'll see that first byte overwriten with 0xCC) and after my patch you'll not see it at all.
Now, I suppose we can stick an explicit perf_event_text_poke() in there after my patch.
We have to delay this patch until after my x86/ftrace rewrite anyway, because otherwise ftrace is hidden too.
And even then, this will then notify us of all text modifications, but what about all the extra text, like ftrace trampolines, k(ret)probe trampolines, optprobe slots, bpf-jit, etc.?
Adrian Hunter
reviewerQuoted 41 lines · Peter Zijlstra
Quoted 40 lines · Adrian Hunter
Record changes to kernel text (i.e. self-modifying code) in order to support tracers like Intel PT decoding through jump labels.
A copy of the running kernel code is needed as a reference point (e.g. from
/proc/kcore). The text poke event records the old bytes and the new bytes so that the event can be processed forwards or backwards.The text poke event has 'flags' to describe when the event is sent. At present the only flag is
PERF_TEXT_POKE_UPDATEwhich indicates the point at which tools should update their reference kernel executable to change the old bytes to the new bytes.Other architectures may wish to emit a pair of text poke events. One before and one after a text poke. In that case,
PERF_TEXT_POKE_UPDATEflag would be set on only one of the pair.In the case of Intel PT tracing, the executable code must be walked to reconstruct the control flow. For x86 a jump label text poke begins:
- write INT3 byte
- IPI-SYNC
At this point the actual control flow will be through the INT3 and handler and not hit the old or new instruction. Intel PT outputs FUP/TIP packets for the INT3, so the flow can still be decoded. Subsequently:
- emitRECORD_TEXT_POKEwith the new instruction
- write instruction tail
- IPI-SYNC
- write first byte
- IPI-SYNC
So before the text poke event timestamp, the decoder will see either the old instruction flow or FUP/TIP of INT3. After the text poke event timestamp, the decoder will see either the new instruction flow or FUP/TIP of INT3. Thus decoders can use the timestamp as the point at which to modify the executable code.Signed-off-by Adrian Hunter <[email protected]>include/uapi/linux/perf_event.h +0
-0@@ -1000,6 +1001,26 @@ enum perf_event_type {*/ PERF_RECORD_BPF_EVENT = 18,
Quoted 16 lines
include/uapi/linux/perf_event.h enum perf_event_type { · line 1016
Quoted 13 lines
include/uapi/linux/perf_event.h +13
-0+/* +* Records changes to kernel text i.e. self-modified code. +* 'flags' has PERF_TEXT_POKE_UPDATE (i.e. bit 0) set to +* indicate to tools to update old bytes to new bytes in the +* executable image. +* 'len' is the number of old bytes, which is the same as the number +* of new bytes. 'bytes' contains the old bytes followed immediately +* by the new bytes. +* +* struct { +* struct perf_event_header header; +* u64 addr; +* u16 flags; What's the purpose of the 'flags' field? We currently only have the 1 flag defined, but what possible other flags are you thinking of?
That was to allow for identifying 'before' and 'after' text poke events.
Quoted 82 lines
Quoted 26 lines
include/uapi/linux/perf_event.h +13
-0+* u16 len; +* u8 bytes[]; +* struct sample_id sample_id; +* }; +*/ +PERF_RECORD_TEXT_POKE =19 , + PERF_RECORD_MAX, /* non-ABI */ };@@ -1041,6 +1062,11 @@ enum perf_callchain_context {#define PERF_AUX_FLAG_PARTIAL 0x04/* record contains gaps */ #define PERF_AUX_FLAG_COLLISION 0x08/* sample collided with another */ +/** +* PERF_RECORD_TEXT_POKE::flags bits +*/ +#define PERF_TEXT_POKE_UPDATE 0x01/* update old bytes to new bytes */ +#define PERF_FLAG_FD_NO_GROUP (1UL << 0)#define PERF_FLAG_FD_OUTPUT (1UL << 1)#define PERF_FLAG_PID_CGROUP (1UL << 2)/* pid=cgroup id, per-cpu mode only */ +void perf_event_text_poke(unsigned long addr, u16 flags, const void *old_bytes,include/uapi/linux/perf_event.h enum perf_callchain_context {
Quoted 10 lines
include/uapi/linux/perf_event.h +10
-0+ const void *new_bytes, size_t len) +{ +struct perf_text_poke_event text_poke_event; +size_t tot, pad; + +if (!atomic_read (&nr_text_poke_events)) +return ; + + tot =sizeof (text_poke_event.flags) + +sizeof (text_poke_event.len) + (len <<1 );Maybe use: 'len * 2', compiler should be smart enough.
Quoted 10 lines
kernel/events/core.c +10
-0+ pad =ALIGN (tot,sizeof (u64)) - tot; + + text_poke_event = (struct perf_text_poke_event ){ + .old_bytes = old_bytes, + .new_bytes = new_bytes, + .pad = pad, + .flags = flags, + .len = len, + .event_id = { + .header = {kernel/events/core.c void perf_event_bpf_event(struct bpf_prog *prog, · line 8157
Quoted 10 lines
kernel/events/core.c +10
-0+ .type = PERF_RECORD_TEXT_POKE, + .misc = PERF_RECORD_MISC_KERNEL, + .size = sizeof(text_poke_event.event_id) + tot + pad, + }, + .addr = addr, + }, + }; + +perf_iterate_sb (perf_event_text_poke_output, &text_poke_event,NULL ); +}Also, I'm thinking this patch has a problem with
arch_unoptimize_kprobe()as it stands today.I've got a patch pending for that:
https://lkml.kernel.org/r/[email protected]
That rewrites that a little, but it will be slightly differently broken after that.
The problem is that while we can (and do) ignore regular kprobes (they use
text_poke()which isn't instrumented, and they don't need to be, because they're always going through INT3), we cannot ignore optprobes.Installing the optprobe works as expected, but unoptimizing is difficult and as it stands your patch will not report the old text (you'll see that first byte overwriten with
0xCC) and after my patch you'll not see it at all.Now, I suppose we can stick an explicit
perf_event_text_poke()in there after my patch.
Quoted 6 lines
We have to delay this patch until after my x86/ftrace rewrite anyway, because otherwise ftrace is hidden too.
And even then, this will then notify us of all text modifications, but what about all the extra text, like ftrace trampolines, k(ret)probe trampolines, optprobe slots, bpf-jit, etc.?
Generally jump labels are used by default, whereas ftrace and probes are more under the control of the user. So just having jump labels done is the main thing.
The tools have some support for copying out bpf-jit, although I am not sure it guarantees to get short-lived code.
Adrian Hunter
reviewerRFC PATCH 2/6 perf dso: Refactor dso_cache__read()
In reply to Adrian HunterRefactor dso_cache__read() to separate populating the cache from copying data from it. This is preparation for adding a cache "write" that will update the data in the cache.
--- tools/perf/util/dso.c | 64 +++++++++++++++++++++++++------------------ 1 file changed, 37 insertions(+), 27 deletions(-)
tools/perf/util/dso.c +37 -27
@@ -768,7 +768,7 @@ dso_cache__free(struct dso *dso)pthread_mutex_unlock (&dso->lock); } -static struct dso_cache *dso_cache__find (struct dso *dso,u64 offset) +static struct dso_cache *__dso_cache__find (struct dso *dso,u64 offset) {const struct rb_root *root = &dso->data.cache;struct rb_node *const *p = &root->rb_node;
@@ -863,54 +863,64 @@ static ssize_t file_read(struct dso *dso, struct machine *machine,return ret; } -static ssize_t -dso_cache__read (struct dso *dso,struct machine *machine, -u64 offset,u8 *data,ssize_t size) +static struct dso_cache *dso_cache__populate (struct dso *dso, +struct machine *machine, +u64 offset,ssize_t *ret) {u64 cache_offset = offset &DSO__DATA_CACHE_MASK ;struct dso_cache *cache;struct dso_cache *old; -ssize_t ret; cache =zalloc (sizeof (*cache) +DSO__DATA_CACHE_SIZE ); -if (!cache) -return -ENOMEM ; +if (!cache) { + *ret = -ENOMEM ; +return NULL ; + }if (dso->binary_type ==DSO_BINARY_TYPE__BPF_PROG_INFO ) - ret =bpf_read (dso, cache_offset, cache->data); + *ret =bpf_read (dso, cache_offset, cache->data);else - ret =file_read (dso, machine, cache_offset, cache->data); + *ret =file_read (dso, machine, cache_offset, cache->data); -if (ret >0 ) { - cache->offset = cache_offset; - cache->size = ret; +if (*ret <=0 ) { +free (cache); +return NULL ; + } - old =dso_cache__insert (dso, cache); -if (old) { -/* we lose the race */ -free (cache); - cache = old; - } + cache->offset = cache_offset; + cache->size = *ret; - ret =dso_cache__memcpy (cache, offset, data, size); + old =dso_cache__insert (dso, cache); +if (old) { +/* we lose the race */ +free (cache); + cache = old; } -if (ret <=0 ) -free (cache); +return cache; +} -return ret; +static struct dso_cache *dso_cache__find (struct dso *dso, +struct machine *machine, +u64 offset, +ssize_t *ret) +{ +struct dso_cache *cache =__dso_cache__find (dso, offset); + +return cache ? cache :dso_cache__populate (dso, machine, offset, ret); }static ssize_t dso_cache_read (struct dso *dso,struct machine *machine,u64 offset,u8 *data,ssize_t size) {struct dso_cache *cache; +ssize_t ret =0 ; - cache =dso_cache__find (dso, offset); -if (cache) -return dso_cache__memcpy (cache, offset, data, size); -else -return dso_cache__read (dso, machine, offset, data, size); + cache =dso_cache__find (dso, machine, offset, &ret); +if (!cache) +return ret; + +return dso_cache__memcpy (cache, offset, data, size); }/*
Signature · 1 line
2.17.1
Re: [PATCH RFC 2/6] perf dso: Refactor dso_cache__read()
Em Fri, Oct 25, 2019 at 03:59:56PM +0300, Adrian Hunter escreveu:
Quoted 3 lines
Refactor
dso_cache__read()to separate populating the cache from copying data from it. This is preparation for adding a cache "write" that will update the data in the cache.
Ditto for 2/6 and 3/6, i.e. applying them now.
- Arnaldo
Quoted 111 lines
Signed-off-by Adrian Hunter <[email protected]>--- tools/perf/util/dso.c | 64 +++++++++++++++++++++++++------------------ 1 file changed, 37 insertions(+), 27 deletions(-)tools/perf/util/dso.c +37
-27@@ -768,7 +768,7 @@ dso_cache__free(struct dso *dso)pthread_mutex_unlock (&dso->lock); } -static struct dso_cache *dso_cache__find (struct dso *dso,u64 offset) +static struct dso_cache *__dso_cache__find (struct dso *dso,u64 offset) {const struct rb_root *root = &dso->data.cache;struct rb_node *const *p = &root->rb_node;@@ -863,54 +863,64 @@ static ssize_t file_read(struct dso *dso, struct machine *machine,return ret; } -static ssize_t -dso_cache__read (struct dso *dso,struct machine *machine, -u64 offset,u8 *data,ssize_t size) +static struct dso_cache *dso_cache__populate (struct dso *dso, +struct machine *machine, +u64 offset,ssize_t *ret) {u64 cache_offset = offset &DSO__DATA_CACHE_MASK ;struct dso_cache *cache;struct dso_cache *old; -ssize_t ret; cache =zalloc (sizeof (*cache) +DSO__DATA_CACHE_SIZE ); -if (!cache) -return -ENOMEM ; +if (!cache) { + *ret = -ENOMEM ; +return NULL ; + }if (dso->binary_type ==DSO_BINARY_TYPE__BPF_PROG_INFO ) - ret =bpf_read (dso, cache_offset, cache->data); + *ret =bpf_read (dso, cache_offset, cache->data);else - ret =file_read (dso, machine, cache_offset, cache->data); + *ret =file_read (dso, machine, cache_offset, cache->data); -if (ret >0 ) { - cache->offset = cache_offset; - cache->size = ret; +if (*ret <=0 ) { +free (cache); +return NULL ; + } - old =dso_cache__insert (dso, cache); -if (old) { -/* we lose the race */ -free (cache); - cache = old; - } + cache->offset = cache_offset; + cache->size = *ret; - ret =dso_cache__memcpy (cache, offset, data, size); + old =dso_cache__insert (dso, cache); +if (old) { +/* we lose the race */ +free (cache); + cache = old; } -if (ret <=0 ) -free (cache); +return cache; +} -return ret; +static struct dso_cache *dso_cache__find (struct dso *dso, +struct machine *machine, +u64 offset, +ssize_t *ret) +{ +struct dso_cache *cache =__dso_cache__find (dso, offset); + +return cache ? cache :dso_cache__populate (dso, machine, offset, ret); }static ssize_t dso_cache_read (struct dso *dso,struct machine *machine,u64 offset,u8 *data,ssize_t size) {struct dso_cache *cache; +ssize_t ret =0 ; - cache =dso_cache__find (dso, offset); -if (cache) -return dso_cache__memcpy (cache, offset, data, size); -else -return dso_cache__read (dso, machine, offset, data, size); + cache =dso_cache__find (dso, machine, offset, &ret); +if (!cache) +return ret; + +return dso_cache__memcpy (cache, offset, data, size); }/* Signature · 1 line
2.17.1
Signature · 1 line
- Arnaldo
Re: [PATCH RFC 2/6] perf dso: Refactor dso_cache__read()
In reply to Adrian HunterOn Fri, Oct 25, 2019 at 03:59:56PM +0300, Adrian Hunter wrote:
SNIP
Quoted 17 lines
tools/perf/util/dso.c +9
-1+} -return ret; +static struct dso_cache *dso_cache__find (struct dso *dso, +struct machine *machine, +u64 offset, +ssize_t *ret) +{ +struct dso_cache *cache =__dso_cache__find (dso, offset); + +return cache ? cache :dso_cache__populate (dso, machine, offset, ret); }static ssize_t dso_cache_read (struct dso *dso,struct machine *machine,u64 offset,u8 *data,ssize_t size) {struct dso_cache *cache;
tools/perf/util/dso.c static ssize_t file_read(struct dso *dso, struct machine *machine, · line 921
Quoted 10 lines
tools/perf/util/dso.c +4
-5+ssize_t ret =0 ; - cache =dso_cache__find (dso, offset); -if (cache) -return dso_cache__memcpy (cache, offset, data, size); -else -return dso_cache__read (dso, machine, offset, data, size); + cache =dso_cache__find (dso, machine, offset, &ret); +if (!cache) +return ret;
why not use the ERR_* macros to get error through the pointer instead of adding extra argument?
jirka
Quoted 8 lines
tools/perf/util/dso.c +2
-0+ +return dso_cache__memcpy (cache, offset, data, size); }/* Signature · 1 line
2.17.1
Adrian Hunter
reviewerRe: [PATCH RFC 2/6] perf dso: Refactor dso_cache__read()
Quoted 23 lines · Jiri Olsa
On Fri, Oct 25, 2019 at 03:59:56PM +0300, Adrian Hunter wrote:
SNIP
Quoted 19 lines
(quoted diff) +10
-1+} - return ret; +static struct dso_cache *dso_cache__find(struct dso *dso, + struct machine *machine, + u64 offset, + ssize_t *ret) +{ + struct dso_cache *cache = __dso_cache__find(dso, offset); + + return cache ? cache : dso_cache__populate(dso, machine, offset, ret); } static ssize_t dso_cache_read(struct dso *dso, struct machine *machine, u64 offset, u8 *data, ssize_t size) { struct dso_cache *cache; + ssize_t ret = 0;
Quoted 12 lines
Review
Quoted 8 lines
(quoted diff) +3
-5- cache = dso_cache__find(dso, offset); - if (cache) - return dso_cache__memcpy(cache, offset, data, size); - else - return dso_cache__read(dso, machine, offset, data, size); + cache = dso_cache__find(dso, machine, offset, &ret); + if (!cache) + return ret;why not use the ERR_* macros to get error through the pointer instead of adding extra argument?
OK, here's the diff for that:
tools/perf/util/dso.c +15 -18
@@ -3,6 +3,7 @@#include <linux/kernel.h> #include <linux/string.h> #include <linux/zalloc.h> +#include <linux/err.h> #include <sys/time.h> #include <sys/resource.h> #include <sys/types.h>
@@ -865,30 +866,29 @@ static ssize_t file_read(struct dso *dso, struct machine *machine,static struct dso_cache *dso_cache__populate (struct dso *dso,struct machine *machine, -u64 offset,ssize_t *ret) +u64 offset) {u64 cache_offset = offset &DSO__DATA_CACHE_MASK ;struct dso_cache *cache;struct dso_cache *old; +ssize_t ret; cache =zalloc (sizeof (*cache) +DSO__DATA_CACHE_SIZE ); -if (!cache) { - *ret = -ENOMEM ; -return NULL ; - } +if (!cache) +return ERR_PTR (-ENOMEM );if (dso->binary_type ==DSO_BINARY_TYPE__BPF_PROG_INFO ) - *ret =bpf_read (dso, cache_offset, cache->data); + ret =bpf_read (dso, cache_offset, cache->data);else - *ret =file_read (dso, machine, cache_offset, cache->data); + ret =file_read (dso, machine, cache_offset, cache->data); -if (*ret <=0 ) { +if (ret <=0 ) {free (cache); -return NULL ; +return ERR_PTR (ret); } cache->offset = cache_offset; - cache->size = *ret; + cache->size = ret; old =dso_cache__insert (dso, cache);if (old) {
@@ -902,23 +902,20 @@ static struct dso_cache *dso_cache__populate(struct dso *dso,static struct dso_cache *dso_cache__find (struct dso *dso,struct machine *machine, -u64 offset, -ssize_t *ret) +u64 offset) {struct dso_cache *cache =__dso_cache__find (dso, offset); -return cache ? cache :dso_cache__populate (dso, machine, offset, ret); +return cache ? cache :dso_cache__populate (dso, machine, offset); }static ssize_t dso_cache_read (struct dso *dso,struct machine *machine,u64 offset,u8 *data,ssize_t size) { -struct dso_cache *cache; -ssize_t ret =0 ; +struct dso_cache *cache =dso_cache__find (dso, machine, offset); - cache =dso_cache__find (dso, machine, offset, &ret); -if (!cache) -return ret; +if (IS_ERR_OR_NULL (cache)) +return PTR_ERR (cache);return dso_cache__memcpy (cache, offset, data, size); }
[tip: perf/core] perf dso: Refactor dso_cache__read()
In reply to Adrian HunterThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 366df72657e0cd6bd072b56a48e63b8d89718f70 Gitweb: https://git.kernel.org/tip/366df72657e0cd6bd072b56a48e63b8d89718f70 Author: Adrian Hunter <[email protected]> AuthorDate: Fri, 25 Oct 2019 15:59:56 +03:00 Committer: Arnaldo Carvalho de Melo <[email protected]> CommitterDate: Wed, 06 Nov 2019 15:43:06 -03:00
perf dso: Refactor dso_cache__read()
Refactor dso_cache__read() to separate populating the cache from copying data from it. This is preparation for adding a cache "write" that will update the data in the cache.
--- tools/perf/util/dso.c | 64 ++++++++++++++++++++++++------------------ 1 file changed, 37 insertions(+), 27 deletions(-)
tools/perf/util/dso.c +37 -27
@@ -768,7 +768,7 @@ dso_cache__free(struct dso *dso)pthread_mutex_unlock (&dso->lock); } -static struct dso_cache *dso_cache__find (struct dso *dso,u64 offset) +static struct dso_cache *__dso_cache__find (struct dso *dso,u64 offset) {const struct rb_root *root = &dso->data.cache;struct rb_node *const *p = &root->rb_node;
@@ -863,54 +863,64 @@ out:return ret; } -static ssize_t -dso_cache__read (struct dso *dso,struct machine *machine, -u64 offset,u8 *data,ssize_t size) +static struct dso_cache *dso_cache__populate (struct dso *dso, +struct machine *machine, +u64 offset,ssize_t *ret) {u64 cache_offset = offset &DSO__DATA_CACHE_MASK ;struct dso_cache *cache;struct dso_cache *old; -ssize_t ret; cache =zalloc (sizeof (*cache) +DSO__DATA_CACHE_SIZE ); -if (!cache) -return -ENOMEM ; +if (!cache) { + *ret = -ENOMEM ; +return NULL ; + }if (dso->binary_type ==DSO_BINARY_TYPE__BPF_PROG_INFO ) - ret =bpf_read (dso, cache_offset, cache->data); + *ret =bpf_read (dso, cache_offset, cache->data);else - ret =file_read (dso, machine, cache_offset, cache->data); + *ret =file_read (dso, machine, cache_offset, cache->data); -if (ret >0 ) { - cache->offset = cache_offset; - cache->size = ret; +if (*ret <=0 ) { +free (cache); +return NULL ; + } - old =dso_cache__insert (dso, cache); -if (old) { -/* we lose the race */ -free (cache); - cache = old; - } + cache->offset = cache_offset; + cache->size = *ret; - ret =dso_cache__memcpy (cache, offset, data, size); + old =dso_cache__insert (dso, cache); +if (old) { +/* we lose the race */ +free (cache); + cache = old; } -if (ret <=0 ) -free (cache); +return cache; +} -return ret; +static struct dso_cache *dso_cache__find (struct dso *dso, +struct machine *machine, +u64 offset, +ssize_t *ret) +{ +struct dso_cache *cache =__dso_cache__find (dso, offset); + +return cache ? cache :dso_cache__populate (dso, machine, offset, ret); }static ssize_t dso_cache_read (struct dso *dso,struct machine *machine,u64 offset,u8 *data,ssize_t size) {struct dso_cache *cache; +ssize_t ret =0 ; - cache =dso_cache__find (dso, offset); -if (cache) -return dso_cache__memcpy (cache, offset, data, size); -else -return dso_cache__read (dso, machine, offset, data, size); + cache =dso_cache__find (dso, machine, offset, &ret); +if (!cache) +return ret; + +return dso_cache__memcpy (cache, offset, data, size); }/*
Adrian Hunter
reviewerRFC PATCH 3/6 perf dso: Add dso__data_write_cache_addr()
In reply to Adrian HunterAdd functions to write into the dso file data cache, but not change the file itself.
--- tools/perf/util/dso.c | 73 ++++++++++++++++++++++++++++++++++--------- tools/perf/util/dso.h | 7 +++++ 2 files changed, 65 insertions(+), 15 deletions(-)
tools/perf/util/dso.c +58 -15
@@ -827,14 +827,16 @@ dso_cache__insert(struct dso *dso, struct dso_cache *new)return cache; } -static ssize_t -dso_cache__memcpy (struct dso_cache *cache,u64 offset, -u8 *data,u64 size) +static ssize_t dso_cache__memcpy (struct dso_cache *cache,u64 offset,u8 *data, +u64 size,bool out) {u64 cache_offset = offset - cache->offset;u64 cache_size =min (cache->size - cache_offset, size); -memcpy (data, cache->data + cache_offset, cache_size); +if (out) +memcpy (data, cache->data + cache_offset, cache_size); +else +memcpy (cache->data + cache_offset, data, cache_size);return cache_size; }
@@ -910,8 +912,8 @@ static struct dso_cache *dso_cache__find(struct dso *dso,return cache ? cache :dso_cache__populate (dso, machine, offset, ret); } -static ssize_t dso_cache_read (struct dso *dso,struct machine *machine, -u64 offset,u8 *data,ssize_t size) +static ssize_t dso_cache_io (struct dso *dso,struct machine *machine, +u64 offset,u8 *data,ssize_t size,bool out) {struct dso_cache *cache;ssize_t ret =0 ;
@@ -920,16 +922,16 @@ static ssize_t dso_cache_read(struct dso *dso, struct machine *machine,if (!cache)return ret; -return dso_cache__memcpy (cache, offset, data, size); +return dso_cache__memcpy (cache, offset, data, size, out); }/* * Reads and caches dso data DSO__DATA_CACHE_SIZE size chunks * in the rb_tree. Any read to already cached data is served -* by cached data. +* by cached data. Writes update the cache only, not the backing file. */ -static ssize_t cached_read (struct dso *dso,struct machine *machine, -u64 offset,u8 *data,ssize_t size) +static ssize_t cached_io (struct dso *dso,struct machine *machine, +u64 offset,u8 *data,ssize_t size,bool out) {ssize_t r =0 ;u8 *p = data;
@@ -937,7 +939,7 @@ static ssize_t cached_read(struct dso *dso, struct machine *machine, do { ssize_t ret; - ret = dso_cache_read(dso, machine, offset, p, size); + ret = dso_cache_io(dso, machine, offset, p, size, out); if (ret < 0) return ret;
@@ -1021,8 +1023,9 @@ off_t dso__data_size(struct dso *dso, struct machine *machine)return dso->data.file_size; } -static ssize_t data_read_offset (struct dso *dso,struct machine *machine, -u64 offset,u8 *data,ssize_t size) +static ssize_t data_read_write_offset (struct dso *dso,struct machine *machine, +u64 offset,u8 *data,ssize_t size, +bool out) {if (dso__data_file_size (dso, machine))return -1 ;
@@ -1034,7 +1037,7 @@ static ssize_t data_read_offset(struct dso *dso, struct machine *machine,if (offset + size < offset)return -1 ; -return cached_read (dso, machine, offset, data, size); +return cached_io (dso, machine, offset, data, size, out); }/**
@@ -1054,7 +1057,7 @@ ssize_t dso__data_read_offset(struct dso *dso, struct machine *machine,if (dso->data.status ==DSO_DATA_STATUS_ERROR )return -1 ; -return data_read_offset (dso, machine, offset, data, size); +return data_read_write_offset (dso, machine, offset, data, size, true); }/**
@@ -1075,6 +1078,46 @@ ssize_t dso__data_read_addr(struct dso *dso, struct map *map,return dso__data_read_offset (dso, machine, offset, data, size); } +/** +* dso__data_write_cache_offs - Write data to dso data cache at file offset +* @dso: dso object +* @machine: machine object +* @offset: file offset +* @data: buffer to write +* @size: size of the @data buffer +* +* Write into the dso file data cache, but do not change the file itself. +*/ +ssize_t dso__data_write_cache_offs (struct dso *dso,struct machine *machine, +u64 offset,const u8 *data_in,ssize_t size) +{ +u8 *data = (u8 *)data_in;/* cast away const to use same fns for r/w */ + +if (dso->data.status ==DSO_DATA_STATUS_ERROR ) +return -1 ; + +return data_read_write_offset (dso, machine, offset, data, size, false); +} + +/** +* dso__data_write_cache_addr - Write data to dso data cache at dso address +* @dso: dso object +* @machine: machine object +* @add: virtual memory address +* @data: buffer to write +* @size: size of the @data buffer +* +* External interface to write into the dso file data cache, but do not change +* the file itself. +*/ +ssize_t dso__data_write_cache_addr (struct dso *dso,struct map *map, +struct machine *machine,u64 addr, +const u8 *data,ssize_t size) +{ +u64 offset = map->map_ip (map, addr); +return dso__data_write_cache_offs (dso, machine, offset, data, size); +} +struct map *dso__new_map (const char *name) {struct map *map =NULL ;
tools/perf/util/dso.h +7 -0
@@ -285,6 +285,8 @@ void dso__set_module_info(struct dso *dso, struct kmod_path *m,* dso__data_size * dso__data_read_offset * dso__data_read_addr +* dso__data_write_cache_offs +* dso__data_write_cache_addr * * Please refer to the dso.c object code for each function and * arguments documentation. Following text tries to explain the
@@ -332,6 +334,11 @@ ssize_t dso__data_read_addr(struct dso *dso, struct map *map, struct machine *machine, u64 addr, u8 *data, ssize_t size);bool dso__data_status_seen (struct dso *dso,enum dso_data_status_seen by); +ssize_t dso__data_write_cache_offs (struct dso *dso,struct machine *machine, +u64 offset,const u8 *data,ssize_t size); +ssize_t dso__data_write_cache_addr (struct dso *dso,struct map *map, +struct machine *machine,u64 addr, +const u8 *data,ssize_t size);struct map *dso__new_map (const char *name); struct dso *machine__findnew_kernel(struct machine *machine, const char *name,
Signature · 1 line
2.17.1
Re: [PATCH RFC 3/6] perf dso: Add dso__data_write_cache_addr()
On Fri, Oct 25, 2019 at 03:59:57PM +0300, Adrian Hunter wrote:
SNIP
Quoted 10 lines
tools/perf/util/dso.c +3
-2-static ssize_t data_read_offset (struct dso *dso,struct machine *machine, -u64 offset,u8 *data,ssize_t size) +static ssize_t data_read_write_offset (struct dso *dso,struct machine *machine, +u64 offset,u8 *data,ssize_t size, +bool out) {if (dso__data_file_size (dso, machine))return -1 ;
tools/perf/util/dso.c static ssize_t data_read_offset(struct dso *dso, struct machine *machine, · line 1040
Quoted 5 lines
tools/perf/util/dso.c +1
-1@@ -1034,7 +1037,7 @@ static ssize_t data_read_offset(struct dso *dso, struct machine *machine,if (offset + size < offset)return -1 ; -return cached_read (dso, machine, offset, data, size); +return cached_io (dso, machine, offset, data, size, out);
you renamed the function as _read_write_ so the bool choosing the opration should be named either read or write.. I had to go all the way down to find out what 'out' means ;-)
jirka
Adrian Hunter
reviewerRe: [PATCH RFC 3/6] perf dso: Add dso__data_write_cache_addr()
Quoted 16 lines · Jiri Olsa
On Fri, Oct 25, 2019 at 03:59:57PM +0300, Adrian Hunter wrote:
SNIPQuoted 13 lines
(quoted diff) +3
-2-static ssize_t data_read_offset(struct dso *dso, struct machine *machine, - u64 offset, u8 *data, ssize_t size) +static ssize_t data_read_write_offset(struct dso *dso, struct machine *machine, + u64 offset, u8 *data, ssize_t size, + bool out) { if (dso__data_file_size(dso, machine)) return -1;@@ -1034,7 +1037,7 @@ static ssize_t data_read_offset(struct dso *dso, struct machine *machine, if (offset + size < offset) return -1;
Quoted 7 lines
Review static ssize_t data_read_offset(struct dso *dso, struct machine *machine, · line 1040
Quoted 2 lines
(quoted diff) +1
-1- return cached_read(dso, machine, offset, data, size); + return cached_io(dso, machine, offset, data, size, out);you renamed the function as _read_write_ so the bool choosing the opration should be named either read or write.. I had to go all the way down to find out what 'out' means ;-)
Arnaldo already applied it, so here is the diff:
tools/perf/util/dso.c +8 -8
@@ -829,12 +829,12 @@ dso_cache__insert(struct dso *dso, struct dso_cache *new) }static ssize_t dso_cache__memcpy (struct dso_cache *cache,u64 offset,u8 *data, -u64 size,bool out) +u64 size,bool read) {u64 cache_offset = offset - cache->offset;u64 cache_size =min (cache->size - cache_offset, size); -if (out) +if (read)memcpy (data, cache->data + cache_offset, cache_size);else memcpy (cache->data + cache_offset, data, cache_size);
@@ -912,14 +912,14 @@ static struct dso_cache *dso_cache__find(struct dso *dso, }static ssize_t dso_cache_io (struct dso *dso,struct machine *machine, -u64 offset,u8 *data,ssize_t size,bool out) +u64 offset,u8 *data,ssize_t size,bool read) {struct dso_cache *cache =dso_cache__find (dso, machine, offset);if (IS_ERR_OR_NULL (cache))return PTR_ERR (cache); -return dso_cache__memcpy (cache, offset, data, size, out); +return dso_cache__memcpy (cache, offset, data, size, read); }/*
@@ -928,7 +928,7 @@ static ssize_t dso_cache_io(struct dso *dso, struct machine *machine,* by cached data. Writes update the cache only, not the backing file. */ static ssize_t cached_io (struct dso *dso,struct machine *machine, -u64 offset,u8 *data,ssize_t size,bool out) +u64 offset,u8 *data,ssize_t size,bool read) {ssize_t r =0 ;u8 *p = data;
@@ -936,7 +936,7 @@ static ssize_t cached_io(struct dso *dso, struct machine *machine, do { ssize_t ret; - ret = dso_cache_io(dso, machine, offset, p, size, out); + ret = dso_cache_io(dso, machine, offset, p, size, read); if (ret < 0) return ret;
@@ -1022,7 +1022,7 @@ off_t dso__data_size(struct dso *dso, struct machine *machine)static ssize_t data_read_write_offset (struct dso *dso,struct machine *machine,u64 offset,u8 *data,ssize_t size, -bool out) +bool read) {if (dso__data_file_size (dso, machine))return -1 ;
@@ -1034,7 +1034,7 @@ static ssize_t data_read_write_offset(struct dso *dso, struct machine *machine,if (offset + size < offset)return -1 ; -return cached_io (dso, machine, offset, data, size, out); +return cached_io (dso, machine, offset, data, size, read); }/**
[tip: perf/core] perf dso: Add dso__data_write_cache_addr()
In reply to Adrian HunterThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: b86a9d918a389162803d833d4dc491fde9b62fa2 Gitweb: https://git.kernel.org/tip/b86a9d918a389162803d833d4dc491fde9b62fa2 Author: Adrian Hunter <[email protected]> AuthorDate: Fri, 25 Oct 2019 15:59:57 +03:00 Committer: Arnaldo Carvalho de Melo <[email protected]> CommitterDate: Wed, 06 Nov 2019 15:43:06 -03:00
perf dso: Add dso__data_write_cache_addr()
Add functions to write into the dso file data cache, but not change the file itself.
--- tools/perf/util/dso.c | 73 +++++++++++++++++++++++++++++++++--------- tools/perf/util/dso.h | 7 ++++- 2 files changed, 65 insertions(+), 15 deletions(-)
tools/perf/util/dso.c +58 -15
@@ -827,14 +827,16 @@ out:return cache; } -static ssize_t -dso_cache__memcpy (struct dso_cache *cache,u64 offset, -u8 *data,u64 size) +static ssize_t dso_cache__memcpy (struct dso_cache *cache,u64 offset,u8 *data, +u64 size,bool out) {u64 cache_offset = offset - cache->offset;u64 cache_size =min (cache->size - cache_offset, size); -memcpy (data, cache->data + cache_offset, cache_size); +if (out) +memcpy (data, cache->data + cache_offset, cache_size); +else +memcpy (cache->data + cache_offset, data, cache_size);return cache_size; }
@@ -910,8 +912,8 @@ static struct dso_cache *dso_cache__find(struct dso *dso,return cache ? cache :dso_cache__populate (dso, machine, offset, ret); } -static ssize_t dso_cache_read (struct dso *dso,struct machine *machine, -u64 offset,u8 *data,ssize_t size) +static ssize_t dso_cache_io (struct dso *dso,struct machine *machine, +u64 offset,u8 *data,ssize_t size,bool out) {struct dso_cache *cache;ssize_t ret =0 ;
@@ -920,16 +922,16 @@ static ssize_t dso_cache_read(struct dso *dso, struct machine *machine,if (!cache)return ret; -return dso_cache__memcpy (cache, offset, data, size); +return dso_cache__memcpy (cache, offset, data, size, out); }/* * Reads and caches dso data DSO__DATA_CACHE_SIZE size chunks * in the rb_tree. Any read to already cached data is served -* by cached data. +* by cached data. Writes update the cache only, not the backing file. */ -static ssize_t cached_read (struct dso *dso,struct machine *machine, -u64 offset,u8 *data,ssize_t size) +static ssize_t cached_io (struct dso *dso,struct machine *machine, +u64 offset,u8 *data,ssize_t size,bool out) {ssize_t r =0 ;u8 *p = data;
@@ -937,7 +939,7 @@ static ssize_t cached_read(struct dso *dso, struct machine *machine, do { ssize_t ret; - ret = dso_cache_read(dso, machine, offset, p, size); + ret = dso_cache_io(dso, machine, offset, p, size, out); if (ret < 0) return ret;
@@ -1021,8 +1023,9 @@ off_t dso__data_size(struct dso *dso, struct machine *machine)return dso->data.file_size; } -static ssize_t data_read_offset (struct dso *dso,struct machine *machine, -u64 offset,u8 *data,ssize_t size) +static ssize_t data_read_write_offset (struct dso *dso,struct machine *machine, +u64 offset,u8 *data,ssize_t size, +bool out) {if (dso__data_file_size (dso, machine))return -1 ;
@@ -1034,7 +1037,7 @@ static ssize_t data_read_offset(struct dso *dso, struct machine *machine,if (offset + size < offset)return -1 ; -return cached_read (dso, machine, offset, data, size); +return cached_io (dso, machine, offset, data, size, out); }/**
@@ -1054,7 +1057,7 @@ ssize_t dso__data_read_offset(struct dso *dso, struct machine *machine,if (dso->data.status ==DSO_DATA_STATUS_ERROR )return -1 ; -return data_read_offset (dso, machine, offset, data, size); +return data_read_write_offset (dso, machine, offset, data, size, true); }/**
@@ -1075,6 +1078,46 @@ ssize_t dso__data_read_addr(struct dso *dso, struct map *map,return dso__data_read_offset (dso, machine, offset, data, size); } +/** +* dso__data_write_cache_offs - Write data to dso data cache at file offset +* @dso: dso object +* @machine: machine object +* @offset: file offset +* @data: buffer to write +* @size: size of the @data buffer +* +* Write into the dso file data cache, but do not change the file itself. +*/ +ssize_t dso__data_write_cache_offs (struct dso *dso,struct machine *machine, +u64 offset,const u8 *data_in,ssize_t size) +{ +u8 *data = (u8 *)data_in;/* cast away const to use same fns for r/w */ + +if (dso->data.status ==DSO_DATA_STATUS_ERROR ) +return -1 ; + +return data_read_write_offset (dso, machine, offset, data, size, false); +} + +/** +* dso__data_write_cache_addr - Write data to dso data cache at dso address +* @dso: dso object +* @machine: machine object +* @add: virtual memory address +* @data: buffer to write +* @size: size of the @data buffer +* +* External interface to write into the dso file data cache, but do not change +* the file itself. +*/ +ssize_t dso__data_write_cache_addr (struct dso *dso,struct map *map, +struct machine *machine,u64 addr, +const u8 *data,ssize_t size) +{ +u64 offset = map->map_ip (map, addr); +return dso__data_write_cache_offs (dso, machine, offset, data, size); +} +struct map *dso__new_map (const char *name) {struct map *map =NULL ;
tools/perf/util/dso.h +7 -0
@@ -285,6 +285,8 @@ void dso__set_module_info(struct dso *dso, struct kmod_path *m,* dso__data_size * dso__data_read_offset * dso__data_read_addr +* dso__data_write_cache_offs +* dso__data_write_cache_addr * * Please refer to the dso.c object code for each function and * arguments documentation. Following text tries to explain the
@@ -332,6 +334,11 @@ ssize_t dso__data_read_addr(struct dso *dso, struct map *map, struct machine *machine, u64 addr, u8 *data, ssize_t size);bool dso__data_status_seen (struct dso *dso,enum dso_data_status_seen by); +ssize_t dso__data_write_cache_offs (struct dso *dso,struct machine *machine, +u64 offset,const u8 *data,ssize_t size); +ssize_t dso__data_write_cache_addr (struct dso *dso,struct map *map, +struct machine *machine,u64 addr, +const u8 *data,ssize_t size);struct map *dso__new_map (const char *name); struct dso *machine__findnew_kernel(struct machine *machine, const char *name,
Adrian Hunter
reviewerRFC PATCH 4/6 perf tools: Add support for PERF_RECORD_TEXT_POKE
In reply to Adrian HunterAdd processing for PERF_RECORD_TEXT_POKE events. When a text poke event is processed, if the PERF_TEXT_POKE_UPDATE flag is set, then the kernel dso data cache is updated with the poked bytes.
--- tools/include/uapi/linux/perf_event.h | 28 +++++++++++++- tools/perf/arch/x86/util/intel-pt.c | 1 + tools/perf/builtin-record.c | 45 +++++++++++++++++++++++ tools/perf/lib/include/perf/event.h | 9 +++++ tools/perf/util/event.c | 39 ++++++++++++++++++++ tools/perf/util/event.h | 5 +++ tools/perf/util/evlist.h | 1 + tools/perf/util/machine.c | 38 +++++++++++++++++++ tools/perf/util/machine.h | 3 ++ tools/perf/util/perf_event_attr_fprintf.c | 1 + tools/perf/util/record.c | 10 +++++ tools/perf/util/record.h | 1 + tools/perf/util/session.c | 22 +++++++++++ tools/perf/util/tool.h | 3 +- 14 files changed, 204 insertions(+), 2 deletions(-)
tools/include/uapi/linux/perf_event.h +27 -1
@@ -375,7 +375,8 @@ struct perf_event_attr { ksymbol : 1, /* include ksymbol events */ bpf_event : 1, /* include bpf events */ aux_output : 1, /* generate AUX records instead of events */ - __reserved_1 : 32; + text_poke : 1, /* include text poke events */ + __reserved_1 : 31;union {__u32 wakeup_events;/* wakeup every n events */
@@ -1000,6 +1001,26 @@ enum perf_event_type {*/ PERF_RECORD_BPF_EVENT =18 , +/* +* Records changes to kernel text i.e. self-modified code. +* 'flags' has PERF_TEXT_POKE_UPDATE (i.e. bit 0) set to +* indicate to tools to update old bytes to new bytes in the +* executable image. +* 'len' is the number of old bytes, which is the same as the number +* of new bytes. 'bytes' contains the old bytes followed immediately +* by the new bytes. +* +* struct { +* struct perf_event_header header; +* u64 addr; +* u16 flags; +* u16 len; +* u8 bytes[]; +* struct sample_id sample_id; +* }; +*/ +PERF_RECORD_TEXT_POKE =19 , + PERF_RECORD_MAX, /* non-ABI */ };
@@ -1041,6 +1062,11 @@ enum perf_callchain_context {#define PERF_AUX_FLAG_PARTIAL 0x04/* record contains gaps */ #define PERF_AUX_FLAG_COLLISION 0x08/* sample collided with another */ +/** +* PERF_RECORD_TEXT_POKE::flags bits +*/ +#define PERF_TEXT_POKE_UPDATE 0x01/* update old bytes to new bytes */ +#define PERF_FLAG_FD_NO_GROUP (1UL << 0)#define PERF_FLAG_FD_OUTPUT (1UL << 1)#define PERF_FLAG_PID_CGROUP (1UL << 2)/* pid=cgroup id, per-cpu mode only */
tools/perf/arch/x86/util/intel-pt.c +1 -0
@@ -17,6 +17,7 @@#include "../../util/event.h" #include "../../util/evlist.h" #include "../../util/evsel.h" +#include "../../util/evsel_config.h" #include "../../util/cpumap.h" #include "../../util/mmap.h" #include <subcmd/parse-options.h>
tools/perf/builtin-record.c +45 -0
57 lines not on this page · raw message
tools/perf/lib/include/perf/event.h +9 -0
@@ -105,6 +105,14 @@ struct perf_record_bpf_event {__u8 tag[BPF_TAG_SIZE ];// prog tag }; +struct perf_record_text_poke_event { +struct perf_event_header header; +__u64 addr; +__u16 flags; +__u16 len; +__u8 bytes[]; +}; +struct perf_record_sample {struct perf_event_header header;__u64 array[];
@@ -360,6 +368,7 @@ union perf_event {struct perf_record_sample sample;struct perf_record_bpf_event bpf;struct perf_record_ksymbol ksymbol; +struct perf_record_text_poke_event text_poke;struct perf_record_header_attr attr;struct perf_record_event_update event_update;struct perf_record_header_event_type event_type;
tools/perf/util/event.c +39 -0
@@ -31,6 +31,7 @@#include "stat.h" #include "session.h" #include "bpf-event.h" +#include "print_binary.h" #include "tool.h" #include "../perf.h"
@@ -54,6 +55,7 @@ static const char *perf_event__names[] = { [PERF_RECORD_NAMESPACES] = "NAMESPACES", [PERF_RECORD_KSYMBOL] = "KSYMBOL", [PERF_RECORD_BPF_EVENT] = "BPF_EVENT", + [PERF_RECORD_TEXT_POKE] = "TEXT_POKE", [PERF_RECORD_HEADER_ATTR] = "ATTR", [PERF_RECORD_HEADER_EVENT_TYPE] = "EVENT_TYPE", [PERF_RECORD_HEADER_TRACING_DATA] = "TRACING_DATA",
@@ -252,6 +254,14 @@ int perf_event__process_bpf(struct perf_tool *tool __maybe_unused,return machine__process_bpf (machine, event, sample); } +int perf_event__process_text_poke (struct perf_tool *tool__maybe_unused , +union perf_event *event, +struct perf_sample *sample, +struct machine *machine) +{ +return machine__process_text_poke (machine, event, sample); +} +size_t perf_event__fprintf_mmap (union perf_event *event,FILE *fp) { return fprintf(fp, " %d/%d: [%#" PRI_lx64 "(%#" PRI_lx64 ") @ %#" PRI_lx64 "]: %c %s\n",
@@ -398,6 +408,32 @@ size_t perf_event__fprintf_bpf(union perf_event *event, FILE *fp) event->bpf.type, event->bpf.flags, event->bpf.id); } +static int text_poke_printer (enum binary_printer_ops op,unsigned int val, +void *extra__maybe_unused ,FILE *fp) +{ +if (op ==BINARY_PRINT_NUM_DATA ) +return fprintf (fp," %02x" , val); +if (op ==BINARY_PRINT_LINE_END ) +return fprintf (fp,"\n" ); +return 0 ; +} + +size_t perf_event__fprintf_text_poke (union perf_event *event,FILE *fp) +{ +size_t ret =fprintf (fp," addr %#" PRI_lx64" flags %#x len %u\n" , + event->text_poke.addr, event->text_poke.flags, + event->text_poke.len); + + ret +=fprintf (fp," old bytes:" ); + ret +=binary__fprintf (event->text_poke.bytes, event->text_poke.len,16 , + text_poke_printer,NULL , fp); + ret +=fprintf (fp," new bytes:" ); + ret +=binary__fprintf (event->text_poke.bytes + event->text_poke.len, + event->text_poke.len,16 , text_poke_printer, +NULL , fp); +return ret; +} +size_t perf_event__fprintf (union perf_event *event,FILE *fp) { size_t ret = fprintf(fp, "PERF_RECORD_%s",
@@ -439,6 +475,9 @@ size_t perf_event__fprintf(union perf_event *event, FILE *fp)case PERF_RECORD_BPF_EVENT : ret +=perf_event__fprintf_bpf (event, fp);break ; +case PERF_RECORD_TEXT_POKE : + ret +=perf_event__fprintf_text_poke (event, fp); +break ;default : ret +=fprintf (fp,"\n" ); }
tools/perf/util/event.h +5 -0
@@ -339,6 +339,10 @@ int perf_event__process_bpf(struct perf_tool *tool,union perf_event *event, struct perf_sample *sample, struct machine *machine); +int perf_event__process_text_poke (struct perf_tool *tool, +union perf_event *event, +struct perf_sample *sample, +struct machine *machine); int perf_event__process(struct perf_tool *tool, union perf_event *event, struct perf_sample *sample,
@@ -372,6 +376,7 @@ size_t perf_event__fprintf_cpu_map(union perf_event *event, FILE *fp);size_t perf_event__fprintf_namespaces (union perf_event *event,FILE *fp);size_t perf_event__fprintf_ksymbol (union perf_event *event,FILE *fp);size_t perf_event__fprintf_bpf (union perf_event *event,FILE *fp); +size_t perf_event__fprintf_text_poke (union perf_event *event,FILE *fp);size_t perf_event__fprintf (union perf_event *event,FILE *fp); int kallsyms__get_function_start(const char *kallsyms_filename,
tools/perf/util/evlist.h +1 -0
@@ -175,6 +175,7 @@ struct callchain_param;void perf_evlist__set_id_pos (struct evlist *evlist);bool perf_can_sample_identifier (void );bool perf_can_record_switch_events (void ); +bool perf_can_record_text_poke_events (void );bool perf_can_record_cpu_wide (void );void perf_evlist__config (struct evlist *evlist,struct record_opts *opts,struct callchain_param *callchain);
tools/perf/util/machine.c +38 -0
@@ -767,6 +767,42 @@ int machine__process_ksymbol(struct machine *machine __maybe_unused,return machine__process_ksymbol_register (machine, event, sample); } +int machine__process_text_poke (struct machine *machine,union perf_event *event, +struct perf_sample *sample__maybe_unused ) +{ +struct map_groups *mg = &machine->kmaps; +struct map *map =map_groups__find (mg, event->text_poke.addr); + +if (dump_trace) +perf_event__fprintf_text_poke (event, stdout); + +if (!(event->text_poke.flags &PERF_TEXT_POKE_UPDATE )) +return 0 ; + +if (map) { +u8 *new_bytes = event->text_poke.bytes + event->text_poke.len; +int ret; + +/* +* Kernel maps might be changed when loading symbols so loading +* must be done prior to using kernel maps. +*/ +map__load (map); + ret =dso__data_write_cache_addr (map->dso, map, machine, + event->text_poke.addr, + new_bytes, + event->text_poke.len); +if (ret != event->text_poke.len) +pr_err ("Failed to write kernel text poke at %#" PRI_lx64"\n" , + event->text_poke.addr); + }else { +pr_err ("Failed to find kernel text poke address map for %#" PRI_lx64"\n" , + event->text_poke.addr); + } + +return 0 ; +} +static void dso__adjust_kmod_long_name (struct dso *dso,const char *filename) {const char *dup_filename;
@@ -1931,6 +1967,8 @@ int machine__process_event(struct machine *machine, union perf_event *event, ret =machine__process_ksymbol (machine, event, sample);break ;case PERF_RECORD_BPF_EVENT : ret =machine__process_bpf (machine, event, sample);break ; +case PERF_RECORD_TEXT_POKE : + ret =machine__process_text_poke (machine, event, sample);break ;default : ret =-1 ;break ;
tools/perf/util/machine.h +3 -0
@@ -134,6 +134,9 @@ int machine__process_mmap2_event(struct machine *machine, union perf_event *evenint machine__process_ksymbol (struct machine *machine,union perf_event *event,struct perf_sample *sample); +int machine__process_text_poke (struct machine *machine, +union perf_event *event, +struct perf_sample *sample);int machine__process_event (struct machine *machine,union perf_event *event,struct perf_sample *sample);
tools/perf/util/perf_event_attr_fprintf.c +1 -0
@@ -143,6 +143,7 @@ int perf_event_attr__fprintf(FILE *fp, struct perf_event_attr *attr,PRINT_ATTRf (sample_regs_intr, p_hex);PRINT_ATTRf (aux_watermark, p_unsigned);PRINT_ATTRf (sample_max_stack, p_unsigned); +PRINT_ATTRf (text_poke, p_unsigned);return ret; }
tools/perf/util/record.c +10 -0
@@ -97,6 +97,11 @@ static void perf_probe_context_switch(struct evsel *evsel) evsel->core.attr.context_switch =1 ; } +static void perf_probe_text_poke (struct evsel *evsel) +{ + evsel->core.attr.text_poke =1 ; +} +bool perf_can_sample_identifier (void ) {return perf_probe_api (perf_probe_sample_identifier);
@@ -112,6 +117,11 @@ bool perf_can_record_switch_events(void)return perf_probe_api (perf_probe_context_switch); } +bool perf_can_record_text_poke_events (void ) +{ +return perf_probe_api (perf_probe_text_poke); +} +bool perf_can_record_cpu_wide (void ) {struct perf_event_attr attr = {
tools/perf/util/record.h +1 -0
@@ -45,6 +45,7 @@ struct record_opts {bool sample_id;bool no_bpf_event;bool kcore; +bool text_poke;unsigned int freq;unsigned int mmap_pages;unsigned int auxtrace_mmap_pages;
tools/perf/util/session.c +22 -0
@@ -489,6 +489,8 @@ void perf_tool__fill_defaults(struct perf_tool *tool) tool->ksymbol = perf_event__process_ksymbol;if (tool->bpf ==NULL ) tool->bpf = perf_event__process_bpf; +if (tool->text_poke ==NULL ) + tool->text_poke = perf_event__process_text_poke;if (tool->read ==NULL ) tool->read = process_event_sample_stub;if (tool->throttle ==NULL )
@@ -658,6 +660,23 @@ static void perf_event__switch_swap(union perf_event *event, bool sample_id_all)swap_sample_id_all (event, &event->context_switch +1 ); } +static void perf_event__text_poke_swap (union perf_event *event,bool sample_id_all) +{ + event->text_poke.addr =bswap_64 (event->text_poke.addr); + event->text_poke.flags =bswap_16 (event->text_poke.flags); + event->text_poke.len =bswap_16 (event->text_poke.len); + +if (sample_id_all) { +size_t len =sizeof (event->text_poke.flags) + +sizeof (event->text_poke.len) + + event->text_poke.len *2 ; +void *data = &event->text_poke.flags; + + data +=PERF_ALIGN (len,sizeof (u64)); +swap_sample_id_all (event, data); + } +} +static void perf_event__throttle_swap (union perf_event *event,bool sample_id_all) {
@@ -930,6 +949,7 @@ static perf_event__swap_op perf_event__swap_ops[] = { [PERF_RECORD_SWITCH] = perf_event__switch_swap, [PERF_RECORD_SWITCH_CPU_WIDE] = perf_event__switch_swap, [PERF_RECORD_NAMESPACES] = perf_event__namespaces_swap, + [PERF_RECORD_TEXT_POKE] = perf_event__text_poke_swap, [PERF_RECORD_HEADER_ATTR] = perf_event__hdr_attr_swap, [PERF_RECORD_HEADER_EVENT_TYPE] = perf_event__event_type_swap, [PERF_RECORD_HEADER_TRACING_DATA] = perf_event__tracing_data_swap,
@@ -1469,6 +1489,8 @@ static int machines__deliver_event(struct machines *machines,return tool->ksymbol (tool, event, sample, machine);case PERF_RECORD_BPF_EVENT :return tool->bpf (tool, event, sample, machine); +case PERF_RECORD_TEXT_POKE : +return tool->text_poke (tool, event, sample, machine);default : ++evlist->stats.nr_unknown_events;return -1 ;
tools/perf/util/tool.h +2 -1
@@ -56,7 +56,8 @@ struct perf_tool { throttle, unthrottle, ksymbol, - bpf; + bpf, + text_poke;event_attr_op attr;event_attr_op event_update;
Signature · 1 line
2.17.1
Adrian Hunter
reviewerRFC PATCH 5/6 perf auxtrace: Add auxtrace_cache__remove()
In reply to Adrian HunterAdd auxtrace_cache__remove(). Intel PT uses an auxtrace_cache to store the results of code-walking, so that the same block of instructions does not have to be decoded repeatedly. However, when there are text poke events, the associated cache entries need to be removed.
--- tools/perf/util/auxtrace.c | 28 ++++++++++++++++++++++++++++ tools/perf/util/auxtrace.h | 1 + 2 files changed, 29 insertions(+)
tools/perf/util/auxtrace.c +28 -0
@@ -1457,6 +1457,34 @@ int auxtrace_cache__add(struct auxtrace_cache *c, u32 key,return 0 ; } +static struct auxtrace_cache_entry *auxtrace_cache__rm (struct auxtrace_cache *c, +u32 key) +{ +struct auxtrace_cache_entry *entry; +struct hlist_head *hlist; +struct hlist_node *n; + +if (!c) +return NULL ; + + hlist = &c->hashtable[hash_32 (key, c->bits)]; +hlist_for_each_entry_safe (entry, n, hlist, hash) { +if (entry->key == key) { +hlist_del (&entry->hash); +return entry; + } + } + +return NULL ; +} + +void auxtrace_cache__remove (struct auxtrace_cache *c,u32 key) +{ +struct auxtrace_cache_entry *entry =auxtrace_cache__rm (c, key); + +auxtrace_cache__free_entry (c, entry); +} +void *auxtrace_cache__lookup (struct auxtrace_cache *c,u32 key) {struct auxtrace_cache_entry *entry;
tools/perf/util/auxtrace.h +1 -0
@@ -489,6 +489,7 @@ void *auxtrace_cache__alloc_entry(struct auxtrace_cache *c);void auxtrace_cache__free_entry (struct auxtrace_cache *c,void *entry);int auxtrace_cache__add (struct auxtrace_cache *c,u32 key,struct auxtrace_cache_entry *entry); +void auxtrace_cache__remove (struct auxtrace_cache *c,u32 key);void *auxtrace_cache__lookup (struct auxtrace_cache *c,u32 key); struct auxtrace_record *auxtrace_record__init(struct evlist *evlist,
Signature · 1 line
2.17.1
Re: [PATCH RFC 5/6] perf auxtrace: Add auxtrace_cache__remove()
Em Fri, Oct 25, 2019 at 03:59:59PM +0300, Adrian Hunter escreveu:
Quoted 4 lines
Add
auxtrace_cache__remove(). Intel PT uses anauxtrace_cacheto store the results of code-walking, so that the same block of instructions does not have to be decoded repeatedly. However, when there are text poke events, the associated cache entries need to be removed.
Applied this one as it is just leg work for the rest, that I'll wait a bit for comments.
- Arnaldo
Quoted 59 lines
Signed-off-by Adrian Hunter <[email protected]>--- tools/perf/util/auxtrace.c | 28 ++++++++++++++++++++++++++++ tools/perf/util/auxtrace.h | 1 + 2 files changed, 29 insertions(+)tools/perf/util/auxtrace.c +28
-0@@ -1457,6 +1457,34 @@ int auxtrace_cache__add(struct auxtrace_cache *c, u32 key,return 0 ; } +static struct auxtrace_cache_entry *auxtrace_cache__rm (struct auxtrace_cache *c, +u32 key) +{ +struct auxtrace_cache_entry *entry; +struct hlist_head *hlist; +struct hlist_node *n; + +if (!c) +return NULL ; + + hlist = &c->hashtable[hash_32 (key, c->bits)]; +hlist_for_each_entry_safe (entry, n, hlist, hash) { +if (entry->key == key) { +hlist_del (&entry->hash); +return entry; + } + } + +return NULL ; +} + +void auxtrace_cache__remove (struct auxtrace_cache *c,u32 key) +{ +struct auxtrace_cache_entry *entry =auxtrace_cache__rm (c, key); + +auxtrace_cache__free_entry (c, entry); +} +void *auxtrace_cache__lookup (struct auxtrace_cache *c,u32 key) {struct auxtrace_cache_entry *entry;tools/perf/util/auxtrace.h +1
-0@@ -489,6 +489,7 @@ void *auxtrace_cache__alloc_entry(struct auxtrace_cache *c);void auxtrace_cache__free_entry (struct auxtrace_cache *c,void *entry);int auxtrace_cache__add (struct auxtrace_cache *c,u32 key,struct auxtrace_cache_entry *entry); +void auxtrace_cache__remove (struct auxtrace_cache *c,u32 key);void *auxtrace_cache__lookup (struct auxtrace_cache *c,u32 key); struct auxtrace_record *auxtrace_record__init(struct evlist *evlist,Signature · 1 line
2.17.1
Signature · 1 line
- Arnaldo
[tip: perf/core] perf auxtrace: Add auxtrace_cache__remove()
In reply to Adrian HunterThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: fd62c1097a0700484fc2cbc9a182f341f30890cd Gitweb: https://git.kernel.org/tip/fd62c1097a0700484fc2cbc9a182f341f30890cd Author: Adrian Hunter <[email protected]> AuthorDate: Fri, 25 Oct 2019 15:59:59 +03:00 Committer: Arnaldo Carvalho de Melo <[email protected]> CommitterDate: Wed, 06 Nov 2019 15:43:06 -03:00
perf auxtrace: Add auxtrace_cache__remove()
Add auxtrace_cache__remove(). Intel PT uses an auxtrace_cache to store the results of code-walking, so that the same block of instructions does not have to be decoded repeatedly. However, when there are text poke events, the associated cache entries need to be removed.
--- tools/perf/util/auxtrace.c | 28 ++++++++++++++++++++++++++++ tools/perf/util/auxtrace.h | 1 + 2 files changed, 29 insertions(+)
tools/perf/util/auxtrace.c +28 -0
@@ -1457,6 +1457,34 @@ int auxtrace_cache__add(struct auxtrace_cache *c, u32 key,return 0 ; } +static struct auxtrace_cache_entry *auxtrace_cache__rm (struct auxtrace_cache *c, +u32 key) +{ +struct auxtrace_cache_entry *entry; +struct hlist_head *hlist; +struct hlist_node *n; + +if (!c) +return NULL ; + + hlist = &c->hashtable[hash_32 (key, c->bits)]; +hlist_for_each_entry_safe (entry, n, hlist, hash) { +if (entry->key == key) { +hlist_del (&entry->hash); +return entry; + } + } + +return NULL ; +} + +void auxtrace_cache__remove (struct auxtrace_cache *c,u32 key) +{ +struct auxtrace_cache_entry *entry =auxtrace_cache__rm (c, key); + +auxtrace_cache__free_entry (c, entry); +} +void *auxtrace_cache__lookup (struct auxtrace_cache *c,u32 key) {struct auxtrace_cache_entry *entry;
tools/perf/util/auxtrace.h +1 -0
@@ -489,6 +489,7 @@ void *auxtrace_cache__alloc_entry(struct auxtrace_cache *c);void auxtrace_cache__free_entry (struct auxtrace_cache *c,void *entry);int auxtrace_cache__add (struct auxtrace_cache *c,u32 key,struct auxtrace_cache_entry *entry); +void auxtrace_cache__remove (struct auxtrace_cache *c,u32 key);void *auxtrace_cache__lookup (struct auxtrace_cache *c,u32 key); struct auxtrace_record *auxtrace_record__init(struct evlist *evlist,
Adrian Hunter
reviewerRFC PATCH 6/6 perf intel-pt: Add support for text poke events
In reply to Adrian HunterSelect text poke events when available and the kernel is being traced. Process text poke events to invalidate entries in Intel PT's instruction cache.
--- tools/perf/arch/x86/util/intel-pt.c | 4 ++ tools/perf/util/intel-pt.c | 71 +++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+)
tools/perf/arch/x86/util/intel-pt.c +4 -0
@@ -754,6 +754,10 @@ static int intel_pt_recording_options(struct auxtrace_record *itr, } } +if (have_timing_info && !intel_pt_evsel->core.attr.exclude_kernel && +perf_can_record_text_poke_events () &&perf_can_record_cpu_wide ()) + opts->text_poke = true; +if (intel_pt_evsel) {/* * To obtain the auxtrace buffer file descriptor, the auxtrace
tools/perf/util/intel-pt.c +71 -0
89 lines not on this page · raw message
Signature · 1 line
2.17.1