perf/x86: Add perf text poke event
Subsystems (2)
Adrian Hunter
reviewercover perf/x86: Add perf text poke event
Hi
Here are patches to add a text poke event to record changes to kernel text (i.e. self-modifying code) in order to support tracers like Intel PT decoding through jump labels.
The first patch makes the kernel change and the subsequent patches are tools changes.
The second patch adds support for updating perf tools' data cache with the changed bytes.
The last patch is an Intel PT specific tools change.
The patches are based on linux-next.
Changes since RFC:
Dropped 'flags' from the new event. The consensus seemed to be that text pokes should emply a scheme similar to x86's INT3 method instead.
dropped tools patches already applied.
Example:
For jump labels, the kernel needs
CONFIG_JUMP_LABEL=y
and also an easy to flip jump label is in sysctl_schedstats() which needs
CONFIG_SCHEDSTATS=y
CONFIG_PROC_SYSCTL=y
Also note the 'sudo perf record' is put into the background which, as written, needs sudo credential caching (otherwise the background task will stop awaiting the sudo password), hence the 'sudo echo' to start.
Before:
$ sudo echo $ sudo perf record -o perf.data.before --kcore -a -e intel_pt//k -m,64M & [1] 1640 $ cat /proc/sys/kernel/sched_schedstats 0 $ sudo bash -c 'echo 1 > /proc/sys/kernel/sched_schedstats' $ cat /proc/sys/kernel/sched_schedstats 1 $ sudo bash -c 'echo 0 > /proc/sys/kernel/sched_schedstats' $ cat /proc/sys/kernel/sched_schedstats 0 $ sudo kill 1640 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 16.635 MB perf.data.before ] $ perf script -i perf.data.before --itrace=e >/dev/null Warning: 1946 instruction trace errors
After:
$ sudo echo $ sudo perf record -o perf.data.after --kcore -a -e intel_pt//k -m,64M & [1] 1882 $ cat /proc/sys/kernel/sched_schedstats 0 $ sudo bash -c 'echo 1 > /proc/sys/kernel/sched_schedstats' $ cat /proc/sys/kernel/sched_schedstats 1 $ sudo bash -c 'echo 0 > /proc/sys/kernel/sched_schedstats' $ cat /proc/sys/kernel/sched_schedstats 0 $ sudo kill 1882 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 10.893 MB perf.data.after ] $ perf script -i perf.data.after --itrace=e $
Adrian Hunter (3):
perf/x86: Add perf text poke event
perf tools: Add support for PERF_RECORD_TEXT_POKE perf intel-pt: Add support for text poke events
arch/x86/kernel/alternative.c | 37 ++++++++++++- include/linux/perf_event.h | 6 +++ include/uapi/linux/perf_event.h | 19 ++++++- kernel/events/core.c | 87 ++++++++++++++++++++++++++++++- tools/include/uapi/linux/perf_event.h | 19 ++++++- tools/perf/arch/x86/util/intel-pt.c | 4 ++ tools/perf/builtin-record.c | 45 ++++++++++++++++ tools/perf/lib/include/perf/event.h | 8 +++ tools/perf/util/event.c | 39 ++++++++++++++ tools/perf/util/event.h | 5 ++ tools/perf/util/evlist.h | 1 + tools/perf/util/intel-pt.c | 71 +++++++++++++++++++++++++ tools/perf/util/machine.c | 34 ++++++++++++ tools/perf/util/machine.h | 3 ++ tools/perf/util/perf_event_attr_fprintf.c | 1 + tools/perf/util/record.c | 10 ++++ tools/perf/util/record.h | 1 + tools/perf/util/session.c | 20 +++++++ tools/perf/util/tool.h | 3 +- 19 files changed, 408 insertions(+), 5 deletions(-)
Regards
Adrian
Adrian Hunter
reviewerPATCH 1/3 perf/x86: Add perf text poke event
Record changes to kernel text (i.e. self-modifying code) in order to support tracers like Intel PT decoding through jump labels.
A copy of the running kernel code is needed as a reference point (e.g. from /proc/kcore). The text poke event records the old bytes and the new bytes so that the event can be processed forwards or backwards.
In the case of Intel PT tracing, the executable code must be walked to reconstruct the control flow. For x86 a jump label text poke begins:
- write INT3 byte
- IPI-SYNC
- write instruction tail
At this point the actual control flow will be through the INT3 and handler and not hit the old or new instruction. Intel PT outputs FUP/TIP packets for the INT3, so the flow can still be decoded. Subsequently:
- emit RECORD_TEXT_POKE with the new instruction
- IPI-SYNC
- write first byte
- IPI-SYNC
So before the text poke event timestamp, the decoder will see either the old instruction flow or FUP/TIP of INT3. After the text poke event timestamp, the decoder will see either the new instruction flow or FUP/TIP of INT3. Thus decoders can use the timestamp as the point at which to modify the executable code.
--- arch/x86/kernel/alternative.c | 37 +++++++++++++- include/linux/perf_event.h | 6 +++ include/uapi/linux/perf_event.h | 19 ++++++- kernel/events/core.c | 87 ++++++++++++++++++++++++++++++++- 4 files changed, 146 insertions(+), 3 deletions(-)
arch/x86/kernel/alternative.c +36 -1
@@ -3,6 +3,7 @@#include <linux/module.h> #include <linux/sched.h> +#include <linux/perf_event.h> #include <linux/mutex.h> #include <linux/list.h> #include <linux/stringify.h>
@@ -946,6 +947,7 @@ struct text_poke_loc {s32 rel32;u8 opcode;const u8 text[POKE_MAX_OPCODE_SIZE ]; +u8 old; }; static struct bp_patching_desc {
@@ -1087,8 +1089,10 @@ static void text_poke_bp_batch(struct text_poke_loc *tp, unsigned int nr_entries/* * First step: add a int3 trap to the address that will be patched. */ -for (i =0 ; i < nr_entries; i++) +for (i =0 ; i < nr_entries; i++) { + tp[i].old = *(u8 *)text_poke_addr (&tp[i]);text_poke (text_poke_addr (&tp[i]), &int3,INT3_INSN_SIZE ); + }text_poke_sync ();
@@ -1096,14 +1100,45 @@ static void text_poke_bp_batch(struct text_poke_loc *tp, unsigned int nr_entries* Second step: update all but the first byte of the patched range. */ for (do_sync =0 , i =0 ; i < nr_entries; i++) { +u8 old[POKE_MAX_OPCODE_SIZE ] = { tp[i].old, };int len =text_opcode_size (tp[i].opcode);if (len -INT3_INSN_SIZE >0 ) { +memcpy (old +INT3_INSN_SIZE , +text_poke_addr (&tp[i]) +INT3_INSN_SIZE , + len -INT3_INSN_SIZE );text_poke (text_poke_addr (&tp[i]) +INT3_INSN_SIZE , (const char *)tp[i].text +INT3_INSN_SIZE , len -INT3_INSN_SIZE ); do_sync++; } + +/* +* Emit a perf event to record the text poke, primarily to +* support Intel PT decoding which must walk the executable code +* to reconstruct the trace. The flow up to here is: +* - write INT3 byte +* - IPI-SYNC +* - write instruction tail +* At this point the actual control flow will be through the +* INT3 and handler and not hit the old or new instruction. +* Intel PT outputs FUP/TIP packets for the INT3, so the flow +* can still be decoded. Subsequently: +* - emit RECORD_TEXT_POKE with the new instruction +* - IPI-SYNC +* - write first byte +* - IPI-SYNC +* So before the text poke event timestamp, the decoder will see +* either the old instruction flow or FUP/TIP of INT3. After the +* text poke event timestamp, the decoder will see either the +* new instruction flow or FUP/TIP of INT3. Thus decoders can +* use the timestamp as the point at which to modify the +* executable code. +* The old instruction is recorded so that the event can be +* processed forwards or backwards. +*/ +perf_event_text_poke ((unsigned long )text_poke_addr (&tp[i]), old, + tp[i].text, len); }if (do_sync) {
include/linux/perf_event.h +6 -0
@@ -1212,6 +1212,8 @@ extern void perf_event_exec(void);extern void perf_event_comm (struct task_struct *tsk,bool exec);extern void perf_event_namespaces (struct task_struct *tsk);extern void perf_event_fork (struct task_struct *tsk); +extern void perf_event_text_poke (unsigned long addr,const void *old_bytes, +const void *new_bytes,size_t len);/* Callchains */ DECLARE_PER_CPU(struct perf_callchain_entry, perf_callchain_entry);
@@ -1462,6 +1464,10 @@ static inline void perf_event_exec(void) { }static inline void perf_event_comm (struct task_struct *tsk,bool exec) { }static inline void perf_event_namespaces (struct task_struct *tsk) { }static inline void perf_event_fork (struct task_struct *tsk) { } +static inline void perf_event_text_poke (unsigned long addr, +const void *old_bytes, +const void *new_bytes, +size_t len) { }static inline void perf_event_init (void ) { }static inline int perf_swevent_get_recursion_context (void ) {return -1 ; }static inline void perf_swevent_put_recursion_context (int rctx) { }
include/uapi/linux/perf_event.h +18 -1
@@ -377,7 +377,8 @@ struct perf_event_attr { ksymbol : 1, /* include ksymbol events */ bpf_event : 1, /* include bpf events */ aux_output : 1, /* generate AUX records instead of events */ - __reserved_1 : 32; + text_poke : 1, /* include text poke events */ + __reserved_1 : 31;union {__u32 wakeup_events;/* wakeup every n events */
@@ -1006,6 +1007,22 @@ enum perf_event_type {*/ PERF_RECORD_BPF_EVENT =18 , +/* +* Records changes to kernel text i.e. self-modified code. +* 'len' is the number of old bytes, which is the same as the number +* of new bytes. 'bytes' contains the old bytes followed immediately +* by the new bytes. +* +* struct { +* struct perf_event_header header; +* u64 addr; +* u16 len; +* u8 bytes[]; +* struct sample_id sample_id; +* }; +*/ +PERF_RECORD_TEXT_POKE =19 , + PERF_RECORD_MAX, /* non-ABI */ };
kernel/events/core.c +86 -1
117 lines not on this page · raw message
Signature · 1 line
2.17.1
Adrian Hunter
reviewerPATCH 2/3 perf tools: Add support for PERF_RECORD_TEXT_POKE
In reply to Adrian HunterAdd processing for PERF_RECORD_TEXT_POKE events. When a text poke event is processed, then the kernel dso data cache is updated with the poked bytes.
--- tools/include/uapi/linux/perf_event.h | 19 +++++++++- tools/perf/builtin-record.c | 45 +++++++++++++++++++++++ tools/perf/lib/include/perf/event.h | 8 ++++ tools/perf/util/event.c | 39 ++++++++++++++++++++ tools/perf/util/event.h | 5 +++ tools/perf/util/evlist.h | 1 + tools/perf/util/machine.c | 34 +++++++++++++++++ tools/perf/util/machine.h | 3 ++ tools/perf/util/perf_event_attr_fprintf.c | 1 + tools/perf/util/record.c | 10 +++++ tools/perf/util/record.h | 1 + tools/perf/util/session.c | 20 ++++++++++ tools/perf/util/tool.h | 3 +- 13 files changed, 187 insertions(+), 2 deletions(-)
tools/include/uapi/linux/perf_event.h +18 -1
@@ -377,7 +377,8 @@ struct perf_event_attr { ksymbol : 1, /* include ksymbol events */ bpf_event : 1, /* include bpf events */ aux_output : 1, /* generate AUX records instead of events */ - __reserved_1 : 32; + text_poke : 1, /* include text poke events */ + __reserved_1 : 31;union {__u32 wakeup_events;/* wakeup every n events */
@@ -1006,6 +1007,22 @@ enum perf_event_type {*/ PERF_RECORD_BPF_EVENT =18 , +/* +* Records changes to kernel text i.e. self-modified code. +* 'len' is the number of old bytes, which is the same as the number +* of new bytes. 'bytes' contains the old bytes followed immediately +* by the new bytes. +* +* struct { +* struct perf_event_header header; +* u64 addr; +* u16 len; +* u8 bytes[]; +* struct sample_id sample_id; +* }; +*/ +PERF_RECORD_TEXT_POKE =19 , + PERF_RECORD_MAX, /* non-ABI */ };
tools/perf/builtin-record.c +45 -0
57 lines not on this page · raw message
tools/perf/lib/include/perf/event.h +8 -0
@@ -105,6 +105,13 @@ struct perf_record_bpf_event {__u8 tag[BPF_TAG_SIZE ];// prog tag }; +struct perf_record_text_poke_event { +struct perf_event_header header; +__u64 addr; +__u16 len; +__u8 bytes[]; +}; +struct perf_record_sample {struct perf_event_header header;__u64 array[];
@@ -360,6 +367,7 @@ union perf_event {struct perf_record_sample sample;struct perf_record_bpf_event bpf;struct perf_record_ksymbol ksymbol; +struct perf_record_text_poke_event text_poke;struct perf_record_header_attr attr;struct perf_record_event_update event_update;struct perf_record_header_event_type event_type;
tools/perf/util/event.c +39 -0
@@ -31,6 +31,7 @@#include "stat.h" #include "session.h" #include "bpf-event.h" +#include "print_binary.h" #include "tool.h" #include "../perf.h"
@@ -54,6 +55,7 @@ static const char *perf_event__names[] = { [PERF_RECORD_NAMESPACES] = "NAMESPACES", [PERF_RECORD_KSYMBOL] = "KSYMBOL", [PERF_RECORD_BPF_EVENT] = "BPF_EVENT", + [PERF_RECORD_TEXT_POKE] = "TEXT_POKE", [PERF_RECORD_HEADER_ATTR] = "ATTR", [PERF_RECORD_HEADER_EVENT_TYPE] = "EVENT_TYPE", [PERF_RECORD_HEADER_TRACING_DATA] = "TRACING_DATA",
@@ -252,6 +254,14 @@ int perf_event__process_bpf(struct perf_tool *tool __maybe_unused,return machine__process_bpf (machine, event, sample); } +int perf_event__process_text_poke (struct perf_tool *tool__maybe_unused , +union perf_event *event, +struct perf_sample *sample, +struct machine *machine) +{ +return machine__process_text_poke (machine, event, sample); +} +size_t perf_event__fprintf_mmap (union perf_event *event,FILE *fp) { return fprintf(fp, " %d/%d: [%#" PRI_lx64 "(%#" PRI_lx64 ") @ %#" PRI_lx64 "]: %c %s\n",
@@ -398,6 +408,32 @@ size_t perf_event__fprintf_bpf(union perf_event *event, FILE *fp) event->bpf.type, event->bpf.flags, event->bpf.id); } +static int text_poke_printer (enum binary_printer_ops op,unsigned int val, +void *extra__maybe_unused ,FILE *fp) +{ +if (op ==BINARY_PRINT_NUM_DATA ) +return fprintf (fp," %02x" , val); +if (op ==BINARY_PRINT_LINE_END ) +return fprintf (fp,"\n" ); +return 0 ; +} + +size_t perf_event__fprintf_text_poke (union perf_event *event,FILE *fp) +{ +size_t ret =fprintf (fp," addr %#" PRI_lx64" len %u\n" , + event->text_poke.addr, + event->text_poke.len); + + ret +=fprintf (fp," old bytes:" ); + ret +=binary__fprintf (event->text_poke.bytes, event->text_poke.len,16 , + text_poke_printer,NULL , fp); + ret +=fprintf (fp," new bytes:" ); + ret +=binary__fprintf (event->text_poke.bytes + event->text_poke.len, + event->text_poke.len,16 , text_poke_printer, +NULL , fp); +return ret; +} +size_t perf_event__fprintf (union perf_event *event,FILE *fp) { size_t ret = fprintf(fp, "PERF_RECORD_%s",
@@ -439,6 +475,9 @@ size_t perf_event__fprintf(union perf_event *event, FILE *fp)case PERF_RECORD_BPF_EVENT : ret +=perf_event__fprintf_bpf (event, fp);break ; +case PERF_RECORD_TEXT_POKE : + ret +=perf_event__fprintf_text_poke (event, fp); +break ;default : ret +=fprintf (fp,"\n" ); }
tools/perf/util/event.h +5 -0
@@ -345,6 +345,10 @@ int perf_event__process_bpf(struct perf_tool *tool,union perf_event *event, struct perf_sample *sample, struct machine *machine); +int perf_event__process_text_poke (struct perf_tool *tool, +union perf_event *event, +struct perf_sample *sample, +struct machine *machine); int perf_event__process(struct perf_tool *tool, union perf_event *event, struct perf_sample *sample,
@@ -378,6 +382,7 @@ size_t perf_event__fprintf_cpu_map(union perf_event *event, FILE *fp);size_t perf_event__fprintf_namespaces (union perf_event *event,FILE *fp);size_t perf_event__fprintf_ksymbol (union perf_event *event,FILE *fp);size_t perf_event__fprintf_bpf (union perf_event *event,FILE *fp); +size_t perf_event__fprintf_text_poke (union perf_event *event,FILE *fp);size_t perf_event__fprintf (union perf_event *event,FILE *fp); int kallsyms__get_function_start(const char *kallsyms_filename,
tools/perf/util/evlist.h +1 -0
@@ -175,6 +175,7 @@ struct callchain_param;void perf_evlist__set_id_pos (struct evlist *evlist);bool perf_can_sample_identifier (void );bool perf_can_record_switch_events (void ); +bool perf_can_record_text_poke_events (void );bool perf_can_record_cpu_wide (void );bool perf_can_aux_sample (void ); void perf_evlist__config(struct evlist *evlist, struct record_opts *opts,
tools/perf/util/machine.c +34 -0
@@ -770,6 +770,38 @@ int machine__process_ksymbol(struct machine *machine __maybe_unused,return machine__process_ksymbol_register (machine, event, sample); } +int machine__process_text_poke (struct machine *machine,union perf_event *event, +struct perf_sample *sample__maybe_unused ) +{ +struct map *map =maps__find (&machine->kmaps, event->text_poke.addr); + +if (dump_trace) +perf_event__fprintf_text_poke (event, stdout); + +if (map) { +u8 *new_bytes = event->text_poke.bytes + event->text_poke.len; +int ret; + +/* +* Kernel maps might be changed when loading symbols so loading +* must be done prior to using kernel maps. +*/ +map__load (map); + ret =dso__data_write_cache_addr (map->dso, map, machine, + event->text_poke.addr, + new_bytes, + event->text_poke.len); +if (ret != event->text_poke.len) +pr_err ("Failed to write kernel text poke at %#" PRI_lx64"\n" , + event->text_poke.addr); + }else { +pr_err ("Failed to find kernel text poke address map for %#" PRI_lx64"\n" , + event->text_poke.addr); + } + +return 0 ; +} +static struct map *machine__addnew_module_map (struct machine *machine,u64 start,const char *filename) {
@@ -1901,6 +1933,8 @@ int machine__process_event(struct machine *machine, union perf_event *event, ret =machine__process_ksymbol (machine, event, sample);break ;case PERF_RECORD_BPF_EVENT : ret =machine__process_bpf (machine, event, sample);break ; +case PERF_RECORD_TEXT_POKE : + ret =machine__process_text_poke (machine, event, sample);break ;default : ret =-1 ;break ;
tools/perf/util/machine.h +3 -0
@@ -135,6 +135,9 @@ int machine__process_mmap2_event(struct machine *machine, union perf_event *evenint machine__process_ksymbol (struct machine *machine,union perf_event *event,struct perf_sample *sample); +int machine__process_text_poke (struct machine *machine, +union perf_event *event, +struct perf_sample *sample);int machine__process_event (struct machine *machine,union perf_event *event,struct perf_sample *sample);
tools/perf/util/perf_event_attr_fprintf.c +1 -0
@@ -144,6 +144,7 @@ int perf_event_attr__fprintf(FILE *fp, struct perf_event_attr *attr,PRINT_ATTRf (aux_watermark, p_unsigned);PRINT_ATTRf (sample_max_stack, p_unsigned);PRINT_ATTRf (aux_sample_size, p_unsigned); +PRINT_ATTRf (text_poke, p_unsigned);return ret; }
tools/perf/util/record.c +10 -0
@@ -97,6 +97,11 @@ static void perf_probe_context_switch(struct evsel *evsel) evsel->core.attr.context_switch =1 ; } +static void perf_probe_text_poke (struct evsel *evsel) +{ + evsel->core.attr.text_poke =1 ; +} +bool perf_can_sample_identifier (void ) {return perf_probe_api (perf_probe_sample_identifier);
@@ -112,6 +117,11 @@ bool perf_can_record_switch_events(void)return perf_probe_api (perf_probe_context_switch); } +bool perf_can_record_text_poke_events (void ) +{ +return perf_probe_api (perf_probe_text_poke); +} +bool perf_can_record_cpu_wide (void ) {struct perf_event_attr attr = {
tools/perf/util/record.h +1 -0
@@ -46,6 +46,7 @@ struct record_opts {bool sample_id;bool no_bpf_event;bool kcore; +bool text_poke;unsigned int freq;unsigned int mmap_pages;unsigned int auxtrace_mmap_pages;
tools/perf/util/session.c +20 -0
@@ -489,6 +489,8 @@ void perf_tool__fill_defaults(struct perf_tool *tool) tool->ksymbol = perf_event__process_ksymbol;if (tool->bpf ==NULL ) tool->bpf = perf_event__process_bpf; +if (tool->text_poke ==NULL ) + tool->text_poke = perf_event__process_text_poke;if (tool->read ==NULL ) tool->read = process_event_sample_stub;if (tool->throttle ==NULL )
@@ -658,6 +660,21 @@ static void perf_event__switch_swap(union perf_event *event, bool sample_id_all)swap_sample_id_all (event, &event->context_switch +1 ); } +static void perf_event__text_poke_swap (union perf_event *event,bool sample_id_all) +{ + event->text_poke.addr =bswap_64 (event->text_poke.addr); + event->text_poke.len =bswap_16 (event->text_poke.len); + +if (sample_id_all) { +size_t len =sizeof (event->text_poke.len) + + event->text_poke.len *2 ; +void *data = &event->text_poke.len; + + data +=PERF_ALIGN (len,sizeof (u64)); +swap_sample_id_all (event, data); + } +} +static void perf_event__throttle_swap (union perf_event *event,bool sample_id_all) {
@@ -931,6 +948,7 @@ static perf_event__swap_op perf_event__swap_ops[] = { [PERF_RECORD_SWITCH] = perf_event__switch_swap, [PERF_RECORD_SWITCH_CPU_WIDE] = perf_event__switch_swap, [PERF_RECORD_NAMESPACES] = perf_event__namespaces_swap, + [PERF_RECORD_TEXT_POKE] = perf_event__text_poke_swap, [PERF_RECORD_HEADER_ATTR] = perf_event__hdr_attr_swap, [PERF_RECORD_HEADER_EVENT_TYPE] = perf_event__event_type_swap, [PERF_RECORD_HEADER_TRACING_DATA] = perf_event__tracing_data_swap,
@@ -1470,6 +1488,8 @@ static int machines__deliver_event(struct machines *machines,return tool->ksymbol (tool, event, sample, machine);case PERF_RECORD_BPF_EVENT :return tool->bpf (tool, event, sample, machine); +case PERF_RECORD_TEXT_POKE : +return tool->text_poke (tool, event, sample, machine);default : ++evlist->stats.nr_unknown_events;return -1 ;
tools/perf/util/tool.h +2 -1
@@ -56,7 +56,8 @@ struct perf_tool { throttle, unthrottle, ksymbol, - bpf; + bpf, + text_poke;event_attr_op attr;event_attr_op event_update;
Signature · 1 line
2.17.1
Adrian Hunter
reviewerPATCH 3/3 perf intel-pt: Add support for text poke events
In reply to Adrian HunterSelect text poke events when available and the kernel is being traced. Process text poke events to invalidate entries in Intel PT's instruction cache.
--- tools/perf/arch/x86/util/intel-pt.c | 4 ++ tools/perf/util/intel-pt.c | 71 +++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+)
tools/perf/arch/x86/util/intel-pt.c +4 -0
@@ -827,6 +827,10 @@ static int intel_pt_recording_options(struct auxtrace_record *itr, } } +if (have_timing_info && !intel_pt_evsel->core.attr.exclude_kernel && +perf_can_record_text_poke_events () &&perf_can_record_cpu_wide ()) + opts->text_poke = true; +if (intel_pt_evsel) {/* * To obtain the auxtrace buffer file descriptor, the auxtrace
tools/perf/util/intel-pt.c +71 -0
89 lines not on this page · raw message
Signature · 1 line
2.17.1
Peter Zijlstra
maintainerQuoted 2 lines · Adrian Hunter
Record changes to kernel text (i.e. self-modifying code) in order to support tracers like Intel PT decoding through jump labels.
I don't get the obsession with just jump-labels, we need a solution all modifying code. The fentry site usage is increasing, and optprobes are also fairly popular with a bunch of people.
Quoted 13 lines
A copy of the running kernel code is needed as a reference point (e.g. from
/proc/kcore). The text poke event records the old bytes and the new bytes so that the event can be processed forwards or backwards.In the case of Intel PT tracing, the executable code must be walked to reconstruct the control flow. For x86 a jump label text poke begins:
- write INT3 byte
- IPI-SYNC
- write instruction tail
At this point the actual control flow will be through the INT3 and handler and not hit the old or new instruction. Intel PT outputs FUP/TIP packets for the INT3, so the flow can still be decoded. Subsequently:
- emitRECORD_TEXT_POKEwith the new instruction
Quoted 8 lines
- IPI-SYNC
- write first byte
- IPI-SYNC
So before the text poke event timestamp, the decoder will see either the old instruction flow or FUP/TIP of INT3. After the text poke event timestamp, the decoder will see either the new instruction flow or FUP/TIP of INT3. Thus decoders can use the timestamp as the point at which to modify the executable code.
I feel a much better justification for the design can be found in the discussion we've had around ARM-CS.
Basically SMP instruction coherency mandates something like this, it is just a happy accident x86 already had all the bits in place.
How is something like:
"Record (single instruction) changes to the kernel text (i.e. self-modifying code) in order to support tracers like Intel PT and ARM CoreSight.
A copy of the running kernel code is needed as a reference point (e.g. from /proc/kcore). The text poke event records the old bytes and the new bytes so that the event can be processed forwards or backwards.
The basic problem is recording the modified instruction in an unambiguous manner given SMP instruction cache (in)coherence. That is, when modifying an instruction concurrently any solution with one or multiple timestamps is not sufficient:
CPU0 CPU1
0
1 write insn A
2 execute insn A
3 sync-I$
4
Due to I$, CPU1 might execute either the old or new A. No matter where we record tracepoints on CPU0, one simply cannot tell what CPU1 will have observed, except that at 0 it must be the old one and at 4 it must be the new one.
To solve this, take inspiration from x86 text poking, which has to solve this exact problem due to variable length instruction encoding and I-fetch windows.
1) overwrite the instruction with a breakpoint and sync I$
This guarantees that that code flow will never hit the target instruction anymore, on any CPU (or rather, it will cause an exception).
2) issue the TEXT_POKE event
3) overwrite the breakpoint with the new instruction and sync I$
Now we know that any execution after the TEXT_POKE event will either observe the breakpoint (and hit the exception) or the new instruction.
So by guarding the TEXT_POKE event with an exception on either side; we can now tell, without doubt, which instruction another CPU will have observed."
?
Quoted 3 lines
Signed-off-by Adrian Hunter <[email protected]>--- arch/x86/kernel/alternative.c | 37 +++++++++++++-
I'm thinking it might make sense to do this x86 part in a separate patch, and just present the generic thing first:
Quoted 11 lines
include/linux/perf_event.h | 6 +++ include/uapi/linux/perf_event.h | 19 ++++++- kernel/events/core.c | 87 ++++++++++++++++++++++++++++++++- 4 files changed, 146 insertions(+), 3 deletions(-)include/uapi/linux/perf_event.h +1
-0@@ -1006,6 +1007,22 @@ enum perf_event_type {*/ PERF_RECORD_BPF_EVENT = 18, +/*
include/uapi/linux/perf_event.h enum perf_event_type { · line 1020
Quoted 10 lines
include/uapi/linux/perf_event.h +10
-0+* Records changes to kernel text i.e. self-modified code. +* 'len' is the number of old bytes, which is the same as the number +* of new bytes. 'bytes' contains the old bytes followed immediately +* by the new bytes. +* +* struct { +* struct perf_event_header header; +* u64 addr; +* u16 len; +* u8 bytes[];
Would it make sense to have something like:
* u16 old_len; * u16 new_len; * u8 old_bytes[old_len]; * u8 new_bytes[new_len];
That would allow using this for (short) trampolines (ftrace, optprobes etc..). {old_len=0, new_len>0} would indicate a new trampoline, while {old_len>0, new_len=0} would indicate the dissapearance of a trampoline.
include/uapi/linux/perf_event.h enum perf_event_type { · line 1027
Quoted 7 lines
include/uapi/linux/perf_event.h +5
-0+* struct sample_id sample_id; +* }; +*/ +PERF_RECORD_TEXT_POKE =19 , + PERF_RECORD_MAX, /* non-ABI */ };
Then the x86 patch, hooking up the event, can also cover kprobes and stuff.
Hunter, Adrian
reviewerQuoted 13 lines
-----Original Message-----
From: Peter Zijlstra <[email protected]>
Sent: Thursday, December 19, 2019 3:09 PM
To: Hunter, Adrian <[email protected]>
Cc: Ingo Molnar <[email protected]>; Borislav Petkov <[email protected]>; H . Peter Anvin <[email protected]>; [email protected]; Mark Rutland <[email protected]>; Alexander Shishkin <[email protected]>; Mathieu Poirier <[email protected]>; Leo Yan <[email protected]>; Arnaldo Carvalho de Melo <[email protected]>; Jiri Olsa <[email protected]>; linux- [email protected]
Subject: Re: [PATCH 1/3] perf/x86: Add perf text poke event
Quoted 7 lines
Quoted 2 lines · Adrian Hunter
Record changes to kernel text (i.e. self-modifying code) in order to support tracers like Intel PT decoding through jump labels.
I don't get the obsession with just jump-labels, we need a solution all modifying code. The fentry site usage is increasing, and optprobes are also fairly popular with a bunch of people.
Yes we need a solution for all modifying code. Jump labels are just a good place to start because they are the biggest problem by far.
Quoted 70 lines
Quoted 5 lines
A copy of the running kernel code is needed as a reference point (e.g. from
/proc/kcore). The text poke event records the old bytes and the new bytes so that the event can be processed forwards or backwards.Quoted 8 lines
In the case of Intel PT tracing, the executable code must be walked to reconstruct the control flow. For x86 a jump label text poke begins:
- write INT3 byte
- IPI-SYNC
- write instruction tail
At this point the actual control flow will be through the INT3 and handler and not hit the old or new instruction. Intel PT outputs FUP/TIP packets for the INT3, so the flow can still be decoded.Subsequently:
Quoted 9 lines
- emit
RECORD_TEXT_POKEwith the new instruction
- IPI-SYNC
- write first byte
- IPI-SYNC
So before the text poke event timestamp, the decoder will see either the old instruction flow or FUP/TIP of INT3. After the text poke event timestamp, the decoder will see either the new instruction flow or FUP/TIP of INT3. Thus decoders can use the timestamp as the point at which to modify the executable code.I feel a much better justification for the design can be found in the discussion we've had around ARM-CS.
Basically SMP instruction coherency mandates something like this, it is just a happy accident x86 already had all the bits in place.
How is something like:
"Record (single instruction) changes to the kernel text (i.e. self-modifying code) in order to support tracers like Intel PT and ARM CoreSight.
A copy of the running kernel code is needed as a reference point (e.g. from
/proc/kcore). The text poke event records the old bytes and the new bytes so that the event can be processed forwards or backwards.The basic problem is recording the modified instruction in an unambiguous manner given SMP instruction cache (in)coherence. That is, when modifying an instruction concurrently any solution with one or multiple timestamps is not sufficient:
CPU0 CPU10
1 write insn A
2 execute insn A
3 sync-I$
4Due to I$, CPU1 might execute either the old or new A. No matter where we record tracepoints on CPU0, one simply cannot tell what CPU1 will have observed, except that at 0 it must be the old one and at 4 it must be the new one.
To solve this, take inspiration from x86 text poking, which has to solve this exact problem due to variable length instruction encoding and I-fetch windows.
1) overwrite the instruction with a breakpoint and sync I$
This guarantees that that code flow will never hit the target instruction anymore, on any CPU (or rather, it will cause an exception).
2) issue the
TEXT_POKEevent3) overwrite the breakpoint with the new instruction and sync I$
Quoted 8 lines
Now we know that any execution after the
TEXT_POKEevent will either observe the breakpoint (and hit the exception) or the new instruction.So by guarding the
TEXT_POKEevent with an exception on either side; we can now tell, without doubt, which instruction another CPU will have observed."?
Ok
Quoted 32 lines
Quoted 4 lines
Signed-off-by Adrian Hunter <[email protected]>--- arch/x86/kernel/alternative.c | 37 +++++++++++++-I'm thinking it might make sense to do this x86 part in a separate patch, and just present the generic thing first:
Quoted 11 lines
include/linux/perf_event.h | 6 +++ include/uapi/linux/perf_event.h | 19 ++++++- kernel/events/core.c | 87 ++++++++++++++++++++++++++++++++- 4 files changed, 146 insertions(+), 3 deletions(-)include/uapi/linux/perf_event.h +1
-0@@ -1006,6 +1007,22 @@ enum perf_event_type {*/ PERF_RECORD_BPF_EVENT = 18, +/* include/uapi/linux/perf_event.h enum perf_event_type { · line 1020
Quoted 10 lines
include/uapi/linux/perf_event.h +10
-0+* Records changes to kernel text i.e. self-modified code. +* 'len' is the number of old bytes, which is the same as the number +* of new bytes. 'bytes' contains the old bytes followed immediately +* by the new bytes. +* +* struct { +* struct perf_event_header header; +* u64 addr; +* u16 len; +* u8 bytes[]; Would it make sense to have something like:
Quoted 8 lines
* u16 old_len; * u16 new_len; * u8 old_bytes[old_len]; * u8 new_bytes[new_len];That would allow using this for (short) trampolines (ftrace, optprobes etc..). {old_len=0, new_len>0} would indicate a new trampoline, while {old_len>0, new_len=0} would indicate the dissapearance of a trampoline.
Yes that makes sense.
Quoted 10 lines
include/uapi/linux/perf_event.h enum perf_event_type { · line 1027
Quoted 8 lines
include/uapi/linux/perf_event.h +5
-0+* struct sample_id sample_id; +* }; +*/ +PERF_RECORD_TEXT_POKE =19 , + PERF_RECORD_MAX, /* non-ABI */ };Then the x86 patch, hooking up the event, can also cover kprobes and stuff.
Ok