Skip to contentSkip to conversation

perf/x86: Add perf text poke event

RFC PATCH Applied Adrian Hunter 8 participants 34 messages Oct 25, 2019 – Nov 12, 2019 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore

Adrian Hunter

author reviewer
#0 lore raw

cover perf/x86: Add perf text poke event

Hi

Here are patches to add a text poke event to record changes to kernel text (i.e. self-modifying code) in order to support tracers like Intel PT decoding through jump labels.

One question is whether this approach can be of use to other architectures, particularly ARM-CS.

The first patch makes the kernel change and the subsequent patches are tools changes.

The first 3 tools patches add support for updating perf tools' data cache with the changed bytes.

The last 2 patches are Intel PT specific tools changes.

The patches are based on Arnaldo's tree perf/core branch.

Example:

For jump labels, the kernel needs
CONFIG_JUMP_LABEL=y
and also an easy to flip jump label is in sysctl_schedstats() which needs
CONFIG_SCHEDSTATS=y
CONFIG_PROC_SYSCTL=y

Also note the 'sudo perf record' is put into the background which, as written, needs sudo credential caching (otherwise the background task will stop awaiting the sudo password), hence the 'sudo echo' to start.

Before:

  $ sudo echo
  $ sudo perf record -o perf.data.before --kcore -a -e intel_pt//k -m,64M &
  [1] 1640
  $ cat /proc/sys/kernel/sched_schedstats
  0
  $ sudo bash -c 'echo 1 > /proc/sys/kernel/sched_schedstats'
  $ cat /proc/sys/kernel/sched_schedstats
  1
  $ sudo bash -c 'echo 0 > /proc/sys/kernel/sched_schedstats'
  $ cat /proc/sys/kernel/sched_schedstats
  0
  $ sudo kill 1640
  [ perf record: Woken up 1 times to write data ]
  [ perf record: Captured and wrote 16.635 MB perf.data.before ]
  $ perf script -i perf.data.before --itrace=e >/dev/null
  Warning:
  1946 instruction trace errors

After:

  $ sudo echo
  $ sudo perf record -o perf.data.after --kcore -a -e intel_pt//k -m,64M &
  [1] 1882
  $ cat /proc/sys/kernel/sched_schedstats
  0
  $ sudo bash -c 'echo 1 > /proc/sys/kernel/sched_schedstats'
  $ cat /proc/sys/kernel/sched_schedstats
  1
  $ sudo bash -c 'echo 0 > /proc/sys/kernel/sched_schedstats'
  $ cat /proc/sys/kernel/sched_schedstats
  0
  $ sudo kill 1882
  [ perf record: Woken up 1 times to write data ]
  [ perf record: Captured and wrote 10.893 MB perf.data.after ]
  $ perf script -i perf.data.after --itrace=e
  $

Adrian Hunter (6):
perf/x86: Add perf text poke event
perf dso: Refactor dso_cache__read() perf dso: Add dso__data_write_cache_addr() perf tools: Add support for PERF_RECORD_TEXT_POKE perf auxtrace: Add auxtrace_cache__remove() perf intel-pt: Add support for text poke events

 arch/x86/include/asm/text-patching.h      |   1 +
 arch/x86/kernel/alternative.c             |  39 ++++++++-
 include/linux/perf_event.h                |   6 ++
 include/uapi/linux/perf_event.h           |  28 ++++++-
 kernel/events/core.c                      |  90 +++++++++++++++++++-
 tools/include/uapi/linux/perf_event.h     |  28 ++++++-
 tools/perf/arch/x86/util/intel-pt.c       |   5 ++
 tools/perf/builtin-record.c               |  45 ++++++++++
 tools/perf/lib/include/perf/event.h       |   9 ++
 tools/perf/util/auxtrace.c                |  28 +++++++
 tools/perf/util/auxtrace.h                |   1 +
 tools/perf/util/dso.c                     | 135 +++++++++++++++++++++---------
 tools/perf/util/dso.h                     |   7 ++
 tools/perf/util/event.c                   |  39 +++++++++
 tools/perf/util/event.h                   |   5 ++
 tools/perf/util/evlist.h                  |   1 +
 tools/perf/util/intel-pt.c                |  71 ++++++++++++++++
 tools/perf/util/machine.c                 |  38 +++++++++
 tools/perf/util/machine.h                 |   3 +
 tools/perf/util/perf_event_attr_fprintf.c |   1 +
 tools/perf/util/record.c                  |  10 +++
 tools/perf/util/record.h                  |   1 +
 tools/perf/util/session.c                 |  22 +++++
 tools/perf/util/tool.h                    |   3 +-
 24 files changed, 570 insertions(+), 46 deletions(-)

Regards
Adrian

author reviewer
#1 lore raw

RFC PATCH 1/6 perf/x86: Add perf text poke event

Record changes to kernel text (i.e. self-modifying code) in order to support tracers like Intel PT decoding through jump labels.

A copy of the running kernel code is needed as a reference point (e.g. from /proc/kcore). The text poke event records the old bytes and the new bytes so that the event can be processed forwards or backwards.

The text poke event has 'flags' to describe when the event is sent. At present the only flag is PERF_TEXT_POKE_UPDATE which indicates the point at which tools should update their reference kernel executable to change the old bytes to the new bytes.

Other architectures may wish to emit a pair of text poke events. One before and one after a text poke. In that case, PERF_TEXT_POKE_UPDATE flag would be set on only one of the pair.

In the case of Intel PT tracing, the executable code must be walked to reconstruct the control flow. For x86 a jump label text poke begins:
- write INT3 byte
- IPI-SYNC
At this point the actual control flow will be through the INT3 and handler and not hit the old or new instruction. Intel PT outputs FUP/TIP packets for the INT3, so the flow can still be decoded. Subsequently:
- emit RECORD_TEXT_POKE with the new instruction
- write instruction tail
- IPI-SYNC
- write first byte
- IPI-SYNC
So before the text poke event timestamp, the decoder will see either the old instruction flow or FUP/TIP of INT3. After the text poke event timestamp, the decoder will see either the new instruction flow or FUP/TIP of INT3. Thus decoders can use the timestamp as the point at which to modify the executable code.

---
 arch/x86/include/asm/text-patching.h |  1 +
 arch/x86/kernel/alternative.c        | 39 +++++++++++-
 include/linux/perf_event.h           |  6 ++
 include/uapi/linux/perf_event.h      | 28 ++++++++-
 kernel/events/core.c                 | 90 +++++++++++++++++++++++++++-
 5 files changed, 161 insertions(+), 3 deletions(-)
arch/x86/include/asm/text-patching.h +1 -0
index 5e8319bb207a..873141765d8e 100644
@@ -30,6 +30,7 @@ struct text_poke_loc {
 	void *addr;
 	size_t len;
 	const char opcode[POKE_MAX_OPCODE_SIZE];
+	u8 old;
 };
 
 extern void text_poke_early(void *addr, const void *opcode, size_t len);
arch/x86/kernel/alternative.c +38 -1
index 9d3a971ea364..9488f0fa7e22 100644
@@ -3,6 +3,7 @@
 
 #include <linux/module.h>
 #include <linux/sched.h>
+#include <linux/perf_event.h>
 #include <linux/mutex.h>
 #include <linux/list.h>
 #include <linux/stringify.h>
@@ -1045,8 +1046,10 @@ void text_poke_bp_batch(struct text_poke_loc *tp, unsigned int nr_entries)
 	/*
 	 * First step: add a int3 trap to the address that will be patched.
 	 */
-	for (i = 0; i < nr_entries; i++)
+	for (i = 0; i < nr_entries; i++) {
+		tp[i].old = *(u8 *)tp[i].addr;
 		text_poke(tp[i].addr, &int3, sizeof(int3));
+	}
 
 	on_each_cpu(do_sync_core, NULL, 1);
 
@@ -1054,12 +1057,46 @@ void text_poke_bp_batch(struct text_poke_loc *tp, unsigned int nr_entries)
 	 * Second step: update all but the first byte of the patched range.
 	 */
 	for (i = 0; i < nr_entries; i++) {
+		u8 old[POKE_MAX_OPCODE_SIZE] = { tp[i].old, };
+
+		memcpy(old + sizeof(int3),
+		       tp[i].addr + sizeof(int3),
+		       tp[i].len - sizeof(int3));
+
 		if (tp[i].len - sizeof(int3) > 0) {
 			text_poke((char *)tp[i].addr + sizeof(int3),
 				  (const char *)tp[i].opcode + sizeof(int3),
 				  tp[i].len - sizeof(int3));
 			patched_all_but_first++;
 		}
+
+		/*
+		 * Emit a perf event to record the text poke, primarily to
+		 * support Intel PT decoding which must walk the executable code
+		 * to reconstruct the trace. The flow up to here is:
+		 *   - write INT3 byte
+		 *   - IPI-SYNC
+		 * At this point the actual control flow will be through the
+		 * INT3 and handler and not hit the old or new instruction.
+		 * Intel PT outputs FUP/TIP packets for the INT3, so the flow
+		 * can still be decoded. Subsequently:
+		 *   - emit RECORD_TEXT_POKE with the new instruction
+		 *   - write instruction tail
+		 *   - IPI-SYNC
+		 *   - write first byte
+		 *   - IPI-SYNC
+		 * So before the text poke event timestamp, the decoder will see
+		 * either the old instruction flow or FUP/TIP of INT3. After the
+		 * text poke event timestamp, the decoder will see either the
+		 * new instruction flow or FUP/TIP of INT3. Thus decoders can
+		 * use the timestamp as the point at which to modify the
+		 * executable code.
+		 * The old instruction is recorded so that the event can be
+		 * processed forwards or backwards.
+		 */
+		perf_event_text_poke((unsigned long)tp[i].addr,
+				     PERF_TEXT_POKE_UPDATE, old, tp[i].opcode,
+				     tp[i].len);
 	}
 
 	if (patched_all_but_first) {
include/linux/perf_event.h +6 -0
index 61448c19a132..db88b7ade8c6 100644
@@ -1183,6 +1183,8 @@ extern void perf_event_exec(void);
 extern void perf_event_comm(struct task_struct *tsk, bool exec);
 extern void perf_event_namespaces(struct task_struct *tsk);
 extern void perf_event_fork(struct task_struct *tsk);
+extern void perf_event_text_poke(unsigned long addr, u16 flags, const void *old_bytes,
+				 const void *new_bytes, size_t len);
 
 /* Callchains */
 DECLARE_PER_CPU(struct perf_callchain_entry, perf_callchain_entry);
@@ -1406,6 +1408,10 @@ static inline void perf_event_exec(void)				{ }
 static inline void perf_event_comm(struct task_struct *tsk, bool exec)	{ }
 static inline void perf_event_namespaces(struct task_struct *tsk)	{ }
 static inline void perf_event_fork(struct task_struct *tsk)		{ }
+static inline void perf_event_text_poke(unsigned long addr, u16 flags,
+					const void *old_bytes,
+					const void *new_bytes,
+					size_t len)			{ }
 static inline void perf_event_init(void)				{ }
 static inline int  perf_swevent_get_recursion_context(void)		{ return -1; }
 static inline void perf_swevent_put_recursion_context(int rctx)		{ }
include/uapi/linux/perf_event.h +27 -1
index bb7b271397a6..c8d1f52a7fce 100644
@@ -375,7 +375,8 @@ struct perf_event_attr {
 				ksymbol        :  1, /* include ksymbol events */
 				bpf_event      :  1, /* include bpf events */
 				aux_output     :  1, /* generate AUX records instead of events */
-				__reserved_1   : 32;
+				text_poke      :  1, /* include text poke events */
+				__reserved_1   : 31;
 
 	union {
 		__u32		wakeup_events;	  /* wakeup every n events */
@@ -1000,6 +1001,26 @@ enum perf_event_type {
 	 */
 	PERF_RECORD_BPF_EVENT			= 18,
 
+	/*
+	 * Records changes to kernel text i.e. self-modified code.
+	 * 'flags' has PERF_TEXT_POKE_UPDATE (i.e. bit 0) set to
+	 * indicate to tools to update old bytes to new bytes in the
+	 * executable image.
+	 * 'len' is the number of old bytes, which is the same as the number
+	 * of new bytes. 'bytes' contains the old bytes followed immediately
+	 * by the new bytes.
+	 *
+	 * struct {
+	 *	struct perf_event_header	header;
+	 *	u64				addr;
+	 *	u16				flags;
+	 *	u16				len;
+	 *	u8				bytes[];
+	 *	struct sample_id		sample_id;
+	 * };
+	 */
+	PERF_RECORD_TEXT_POKE			= 19,
+
 	PERF_RECORD_MAX,			/* non-ABI */
 };
 
@@ -1041,6 +1062,11 @@ enum perf_callchain_context {
 #define PERF_AUX_FLAG_PARTIAL		0x04	/* record contains gaps */
 #define PERF_AUX_FLAG_COLLISION		0x08	/* sample collided with another */
 
+/**
+ * PERF_RECORD_TEXT_POKE::flags bits
+ */
+#define PERF_TEXT_POKE_UPDATE		0x01	/* update old bytes to new bytes */
+
 #define PERF_FLAG_FD_NO_GROUP		(1UL << 0)
 #define PERF_FLAG_FD_OUTPUT		(1UL << 1)
 #define PERF_FLAG_PID_CGROUP		(1UL << 2) /* pid=cgroup id, per-cpu mode only */
kernel/events/core.c +89 -1

120 lines not on this page · raw message

Signature · 1 line
2.17.1
#2 lore raw

Hi Adrian,

Quoted 7 lines · Adrian Hunter

Record changes to kernel text (i.e. self-modifying code) in order to support tracers like Intel PT decoding through jump labels.

A copy of the running kernel code is needed as a reference point (e.g. from /proc/kcore). The text poke event records the old bytes and the new bytes so that the event can be processed forwards or backwards.

Quoted 8 lines

The text poke event has 'flags' to describe when the event is sent. At present the only flag is PERF_TEXT_POKE_UPDATE which indicates the point at which tools should update their reference kernel executable to change the old bytes to the new bytes.

Other architectures may wish to emit a pair of text poke events. One before and one after a text poke. In that case, PERF_TEXT_POKE_UPDATE flag would be set on only one of the pair.

Thanks a lot for the patch set.

Below is my understanding for implementation 'emit a pair of text poke events' as you mentioned:

Since Arm64 instruction is fixed size, it doesn't need to rely on INT3 liked mechanism to replace instructions and directly uses two operations to alter instruction (modify instruction and flush icache line). So Arm64 has no chance to send perf event in the middle of altering instruction.

Thus we can send pair of text poke events in the kernel:

perf_event_text_poke(PERF_TEXT_POKE_UPDATE_PREV)

Change instruction
Flush icache

perf_event_text_poke(PERF_TEXT_POKE_UPDATE_POST)

In the userspace, if perf tool detects the instruction is changed from nop to branch, we need to update dso cache for the 'PERF_TEXT_POKE_UPDATE_PREV' event; if detect the instruction is changed from branch to nop, we need to update dso cache for 'PERF_TEXT_POKE_UPDATE_POST' event. The main idea is to ensure the branch instructions can be safely contained in the dso file and any branch samples can read out correct branch instruction.

Could you confirm this is the same with your understanding? Or I miss anything? I personally even think the pair events can be used for different arches (e.g. the solution can be reused on Arm64/x86, etc).

Quoted 8 lines

In the case of Intel PT tracing, the executable code must be walked to reconstruct the control flow. For x86 a jump label text poke begins:
- write INT3 byte
- IPI-SYNC
At this point the actual control flow will be through the INT3 and handler and not hit the old or new instruction. Intel PT outputs FUP/TIP packets for the INT3, so the flow can still be decoded. Subsequently:
- emit RECORD_TEXT_POKE with the new instruction

s/RECORD_TEXT_POKE/PERF_TEXT_POKE_UPDATE

And I think emit the perf event is after the item 'write instruction tail' but not before it. So the commit log is not consistent with the change in the patch.

Quoted 7 lines

- write instruction tail
- IPI-SYNC
- write first byte
- IPI-SYNC
So before the text poke event timestamp, the decoder will see either the old instruction flow or FUP/TIP of INT3. After the text poke event timestamp, the decoder will see either the new instruction flow or FUP/TIP

I will check internally for timestamp on Arm/Arm64 ...

Thanks,
Leo Yan

Quoted 304 lines

of INT3. Thus decoders can use the timestamp as the point at which to modify the executable code.

---
 arch/x86/include/asm/text-patching.h |  1 +
 arch/x86/kernel/alternative.c        | 39 +++++++++++-
 include/linux/perf_event.h           |  6 ++
 include/uapi/linux/perf_event.h      | 28 ++++++++-
 kernel/events/core.c                 | 90 +++++++++++++++++++++++++++-
 5 files changed, 161 insertions(+), 3 deletions(-)
arch/x86/include/asm/text-patching.h +1 -0
index 5e8319bb207a..873141765d8e 100644
@@ -30,6 +30,7 @@ struct text_poke_loc {
 	void *addr;
 	size_t len;
 	const char opcode[POKE_MAX_OPCODE_SIZE];
+	u8 old;
 };
 
 extern void text_poke_early(void *addr, const void *opcode, size_t len);
arch/x86/kernel/alternative.c +38 -1
index 9d3a971ea364..9488f0fa7e22 100644
@@ -3,6 +3,7 @@
 
 #include <linux/module.h>
 #include <linux/sched.h>
+#include <linux/perf_event.h>
 #include <linux/mutex.h>
 #include <linux/list.h>
 #include <linux/stringify.h>
@@ -1045,8 +1046,10 @@ void text_poke_bp_batch(struct text_poke_loc *tp, unsigned int nr_entries)
 	/*
 	 * First step: add a int3 trap to the address that will be patched.
 	 */
-	for (i = 0; i < nr_entries; i++)
+	for (i = 0; i < nr_entries; i++) {
+		tp[i].old = *(u8 *)tp[i].addr;
 		text_poke(tp[i].addr, &int3, sizeof(int3));
+	}
 
 	on_each_cpu(do_sync_core, NULL, 1);
 
@@ -1054,12 +1057,46 @@ void text_poke_bp_batch(struct text_poke_loc *tp, unsigned int nr_entries)
 	 * Second step: update all but the first byte of the patched range.
 	 */
 	for (i = 0; i < nr_entries; i++) {
+		u8 old[POKE_MAX_OPCODE_SIZE] = { tp[i].old, };
+
+		memcpy(old + sizeof(int3),
+		       tp[i].addr + sizeof(int3),
+		       tp[i].len - sizeof(int3));
+
 		if (tp[i].len - sizeof(int3) > 0) {
 			text_poke((char *)tp[i].addr + sizeof(int3),
 				  (const char *)tp[i].opcode + sizeof(int3),
 				  tp[i].len - sizeof(int3));
 			patched_all_but_first++;
 		}
+
+		/*
+		 * Emit a perf event to record the text poke, primarily to
+		 * support Intel PT decoding which must walk the executable code
+		 * to reconstruct the trace. The flow up to here is:
+		 *   - write INT3 byte
+		 *   - IPI-SYNC
+		 * At this point the actual control flow will be through the
+		 * INT3 and handler and not hit the old or new instruction.
+		 * Intel PT outputs FUP/TIP packets for the INT3, so the flow
+		 * can still be decoded. Subsequently:
+		 *   - emit RECORD_TEXT_POKE with the new instruction
+		 *   - write instruction tail
+		 *   - IPI-SYNC
+		 *   - write first byte
+		 *   - IPI-SYNC
+		 * So before the text poke event timestamp, the decoder will see
+		 * either the old instruction flow or FUP/TIP of INT3. After the
+		 * text poke event timestamp, the decoder will see either the
+		 * new instruction flow or FUP/TIP of INT3. Thus decoders can
+		 * use the timestamp as the point at which to modify the
+		 * executable code.
+		 * The old instruction is recorded so that the event can be
+		 * processed forwards or backwards.
+		 */
+		perf_event_text_poke((unsigned long)tp[i].addr,
+				     PERF_TEXT_POKE_UPDATE, old, tp[i].opcode,
+				     tp[i].len);
 	}
 
 	if (patched_all_but_first) {
include/linux/perf_event.h +6 -0
index 61448c19a132..db88b7ade8c6 100644
@@ -1183,6 +1183,8 @@ extern void perf_event_exec(void);
 extern void perf_event_comm(struct task_struct *tsk, bool exec);
 extern void perf_event_namespaces(struct task_struct *tsk);
 extern void perf_event_fork(struct task_struct *tsk);
+extern void perf_event_text_poke(unsigned long addr, u16 flags, const void *old_bytes,
+				 const void *new_bytes, size_t len);
 
 /* Callchains */
 DECLARE_PER_CPU(struct perf_callchain_entry, perf_callchain_entry);
@@ -1406,6 +1408,10 @@ static inline void perf_event_exec(void)				{ }
 static inline void perf_event_comm(struct task_struct *tsk, bool exec)	{ }
 static inline void perf_event_namespaces(struct task_struct *tsk)	{ }
 static inline void perf_event_fork(struct task_struct *tsk)		{ }
+static inline void perf_event_text_poke(unsigned long addr, u16 flags,
+					const void *old_bytes,
+					const void *new_bytes,
+					size_t len)			{ }
 static inline void perf_event_init(void)				{ }
 static inline int  perf_swevent_get_recursion_context(void)		{ return -1; }
 static inline void perf_swevent_put_recursion_context(int rctx)		{ }
include/uapi/linux/perf_event.h +27 -1
index bb7b271397a6..c8d1f52a7fce 100644
@@ -375,7 +375,8 @@ struct perf_event_attr {
 				ksymbol        :  1, /* include ksymbol events */
 				bpf_event      :  1, /* include bpf events */
 				aux_output     :  1, /* generate AUX records instead of events */
-				__reserved_1   : 32;
+				text_poke      :  1, /* include text poke events */
+				__reserved_1   : 31;
 
 	union {
 		__u32		wakeup_events;	  /* wakeup every n events */
@@ -1000,6 +1001,26 @@ enum perf_event_type {
 	 */
 	PERF_RECORD_BPF_EVENT			= 18,
 
+	/*
+	 * Records changes to kernel text i.e. self-modified code.
+	 * 'flags' has PERF_TEXT_POKE_UPDATE (i.e. bit 0) set to
+	 * indicate to tools to update old bytes to new bytes in the
+	 * executable image.
+	 * 'len' is the number of old bytes, which is the same as the number
+	 * of new bytes. 'bytes' contains the old bytes followed immediately
+	 * by the new bytes.
+	 *
+	 * struct {
+	 *	struct perf_event_header	header;
+	 *	u64				addr;
+	 *	u16				flags;
+	 *	u16				len;
+	 *	u8				bytes[];
+	 *	struct sample_id		sample_id;
+	 * };
+	 */
+	PERF_RECORD_TEXT_POKE			= 19,
+
 	PERF_RECORD_MAX,			/* non-ABI */
 };
 
@@ -1041,6 +1062,11 @@ enum perf_callchain_context {
 #define PERF_AUX_FLAG_PARTIAL		0x04	/* record contains gaps */
 #define PERF_AUX_FLAG_COLLISION		0x08	/* sample collided with another */
 
+/**
+ * PERF_RECORD_TEXT_POKE::flags bits
+ */
+#define PERF_TEXT_POKE_UPDATE		0x01	/* update old bytes to new bytes */
+
 #define PERF_FLAG_FD_NO_GROUP		(1UL << 0)
 #define PERF_FLAG_FD_OUTPUT		(1UL << 1)
 #define PERF_FLAG_PID_CGROUP		(1UL << 2) /* pid=cgroup id, per-cpu mode only */
kernel/events/core.c +89 -1
index 9ec0b0bfddbd..666e094e8ed9 100644
@@ -386,6 +386,7 @@ static atomic_t nr_freq_events __read_mostly;
 static atomic_t nr_switch_events __read_mostly;
 static atomic_t nr_ksymbol_events __read_mostly;
 static atomic_t nr_bpf_events __read_mostly;
+static atomic_t nr_text_poke_events __read_mostly;
 
 static LIST_HEAD(pmus);
 static DEFINE_MUTEX(pmus_lock);
@@ -4351,7 +4352,7 @@ static bool is_sb_event(struct perf_event *event)
 	if (attr->mmap || attr->mmap_data || attr->mmap2 ||
 	    attr->comm || attr->comm_exec ||
 	    attr->task || attr->ksymbol ||
-	    attr->context_switch ||
+	    attr->context_switch || attr->text_poke ||
 	    attr->bpf_event)
 		return true;
 	return false;
@@ -4425,6 +4426,8 @@ static void unaccount_event(struct perf_event *event)
 		atomic_dec(&nr_ksymbol_events);
 	if (event->attr.bpf_event)
 		atomic_dec(&nr_bpf_events);
+	if (event->attr.text_poke)
+		atomic_dec(&nr_text_poke_events);
 
 	if (dec) {
 		if (!atomic_add_unless(&perf_sched_count, -1, 1))
@@ -8070,6 +8073,89 @@ void perf_event_bpf_event(struct bpf_prog *prog,
 	perf_iterate_sb(perf_event_bpf_output, &bpf_event, NULL);
 }
 
+struct perf_text_poke_event {
+	const void		*old_bytes;
+	const void		*new_bytes;
+	size_t			pad;
+	u16			flags;
+	u16			len;
+
+	struct {
+		struct perf_event_header	header;
+
+		u64				addr;
+	} event_id;
+};
+
+static int perf_event_text_poke_match(struct perf_event *event)
+{
+	return event->attr.text_poke;
+}
+
+static void perf_event_text_poke_output(struct perf_event *event, void *data)
+{
+	struct perf_text_poke_event *text_poke_event = data;
+	struct perf_output_handle handle;
+	struct perf_sample_data sample;
+	u64 padding = 0;
+	int ret;
+
+	if (!perf_event_text_poke_match(event))
+		return;
+
+	perf_event_header__init_id(&text_poke_event->event_id.header, &sample, event);
+
+	ret = perf_output_begin(&handle, event, text_poke_event->event_id.header.size);
+	if (ret)
+		return;
+
+	perf_output_put(&handle, text_poke_event->event_id);
+	perf_output_put(&handle, text_poke_event->flags);
+	perf_output_put(&handle, text_poke_event->len);
+
+	__output_copy(&handle, text_poke_event->old_bytes, text_poke_event->len);
+	__output_copy(&handle, text_poke_event->new_bytes, text_poke_event->len);
+
+	if (text_poke_event->pad)
+		__output_copy(&handle, &padding, text_poke_event->pad);
+
+	perf_event__output_id_sample(event, &handle, &sample);
+
+	perf_output_end(&handle);
+}
+
+void perf_event_text_poke(unsigned long addr, u16 flags, const void *old_bytes,
+			  const void *new_bytes, size_t len)
+{
+	struct perf_text_poke_event text_poke_event;
+	size_t tot, pad;
+
+	if (!atomic_read(&nr_text_poke_events))
+		return;
+
+	tot  = sizeof(text_poke_event.flags) +
+	       sizeof(text_poke_event.len) + (len << 1);
+	pad  = ALIGN(tot, sizeof(u64)) - tot;
+
+	text_poke_event = (struct perf_text_poke_event){
+		.old_bytes    = old_bytes,
+		.new_bytes    = new_bytes,
+		.pad          = pad,
+		.flags        = flags,
+		.len          = len,
+		.event_id  = {
+			.header = {
+				.type = PERF_RECORD_TEXT_POKE,
+				.misc = PERF_RECORD_MISC_KERNEL,
+				.size = sizeof(text_poke_event.event_id) + tot + pad,
+			},
+			.addr = addr,
+		},
+	};
+
+	perf_iterate_sb(perf_event_text_poke_output, &text_poke_event, NULL);
+}
+
 void perf_event_itrace_started(struct perf_event *event)
 {
 	event->attach_state |= PERF_ATTACH_ITRACE;
@@ -10356,6 +10442,8 @@ static void account_event(struct perf_event *event)
 		atomic_inc(&nr_ksymbol_events);
 	if (event->attr.bpf_event)
 		atomic_inc(&nr_bpf_events);
+	if (event->attr.text_poke)
+		atomic_inc(&nr_text_poke_events);
 
 	if (inc) {
 		/*
Signature · 1 line
2.17.1
maintainer
#3 lore raw
Quoted 21 lines · Leo Yan

Hi Adrian,

Quoted 7 lines · Adrian Hunter

Record changes to kernel text (i.e. self-modifying code) in order to support tracers like Intel PT decoding through jump labels.

A copy of the running kernel code is needed as a reference point (e.g. from /proc/kcore). The text poke event records the old bytes and the new bytes so that the event can be processed forwards or backwards.

Quoted 8 lines

The text poke event has 'flags' to describe when the event is sent. At present the only flag is PERF_TEXT_POKE_UPDATE which indicates the point at which tools should update their reference kernel executable to change the old bytes to the new bytes.

Other architectures may wish to emit a pair of text poke events. One before and one after a text poke. In that case, PERF_TEXT_POKE_UPDATE flag would be set on only one of the pair.

Thanks a lot for the patch set.

Quoted 8 lines

Below is my understanding for implementation 'emit a pair of text poke events' as you mentioned:

Since Arm64 instruction is fixed size, it doesn't need to rely on INT3 liked mechanism to replace instructions and directly uses two operations to alter instruction (modify instruction and flush icache line). So Arm64 has no chance to send perf event in the middle of altering instruction.

Right.

Quoted 11 lines

Thus we can send pair of text poke events in the kernel:

perf_event_text_poke(PERF_TEXT_POKE_UPDATE_PREV)

Change instruction
Flush icache

perf_event_text_poke(PERF_TEXT_POKE_UPDATE_POST)

In the userspace, if perf tool detects the instruction is changed from nop to branch,

There is _far_ more text poking than just jump_label's NOP/JMP transitions. Ftrace also does NOP/CALL, CALL/CALL, and the static_call infrastructure that I posted here:

https://lkml.kernel.org/r/[email protected]

Has: JMP/RET, JMP/JMP and if it has inline patching support: NOP/CALL, CALL/CALL, patching.

Anyway, the below argument doesn't care much, it works for NOP/JMP just fine.

Quoted 10 lines

we need to update dso cache for the 'PERF_TEXT_POKE_UPDATE_PREV' event; if detect the instruction is changed from branch to nop, we need to update dso cache for 'PERF_TEXT_POKE_UPDATE_POST' event. The main idea is to ensure the branch instructions can be safely contained in the dso file and any branch samples can read out correct branch instruction.

Could you confirm this is the same with your understanding? Or I miss anything? I personally even think the pair events can be used for different arches (e.g. the solution can be reused on Arm64/x86, etc).

So the problem we have with PT is that it is a bit-stream of branch taken/not-taken decisions. In order to decode that we need to have an accurate view of the unconditional code flow.

Both NOP/JMP are unconditional and we need to exactly know which of the two was encountered.

With your scheme, I don't see how we can ever actually know that. When we get the PRE event, all we really know is that we're going to change a specific instruction into another. And at the POST event we know it has been done. But in between these two events, we have no clue which of the two instructions is live on which CPU (two CPUs might in fact have a different live instruction at the same time).

This means we _cannot_ unambiguously decode a taken/not-taken decision stream.

Does CS have this same problem, and how would the PRE/POST events help with that?

So our (x86) horrible (variable instruction length induced) complexity for text poking is actually in our advantage this one time. The exception and exception-return paths allow us to unambiguously know what happens around the time of poking.

#4 lore raw

Hi Peter,

[ + Mike Leach ]

Quoted 45 lines · Peter Zijlstra
Quoted 21 lines · Leo Yan

Hi Adrian,

Quoted 7 lines · Adrian Hunter

Record changes to kernel text (i.e. self-modifying code) in order to support tracers like Intel PT decoding through jump labels.

A copy of the running kernel code is needed as a reference point (e.g. from /proc/kcore). The text poke event records the old bytes and the new bytes so that the event can be processed forwards or backwards.

Quoted 8 lines

The text poke event has 'flags' to describe when the event is sent. At present the only flag is PERF_TEXT_POKE_UPDATE which indicates the point at which tools should update their reference kernel executable to change the old bytes to the new bytes.

Other architectures may wish to emit a pair of text poke events. One before and one after a text poke. In that case, PERF_TEXT_POKE_UPDATE flag would be set on only one of the pair.

Thanks a lot for the patch set.

Quoted 8 lines

Below is my understanding for implementation 'emit a pair of text poke events' as you mentioned:

Since Arm64 instruction is fixed size, it doesn't need to rely on INT3 liked mechanism to replace instructions and directly uses two operations to alter instruction (modify instruction and flush icache line). So Arm64 has no chance to send perf event in the middle of altering instruction.

Right.

Quoted 11 lines

Thus we can send pair of text poke events in the kernel:

perf_event_text_poke(PERF_TEXT_POKE_UPDATE_PREV)

Change instruction
Flush icache

perf_event_text_poke(PERF_TEXT_POKE_UPDATE_POST)

In the userspace, if perf tool detects the instruction is changed from nop to branch,

Quoted 8 lines

There is _far_ more text poking than just jump_label's NOP/JMP transitions. Ftrace also does NOP/CALL, CALL/CALL, and the static_call infrastructure that I posted here:

https://lkml.kernel.org/r/[email protected]

Has: JMP/RET, JMP/JMP and if it has inline patching support: NOP/CALL, CALL/CALL, patching.

Thanks for the info. I took a bit time to look your patch set and Steven's patch set for dynamic function, though don't completely understand, but get more sense for the context.

Quoted 2 lines

Anyway, the below argument doesn't care much, it works for NOP/JMP just fine.

We can support NOP/JMP case as the first step, but later should can extend to support other transitions.

Quoted 11 lines
Quoted 10 lines

we need to update dso cache for the 'PERF_TEXT_POKE_UPDATE_PREV' event; if detect the instruction is changed from branch to nop, we need to update dso cache for 'PERF_TEXT_POKE_UPDATE_POST' event. The main idea is to ensure the branch instructions can be safely contained in the dso file and any branch samples can read out correct branch instruction.

Could you confirm this is the same with your understanding? Or I miss anything? I personally even think the pair events can be used for different arches (e.g. the solution can be reused on Arm64/x86, etc).

Quoted 6 lines

So the problem we have with PT is that it is a bit-stream of branch taken/not-taken decisions. In order to decode that we need to have an accurate view of the unconditional code flow.

Both NOP/JMP are unconditional and we need to exactly know which of the two was encountered.

If I understand correctly, PT decoder needs to read out instructions from dso and decide the instruction type (NOP or JMP), and finally generate the accurate code flow.

So PT decoder relies on (cached) DSO for decoding. As I know, this might be different from Arm CS, since Arm CS decoder is merely generate packets and it doesn't need to rely on DSO for decoding.

I think my answer is not very convinced, in case I mislead, loop Mike to confirm for this.

Quoted 7 lines

With your scheme, I don't see how we can ever actually know that. When we get the PRE event, all we really know is that we're going to change a specific instruction into another. And at the POST event we know it has been done. But in between these two events, we have no clue which of the two instructions is live on which CPU (two CPUs might in fact have a different live instruction at the same time).

Quoted 5 lines

This means we _cannot_ unambiguously decode a taken/not-taken decision stream.

Does CS have this same problem, and how would the PRE/POST events help with that?

My purpose is to use PRE event and POST event to update cached DSO, thus perf tool can read out 'correct' instructions and fill them into instruction/branch samples. On the other hand, as mentioned, I don't aware the instruction read out from DSO will be used for Arm CS decoder; Arm CS also reads the instructions from DSO, usually it's used to decide instruction size or decide the sample flags (flags for CALL/RETURN/hw int, etc...), but it isn't used for decoder.

@Mike, please help confirm for this as well, from Arm CoreSight's decoder pespective.

Quoted 4 lines

So our (x86) horrible (variable instruction length induced) complexity for text poking is actually in our advantage this one time. The exception and exception-return paths allow us to unambiguously know what happens around the time of poking.

Thanks a lot for the info!
Leo.

#5 lore raw

Hi Leo,

Quoted 92 lines · Leo Yan

Hi Peter,

[ + Mike Leach ]

Quoted 45 lines · Peter Zijlstra
Quoted 21 lines · Leo Yan

Hi Adrian,

Quoted 7 lines · Adrian Hunter

Record changes to kernel text (i.e. self-modifying code) in order to support tracers like Intel PT decoding through jump labels.

A copy of the running kernel code is needed as a reference point (e.g. from /proc/kcore). The text poke event records the old bytes and the new bytes so that the event can be processed forwards or backwards.

Quoted 8 lines

The text poke event has 'flags' to describe when the event is sent. At present the only flag is PERF_TEXT_POKE_UPDATE which indicates the point at which tools should update their reference kernel executable to change the old bytes to the new bytes.

Other architectures may wish to emit a pair of text poke events. One before and one after a text poke. In that case, PERF_TEXT_POKE_UPDATE flag would be set on only one of the pair.

Thanks a lot for the patch set.

Quoted 8 lines

Below is my understanding for implementation 'emit a pair of text poke events' as you mentioned:

Since Arm64 instruction is fixed size, it doesn't need to rely on INT3 liked mechanism to replace instructions and directly uses two operations to alter instruction (modify instruction and flush icache line). So Arm64 has no chance to send perf event in the middle of altering instruction.

Right.

Quoted 11 lines

Thus we can send pair of text poke events in the kernel:

perf_event_text_poke(PERF_TEXT_POKE_UPDATE_PREV)

Change instruction
Flush icache

perf_event_text_poke(PERF_TEXT_POKE_UPDATE_POST)

In the userspace, if perf tool detects the instruction is changed from nop to branch,

Quoted 8 lines

There is _far_ more text poking than just jump_label's NOP/JMP transitions. Ftrace also does NOP/CALL, CALL/CALL, and the static_call infrastructure that I posted here:

https://lkml.kernel.org/r/[email protected]

Has: JMP/RET, JMP/JMP and if it has inline patching support: NOP/CALL, CALL/CALL, patching.

Thanks for the info. I took a bit time to look your patch set and Steven's patch set for dynamic function, though don't completely understand, but get more sense for the context.

Quoted 2 lines

Anyway, the below argument doesn't care much, it works for NOP/JMP just fine.

We can support NOP/JMP case as the first step, but later should can extend to support other transitions.

Quoted 11 lines
Quoted 10 lines

we need to update dso cache for the 'PERF_TEXT_POKE_UPDATE_PREV' event; if detect the instruction is changed from branch to nop, we need to update dso cache for 'PERF_TEXT_POKE_UPDATE_POST' event. The main idea is to ensure the branch instructions can be safely contained in the dso file and any branch samples can read out correct branch instruction.

Could you confirm this is the same with your understanding? Or I miss anything? I personally even think the pair events can be used for different arches (e.g. the solution can be reused on Arm64/x86, etc).

Quoted 6 lines

So the problem we have with PT is that it is a bit-stream of branch taken/not-taken decisions. In order to decode that we need to have an accurate view of the unconditional code flow.

Both NOP/JMP are unconditional and we need to exactly know which of the two was encountered.

If I understand correctly, PT decoder needs to read out instructions from dso and decide the instruction type (NOP or JMP), and finally generate the accurate code flow.

Quoted 7 lines

So PT decoder relies on (cached) DSO for decoding. As I know, this might be different from Arm CS, since Arm CS decoder is merely generate packets and it doesn't need to rely on DSO for decoding.

I think my answer is not very convinced, in case I mislead, loop Mike to confirm for this.

The CoreSight decoder needs the same information. When a branch / call instruction is traced an atom is generated in the trace to determine if that call as taken or not.
During the decode process the decoder walks the code image to match up atoms to call / branch instructions - which means for correct trace decode we must have an accurate code image as executed, as otherwise trace decode will go wrong.
Within perf, the OpenCSD decoder will call back into perf to ask for the memory image for a given address - which perf will supply from the list of .so / executable files it has in the .debug directory.

Self modifying code presents an issue in this respect.

Correlation of the modification information with the captured trace becomes the primary concern.

Regards

Mike

Quoted 31 lines
Quoted 7 lines

With your scheme, I don't see how we can ever actually know that. When we get the PRE event, all we really know is that we're going to change a specific instruction into another. And at the POST event we know it has been done. But in between these two events, we have no clue which of the two instructions is live on which CPU (two CPUs might in fact have a different live instruction at the same time).

Quoted 5 lines

This means we _cannot_ unambiguously decode a taken/not-taken decision stream.

Does CS have this same problem, and how would the PRE/POST events help with that?

My purpose is to use PRE event and POST event to update cached DSO, thus perf tool can read out 'correct' instructions and fill them into instruction/branch samples. On the other hand, as mentioned, I don't aware the instruction read out from DSO will be used for Arm CS decoder; Arm CS also reads the instructions from DSO, usually it's used to decide instruction size or decide the sample flags (flags for CALL/RETURN/hw int, etc...), but it isn't used for decoder.

@Mike, please help confirm for this as well, from Arm CoreSight's decoder pespective.

Quoted 4 lines

So our (x86) horrible (variable instruction length induced) complexity for text poking is actually in our advantage this one time. The exception and exception-return paths allow us to unambiguously know what happens around the time of poking.

Thanks a lot for the info!
Leo.

Signature · 3 lines
Mike Leach
Principal Engineer, ARM Ltd.
Manchester Design Centre. UK
maintainer
#6 lore raw
In reply to Leo Yan
Quoted 8 lines · Leo Yan
Quoted 4 lines · Peter Zijlstra

On Wed, Oct 30, 2019 at 06:47:47PM +0800, Leo Yan wrote:

Anyway, the below argument doesn't care much, it works for NOP/JMP just fine.

We can support NOP/JMP case as the first step, but later should can extend to support other transitions.

Since all instructions (with the possible exception of RET) are unconditional branch instructions: NOP, JMP, CALL. It makes no read difference to the argument below.

( I'm thinking RET might be special in that it reads the return address from the stack and therefore must emit the whole IP into the stream, as we cannot know the stack state )

Quoted 18 lines
Quoted 11 lines
Quoted 10 lines

we need to update dso cache for the 'PERF_TEXT_POKE_UPDATE_PREV' event; if detect the instruction is changed from branch to nop, we need to update dso cache for 'PERF_TEXT_POKE_UPDATE_POST' event. The main idea is to ensure the branch instructions can be safely contained in the dso file and any branch samples can read out correct branch instruction.

Could you confirm this is the same with your understanding? Or I miss anything? I personally even think the pair events can be used for different arches (e.g. the solution can be reused on Arm64/x86, etc).

Quoted 6 lines

So the problem we have with PT is that it is a bit-stream of branch taken/not-taken decisions. In order to decode that we need to have an accurate view of the unconditional code flow.

Both NOP/JMP are unconditional and we need to exactly know which of the two was encountered.

Quoted 7 lines

If I understand correctly, PT decoder needs to read out instructions from dso and decide the instruction type (NOP or JMP), and finally generate the accurate code flow.

So PT decoder relies on (cached) DSO for decoding. As I know, this might be different from Arm CS, since Arm CS decoder is merely generate packets and it doesn't need to rely on DSO for decoding.

Given a start point (from a start or sync packet) we scan the instruction stream forward until the first conditional branch instruction. Then we consume the next available branch decision bit to know where to continue.

So yes, we need to have a correct text image available for this to work.

Quoted 10 lines
Quoted 10 lines

With your scheme, I don't see how we can ever actually know that. When we get the PRE event, all we really know is that we're going to change a specific instruction into another. And at the POST event we know it has been done. But in between these two events, we have no clue which of the two instructions is live on which CPU (two CPUs might in fact have a different live instruction at the same time).

This means we _cannot_ unambiguously decode a taken/not-taken decision stream.

Quoted 6 lines
Quoted 2 lines

Does CS have this same problem, and how would the PRE/POST events help with that?

My purpose is to use PRE event and POST event to update cached DSO, thus perf tool can read out 'correct' instructions and fill them into instruction/branch samples.

The thing is, as I argued, the instruction state between PRE and POST is ambiguous. This makes it impossible to decode the branch decision stream.

Suppose CPU0 emits the PRE event at T1 and the POST event at T5, but we have CPU1 covering the instruction at T3.

How do you decide where CPU1 goes and what the next conditional branch is?

#7 lore raw

[ + Will, Mark ]

Quoted 10 lines · Peter Zijlstra
Quoted 8 lines · Leo Yan
Quoted 4 lines · Peter Zijlstra

On Wed, Oct 30, 2019 at 06:47:47PM +0800, Leo Yan wrote:

Anyway, the below argument doesn't care much, it works for NOP/JMP just fine.

We can support NOP/JMP case as the first step, but later should can extend to support other transitions.

Quoted 7 lines

Since all instructions (with the possible exception of RET) are unconditional branch instructions: NOP, JMP, CALL. It makes no read difference to the argument below.

( I'm thinking RET might be special in that it reads the return address from the stack and therefore must emit the whole IP into the stream, as we cannot know the stack state )

To be honest, I don't have knowledge what's the exactly format for 'ret' in CoreSight trace; so would like to leave this to Mike.

Since Mike has confirmed that Arm CoreSight also needs the accurate code for decoding branch/call instructions, it's no doubt that we need to implement the same mechanism at here to update DSO for the accurate code flow. So the question is how to do this on Arm/Arm64 :)

Before move farward, I'd like to step back to describe clearly what's current problem on Arm64 and check one question for jump label:

I checked the kernel code, both kprobe and ftrace both uses stop_machine() to alter instructions, since all CPUs run into stop machine's synchronization, there have no race condition between instructions transition and CPUs execte the altered instruction; thus it's safe for kprobe and ftrace to use perf event PERF_TEXT_POKE_UPDATE to notify instruction transition and can allow us to read out 'correct' instruction for decoder.

But for jump label, it doesn't use the stop_machine() and perf event PERF_TEXT_POKE_UPDATE will introduce race condition as below (Let's see the example for transition from nop to branch):

CPU0 CPU1
NOP instruction
`-> static_key_enable()

        `-> aarch64_insn_patch_text_nosync()
             `-> perf event PERF_TEXT_POKE_UPDATE
                                                     -> Execute nop
                                                        instruction
             `-> aarch64_insn_write()
             `-> __flush_icache_range()

Since x86 platform have INT3 as a mediate state, it can avoid the race condition between CPU0 (who is do transition) and other CPUs (who is possible to execute nop/branch).

Quoted 54 lines
Quoted 18 lines
Quoted 11 lines
Quoted 10 lines

we need to update dso cache for the 'PERF_TEXT_POKE_UPDATE_PREV' event; if detect the instruction is changed from branch to nop, we need to update dso cache for 'PERF_TEXT_POKE_UPDATE_POST' event. The main idea is to ensure the branch instructions can be safely contained in the dso file and any branch samples can read out correct branch instruction.

Could you confirm this is the same with your understanding? Or I miss anything? I personally even think the pair events can be used for different arches (e.g. the solution can be reused on Arm64/x86, etc).

Quoted 6 lines

So the problem we have with PT is that it is a bit-stream of branch taken/not-taken decisions. In order to decode that we need to have an accurate view of the unconditional code flow.

Both NOP/JMP are unconditional and we need to exactly know which of the two was encountered.

Quoted 7 lines

If I understand correctly, PT decoder needs to read out instructions from dso and decide the instruction type (NOP or JMP), and finally generate the accurate code flow.

So PT decoder relies on (cached) DSO for decoding. As I know, this might be different from Arm CS, since Arm CS decoder is merely generate packets and it doesn't need to rely on DSO for decoding.

Given a start point (from a start or sync packet) we scan the instruction stream forward until the first conditional branch instruction. Then we consume the next available branch decision bit to know where to continue.

So yes, we need to have a correct text image available for this to work.

Quoted 10 lines
Quoted 10 lines

With your scheme, I don't see how we can ever actually know that. When we get the PRE event, all we really know is that we're going to change a specific instruction into another. And at the POST event we know it has been done. But in between these two events, we have no clue which of the two instructions is live on which CPU (two CPUs might in fact have a different live instruction at the same time).

This means we _cannot_ unambiguously decode a taken/not-taken decision stream.

Quoted 6 lines
Quoted 2 lines

Does CS have this same problem, and how would the PRE/POST events help with that?

My purpose is to use PRE event and POST event to update cached DSO, thus perf tool can read out 'correct' instructions and fill them into instruction/branch samples.

The thing is, as I argued, the instruction state between PRE and POST is ambiguous. This makes it impossible to decode the branch decision stream.

Quoted 5 lines

Suppose CPU0 emits the PRE event at T1 and the POST event at T5, but we have CPU1 covering the instruction at T3.

How do you decide where CPU1 goes and what the next conditional branch is?

Sorry for my not well thought.

I agree that T3 is an uncertain state with below flow:

CPU0 CPU1
perf event PERF_TEXT_POKE_UPDATE_PRE -> T1

    Int3 / NOP                                       -> T3

    Int3 / branch                                    -> T3'

perf event PERF_TEXT_POKE_UPDATE_POST -> T5

Except if the trace has extra info and can use old/new instructions combination for analysis, otherwise PRE/POST pair events aren't helpful for resolve this issue (if trace decoder can do this, then the change in kernel will be much simpler).

Below are two potential options we can use on Arm64 platform:

- Change to use stop_machine() for jump label; this might introduce performance issue if jump label is altered frequently.

To mitigate the impaction, we can only use stop_machine() when detect the perf events are enabled, otherwise will rollback to use the old code path.

- We can use breakpoint to emulate the similiar flow with x86's int3, thus we can dismiss the race condition between one CPU alters instruction and other CPUs run into the alternative instruction.

@Will, @Mark, could you help review this? Appreciate any comments and suggestions. And please let me know if you want to consolidate related works with your side (or as you know if there have ongoing discussion or someone works on this).

Thanks,
Leo Yan

maintainer
#8 lore raw
Quoted 5 lines · Leo Yan

Before move farward, I'd like to step back to describe clearly what's current problem on Arm64 and check one question for jump label:

I checked the kernel code, both kprobe and ftrace both uses stop_machine() to alter instructions,

That's not currect for Aargh64, see aarch64_insn_patch_text_nosync(), which is used in both ftrace and jump_label.

Quoted 6 lines

since all CPUs run into stop
machine's synchronization, there have no race condition between instructions transition and CPUs execte the altered instruction; thus it's safe for kprobe and ftrace to use perf event PERF_TEXT_POKE_UPDATE to notify instruction transition and can allow us to read out 'correct' instruction for decoder.

Agreed, IFF patching happens using stop_machine(), things are easy. ARM is (so far) exclusively using stop_machine() based text_poking, although the last time I spoke to Will about this, he said the _nosync stuff is possible on 32bit too, just nobody has bothered implementing it.

Quoted 14 lines

But for jump label, it doesn't use the stop_machine() and perf event PERF_TEXT_POKE_UPDATE will introduce race condition as below (Let's see the example for transition from nop to branch):

CPU0 CPU1
NOP instruction
`-> static_key_enable()

        `-> aarch64_insn_patch_text_nosync()
             `-> perf event PERF_TEXT_POKE_UPDATE
                                                     -> Execute nop
                                                        instruction
             `-> aarch64_insn_write()
             `-> __flush_icache_range()
Quoted 3 lines

Since x86 platform have INT3 as a mediate state, it can avoid the race condition between CPU0 (who is do transition) and other CPUs (who is possible to execute nop/branch).

Ah, you found the _nosync thing in jump_label, here's the one in ftrace:

arch/arm64/kernel/ftrace.c: if (aarch64_insn_patch_text_nosync((void *)pc, new))

And yes, this is racy.

Quoted 37 lines
Quoted 9 lines

The thing is, as I argued, the instruction state between PRE and POST is ambiguous. This makes it impossible to decode the branch decision stream.

Suppose CPU0 emits the PRE event at T1 and the POST event at T5, but we have CPU1 covering the instruction at T3.

How do you decide where CPU1 goes and what the next conditional branch is?

Sorry for my not well thought.

I agree that T3 is an uncertain state with below flow:

CPU0 CPU1
perf event PERF_TEXT_POKE_UPDATE_PRE -> T1

    Int3 / NOP                                       -> T3

    Int3 / branch                                    -> T3'

perf event PERF_TEXT_POKE_UPDATE_POST -> T5

Except if the trace has extra info and can use old/new instructions combination for analysis, otherwise PRE/POST pair events aren't helpful for resolve this issue (if trace decoder can do this, then the change in kernel will be much simpler).

Below are two potential options we can use on Arm64 platform:

- Change to use stop_machine() for jump label; this might introduce performance issue if jump label is altered frequently.

To mitigate the impaction, we can only use stop_machine() when detect the perf events are enabled, otherwise will rollback to use the old code path.

Quoted 8 lines

- We can use breakpoint to emulate the similiar flow with x86's int3, thus we can dismiss the race condition between one CPU alters instruction and other CPUs run into the alternative instruction.

@Will, @Mark, could you help review this? Appreciate any comments and suggestions. And please let me know if you want to consolidate related works with your side (or as you know if there have ongoing discussion or someone works on this).

Given people are building larger Aargh64 machines (I've heard about 100+ CPUs already), I'm thinking the 3rd option is the most performant.

But yes, as you mention earlier, we can make this optional on the TEXT_POKE_UPDATE event being in use.

I'm thinking something along the lines of:

static uintptr_t nosync_addr;
static u32 nosync_insn;

int __kprobes aarch64_insn_patch_text_nosync(void *addr, u32 insn)

{
	const u32 break = // some_breakpoint_insn;
	uintptr_t tp = (uintptr_t)addr;
	int ret;

	lockdep_assert_held(&text_mutex);

	/* A64 instructions must be word aligned */
	if (tp & 0x3)
		return -EINVAL;

	if (perf_text_poke_update_enabled()) {

		nosync_insn = insn;
		smp_store_release(&nosync_addr, tp);

		ret = aarch64_insn_write(addr, break);
		if (ret == 0)
			__flush_icache_range(tp, tp + AARCH64_INSN_SIZE);

		perf_event_text_poke(....);
	}

	ret = aarch64_insn_write(addr, insn);
	if (ret == 0)
		__flush_icache_range(tp, tp + AARCH64_INSN_SIZE);

	return ret;
}

And have the 'break' handler do:

aarch64_insn_break_handler(struct pt_regs *regs)

{
	unsigned long addr = smp_load_acquire(&nosync_addr);
	u32 insn = nosync_insn;

	if (regs->ip != addr)
		return;

	// emulate @insn
}

I understood from Will the whole nosync scheme only works for a limited set of instructions, but you only have to implement emulation for the actual instructions used of course.

(which is what we do on x86)

Does this sound workable?

maintainer
#9 lore raw
Quoted 24 lines · Peter Zijlstra

I'm thinking something along the lines of:

static uintptr_t nosync_addr;
static u32 nosync_insn;

int __kprobes aarch64_insn_patch_text_nosync(void *addr, u32 insn)

{
	const u32 break = // some_breakpoint_insn;
	uintptr_t tp = (uintptr_t)addr;
	int ret;

	lockdep_assert_held(&text_mutex);

	/* A64 instructions must be word aligned */
	if (tp & 0x3)
		return -EINVAL;

	if (perf_text_poke_update_enabled()) {

		nosync_insn = insn;
		smp_store_release(&nosync_addr, tp);

		ret = aarch64_insn_write(addr, break);
		if (ret == 0)
Quoted 8 lines
			__flush_icache_range(tp, tp + AARCH64_INSN_SIZE);

		perf_event_text_poke(....);
	}

	ret = aarch64_insn_write(addr, insn);
	if (ret == 0)
		__flush_icache_range(tp, tp + AARCH64_INSN_SIZE);
	// barrier that guarantees iflush completion ? dsb(osh) ?

	WRITE_ONCE(nosync_addr, NULL);
Quoted 23 lines
	return ret;
}

And have the 'break' handler do:

aarch64_insn_break_handler(struct pt_regs *regs)

{
	unsigned long addr = smp_load_acquire(&nosync_addr);
	u32 insn = nosync_insn;

	if (regs->ip != addr)
		return;

	// emulate @insn
}

I understood from Will the whole nosync scheme only works for a limited set of instructions, but you only have to implement emulation for the actual instructions used of course.

(which is what we do on x86)

Does this sound workable?

#10 lore raw
In reply to Peter Zijlstra

Hi Peter,

Quoted 10 lines · Peter Zijlstra
Quoted 5 lines · Leo Yan

Before move farward, I'd like to step back to describe clearly what's current problem on Arm64 and check one question for jump label:

I checked the kernel code, both kprobe and ftrace both uses stop_machine() to alter instructions,

That's not currect for Aargh64, see aarch64_insn_patch_text_nosync(), which is used in both ftrace and jump_label.

Thanks for pointing out this.

Agree, for ftrace, it's complex and there have multiple pathes to use aarch64_insn_patch_text_nosync().

Below flow uses stop_machine() in ftrace:

  ftrace_run_stop_machine()
   stop_machine(__ftrace_modify_code, &command, NULL);
     __ftrace_modify_code()
       ftrace_modify_all_code()
         ftrace_update_ftrace_func()
           ftrace_modify_code()
             aarch64_insn_patch_text_nosync()

Below flow doesn't use stop_machine() in ftrace:

prepare_coming_module()
ftrace_module_enable()
process_cached_mods()
process_mod_list()
ftrace_hash_move_and_update_ops()
ftrace_ops_update_code()
ftrace_ops_update_code()
ftrace_run_modify_code()
ftrace_run_update_code()
arch_ftrace_update_code()
ftrace_modify_all_code()
ftrace_update_ftrace_func()
ftrace_modify_code()
aarch64_insn_patch_text_nosync()

Actually, there have other flows also will call into aarch64_insn_patch_text_nosync(), so at least we cannot say all ftrace flow uses stop_machine() to alter instructions.

Quoted 137 lines
Quoted 6 lines

since all CPUs run into stop
machine's synchronization, there have no race condition between instructions transition and CPUs execte the altered instruction; thus it's safe for kprobe and ftrace to use perf event PERF_TEXT_POKE_UPDATE to notify instruction transition and can allow us to read out 'correct' instruction for decoder.

Agreed, IFF patching happens using stop_machine(), things are easy. ARM is (so far) exclusively using stop_machine() based text_poking, although the last time I spoke to Will about this, he said the _nosync stuff is possible on 32bit too, just nobody has bothered implementing it.

Quoted 14 lines

But for jump label, it doesn't use the stop_machine() and perf event PERF_TEXT_POKE_UPDATE will introduce race condition as below (Let's see the example for transition from nop to branch):

CPU0 CPU1
NOP instruction
`-> static_key_enable()

        `-> aarch64_insn_patch_text_nosync()
             `-> perf event PERF_TEXT_POKE_UPDATE
                                                     -> Execute nop
                                                        instruction
             `-> aarch64_insn_write()
             `-> __flush_icache_range()
Quoted 3 lines

Since x86 platform have INT3 as a mediate state, it can avoid the race condition between CPU0 (who is do transition) and other CPUs (who is possible to execute nop/branch).

Ah, you found the _nosync thing in jump_label, here's the one in ftrace:

arch/arm64/kernel/ftrace.c: if (aarch64_insn_patch_text_nosync((void *)pc, new))

And yes, this is racy.

Quoted 37 lines
Quoted 9 lines

The thing is, as I argued, the instruction state between PRE and POST is ambiguous. This makes it impossible to decode the branch decision stream.

Suppose CPU0 emits the PRE event at T1 and the POST event at T5, but we have CPU1 covering the instruction at T3.

How do you decide where CPU1 goes and what the next conditional branch is?

Sorry for my not well thought.

I agree that T3 is an uncertain state with below flow:

CPU0 CPU1
perf event PERF_TEXT_POKE_UPDATE_PRE -> T1

    Int3 / NOP                                       -> T3

    Int3 / branch                                    -> T3'

perf event PERF_TEXT_POKE_UPDATE_POST -> T5

Except if the trace has extra info and can use old/new instructions combination for analysis, otherwise PRE/POST pair events aren't helpful for resolve this issue (if trace decoder can do this, then the change in kernel will be much simpler).

Below are two potential options we can use on Arm64 platform:

- Change to use stop_machine() for jump label; this might introduce performance issue if jump label is altered frequently.

To mitigate the impaction, we can only use stop_machine() when detect the perf events are enabled, otherwise will rollback to use the old code path.

Quoted 8 lines

- We can use breakpoint to emulate the similiar flow with x86's int3, thus we can dismiss the race condition between one CPU alters instruction and other CPUs run into the alternative instruction.

@Will, @Mark, could you help review this? Appreciate any comments and suggestions. And please let me know if you want to consolidate related works with your side (or as you know if there have ongoing discussion or someone works on this).

Given people are building larger Aargh64 machines (I've heard about 100+ CPUs already), I'm thinking the 3rd option is the most performant.

But yes, as you mention earlier, we can make this optional on the TEXT_POKE_UPDATE event being in use.

I'm thinking something along the lines of:

static uintptr_t nosync_addr;
static u32 nosync_insn;

int __kprobes aarch64_insn_patch_text_nosync(void *addr, u32 insn)

{
	const u32 break = // some_breakpoint_insn;
	uintptr_t tp = (uintptr_t)addr;
	int ret;

	lockdep_assert_held(&text_mutex);

	/* A64 instructions must be word aligned */
	if (tp & 0x3)
		return -EINVAL;

	if (perf_text_poke_update_enabled()) {

		nosync_insn = insn;
		smp_store_release(&nosync_addr, tp);

		ret = aarch64_insn_write(addr, break);
		if (ret == 0)
			__flush_icache_range(tp, tp + AARCH64_INSN_SIZE);

		perf_event_text_poke(....);
	}

	ret = aarch64_insn_write(addr, insn);
	if (ret == 0)
		__flush_icache_range(tp, tp + AARCH64_INSN_SIZE);

	return ret;
}

And have the 'break' handler do:

aarch64_insn_break_handler(struct pt_regs *regs)

{
	unsigned long addr = smp_load_acquire(&nosync_addr);
	u32 insn = nosync_insn;

	if (regs->ip != addr)
		return;

	// emulate @insn
}
Quoted 7 lines

I understood from Will the whole nosync scheme only works for a limited set of instructions, but you only have to implement emulation for the actual instructions used of course.

(which is what we do on x86)

Does this sound workable?

Very appreciate for the posted code (and another minor fixing in your next replying), the logic is quite clear.

Will do prototype for this, at the meantime, I'd like to give a bit more time for Will (or other Arm maintainers) to review this.

Thanks,
Leo Yan

#11 lore raw

On Mon, Nov 04, 2019 at 10:23:46AM +0800, Leo Yan wrote:

[...]

Quoted 125 lines
Quoted 125 lines
Quoted 14 lines

But for jump label, it doesn't use the stop_machine() and perf event PERF_TEXT_POKE_UPDATE will introduce race condition as below (Let's see the example for transition from nop to branch):

CPU0 CPU1
NOP instruction
`-> static_key_enable()

        `-> aarch64_insn_patch_text_nosync()
             `-> perf event PERF_TEXT_POKE_UPDATE
                                                     -> Execute nop
                                                        instruction
             `-> aarch64_insn_write()
             `-> __flush_icache_range()
Quoted 3 lines

Since x86 platform have INT3 as a mediate state, it can avoid the race condition between CPU0 (who is do transition) and other CPUs (who is possible to execute nop/branch).

Ah, you found the _nosync thing in jump_label, here's the one in ftrace:

arch/arm64/kernel/ftrace.c: if (aarch64_insn_patch_text_nosync((void *)pc, new))

And yes, this is racy.

Quoted 37 lines
Quoted 9 lines

The thing is, as I argued, the instruction state between PRE and POST is ambiguous. This makes it impossible to decode the branch decision stream.

Suppose CPU0 emits the PRE event at T1 and the POST event at T5, but we have CPU1 covering the instruction at T3.

How do you decide where CPU1 goes and what the next conditional branch is?

Sorry for my not well thought.

I agree that T3 is an uncertain state with below flow:

CPU0 CPU1
perf event PERF_TEXT_POKE_UPDATE_PRE -> T1

    Int3 / NOP                                       -> T3

    Int3 / branch                                    -> T3'

perf event PERF_TEXT_POKE_UPDATE_POST -> T5

Except if the trace has extra info and can use old/new instructions combination for analysis, otherwise PRE/POST pair events aren't helpful for resolve this issue (if trace decoder can do this, then the change in kernel will be much simpler).

Below are two potential options we can use on Arm64 platform:

- Change to use stop_machine() for jump label; this might introduce performance issue if jump label is altered frequently.

To mitigate the impaction, we can only use stop_machine() when detect the perf events are enabled, otherwise will rollback to use the old code path.

Quoted 8 lines

- We can use breakpoint to emulate the similiar flow with x86's int3, thus we can dismiss the race condition between one CPU alters instruction and other CPUs run into the alternative instruction.

@Will, @Mark, could you help review this? Appreciate any comments and suggestions. And please let me know if you want to consolidate related works with your side (or as you know if there have ongoing discussion or someone works on this).

Given people are building larger Aargh64 machines (I've heard about 100+ CPUs already), I'm thinking the 3rd option is the most performant.

But yes, as you mention earlier, we can make this optional on the TEXT_POKE_UPDATE event being in use.

I'm thinking something along the lines of:

static uintptr_t nosync_addr;
static u32 nosync_insn;

int __kprobes aarch64_insn_patch_text_nosync(void *addr, u32 insn)

{
	const u32 break = // some_breakpoint_insn;
	uintptr_t tp = (uintptr_t)addr;
	int ret;

	lockdep_assert_held(&text_mutex);

	/* A64 instructions must be word aligned */
	if (tp & 0x3)
		return -EINVAL;

	if (perf_text_poke_update_enabled()) {

		nosync_insn = insn;
		smp_store_release(&nosync_addr, tp);

		ret = aarch64_insn_write(addr, break);
		if (ret == 0)
			__flush_icache_range(tp, tp + AARCH64_INSN_SIZE);

		perf_event_text_poke(....);
	}

	ret = aarch64_insn_write(addr, insn);
	if (ret == 0)
		__flush_icache_range(tp, tp + AARCH64_INSN_SIZE);

	return ret;
}

And have the 'break' handler do:

aarch64_insn_break_handler(struct pt_regs *regs)

{
	unsigned long addr = smp_load_acquire(&nosync_addr);
	u32 insn = nosync_insn;

	if (regs->ip != addr)
		return;

	// emulate @insn
}
Quoted 7 lines
Quoted 7 lines

I understood from Will the whole nosync scheme only works for a limited set of instructions, but you only have to implement emulation for the actual instructions used of course.

(which is what we do on x86)

Does this sound workable?

I will update some status for prototype (the code has been uploaded into git server [1]) and found some issues for text poke perf event on arm64. These issues are mainly related with arch64 architecture.

- The first issue is for the nosync instruction emulation. On arm64, some instructions need to be single stepped and some instructions is emulated. Especially, after I read the code for kprobe implementation on Arm64. So the main idea for prototyping is to use the almos same method with kprobe for nosync instruction.

As result, there have duplicated code between kprobe and text poke handling. If this is the right way to move forward, we need to refactor the common code so that kprobe and text poke can share with each other.

- The second issue is race condition between the CPU alters instructions and other CPUs hit the altered instructions (and breakpointed).

Peter's suggestion uses global variables 'nosync_insn' and 'nosync_addr' to record the altered instruction. But from the testing I found for single static key, usually it will change for multiple address at once.

So this might cause the issue is: CPU(a) will loop to alter instructions for different address (sometimes the opcode also is different for different address), at the meantime, CPU(b) hits an altered instruction and handle exception for the breakpoint, if CPU(a) is continuing to alter instruction for the next address, thne CPU(a) might wrongly to use the value from 'nosync_insn' and 'nosync_addr'.

Simply to say, we cannot only support single nosync instruction but need to support multiple nosync instructions in the loop.

- The third issue is might be nested issue. E.g. for the injected break instruction, we have no method to pass perf event for this instruction; and if we connect with the first issue, if the instruction is single stepped with slot (the slot is allocated with get_insn_slot()), we cannot to allow the perf user space to know the instructions which are executed in the slots.

I am not sure if I think too complex here. But seems to me, we inject more instructions for poke text event, and if these injected instructions are executed but the userspace decoder has no way to pass the related info.

Just clarify, I am fine for merging this patch set, but we might consider more what's the best way on Arm64. Welcome any public comments and suggestions; I will sync internally for how to follow up this functionality.

Thanks,
Leo Yan

[1] https://git.linaro.org/people/leo.yan/linux-coresight.git/log/?h=arm64_perf_text_pork_prototype

maintainer
#12 lore raw
Quoted 9 lines · Leo Yan

I will update some status for prototype (the code has been uploaded into git server [1]) and found some issues for text poke perf event on arm64. These issues are mainly related with arch64 architecture.

- The first issue is for the nosync instruction emulation. On arm64, some instructions need to be single stepped and some instructions is emulated. Especially, after I read the code for kprobe implementation on Arm64. So the main idea for prototyping is to use the almos same method with kprobe for nosync instruction.

This makes no sense to me what so ever. What actual instructions are patched with _nosync() ? ftrace/jump_label only use 'NOP/JMP/CALL'

For NOP you can advance regs->ip, for JMP you can adjust regs->ip, for CALL you adjust regs->ip and regs->r14 (IIUC you do something like:
regs->r14 = regs->ip+4; regs->ip = func;)

(FWIW emulating CALL on x86 is fun because we get to PUSH something on the stack, let me know if you want to see the patches that were required to make that happen :-)

Quoted 17 lines

- The second issue is race condition between the CPU alters instructions and other CPUs hit the altered instructions (and breakpointed).

Peter's suggestion uses global variables 'nosync_insn' and 'nosync_addr' to record the altered instruction. But from the testing I found for single static key, usually it will change for multiple address at once.

So this might cause the issue is: CPU(a) will loop to alter instructions for different address (sometimes the opcode also is different for different address), at the meantime, CPU(b) hits an altered instruction and handle exception for the breakpoint, if CPU(a) is continuing to alter instruction for the next address, thne CPU(a) might wrongly to use the value from 'nosync_insn' and 'nosync_addr'.

Quoted 2 lines

Simply to say, we cannot only support single nosync instruction but need to support multiple nosync instructions in the loop.

On x86 all actual text poking is serialized by text_mutex.

Quoted 11 lines

- The third issue is might be nested issue. E.g. for the injected break instruction, we have no method to pass perf event for this instruction; and if we connect with the first issue, if the instruction is single stepped with slot (the slot is allocated with get_insn_slot()), we cannot to allow the perf user space to know the instructions which are executed in the slots.

I am not sure if I think too complex here. But seems to me, we inject more instructions for poke text event, and if these injected instructions are executed but the userspace decoder has no way to pass the related info.

That's a misunderstanding, the text_poke event is a side-band event and as such delivered to all events that expressed interest in it. You don't need any exception to event mapping yourself.

Quoted 4 lines

Just clarify, I am fine for merging this patch set, but we might consider more what's the best way on Arm64. Welcome any public comments and suggestions; I will sync internally for how to follow up this functionality.

Thanks!

#13 lore raw

As a disclaimer: I'm having a hard time understanding what this thread is about.

Quoted 6 lines · Peter Zijlstra
Quoted 4 lines · Leo Yan

I will update some status for prototype (the code has been uploaded into git server [1]) and found some issues for text poke perf event on arm64. These issues are mainly related with arch64 architecture.

Quoted 8 lines
Quoted 5 lines

- The first issue is for the nosync instruction emulation. On arm64, some instructions need to be single stepped and some instructions is emulated. Especially, after I read the code for kprobe implementation on Arm64. So the main idea for prototyping is to use the almos same method with kprobe for nosync instruction.

This makes no sense to me what so ever. What actual instructions are patched with _nosync() ? ftrace/jump_label only use 'NOP/JMP/CALL'

'_nosync()' can be used to patch the following instructions:

	B		(JMP)
	BL		(CALL)
	BRK		(Breakpoint)
	SVC, HVC, SMC	(System calls)
	NOP
	ISB		(Pipe flush)

Our kprobes implementation prefers to single-step the instruction in an XOL buffer (we should do this by placing a BRK at the end of the buffer, but we currently use hardware step which is overkill imo). For instructions that are performing a PC-relative operation (e.g. an immediate branch), we can't run from the XOL buffer because we'd go to the wrong place; these instructions are therefore emulated in software...

Quoted 3 lines

For NOP you can advance regs->ip, for JMP you can adjust regs->ip, for CALL you adjust regs->ip and regs->r14 (IIUC you do something like:
regs->r14 = regs->ip+4; regs->ip = func;)

... in a manner similar to what you describe here. See simulate_b_bl(), although it's thankfully simpler than what x86 seems to have to do. This approach means we can avoid emulating the vast majority of the instruction set.

Quoted 3 lines

(FWIW emulating CALL on x86 is fun because we get to PUSH something on the stack, let me know if you want to see the patches that were required to make that happen :-)

No thanks ;)

Quoted 17 lines
Quoted 17 lines

- The second issue is race condition between the CPU alters instructions and other CPUs hit the altered instructions (and breakpointed).

Peter's suggestion uses global variables 'nosync_insn' and 'nosync_addr' to record the altered instruction. But from the testing I found for single static key, usually it will change for multiple address at once.

So this might cause the issue is: CPU(a) will loop to alter instructions for different address (sometimes the opcode also is different for different address), at the meantime, CPU(b) hits an altered instruction and handle exception for the breakpoint, if CPU(a) is continuing to alter instruction for the next address, thne CPU(a) might wrongly to use the value from 'nosync_insn' and 'nosync_addr'.

Quoted 4 lines
Quoted 2 lines

Simply to say, we cannot only support single nosync instruction but need to support multiple nosync instructions in the loop.

On x86 all actual text poking is serialized by text_mutex.

On arm64, we patch a *lot* and I think stop_machine() is the only safe choice for things like the alternatives, where we patch all sorts of things (cache maintenance, I/O accessors, atomics (until recently), user accessors. Even then, we have to provide our own synchronisation in __apply_alternatives_multi_stop() and use special cache-flushing (clean_dcache_range_nopatch()) to avoid the possibility of running the code that we're modifying. Outside of the alternatives, I'd still be wary about recursive debug exceptions if we were to patch jump labels with breakpoints.

Backing up though, I think I'm missing details about what this thread is trying to achieve. You're adding perf events so that coresight trace can take into account modifications of the kernel text, right? If so:

* Does this need to take into account more than just jump_label()?
* What consistency guarantees are needed by userspace? It's not clear to me how it correlates the new event with execution on other CPUs. Is this using timestamps or something else?
* What about module loading?

Finally, the whole point of the '_nosync()' stuff is that we don't have to synchronise. Therefore, there is a window where you really can't tell whether the instruction has been updated from the perspective of another CPU.

Will

maintainer
#14 lore raw
Quoted 3 lines · Will Deacon

Backing up though, I think I'm missing details about what this thread is trying to achieve. You're adding perf events so that coresight trace can take into account modifications of the kernel text, right?

Yes, because ARM-CS / Intel-PT need to know the exact text at any one time in order to correctly decode their traces.

Quoted 3 lines

If so:

* Does this need to take into account more than just jump_label()?

jump_label seems to be the initial target Adrian set, but yes, it needs to cover 'everything'.

That is, all single instruction patching crud around:

- optimized kprobes
(regular kprobes are exempt because they rely on exceptions)
- jump_labels
- static_call (still pending but still)
- ftrace fentry

We also need a solution for whole new chunks of text:

- modules
- ftrace trampolines
- optprobe trampolines
- JIT stuff

but that is so far not included; I had some ideas about a /dev/mem based interface that would report new ranges and wait for acks (from open file-desc) before freeing them.

Quoted 3 lines

* What consistency guarantees are needed by userspace? It's not clear to me how it correlates the new event with execution on other CPUs. Is this using timestamps or something else?

Something else :-)

Both CS/PT basically have a bit-stream encoding of taken / not-taken decisions. To decode they read the text until a conditional branch-point, then consume one bit from the stream and continue.

(note how unconditional branches -- jump_labels -- are expected to be correct in this scheme)

This means they need an exact representation of the text to be able to accurately decode.

This means timestamps or PRE/POST modify events are not acceptible. Because until the I-FLUSH has completed we do not know which CPU has which version of the instruction.

Instead we rely on exceptions; exceptions are differently encoded in the CS/PT data streams.

The scheme used is:

- overwrite target instruction with an exception (INT3 on x86, BRK on arm)
- sync (IPI broadcast CPUID or I-FLUSH completion)

at this point we know the instruction _will_ trap and CS/PT can observe this alternate flow. That is, the exception handler will emulate the instruction.

- emit the TEXT_POKE event with both the old and new instruction included

- overwrite the target instruction with the new instruction
- sync

at this point the new instruction should be valid.

Using this scheme we can at all times follow the instruction flow. Either it is an exception and the exception encoding helps us navigate, or, on either size, we'll know the old/new instruction.

Quoted 1 line

* What about module loading?

I mentioned that above; that is still pending.

Quoted 4 lines

Finally, the whole point of the '_nosync()' stuff is that we don't have to synchronise. Therefore, there is a window where you really can't tell whether the instruction has been updated from the perspective of another CPU.

Right, and much of that is preserved with the above scheme, nowhere do you have to wait/disturb other CPUs, except for the I-FLUSH completion.

Does this help?

#15 lore raw
Quoted 9 lines · Peter Zijlstra
Quoted 3 lines · Will Deacon

Backing up though, I think I'm missing details about what this thread is trying to achieve. You're adding perf events so that coresight trace can take into account modifications of the kernel text, right?

Yes, because ARM-CS / Intel-PT need to know the exact text at any one time in order to correctly decode their traces.

Quoted 6 lines
Quoted 3 lines

If so:

* Does this need to take into account more than just jump_label()?

jump_label seems to be the initial target Adrian set, but yes, it needs to cover 'everything'.

Including alternatives, which are what get me worried since the potential for recursion is pretty high there (on arm64, at least).

Quoted 10 lines

That is, all single instruction patching crud around:

- optimized kprobes
(regular kprobes are exempt because they rely on exceptions)
- jump_labels
- static_call (still pending but still)
- ftrace fentry

We also need a solution for whole new chunks of text:

Quoted 8 lines

- modules
- ftrace trampolines
- optprobe trampolines
- JIT stuff

but that is so far not included; I had some ideas about a /dev/mem based interface that would report new ranges and wait for acks (from open file-desc) before freeing them.

I think that it would be nice not to end up with two very different interfaces for this. But I see this is still just an RFC, so maybe the full picture will emerge once we solve more of these use-cases.

Quoted 20 lines
Quoted 3 lines

* What consistency guarantees are needed by userspace? It's not clear to me how it correlates the new event with execution on other CPUs. Is this using timestamps or something else?

Something else :-)

Both CS/PT basically have a bit-stream encoding of taken / not-taken decisions. To decode they read the text until a conditional branch-point, then consume one bit from the stream and continue.

(note how unconditional branches -- jump_labels -- are expected to be correct in this scheme)

This means they need an exact representation of the text to be able to accurately decode.

This means timestamps or PRE/POST modify events are not acceptible. Because until the I-FLUSH has completed we do not know which CPU has which version of the instruction.

Quoted 7 lines

Instead we rely on exceptions; exceptions are differently encoded in the CS/PT data streams.

The scheme used is:

- overwrite target instruction with an exception (INT3 on x86, BRK on arm)
- sync (IPI broadcast CPUID or I-FLUSH completion)

Hmm. Wouldn't this sync also need to drain the trace buffers for all other CPUs so that we make sure that the upcoming TEXT_POKE event occurs after all prior trace data, which could've been from before the breakpoint was installed?

Quoted 25 lines

at this point we know the instruction _will_ trap and CS/PT can observe this alternate flow. That is, the exception handler will emulate the instruction.

- emit the TEXT_POKE event with both the old and new instruction included

- overwrite the target instruction with the new instruction
- sync

at this point the new instruction should be valid.

Using this scheme we can at all times follow the instruction flow. Either it is an exception and the exception encoding helps us navigate, or, on either size, we'll know the old/new instruction.

Quoted 1 line

* What about module loading?

I mentioned that above; that is still pending.

Quoted 4 lines

Finally, the whole point of the '_nosync()' stuff is that we don't have to synchronise. Therefore, there is a window where you really can't tell whether the instruction has been updated from the perspective of another CPU.

Quoted 4 lines

Right, and much of that is preserved with the above scheme, nowhere do you have to wait/disturb other CPUs, except for the I-FLUSH completion.

Does this help?

Yes, thank you for explaining it!

Will

maintainer
#16 lore raw
Quoted 12 lines · Will Deacon
Quoted 11 lines · Peter Zijlstra
Quoted 3 lines · Will Deacon

Backing up though, I think I'm missing details about what this thread is trying to achieve. You're adding perf events so that coresight trace can take into account modifications of the kernel text, right?

Yes, because ARM-CS / Intel-PT need to know the exact text at any one time in order to correctly decode their traces.

Quoted 2 lines

If so:

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

* Does this need to take into account more than just jump_label()?

jump_label seems to be the initial target Adrian set, but yes, it needs to cover 'everything'.

Including alternatives, which are what get me worried since the potential for recursion is pretty high there (on arm64, at least).

So I had not considered alternatives because they're typically ran once at boot (and module load) and never seen again. That would make them just part of loading new text.

But you mentioned wanting to run them at hotplug time... which is more 'interresting'.

Quoted 15 lines
Quoted 15 lines

That is, all single instruction patching crud around:

- optimized kprobes
(regular kprobes are exempt because they rely on exceptions)
- jump_labels
- static_call (still pending but still)
- ftrace fentry

We also need a solution for whole new chunks of text:

- modules
- ftrace trampolines
- optprobe trampolines
- JIT stuff

Quoted 7 lines
Quoted 3 lines

but that is so far not included; I had some ideas about a /dev/mem based interface that would report new ranges and wait for acks (from open file-desc) before freeing them.

I think that it would be nice not to end up with two very different interfaces for this. But I see this is still just an RFC, so maybe the full picture will emerge once we solve more of these use-cases.

The general distinction is between new text mappings and changing them once they exist.

In general a text mapping is large and doesn't change (much). Once you get an event it exist you can copy it out at your convenience, all you really need to make sure of it that it doesn't dissapear before you've completed your copy.

OTOH dynamic text like jump_labels can happen quite frequently and we cannot wait for all observers to have observed/copied the new state before we allow changing it again -- ie. we need a buffered event.

So we don't want to stick whole new text things into a buffer (a module might be larger than the buffer) but we cannot be lazy with text updates.

That is, yes it sucks, but these are two different cases.

Quoted 5 lines
Quoted 5 lines

Instead we rely on exceptions; exceptions are differently encoded in the CS/PT data streams.

The scheme used is:

Quoted 7 lines
Quoted 2 lines

- overwrite target instruction with an exception (INT3 on x86, BRK on arm)
- sync (IPI broadcast CPUID or I-FLUSH completion)

Hmm. Wouldn't this sync also need to drain the trace buffers for all other CPUs so that we make sure that the upcoming TEXT_POKE event occurs after all prior trace data, which could've been from before the breakpoint was installed?

All we need to ensure is that the breakpoint is visible before the event. That way we know that before the event we have the old instruction, after the event we have the new instruction, and any ambiguity must be resolved with exception packets.

That is, if there is concurrency, the trace buffer will be 'flushed' by the exception. If there is no concurrency, we don't care and can assume old/new depending on timestamps relative to the event.

Quoted 15 lines
Quoted 15 lines

at this point we know the instruction _will_ trap and CS/PT can observe this alternate flow. That is, the exception handler will emulate the instruction.

- emit the TEXT_POKE event with both the old and new instruction included

- overwrite the target instruction with the new instruction
- sync

at this point the new instruction should be valid.

Using this scheme we can at all times follow the instruction flow. Either it is an exception and the exception encoding helps us navigate, or, on either size, we'll know the old/new instruction.

A message that is not in the archive (sent off-list or not yet received): CAJ9a7VgZH7g=rFDpKf=FzEcyBVLS_WjqbrqtRnjOi7WOY4st+w@mail.gmail.com

maintainer
#18 lore raw
Quoted 14 lines · Mike Leach
Quoted 11 lines · Leo Yan
Quoted 7 lines

Since all instructions (with the possible exception of RET) are unconditional branch instructions: NOP, JMP, CALL. It makes no read difference to the argument below.

( I'm thinking RET might be special in that it reads the return address from the stack and therefore must emit the whole IP into the stream, as we cannot know the stack state )

To be honest, I don't have knowledge what's the exactly format for 'ret' in CoreSight trace; so would like to leave this to Mike.

Quoted 8 lines

For ETM trace we do not have to output the entire address into he stream if:
- address compression allows us to emit only the changed ls bit from the last address.
- the address is identical to one of the last three addresses emitted ( we just emit a ‘same address encoding’
- we are using return stack compression and the address is top of the return stack (we emit nothing and the decoder gets the address from its own mode, of the return stack)

Cute. I don't actually know what PT does, but I figured there had to at least be the option to provide more information.

maintainer
#19 lore raw
In reply to Adrian Hunter
Quoted 43 lines · Adrian Hunter

Record changes to kernel text (i.e. self-modifying code) in order to support tracers like Intel PT decoding through jump labels.

A copy of the running kernel code is needed as a reference point (e.g. from /proc/kcore). The text poke event records the old bytes and the new bytes so that the event can be processed forwards or backwards.

The text poke event has 'flags' to describe when the event is sent. At present the only flag is PERF_TEXT_POKE_UPDATE which indicates the point at which tools should update their reference kernel executable to change the old bytes to the new bytes.

Other architectures may wish to emit a pair of text poke events. One before and one after a text poke. In that case, PERF_TEXT_POKE_UPDATE flag would be set on only one of the pair.

In the case of Intel PT tracing, the executable code must be walked to reconstruct the control flow. For x86 a jump label text poke begins:
- write INT3 byte
- IPI-SYNC
At this point the actual control flow will be through the INT3 and handler and not hit the old or new instruction. Intel PT outputs FUP/TIP packets for the INT3, so the flow can still be decoded. Subsequently:
- emit RECORD_TEXT_POKE with the new instruction
- write instruction tail
- IPI-SYNC
- write first byte
- IPI-SYNC
So before the text poke event timestamp, the decoder will see either the old instruction flow or FUP/TIP of INT3. After the text poke event timestamp, the decoder will see either the new instruction flow or FUP/TIP of INT3. Thus decoders can use the timestamp as the point at which to modify the executable code.

include/uapi/linux/perf_event.h +3 -0
@@ -1000,6 +1001,26 @@ enum perf_event_type {
 	 */
 	PERF_RECORD_BPF_EVENT			= 18,
 
+	/*
+	 * Records changes to kernel text i.e. self-modified code.
+	 * 'flags' has PERF_TEXT_POKE_UPDATE (i.e. bit 0) set to

include/uapi/linux/perf_event.h enum perf_event_type { · line 1016

Quoted 10 lines
include/uapi/linux/perf_event.h +10 -0
+	 * indicate to tools to update old bytes to new bytes in the
+	 * executable image.
+	 * 'len' is the number of old bytes, which is the same as the number
+	 * of new bytes. 'bytes' contains the old bytes followed immediately
+	 * by the new bytes.
+	 *
+	 * struct {
+	 *	struct perf_event_header	header;
+	 *	u64				addr;
+	 *	u16				flags;

What's the purpose of the 'flags' field? We currently only have the 1 flag defined, but what possible other flags are you thinking of?

Quoted 25 lines
include/uapi/linux/perf_event.h +13 -0
+	 *	u16				len;
+	 *	u8				bytes[];
+	 *	struct sample_id		sample_id;
+	 * };
+	 */
+	PERF_RECORD_TEXT_POKE			= 19,
+
 	PERF_RECORD_MAX,			/* non-ABI */
 };
 
@@ -1041,6 +1062,11 @@ enum perf_callchain_context {
 #define PERF_AUX_FLAG_PARTIAL		0x04	/* record contains gaps */
 #define PERF_AUX_FLAG_COLLISION		0x08	/* sample collided with another */
 
+/**
+ * PERF_RECORD_TEXT_POKE::flags bits
+ */
+#define PERF_TEXT_POKE_UPDATE		0x01	/* update old bytes to new bytes */
+
 #define PERF_FLAG_FD_NO_GROUP		(1UL << 0)
 #define PERF_FLAG_FD_OUTPUT		(1UL << 1)
 #define PERF_FLAG_PID_CGROUP		(1UL << 2) /* pid=cgroup id, per-cpu mode only */
 
 
+void perf_event_text_poke(unsigned long addr, u16 flags, const void *old_bytes,

include/uapi/linux/perf_event.h enum perf_callchain_context {

Quoted 10 lines
include/uapi/linux/perf_event.h +10 -0
+			  const void *new_bytes, size_t len)
+{
+	struct perf_text_poke_event text_poke_event;
+	size_t tot, pad;
+
+	if (!atomic_read(&nr_text_poke_events))
+		return;
+
+	tot  = sizeof(text_poke_event.flags) +
+	       sizeof(text_poke_event.len) + (len << 1);

Maybe use: 'len * 2', compiler should be smart enough.

Quoted 10 lines
kernel/events/core.c +10 -0
+	pad  = ALIGN(tot, sizeof(u64)) - tot;
+
+	text_poke_event = (struct perf_text_poke_event){
+		.old_bytes    = old_bytes,
+		.new_bytes    = new_bytes,
+		.pad          = pad,
+		.flags        = flags,
+		.len          = len,
+		.event_id  = {
+			.header = {

kernel/events/core.c void perf_event_bpf_event(struct bpf_prog *prog, · line 8157

Quoted 10 lines
kernel/events/core.c +10 -0
+				.type = PERF_RECORD_TEXT_POKE,
+				.misc = PERF_RECORD_MISC_KERNEL,
+				.size = sizeof(text_poke_event.event_id) + tot + pad,
+			},
+			.addr = addr,
+		},
+	};
+
+	perf_iterate_sb(perf_event_text_poke_output, &text_poke_event, NULL);
+}

Also, I'm thinking this patch has a problem with
arch_unoptimize_kprobe() as it stands today.

I've got a patch pending for that:

https://lkml.kernel.org/r/[email protected]

That rewrites that a little, but it will be slightly differently broken after that.

The problem is that while we can (and do) ignore regular kprobes (they use text_poke() which isn't instrumented, and they don't need to be, because they're always going through INT3), we cannot ignore optprobes.

Installing the optprobe works as expected, but unoptimizing is difficult and as it stands your patch will not report the old text (you'll see that first byte overwriten with 0xCC) and after my patch you'll not see it at all.

Now, I suppose we can stick an explicit perf_event_text_poke() in there after my patch.

We have to delay this patch until after my x86/ftrace rewrite anyway, because otherwise ftrace is hidden too.

And even then, this will then notify us of all text modifications, but what about all the extra text, like ftrace trampolines, k(ret)probe trampolines, optprobe slots, bpf-jit, etc.?

author reviewer
#20 lore raw
Quoted 41 lines · Peter Zijlstra
Quoted 40 lines · Adrian Hunter

Record changes to kernel text (i.e. self-modifying code) in order to support tracers like Intel PT decoding through jump labels.

A copy of the running kernel code is needed as a reference point (e.g. from /proc/kcore). The text poke event records the old bytes and the new bytes so that the event can be processed forwards or backwards.

The text poke event has 'flags' to describe when the event is sent. At present the only flag is PERF_TEXT_POKE_UPDATE which indicates the point at which tools should update their reference kernel executable to change the old bytes to the new bytes.

Other architectures may wish to emit a pair of text poke events. One before and one after a text poke. In that case, PERF_TEXT_POKE_UPDATE flag would be set on only one of the pair.

In the case of Intel PT tracing, the executable code must be walked to reconstruct the control flow. For x86 a jump label text poke begins:
- write INT3 byte
- IPI-SYNC
At this point the actual control flow will be through the INT3 and handler and not hit the old or new instruction. Intel PT outputs FUP/TIP packets for the INT3, so the flow can still be decoded. Subsequently:
- emit RECORD_TEXT_POKE with the new instruction
- write instruction tail
- IPI-SYNC
- write first byte
- IPI-SYNC
So before the text poke event timestamp, the decoder will see either the old instruction flow or FUP/TIP of INT3. After the text poke event timestamp, the decoder will see either the new instruction flow or FUP/TIP of INT3. Thus decoders can use the timestamp as the point at which to modify the executable code.

include/uapi/linux/perf_event.h +0 -0
@@ -1000,6 +1001,26 @@ enum perf_event_type {
 	 */
 	PERF_RECORD_BPF_EVENT			= 18,
 
Quoted 16 lines

include/uapi/linux/perf_event.h enum perf_event_type { · line 1016

Quoted 13 lines
include/uapi/linux/perf_event.h +13 -0
+	/*
+	 * Records changes to kernel text i.e. self-modified code.
+	 * 'flags' has PERF_TEXT_POKE_UPDATE (i.e. bit 0) set to
+	 * indicate to tools to update old bytes to new bytes in the
+	 * executable image.
+	 * 'len' is the number of old bytes, which is the same as the number
+	 * of new bytes. 'bytes' contains the old bytes followed immediately
+	 * by the new bytes.
+	 *
+	 * struct {
+	 *	struct perf_event_header	header;
+	 *	u64				addr;
+	 *	u16				flags;

What's the purpose of the 'flags' field? We currently only have the 1 flag defined, but what possible other flags are you thinking of?

That was to allow for identifying 'before' and 'after' text poke events.

Quoted 82 lines
Quoted 26 lines
include/uapi/linux/perf_event.h +13 -0
+	 *	u16				len;
+	 *	u8				bytes[];
+	 *	struct sample_id		sample_id;
+	 * };
+	 */
+	PERF_RECORD_TEXT_POKE			= 19,
+
 	PERF_RECORD_MAX,			/* non-ABI */
 };
 
@@ -1041,6 +1062,11 @@ enum perf_callchain_context {
 #define PERF_AUX_FLAG_PARTIAL		0x04	/* record contains gaps */
 #define PERF_AUX_FLAG_COLLISION		0x08	/* sample collided with another */
 
+/**
+ * PERF_RECORD_TEXT_POKE::flags bits
+ */
+#define PERF_TEXT_POKE_UPDATE		0x01	/* update old bytes to new bytes */
+
 #define PERF_FLAG_FD_NO_GROUP		(1UL << 0)
 #define PERF_FLAG_FD_OUTPUT		(1UL << 1)
 #define PERF_FLAG_PID_CGROUP		(1UL << 2) /* pid=cgroup id, per-cpu mode only */
 
 
+void perf_event_text_poke(unsigned long addr, u16 flags, const void *old_bytes,

include/uapi/linux/perf_event.h enum perf_callchain_context {

Quoted 10 lines
include/uapi/linux/perf_event.h +10 -0
+			  const void *new_bytes, size_t len)
+{
+	struct perf_text_poke_event text_poke_event;
+	size_t tot, pad;
+
+	if (!atomic_read(&nr_text_poke_events))
+		return;
+
+	tot  = sizeof(text_poke_event.flags) +
+	       sizeof(text_poke_event.len) + (len << 1);

Maybe use: 'len * 2', compiler should be smart enough.

Quoted 10 lines
kernel/events/core.c +10 -0
+	pad  = ALIGN(tot, sizeof(u64)) - tot;
+
+	text_poke_event = (struct perf_text_poke_event){
+		.old_bytes    = old_bytes,
+		.new_bytes    = new_bytes,
+		.pad          = pad,
+		.flags        = flags,
+		.len          = len,
+		.event_id  = {
+			.header = {

kernel/events/core.c void perf_event_bpf_event(struct bpf_prog *prog, · line 8157

Quoted 10 lines
kernel/events/core.c +10 -0
+				.type = PERF_RECORD_TEXT_POKE,
+				.misc = PERF_RECORD_MISC_KERNEL,
+				.size = sizeof(text_poke_event.event_id) + tot + pad,
+			},
+			.addr = addr,
+		},
+	};
+
+	perf_iterate_sb(perf_event_text_poke_output, &text_poke_event, NULL);
+}

Also, I'm thinking this patch has a problem with
arch_unoptimize_kprobe() as it stands today.

I've got a patch pending for that:

https://lkml.kernel.org/r/[email protected]

That rewrites that a little, but it will be slightly differently broken after that.

The problem is that while we can (and do) ignore regular kprobes (they use text_poke() which isn't instrumented, and they don't need to be, because they're always going through INT3), we cannot ignore optprobes.

Installing the optprobe works as expected, but unoptimizing is difficult and as it stands your patch will not report the old text (you'll see that first byte overwriten with 0xCC) and after my patch you'll not see it at all.

Now, I suppose we can stick an explicit perf_event_text_poke() in there after my patch.

Quoted 6 lines

We have to delay this patch until after my x86/ftrace rewrite anyway, because otherwise ftrace is hidden too.

And even then, this will then notify us of all text modifications, but what about all the extra text, like ftrace trampolines, k(ret)probe trampolines, optprobe slots, bpf-jit, etc.?

Generally jump labels are used by default, whereas ftrace and probes are more under the control of the user. So just having jump labels done is the main thing.

The tools have some support for copying out bpf-jit, although I am not sure it guarantees to get short-lived code.

author reviewer
#21 lore raw

RFC PATCH 2/6 perf dso: Refactor dso_cache__read()

In reply to Adrian Hunter

Refactor dso_cache__read() to separate populating the cache from copying data from it. This is preparation for adding a cache "write" that will update the data in the cache.

---
 tools/perf/util/dso.c | 64 +++++++++++++++++++++++++------------------
 1 file changed, 37 insertions(+), 27 deletions(-)
tools/perf/util/dso.c +37 -27
index e11ddf86f2b3..460330d125b6 100644
@@ -768,7 +768,7 @@ dso_cache__free(struct dso *dso)
 	pthread_mutex_unlock(&dso->lock);
 }
 
-static struct dso_cache *dso_cache__find(struct dso *dso, u64 offset)
+static struct dso_cache *__dso_cache__find(struct dso *dso, u64 offset)
 {
 	const struct rb_root *root = &dso->data.cache;
 	struct rb_node * const *p = &root->rb_node;
@@ -863,54 +863,64 @@ static ssize_t file_read(struct dso *dso, struct machine *machine,
 	return ret;
 }
 
-static ssize_t
-dso_cache__read(struct dso *dso, struct machine *machine,
-		u64 offset, u8 *data, ssize_t size)
+static struct dso_cache *dso_cache__populate(struct dso *dso,
+					     struct machine *machine,
+					     u64 offset, ssize_t *ret)
 {
 	u64 cache_offset = offset & DSO__DATA_CACHE_MASK;
 	struct dso_cache *cache;
 	struct dso_cache *old;
-	ssize_t ret;
 
 	cache = zalloc(sizeof(*cache) + DSO__DATA_CACHE_SIZE);
-	if (!cache)
-		return -ENOMEM;
+	if (!cache) {
+		*ret = -ENOMEM;
+		return NULL;
+	}
 
 	if (dso->binary_type == DSO_BINARY_TYPE__BPF_PROG_INFO)
-		ret = bpf_read(dso, cache_offset, cache->data);
+		*ret = bpf_read(dso, cache_offset, cache->data);
 	else
-		ret = file_read(dso, machine, cache_offset, cache->data);
+		*ret = file_read(dso, machine, cache_offset, cache->data);
 
-	if (ret > 0) {
-		cache->offset = cache_offset;
-		cache->size   = ret;
+	if (*ret <= 0) {
+		free(cache);
+		return NULL;
+	}
 
-		old = dso_cache__insert(dso, cache);
-		if (old) {
-			/* we lose the race */
-			free(cache);
-			cache = old;
-		}
+	cache->offset = cache_offset;
+	cache->size   = *ret;
 
-		ret = dso_cache__memcpy(cache, offset, data, size);
+	old = dso_cache__insert(dso, cache);
+	if (old) {
+		/* we lose the race */
+		free(cache);
+		cache = old;
 	}
 
-	if (ret <= 0)
-		free(cache);
+	return cache;
+}
 
-	return ret;
+static struct dso_cache *dso_cache__find(struct dso *dso,
+					 struct machine *machine,
+					 u64 offset,
+					 ssize_t *ret)
+{
+	struct dso_cache *cache = __dso_cache__find(dso, offset);
+
+	return cache ? cache : dso_cache__populate(dso, machine, offset, ret);
 }
 
 static ssize_t dso_cache_read(struct dso *dso, struct machine *machine,
 			      u64 offset, u8 *data, ssize_t size)
 {
 	struct dso_cache *cache;
+	ssize_t ret = 0;
 
-	cache = dso_cache__find(dso, offset);
-	if (cache)
-		return dso_cache__memcpy(cache, offset, data, size);
-	else
-		return dso_cache__read(dso, machine, offset, data, size);
+	cache = dso_cache__find(dso, machine, offset, &ret);
+	if (!cache)
+		return ret;
+
+	return dso_cache__memcpy(cache, offset, data, size);
 }
 
 /*
Signature · 1 line
2.17.1
#22 lore raw

Re: [PATCH RFC 2/6] perf dso: Refactor dso_cache__read()

Em Fri, Oct 25, 2019 at 03:59:56PM +0300, Adrian Hunter escreveu:

Quoted 3 lines

Refactor dso_cache__read() to separate populating the cache from copying data from it. This is preparation for adding a cache "write" that will update the data in the cache.

Ditto for 2/6 and 3/6, i.e. applying them now.

- Arnaldo

Quoted 111 lines
---
 tools/perf/util/dso.c | 64 +++++++++++++++++++++++++------------------
 1 file changed, 37 insertions(+), 27 deletions(-)
tools/perf/util/dso.c +37 -27
index e11ddf86f2b3..460330d125b6 100644
@@ -768,7 +768,7 @@ dso_cache__free(struct dso *dso)
 	pthread_mutex_unlock(&dso->lock);
 }
 
-static struct dso_cache *dso_cache__find(struct dso *dso, u64 offset)
+static struct dso_cache *__dso_cache__find(struct dso *dso, u64 offset)
 {
 	const struct rb_root *root = &dso->data.cache;
 	struct rb_node * const *p = &root->rb_node;
@@ -863,54 +863,64 @@ static ssize_t file_read(struct dso *dso, struct machine *machine,
 	return ret;
 }
 
-static ssize_t
-dso_cache__read(struct dso *dso, struct machine *machine,
-		u64 offset, u8 *data, ssize_t size)
+static struct dso_cache *dso_cache__populate(struct dso *dso,
+					     struct machine *machine,
+					     u64 offset, ssize_t *ret)
 {
 	u64 cache_offset = offset & DSO__DATA_CACHE_MASK;
 	struct dso_cache *cache;
 	struct dso_cache *old;
-	ssize_t ret;
 
 	cache = zalloc(sizeof(*cache) + DSO__DATA_CACHE_SIZE);
-	if (!cache)
-		return -ENOMEM;
+	if (!cache) {
+		*ret = -ENOMEM;
+		return NULL;
+	}
 
 	if (dso->binary_type == DSO_BINARY_TYPE__BPF_PROG_INFO)
-		ret = bpf_read(dso, cache_offset, cache->data);
+		*ret = bpf_read(dso, cache_offset, cache->data);
 	else
-		ret = file_read(dso, machine, cache_offset, cache->data);
+		*ret = file_read(dso, machine, cache_offset, cache->data);
 
-	if (ret > 0) {
-		cache->offset = cache_offset;
-		cache->size   = ret;
+	if (*ret <= 0) {
+		free(cache);
+		return NULL;
+	}
 
-		old = dso_cache__insert(dso, cache);
-		if (old) {
-			/* we lose the race */
-			free(cache);
-			cache = old;
-		}
+	cache->offset = cache_offset;
+	cache->size   = *ret;
 
-		ret = dso_cache__memcpy(cache, offset, data, size);
+	old = dso_cache__insert(dso, cache);
+	if (old) {
+		/* we lose the race */
+		free(cache);
+		cache = old;
 	}
 
-	if (ret <= 0)
-		free(cache);
+	return cache;
+}
 
-	return ret;
+static struct dso_cache *dso_cache__find(struct dso *dso,
+					 struct machine *machine,
+					 u64 offset,
+					 ssize_t *ret)
+{
+	struct dso_cache *cache = __dso_cache__find(dso, offset);
+
+	return cache ? cache : dso_cache__populate(dso, machine, offset, ret);
 }
 
 static ssize_t dso_cache_read(struct dso *dso, struct machine *machine,
 			      u64 offset, u8 *data, ssize_t size)
 {
 	struct dso_cache *cache;
+	ssize_t ret = 0;
 
-	cache = dso_cache__find(dso, offset);
-	if (cache)
-		return dso_cache__memcpy(cache, offset, data, size);
-	else
-		return dso_cache__read(dso, machine, offset, data, size);
+	cache = dso_cache__find(dso, machine, offset, &ret);
+	if (!cache)
+		return ret;
+
+	return dso_cache__memcpy(cache, offset, data, size);
 }
 
 /*
Signature · 1 line
2.17.1
Signature · 1 line
- Arnaldo
#23 lore raw

Re: [PATCH RFC 2/6] perf dso: Refactor dso_cache__read()

In reply to Adrian Hunter

On Fri, Oct 25, 2019 at 03:59:56PM +0300, Adrian Hunter wrote:

SNIP

Quoted 17 lines
tools/perf/util/dso.c +9 -1
+}
 
-	return ret;
+static struct dso_cache *dso_cache__find(struct dso *dso,
+					 struct machine *machine,
+					 u64 offset,
+					 ssize_t *ret)
+{
+	struct dso_cache *cache = __dso_cache__find(dso, offset);
+
+	return cache ? cache : dso_cache__populate(dso, machine, offset, ret);
 }
 
 static ssize_t dso_cache_read(struct dso *dso, struct machine *machine,
 			      u64 offset, u8 *data, ssize_t size)
 {
 	struct dso_cache *cache;

tools/perf/util/dso.c static ssize_t file_read(struct dso *dso, struct machine *machine, · line 921

Quoted 10 lines
tools/perf/util/dso.c +4 -5
+	ssize_t ret = 0;
 
-	cache = dso_cache__find(dso, offset);
-	if (cache)
-		return dso_cache__memcpy(cache, offset, data, size);
-	else
-		return dso_cache__read(dso, machine, offset, data, size);
+	cache = dso_cache__find(dso, machine, offset, &ret);
+	if (!cache)
+		return ret;

why not use the ERR_* macros to get error through the pointer instead of adding extra argument?

jirka

Quoted 8 lines
tools/perf/util/dso.c +2 -0
+
+	return dso_cache__memcpy(cache, offset, data, size);
 }
 
 /*
Signature · 1 line
2.17.1
author reviewer
#24 lore raw

Re: [PATCH RFC 2/6] perf dso: Refactor dso_cache__read()

Quoted 23 lines · Jiri Olsa

On Fri, Oct 25, 2019 at 03:59:56PM +0300, Adrian Hunter wrote:

SNIP

Quoted 19 lines
(quoted diff) +10 -1
+}
 
-	return ret;
+static struct dso_cache *dso_cache__find(struct dso *dso,
+					 struct machine *machine,
+					 u64 offset,
+					 ssize_t *ret)
+{
+	struct dso_cache *cache = __dso_cache__find(dso, offset);
+
+	return cache ? cache : dso_cache__populate(dso, machine, offset, ret);
 }
 
 static ssize_t dso_cache_read(struct dso *dso, struct machine *machine,
 			      u64 offset, u8 *data, ssize_t size)
 {
 	struct dso_cache *cache;
+	ssize_t ret = 0;
Quoted 12 lines

Review

Quoted 8 lines
(quoted diff) +3 -5
-	cache = dso_cache__find(dso, offset);
-	if (cache)
-		return dso_cache__memcpy(cache, offset, data, size);
-	else
-		return dso_cache__read(dso, machine, offset, data, size);
+	cache = dso_cache__find(dso, machine, offset, &ret);
+	if (!cache)
+		return ret;

why not use the ERR_* macros to get error through the pointer instead of adding extra argument?

OK, here's the diff for that:

tools/perf/util/dso.c +15 -18
index 460330d125b6..272545624fbe 100644
@@ -3,6 +3,7 @@
 #include <linux/kernel.h>
 #include <linux/string.h>
 #include <linux/zalloc.h>
+#include <linux/err.h>
 #include <sys/time.h>
 #include <sys/resource.h>
 #include <sys/types.h>
@@ -865,30 +866,29 @@ static ssize_t file_read(struct dso *dso, struct machine *machine,
 
 static struct dso_cache *dso_cache__populate(struct dso *dso,
 					     struct machine *machine,
-					     u64 offset, ssize_t *ret)
+					     u64 offset)
 {
 	u64 cache_offset = offset & DSO__DATA_CACHE_MASK;
 	struct dso_cache *cache;
 	struct dso_cache *old;
+	ssize_t ret;
 
 	cache = zalloc(sizeof(*cache) + DSO__DATA_CACHE_SIZE);
-	if (!cache) {
-		*ret = -ENOMEM;
-		return NULL;
-	}
+	if (!cache)
+		return ERR_PTR(-ENOMEM);
 
 	if (dso->binary_type == DSO_BINARY_TYPE__BPF_PROG_INFO)
-		*ret = bpf_read(dso, cache_offset, cache->data);
+		ret = bpf_read(dso, cache_offset, cache->data);
 	else
-		*ret = file_read(dso, machine, cache_offset, cache->data);
+		ret = file_read(dso, machine, cache_offset, cache->data);
 
-	if (*ret <= 0) {
+	if (ret <= 0) {
 		free(cache);
-		return NULL;
+		return ERR_PTR(ret);
 	}
 
 	cache->offset = cache_offset;
-	cache->size   = *ret;
+	cache->size   = ret;
 
 	old = dso_cache__insert(dso, cache);
 	if (old) {
@@ -902,23 +902,20 @@ static struct dso_cache *dso_cache__populate(struct dso *dso,
 
 static struct dso_cache *dso_cache__find(struct dso *dso,
 					 struct machine *machine,
-					 u64 offset,
-					 ssize_t *ret)
+					 u64 offset)
 {
 	struct dso_cache *cache = __dso_cache__find(dso, offset);
 
-	return cache ? cache : dso_cache__populate(dso, machine, offset, ret);
+	return cache ? cache : dso_cache__populate(dso, machine, offset);
 }
 
 static ssize_t dso_cache_read(struct dso *dso, struct machine *machine,
 			      u64 offset, u8 *data, ssize_t size)
 {
-	struct dso_cache *cache;
-	ssize_t ret = 0;
+	struct dso_cache *cache = dso_cache__find(dso, machine, offset);
 
-	cache = dso_cache__find(dso, machine, offset, &ret);
-	if (!cache)
-		return ret;
+	if (IS_ERR_OR_NULL(cache))
+		return PTR_ERR(cache);
 
 	return dso_cache__memcpy(cache, offset, data, size);
 }
merge bot Applied
#25 lore raw

[tip: perf/core] perf dso: Refactor dso_cache__read()

In reply to Adrian Hunter
The following commit has been merged into the perf/core branch of tip:

The following commit has been merged into the perf/core branch of tip:

Commit-ID:     366df72657e0cd6bd072b56a48e63b8d89718f70
Gitweb:        https://git.kernel.org/tip/366df72657e0cd6bd072b56a48e63b8d89718f70
Author:        Adrian Hunter <[email protected]>
AuthorDate:    Fri, 25 Oct 2019 15:59:56 +03:00
Committer:     Arnaldo Carvalho de Melo <[email protected]>
CommitterDate: Wed, 06 Nov 2019 15:43:06 -03:00

perf dso: Refactor dso_cache__read()

Refactor dso_cache__read() to separate populating the cache from copying data from it. This is preparation for adding a cache "write" that will update the data in the cache.

---
 tools/perf/util/dso.c | 64 ++++++++++++++++++++++++------------------
 1 file changed, 37 insertions(+), 27 deletions(-)
tools/perf/util/dso.c +37 -27
index e11ddf8..460330d 100644
@@ -768,7 +768,7 @@ dso_cache__free(struct dso *dso)
 	pthread_mutex_unlock(&dso->lock);
 }
 
-static struct dso_cache *dso_cache__find(struct dso *dso, u64 offset)
+static struct dso_cache *__dso_cache__find(struct dso *dso, u64 offset)
 {
 	const struct rb_root *root = &dso->data.cache;
 	struct rb_node * const *p = &root->rb_node;
@@ -863,54 +863,64 @@ out:
 	return ret;
 }
 
-static ssize_t
-dso_cache__read(struct dso *dso, struct machine *machine,
-		u64 offset, u8 *data, ssize_t size)
+static struct dso_cache *dso_cache__populate(struct dso *dso,
+					     struct machine *machine,
+					     u64 offset, ssize_t *ret)
 {
 	u64 cache_offset = offset & DSO__DATA_CACHE_MASK;
 	struct dso_cache *cache;
 	struct dso_cache *old;
-	ssize_t ret;
 
 	cache = zalloc(sizeof(*cache) + DSO__DATA_CACHE_SIZE);
-	if (!cache)
-		return -ENOMEM;
+	if (!cache) {
+		*ret = -ENOMEM;
+		return NULL;
+	}
 
 	if (dso->binary_type == DSO_BINARY_TYPE__BPF_PROG_INFO)
-		ret = bpf_read(dso, cache_offset, cache->data);
+		*ret = bpf_read(dso, cache_offset, cache->data);
 	else
-		ret = file_read(dso, machine, cache_offset, cache->data);
+		*ret = file_read(dso, machine, cache_offset, cache->data);
 
-	if (ret > 0) {
-		cache->offset = cache_offset;
-		cache->size   = ret;
+	if (*ret <= 0) {
+		free(cache);
+		return NULL;
+	}
 
-		old = dso_cache__insert(dso, cache);
-		if (old) {
-			/* we lose the race */
-			free(cache);
-			cache = old;
-		}
+	cache->offset = cache_offset;
+	cache->size   = *ret;
 
-		ret = dso_cache__memcpy(cache, offset, data, size);
+	old = dso_cache__insert(dso, cache);
+	if (old) {
+		/* we lose the race */
+		free(cache);
+		cache = old;
 	}
 
-	if (ret <= 0)
-		free(cache);
+	return cache;
+}
 
-	return ret;
+static struct dso_cache *dso_cache__find(struct dso *dso,
+					 struct machine *machine,
+					 u64 offset,
+					 ssize_t *ret)
+{
+	struct dso_cache *cache = __dso_cache__find(dso, offset);
+
+	return cache ? cache : dso_cache__populate(dso, machine, offset, ret);
 }
 
 static ssize_t dso_cache_read(struct dso *dso, struct machine *machine,
 			      u64 offset, u8 *data, ssize_t size)
 {
 	struct dso_cache *cache;
+	ssize_t ret = 0;
 
-	cache = dso_cache__find(dso, offset);
-	if (cache)
-		return dso_cache__memcpy(cache, offset, data, size);
-	else
-		return dso_cache__read(dso, machine, offset, data, size);
+	cache = dso_cache__find(dso, machine, offset, &ret);
+	if (!cache)
+		return ret;
+
+	return dso_cache__memcpy(cache, offset, data, size);
 }
 
 /*
author reviewer
#26 lore raw

RFC PATCH 3/6 perf dso: Add dso__data_write_cache_addr()

In reply to Adrian Hunter

Add functions to write into the dso file data cache, but not change the file itself.

---
 tools/perf/util/dso.c | 73 ++++++++++++++++++++++++++++++++++---------
 tools/perf/util/dso.h |  7 +++++
 2 files changed, 65 insertions(+), 15 deletions(-)
tools/perf/util/dso.c +58 -15
index 460330d125b6..0f1b77275a86 100644
@@ -827,14 +827,16 @@ dso_cache__insert(struct dso *dso, struct dso_cache *new)
 	return cache;
 }
 
-static ssize_t
-dso_cache__memcpy(struct dso_cache *cache, u64 offset,
-		  u8 *data, u64 size)
+static ssize_t dso_cache__memcpy(struct dso_cache *cache, u64 offset, u8 *data,
+				 u64 size, bool out)
 {
 	u64 cache_offset = offset - cache->offset;
 	u64 cache_size   = min(cache->size - cache_offset, size);
 
-	memcpy(data, cache->data + cache_offset, cache_size);
+	if (out)
+		memcpy(data, cache->data + cache_offset, cache_size);
+	else
+		memcpy(cache->data + cache_offset, data, cache_size);
 	return cache_size;
 }
 
@@ -910,8 +912,8 @@ static struct dso_cache *dso_cache__find(struct dso *dso,
 	return cache ? cache : dso_cache__populate(dso, machine, offset, ret);
 }
 
-static ssize_t dso_cache_read(struct dso *dso, struct machine *machine,
-			      u64 offset, u8 *data, ssize_t size)
+static ssize_t dso_cache_io(struct dso *dso, struct machine *machine,
+			    u64 offset, u8 *data, ssize_t size, bool out)
 {
 	struct dso_cache *cache;
 	ssize_t ret = 0;
@@ -920,16 +922,16 @@ static ssize_t dso_cache_read(struct dso *dso, struct machine *machine,
 	if (!cache)
 		return ret;
 
-	return dso_cache__memcpy(cache, offset, data, size);
+	return dso_cache__memcpy(cache, offset, data, size, out);
 }
 
 /*
  * Reads and caches dso data DSO__DATA_CACHE_SIZE size chunks
  * in the rb_tree. Any read to already cached data is served
- * by cached data.
+ * by cached data. Writes update the cache only, not the backing file.
  */
-static ssize_t cached_read(struct dso *dso, struct machine *machine,
-			   u64 offset, u8 *data, ssize_t size)
+static ssize_t cached_io(struct dso *dso, struct machine *machine,
+			 u64 offset, u8 *data, ssize_t size, bool out)
 {
 	ssize_t r = 0;
 	u8 *p = data;
@@ -937,7 +939,7 @@ static ssize_t cached_read(struct dso *dso, struct machine *machine,
 	do {
 		ssize_t ret;
 
-		ret = dso_cache_read(dso, machine, offset, p, size);
+		ret = dso_cache_io(dso, machine, offset, p, size, out);
 		if (ret < 0)
 			return ret;
 
@@ -1021,8 +1023,9 @@ off_t dso__data_size(struct dso *dso, struct machine *machine)
 	return dso->data.file_size;
 }
 
-static ssize_t data_read_offset(struct dso *dso, struct machine *machine,
-				u64 offset, u8 *data, ssize_t size)
+static ssize_t data_read_write_offset(struct dso *dso, struct machine *machine,
+				      u64 offset, u8 *data, ssize_t size,
+				      bool out)
 {
 	if (dso__data_file_size(dso, machine))
 		return -1;
@@ -1034,7 +1037,7 @@ static ssize_t data_read_offset(struct dso *dso, struct machine *machine,
 	if (offset + size < offset)
 		return -1;
 
-	return cached_read(dso, machine, offset, data, size);
+	return cached_io(dso, machine, offset, data, size, out);
 }
 
 /**
@@ -1054,7 +1057,7 @@ ssize_t dso__data_read_offset(struct dso *dso, struct machine *machine,
 	if (dso->data.status == DSO_DATA_STATUS_ERROR)
 		return -1;
 
-	return data_read_offset(dso, machine, offset, data, size);
+	return data_read_write_offset(dso, machine, offset, data, size, true);
 }
 
 /**
@@ -1075,6 +1078,46 @@ ssize_t dso__data_read_addr(struct dso *dso, struct map *map,
 	return dso__data_read_offset(dso, machine, offset, data, size);
 }
 
+/**
+ * dso__data_write_cache_offs - Write data to dso data cache at file offset
+ * @dso: dso object
+ * @machine: machine object
+ * @offset: file offset
+ * @data: buffer to write
+ * @size: size of the @data buffer
+ *
+ * Write into the dso file data cache, but do not change the file itself.
+ */
+ssize_t dso__data_write_cache_offs(struct dso *dso, struct machine *machine,
+				   u64 offset, const u8 *data_in, ssize_t size)
+{
+	u8 *data = (u8 *)data_in; /* cast away const to use same fns for r/w */
+
+	if (dso->data.status == DSO_DATA_STATUS_ERROR)
+		return -1;
+
+	return data_read_write_offset(dso, machine, offset, data, size, false);
+}
+
+/**
+ * dso__data_write_cache_addr - Write data to dso data cache at dso address
+ * @dso: dso object
+ * @machine: machine object
+ * @add: virtual memory address
+ * @data: buffer to write
+ * @size: size of the @data buffer
+ *
+ * External interface to write into the dso file data cache, but do not change
+ * the file itself.
+ */
+ssize_t dso__data_write_cache_addr(struct dso *dso, struct map *map,
+				   struct machine *machine, u64 addr,
+				   const u8 *data, ssize_t size)
+{
+	u64 offset = map->map_ip(map, addr);
+	return dso__data_write_cache_offs(dso, machine, offset, data, size);
+}
+
 struct map *dso__new_map(const char *name)
 {
 	struct map *map = NULL;
tools/perf/util/dso.h +7 -0
index e4dddb76770d..2f1fcbc6fead 100644
@@ -285,6 +285,8 @@ void dso__set_module_info(struct dso *dso, struct kmod_path *m,
  *   dso__data_size
  *   dso__data_read_offset
  *   dso__data_read_addr
+ *   dso__data_write_cache_offs
+ *   dso__data_write_cache_addr
  *
  * Please refer to the dso.c object code for each function and
  * arguments documentation. Following text tries to explain the
@@ -332,6 +334,11 @@ ssize_t dso__data_read_addr(struct dso *dso, struct map *map,
 			    struct machine *machine, u64 addr,
 			    u8 *data, ssize_t size);
 bool dso__data_status_seen(struct dso *dso, enum dso_data_status_seen by);
+ssize_t dso__data_write_cache_offs(struct dso *dso, struct machine *machine,
+				   u64 offset, const u8 *data, ssize_t size);
+ssize_t dso__data_write_cache_addr(struct dso *dso, struct map *map,
+				   struct machine *machine, u64 addr,
+				   const u8 *data, ssize_t size);
 
 struct map *dso__new_map(const char *name);
 struct dso *machine__findnew_kernel(struct machine *machine, const char *name,
Signature · 1 line
2.17.1
#27 lore raw

Re: [PATCH RFC 3/6] perf dso: Add dso__data_write_cache_addr()

On Fri, Oct 25, 2019 at 03:59:57PM +0300, Adrian Hunter wrote:
SNIP

Quoted 10 lines
tools/perf/util/dso.c +3 -2
-static ssize_t data_read_offset(struct dso *dso, struct machine *machine,
-				u64 offset, u8 *data, ssize_t size)
+static ssize_t data_read_write_offset(struct dso *dso, struct machine *machine,
+				      u64 offset, u8 *data, ssize_t size,
+				      bool out)
 {
 	if (dso__data_file_size(dso, machine))
 		return -1;

tools/perf/util/dso.c static ssize_t data_read_offset(struct dso *dso, struct machine *machine, · line 1040

Quoted 5 lines
tools/perf/util/dso.c +1 -1
@@ -1034,7 +1037,7 @@ static ssize_t data_read_offset(struct dso *dso, struct machine *machine,
 	if (offset + size < offset)
 		return -1;
 
-	return cached_read(dso, machine, offset, data, size);
+	return cached_io(dso, machine, offset, data, size, out);

you renamed the function as _read_write_ so the bool choosing the opration should be named either read or write.. I had to go all the way down to find out what 'out' means ;-)

jirka

author reviewer
#28 lore raw

Re: [PATCH RFC 3/6] perf dso: Add dso__data_write_cache_addr()

Quoted 16 lines · Jiri Olsa

On Fri, Oct 25, 2019 at 03:59:57PM +0300, Adrian Hunter wrote:
SNIP

Quoted 13 lines
(quoted diff) +3 -2
-static ssize_t data_read_offset(struct dso *dso, struct machine *machine,
-				u64 offset, u8 *data, ssize_t size)
+static ssize_t data_read_write_offset(struct dso *dso, struct machine *machine,
+				      u64 offset, u8 *data, ssize_t size,
+				      bool out)
 {
 	if (dso__data_file_size(dso, machine))
 		return -1;
@@ -1034,7 +1037,7 @@ static ssize_t data_read_offset(struct dso *dso, struct machine *machine,
 	if (offset + size < offset)
 		return -1;
 
Quoted 7 lines

Review static ssize_t data_read_offset(struct dso *dso, struct machine *machine, · line 1040

Quoted 2 lines
(quoted diff) +1 -1
-	return cached_read(dso, machine, offset, data, size);
+	return cached_io(dso, machine, offset, data, size, out);

you renamed the function as _read_write_ so the bool choosing the opration should be named either read or write.. I had to go all the way down to find out what 'out' means ;-)

Arnaldo already applied it, so here is the diff:

tools/perf/util/dso.c +8 -8
index 12ab26baba44..505ba78eda3c 100644
@@ -829,12 +829,12 @@ dso_cache__insert(struct dso *dso, struct dso_cache *new)
 }
 
 static ssize_t dso_cache__memcpy(struct dso_cache *cache, u64 offset, u8 *data,
-				 u64 size, bool out)
+				 u64 size, bool read)
 {
 	u64 cache_offset = offset - cache->offset;
 	u64 cache_size   = min(cache->size - cache_offset, size);
 
-	if (out)
+	if (read)
 		memcpy(data, cache->data + cache_offset, cache_size);
 	else
 		memcpy(cache->data + cache_offset, data, cache_size);
@@ -912,14 +912,14 @@ static struct dso_cache *dso_cache__find(struct dso *dso,
 }
 
 static ssize_t dso_cache_io(struct dso *dso, struct machine *machine,
-			    u64 offset, u8 *data, ssize_t size, bool out)
+			    u64 offset, u8 *data, ssize_t size, bool read)
 {
 	struct dso_cache *cache = dso_cache__find(dso, machine, offset);
 
 	if (IS_ERR_OR_NULL(cache))
 		return PTR_ERR(cache);
 
-	return dso_cache__memcpy(cache, offset, data, size, out);
+	return dso_cache__memcpy(cache, offset, data, size, read);
 }
 
 /*
@@ -928,7 +928,7 @@ static ssize_t dso_cache_io(struct dso *dso, struct machine *machine,
  * by cached data. Writes update the cache only, not the backing file.
  */
 static ssize_t cached_io(struct dso *dso, struct machine *machine,
-			 u64 offset, u8 *data, ssize_t size, bool out)
+			 u64 offset, u8 *data, ssize_t size, bool read)
 {
 	ssize_t r = 0;
 	u8 *p = data;
@@ -936,7 +936,7 @@ static ssize_t cached_io(struct dso *dso, struct machine *machine,
 	do {
 		ssize_t ret;
 
-		ret = dso_cache_io(dso, machine, offset, p, size, out);
+		ret = dso_cache_io(dso, machine, offset, p, size, read);
 		if (ret < 0)
 			return ret;
 
@@ -1022,7 +1022,7 @@ off_t dso__data_size(struct dso *dso, struct machine *machine)
 
 static ssize_t data_read_write_offset(struct dso *dso, struct machine *machine,
 				      u64 offset, u8 *data, ssize_t size,
-				      bool out)
+				      bool read)
 {
 	if (dso__data_file_size(dso, machine))
 		return -1;
@@ -1034,7 +1034,7 @@ static ssize_t data_read_write_offset(struct dso *dso, struct machine *machine,
 	if (offset + size < offset)
 		return -1;
 
-	return cached_io(dso, machine, offset, data, size, out);
+	return cached_io(dso, machine, offset, data, size, read);
 }
 
 /**
merge bot Applied
#29 lore raw

[tip: perf/core] perf dso: Add dso__data_write_cache_addr()

In reply to Adrian Hunter
The following commit has been merged into the perf/core branch of tip:

The following commit has been merged into the perf/core branch of tip:

Commit-ID:     b86a9d918a389162803d833d4dc491fde9b62fa2
Gitweb:        https://git.kernel.org/tip/b86a9d918a389162803d833d4dc491fde9b62fa2
Author:        Adrian Hunter <[email protected]>
AuthorDate:    Fri, 25 Oct 2019 15:59:57 +03:00
Committer:     Arnaldo Carvalho de Melo <[email protected]>
CommitterDate: Wed, 06 Nov 2019 15:43:06 -03:00

perf dso: Add dso__data_write_cache_addr()

Add functions to write into the dso file data cache, but not change the file itself.

---
 tools/perf/util/dso.c | 73 +++++++++++++++++++++++++++++++++---------
 tools/perf/util/dso.h |  7 ++++-
 2 files changed, 65 insertions(+), 15 deletions(-)
tools/perf/util/dso.c +58 -15
index 460330d..0f1b772 100644
@@ -827,14 +827,16 @@ out:
 	return cache;
 }
 
-static ssize_t
-dso_cache__memcpy(struct dso_cache *cache, u64 offset,
-		  u8 *data, u64 size)
+static ssize_t dso_cache__memcpy(struct dso_cache *cache, u64 offset, u8 *data,
+				 u64 size, bool out)
 {
 	u64 cache_offset = offset - cache->offset;
 	u64 cache_size   = min(cache->size - cache_offset, size);
 
-	memcpy(data, cache->data + cache_offset, cache_size);
+	if (out)
+		memcpy(data, cache->data + cache_offset, cache_size);
+	else
+		memcpy(cache->data + cache_offset, data, cache_size);
 	return cache_size;
 }
 
@@ -910,8 +912,8 @@ static struct dso_cache *dso_cache__find(struct dso *dso,
 	return cache ? cache : dso_cache__populate(dso, machine, offset, ret);
 }
 
-static ssize_t dso_cache_read(struct dso *dso, struct machine *machine,
-			      u64 offset, u8 *data, ssize_t size)
+static ssize_t dso_cache_io(struct dso *dso, struct machine *machine,
+			    u64 offset, u8 *data, ssize_t size, bool out)
 {
 	struct dso_cache *cache;
 	ssize_t ret = 0;
@@ -920,16 +922,16 @@ static ssize_t dso_cache_read(struct dso *dso, struct machine *machine,
 	if (!cache)
 		return ret;
 
-	return dso_cache__memcpy(cache, offset, data, size);
+	return dso_cache__memcpy(cache, offset, data, size, out);
 }
 
 /*
  * Reads and caches dso data DSO__DATA_CACHE_SIZE size chunks
  * in the rb_tree. Any read to already cached data is served
- * by cached data.
+ * by cached data. Writes update the cache only, not the backing file.
  */
-static ssize_t cached_read(struct dso *dso, struct machine *machine,
-			   u64 offset, u8 *data, ssize_t size)
+static ssize_t cached_io(struct dso *dso, struct machine *machine,
+			 u64 offset, u8 *data, ssize_t size, bool out)
 {
 	ssize_t r = 0;
 	u8 *p = data;
@@ -937,7 +939,7 @@ static ssize_t cached_read(struct dso *dso, struct machine *machine,
 	do {
 		ssize_t ret;
 
-		ret = dso_cache_read(dso, machine, offset, p, size);
+		ret = dso_cache_io(dso, machine, offset, p, size, out);
 		if (ret < 0)
 			return ret;
 
@@ -1021,8 +1023,9 @@ off_t dso__data_size(struct dso *dso, struct machine *machine)
 	return dso->data.file_size;
 }
 
-static ssize_t data_read_offset(struct dso *dso, struct machine *machine,
-				u64 offset, u8 *data, ssize_t size)
+static ssize_t data_read_write_offset(struct dso *dso, struct machine *machine,
+				      u64 offset, u8 *data, ssize_t size,
+				      bool out)
 {
 	if (dso__data_file_size(dso, machine))
 		return -1;
@@ -1034,7 +1037,7 @@ static ssize_t data_read_offset(struct dso *dso, struct machine *machine,
 	if (offset + size < offset)
 		return -1;
 
-	return cached_read(dso, machine, offset, data, size);
+	return cached_io(dso, machine, offset, data, size, out);
 }
 
 /**
@@ -1054,7 +1057,7 @@ ssize_t dso__data_read_offset(struct dso *dso, struct machine *machine,
 	if (dso->data.status == DSO_DATA_STATUS_ERROR)
 		return -1;
 
-	return data_read_offset(dso, machine, offset, data, size);
+	return data_read_write_offset(dso, machine, offset, data, size, true);
 }
 
 /**
@@ -1075,6 +1078,46 @@ ssize_t dso__data_read_addr(struct dso *dso, struct map *map,
 	return dso__data_read_offset(dso, machine, offset, data, size);
 }
 
+/**
+ * dso__data_write_cache_offs - Write data to dso data cache at file offset
+ * @dso: dso object
+ * @machine: machine object
+ * @offset: file offset
+ * @data: buffer to write
+ * @size: size of the @data buffer
+ *
+ * Write into the dso file data cache, but do not change the file itself.
+ */
+ssize_t dso__data_write_cache_offs(struct dso *dso, struct machine *machine,
+				   u64 offset, const u8 *data_in, ssize_t size)
+{
+	u8 *data = (u8 *)data_in; /* cast away const to use same fns for r/w */
+
+	if (dso->data.status == DSO_DATA_STATUS_ERROR)
+		return -1;
+
+	return data_read_write_offset(dso, machine, offset, data, size, false);
+}
+
+/**
+ * dso__data_write_cache_addr - Write data to dso data cache at dso address
+ * @dso: dso object
+ * @machine: machine object
+ * @add: virtual memory address
+ * @data: buffer to write
+ * @size: size of the @data buffer
+ *
+ * External interface to write into the dso file data cache, but do not change
+ * the file itself.
+ */
+ssize_t dso__data_write_cache_addr(struct dso *dso, struct map *map,
+				   struct machine *machine, u64 addr,
+				   const u8 *data, ssize_t size)
+{
+	u64 offset = map->map_ip(map, addr);
+	return dso__data_write_cache_offs(dso, machine, offset, data, size);
+}
+
 struct map *dso__new_map(const char *name)
 {
 	struct map *map = NULL;
tools/perf/util/dso.h +7 -0
index e4dddb7..2f1fcbc 100644
@@ -285,6 +285,8 @@ void dso__set_module_info(struct dso *dso, struct kmod_path *m,
  *   dso__data_size
  *   dso__data_read_offset
  *   dso__data_read_addr
+ *   dso__data_write_cache_offs
+ *   dso__data_write_cache_addr
  *
  * Please refer to the dso.c object code for each function and
  * arguments documentation. Following text tries to explain the
@@ -332,6 +334,11 @@ ssize_t dso__data_read_addr(struct dso *dso, struct map *map,
 			    struct machine *machine, u64 addr,
 			    u8 *data, ssize_t size);
 bool dso__data_status_seen(struct dso *dso, enum dso_data_status_seen by);
+ssize_t dso__data_write_cache_offs(struct dso *dso, struct machine *machine,
+				   u64 offset, const u8 *data, ssize_t size);
+ssize_t dso__data_write_cache_addr(struct dso *dso, struct map *map,
+				   struct machine *machine, u64 addr,
+				   const u8 *data, ssize_t size);
 
 struct map *dso__new_map(const char *name);
 struct dso *machine__findnew_kernel(struct machine *machine, const char *name,
author reviewer
#30 lore raw

RFC PATCH 4/6 perf tools: Add support for PERF_RECORD_TEXT_POKE

In reply to Adrian Hunter

Add processing for PERF_RECORD_TEXT_POKE events. When a text poke event is processed, if the PERF_TEXT_POKE_UPDATE flag is set, then the kernel dso data cache is updated with the poked bytes.

---
 tools/include/uapi/linux/perf_event.h     | 28 +++++++++++++-
 tools/perf/arch/x86/util/intel-pt.c       |  1 +
 tools/perf/builtin-record.c               | 45 +++++++++++++++++++++++
 tools/perf/lib/include/perf/event.h       |  9 +++++
 tools/perf/util/event.c                   | 39 ++++++++++++++++++++
 tools/perf/util/event.h                   |  5 +++
 tools/perf/util/evlist.h                  |  1 +
 tools/perf/util/machine.c                 | 38 +++++++++++++++++++
 tools/perf/util/machine.h                 |  3 ++
 tools/perf/util/perf_event_attr_fprintf.c |  1 +
 tools/perf/util/record.c                  | 10 +++++
 tools/perf/util/record.h                  |  1 +
 tools/perf/util/session.c                 | 22 +++++++++++
 tools/perf/util/tool.h                    |  3 +-
 14 files changed, 204 insertions(+), 2 deletions(-)
tools/include/uapi/linux/perf_event.h +27 -1
index bb7b271397a6..c8d1f52a7fce 100644
@@ -375,7 +375,8 @@ struct perf_event_attr {
 				ksymbol        :  1, /* include ksymbol events */
 				bpf_event      :  1, /* include bpf events */
 				aux_output     :  1, /* generate AUX records instead of events */
-				__reserved_1   : 32;
+				text_poke      :  1, /* include text poke events */
+				__reserved_1   : 31;
 
 	union {
 		__u32		wakeup_events;	  /* wakeup every n events */
@@ -1000,6 +1001,26 @@ enum perf_event_type {
 	 */
 	PERF_RECORD_BPF_EVENT			= 18,
 
+	/*
+	 * Records changes to kernel text i.e. self-modified code.
+	 * 'flags' has PERF_TEXT_POKE_UPDATE (i.e. bit 0) set to
+	 * indicate to tools to update old bytes to new bytes in the
+	 * executable image.
+	 * 'len' is the number of old bytes, which is the same as the number
+	 * of new bytes. 'bytes' contains the old bytes followed immediately
+	 * by the new bytes.
+	 *
+	 * struct {
+	 *	struct perf_event_header	header;
+	 *	u64				addr;
+	 *	u16				flags;
+	 *	u16				len;
+	 *	u8				bytes[];
+	 *	struct sample_id		sample_id;
+	 * };
+	 */
+	PERF_RECORD_TEXT_POKE			= 19,
+
 	PERF_RECORD_MAX,			/* non-ABI */
 };
 
@@ -1041,6 +1062,11 @@ enum perf_callchain_context {
 #define PERF_AUX_FLAG_PARTIAL		0x04	/* record contains gaps */
 #define PERF_AUX_FLAG_COLLISION		0x08	/* sample collided with another */
 
+/**
+ * PERF_RECORD_TEXT_POKE::flags bits
+ */
+#define PERF_TEXT_POKE_UPDATE		0x01	/* update old bytes to new bytes */
+
 #define PERF_FLAG_FD_NO_GROUP		(1UL << 0)
 #define PERF_FLAG_FD_OUTPUT		(1UL << 1)
 #define PERF_FLAG_PID_CGROUP		(1UL << 2) /* pid=cgroup id, per-cpu mode only */
tools/perf/arch/x86/util/intel-pt.c +1 -0
index d6d26256915f..a81ce60876f6 100644
@@ -17,6 +17,7 @@
 #include "../../util/event.h"
 #include "../../util/evlist.h"
 #include "../../util/evsel.h"
+#include "../../util/evsel_config.h"
 #include "../../util/cpumap.h"
 #include "../../util/mmap.h"
 #include <subcmd/parse-options.h>
tools/perf/builtin-record.c +45 -0

57 lines not on this page · raw message

tools/perf/lib/include/perf/event.h +9 -0
index 18106899cb4e..0127125eefd7 100644
@@ -105,6 +105,14 @@ struct perf_record_bpf_event {
 	__u8			 tag[BPF_TAG_SIZE];  // prog tag
 };
 
+struct perf_record_text_poke_event {
+	struct perf_event_header header;
+	__u64			addr;
+	__u16			flags;
+	__u16			len;
+	__u8			bytes[];
+};
+
 struct perf_record_sample {
 	struct perf_event_header header;
 	__u64			 array[];
@@ -360,6 +368,7 @@ union perf_event {
 	struct perf_record_sample		sample;
 	struct perf_record_bpf_event		bpf;
 	struct perf_record_ksymbol		ksymbol;
+	struct perf_record_text_poke_event	text_poke;
 	struct perf_record_header_attr		attr;
 	struct perf_record_event_update		event_update;
 	struct perf_record_header_event_type	event_type;
tools/perf/util/event.c +39 -0
index fc1e5a991008..b1e59fc3bce9 100644
@@ -31,6 +31,7 @@
 #include "stat.h"
 #include "session.h"
 #include "bpf-event.h"
+#include "print_binary.h"
 #include "tool.h"
 #include "../perf.h"
 
@@ -54,6 +55,7 @@ static const char *perf_event__names[] = {
 	[PERF_RECORD_NAMESPACES]		= "NAMESPACES",
 	[PERF_RECORD_KSYMBOL]			= "KSYMBOL",
 	[PERF_RECORD_BPF_EVENT]			= "BPF_EVENT",
+	[PERF_RECORD_TEXT_POKE]			= "TEXT_POKE",
 	[PERF_RECORD_HEADER_ATTR]		= "ATTR",
 	[PERF_RECORD_HEADER_EVENT_TYPE]		= "EVENT_TYPE",
 	[PERF_RECORD_HEADER_TRACING_DATA]	= "TRACING_DATA",
@@ -252,6 +254,14 @@ int perf_event__process_bpf(struct perf_tool *tool __maybe_unused,
 	return machine__process_bpf(machine, event, sample);
 }
 
+int perf_event__process_text_poke(struct perf_tool *tool __maybe_unused,
+				  union perf_event *event,
+				  struct perf_sample *sample,
+				  struct machine *machine)
+{
+	return machine__process_text_poke(machine, event, sample);
+}
+
 size_t perf_event__fprintf_mmap(union perf_event *event, FILE *fp)
 {
 	return fprintf(fp, " %d/%d: [%#" PRI_lx64 "(%#" PRI_lx64 ") @ %#" PRI_lx64 "]: %c %s\n",
@@ -398,6 +408,32 @@ size_t perf_event__fprintf_bpf(union perf_event *event, FILE *fp)
 		       event->bpf.type, event->bpf.flags, event->bpf.id);
 }
 
+static int text_poke_printer(enum binary_printer_ops op, unsigned int val,
+			     void *extra __maybe_unused, FILE *fp)
+{
+	if (op == BINARY_PRINT_NUM_DATA)
+		return fprintf(fp, " %02x", val);
+	if (op == BINARY_PRINT_LINE_END)
+		return fprintf(fp, "\n");
+	return 0;
+}
+
+size_t perf_event__fprintf_text_poke(union perf_event *event, FILE *fp)
+{
+	size_t ret = fprintf(fp, " addr %#" PRI_lx64 " flags %#x len %u\n",
+			     event->text_poke.addr, event->text_poke.flags,
+			     event->text_poke.len);
+
+	ret += fprintf(fp, "     old bytes:");
+	ret += binary__fprintf(event->text_poke.bytes, event->text_poke.len, 16,
+			       text_poke_printer, NULL, fp);
+	ret += fprintf(fp, "     new bytes:");
+	ret += binary__fprintf(event->text_poke.bytes + event->text_poke.len,
+			       event->text_poke.len, 16, text_poke_printer,
+			       NULL, fp);
+	return ret;
+}
+
 size_t perf_event__fprintf(union perf_event *event, FILE *fp)
 {
 	size_t ret = fprintf(fp, "PERF_RECORD_%s",
@@ -439,6 +475,9 @@ size_t perf_event__fprintf(union perf_event *event, FILE *fp)
 	case PERF_RECORD_BPF_EVENT:
 		ret += perf_event__fprintf_bpf(event, fp);
 		break;
+	case PERF_RECORD_TEXT_POKE:
+		ret += perf_event__fprintf_text_poke(event, fp);
+		break;
 	default:
 		ret += fprintf(fp, "\n");
 	}
tools/perf/util/event.h +5 -0
index a0a0c91cde4a..f0f4f4db6e22 100644
@@ -339,6 +339,10 @@ int perf_event__process_bpf(struct perf_tool *tool,
 			    union perf_event *event,
 			    struct perf_sample *sample,
 			    struct machine *machine);
+int perf_event__process_text_poke(struct perf_tool *tool,
+				  union perf_event *event,
+				  struct perf_sample *sample,
+				  struct machine *machine);
 int perf_event__process(struct perf_tool *tool,
 			union perf_event *event,
 			struct perf_sample *sample,
@@ -372,6 +376,7 @@ size_t perf_event__fprintf_cpu_map(union perf_event *event, FILE *fp);
 size_t perf_event__fprintf_namespaces(union perf_event *event, FILE *fp);
 size_t perf_event__fprintf_ksymbol(union perf_event *event, FILE *fp);
 size_t perf_event__fprintf_bpf(union perf_event *event, FILE *fp);
+size_t perf_event__fprintf_text_poke(union perf_event *event, FILE *fp);
 size_t perf_event__fprintf(union perf_event *event, FILE *fp);
 
 int kallsyms__get_function_start(const char *kallsyms_filename,
tools/perf/util/evlist.h +1 -0
index 13051409fd22..a2b9fa8fb281 100644
@@ -175,6 +175,7 @@ struct callchain_param;
 void perf_evlist__set_id_pos(struct evlist *evlist);
 bool perf_can_sample_identifier(void);
 bool perf_can_record_switch_events(void);
+bool perf_can_record_text_poke_events(void);
 bool perf_can_record_cpu_wide(void);
 void perf_evlist__config(struct evlist *evlist, struct record_opts *opts,
 			 struct callchain_param *callchain);
tools/perf/util/machine.c +38 -0
index 70a9f8716a4b..edf0327093b8 100644
@@ -767,6 +767,42 @@ int machine__process_ksymbol(struct machine *machine __maybe_unused,
 	return machine__process_ksymbol_register(machine, event, sample);
 }
 
+int machine__process_text_poke(struct machine *machine, union perf_event *event,
+			       struct perf_sample *sample __maybe_unused)
+{
+	struct map_groups *mg = &machine->kmaps;
+	struct map *map = map_groups__find(mg, event->text_poke.addr);
+
+	if (dump_trace)
+		perf_event__fprintf_text_poke(event, stdout);
+
+	if (!(event->text_poke.flags & PERF_TEXT_POKE_UPDATE))
+		return 0;
+
+	if (map) {
+		u8 *new_bytes = event->text_poke.bytes + event->text_poke.len;
+		int ret;
+
+		/*
+		 * Kernel maps might be changed when loading symbols so loading
+		 * must be done prior to using kernel maps.
+		 */
+		map__load(map);
+		ret = dso__data_write_cache_addr(map->dso, map, machine,
+						 event->text_poke.addr,
+						 new_bytes,
+						 event->text_poke.len);
+		if (ret != event->text_poke.len)
+			pr_err("Failed to write kernel text poke at %#" PRI_lx64 "\n",
+			       event->text_poke.addr);
+	} else {
+		pr_err("Failed to find kernel text poke address map for %#" PRI_lx64 "\n",
+		       event->text_poke.addr);
+	}
+
+	return 0;
+}
+
 static void dso__adjust_kmod_long_name(struct dso *dso, const char *filename)
 {
 	const char *dup_filename;
@@ -1931,6 +1967,8 @@ int machine__process_event(struct machine *machine, union perf_event *event,
 		ret = machine__process_ksymbol(machine, event, sample); break;
 	case PERF_RECORD_BPF_EVENT:
 		ret = machine__process_bpf(machine, event, sample); break;
+	case PERF_RECORD_TEXT_POKE:
+		ret = machine__process_text_poke(machine, event, sample); break;
 	default:
 		ret = -1;
 		break;
tools/perf/util/machine.h +3 -0
index 18e13c0ccd6a..af936ec01c46 100644
@@ -134,6 +134,9 @@ int machine__process_mmap2_event(struct machine *machine, union perf_event *even
 int machine__process_ksymbol(struct machine *machine,
 			     union perf_event *event,
 			     struct perf_sample *sample);
+int machine__process_text_poke(struct machine *machine,
+			       union perf_event *event,
+			       struct perf_sample *sample);
 int machine__process_event(struct machine *machine, union perf_event *event,
 				struct perf_sample *sample);
 
tools/perf/util/perf_event_attr_fprintf.c +1 -0
index d4ad3f04923a..0134ffd27b23 100644
@@ -143,6 +143,7 @@ int perf_event_attr__fprintf(FILE *fp, struct perf_event_attr *attr,
 	PRINT_ATTRf(sample_regs_intr, p_hex);
 	PRINT_ATTRf(aux_watermark, p_unsigned);
 	PRINT_ATTRf(sample_max_stack, p_unsigned);
+	PRINT_ATTRf(text_poke, p_unsigned);
 
 	return ret;
 }
tools/perf/util/record.c +10 -0
index 8579505c29a4..a58d01b65078 100644
@@ -97,6 +97,11 @@ static void perf_probe_context_switch(struct evsel *evsel)
 	evsel->core.attr.context_switch = 1;
 }
 
+static void perf_probe_text_poke(struct evsel *evsel)
+{
+	evsel->core.attr.text_poke = 1;
+}
+
 bool perf_can_sample_identifier(void)
 {
 	return perf_probe_api(perf_probe_sample_identifier);
@@ -112,6 +117,11 @@ bool perf_can_record_switch_events(void)
 	return perf_probe_api(perf_probe_context_switch);
 }
 
+bool perf_can_record_text_poke_events(void)
+{
+	return perf_probe_api(perf_probe_text_poke);
+}
+
 bool perf_can_record_cpu_wide(void)
 {
 	struct perf_event_attr attr = {
tools/perf/util/record.h +1 -0
index 948bbcf9aef3..acc04d873059 100644
@@ -45,6 +45,7 @@ struct record_opts {
 	bool	      sample_id;
 	bool	      no_bpf_event;
 	bool	      kcore;
+	bool	      text_poke;
 	unsigned int  freq;
 	unsigned int  mmap_pages;
 	unsigned int  auxtrace_mmap_pages;
tools/perf/util/session.c +22 -0
index f07b8ecb91bc..e2c0e72701b0 100644
@@ -489,6 +489,8 @@ void perf_tool__fill_defaults(struct perf_tool *tool)
 		tool->ksymbol = perf_event__process_ksymbol;
 	if (tool->bpf == NULL)
 		tool->bpf = perf_event__process_bpf;
+	if (tool->text_poke == NULL)
+		tool->text_poke = perf_event__process_text_poke;
 	if (tool->read == NULL)
 		tool->read = process_event_sample_stub;
 	if (tool->throttle == NULL)
@@ -658,6 +660,23 @@ static void perf_event__switch_swap(union perf_event *event, bool sample_id_all)
 		swap_sample_id_all(event, &event->context_switch + 1);
 }
 
+static void perf_event__text_poke_swap(union perf_event *event, bool sample_id_all)
+{
+	event->text_poke.addr  = bswap_64(event->text_poke.addr);
+	event->text_poke.flags = bswap_16(event->text_poke.flags);
+	event->text_poke.len   = bswap_16(event->text_poke.len);
+
+	if (sample_id_all) {
+		size_t len = sizeof(event->text_poke.flags) +
+			     sizeof(event->text_poke.len) +
+			     event->text_poke.len * 2;
+		void *data = &event->text_poke.flags;
+
+		data += PERF_ALIGN(len, sizeof(u64));
+		swap_sample_id_all(event, data);
+	}
+}
+
 static void perf_event__throttle_swap(union perf_event *event,
 				      bool sample_id_all)
 {
@@ -930,6 +949,7 @@ static perf_event__swap_op perf_event__swap_ops[] = {
 	[PERF_RECORD_SWITCH]		  = perf_event__switch_swap,
 	[PERF_RECORD_SWITCH_CPU_WIDE]	  = perf_event__switch_swap,
 	[PERF_RECORD_NAMESPACES]	  = perf_event__namespaces_swap,
+	[PERF_RECORD_TEXT_POKE]		  = perf_event__text_poke_swap,
 	[PERF_RECORD_HEADER_ATTR]	  = perf_event__hdr_attr_swap,
 	[PERF_RECORD_HEADER_EVENT_TYPE]	  = perf_event__event_type_swap,
 	[PERF_RECORD_HEADER_TRACING_DATA] = perf_event__tracing_data_swap,
@@ -1469,6 +1489,8 @@ static int machines__deliver_event(struct machines *machines,
 		return tool->ksymbol(tool, event, sample, machine);
 	case PERF_RECORD_BPF_EVENT:
 		return tool->bpf(tool, event, sample, machine);
+	case PERF_RECORD_TEXT_POKE:
+		return tool->text_poke(tool, event, sample, machine);
 	default:
 		++evlist->stats.nr_unknown_events;
 		return -1;
tools/perf/util/tool.h +2 -1
index 2abbf668b8de..006182923bbe 100644
@@ -56,7 +56,8 @@ struct perf_tool {
 			throttle,
 			unthrottle,
 			ksymbol,
-			bpf;
+			bpf,
+			text_poke;
 
 	event_attr_op	attr;
 	event_attr_op	event_update;
Signature · 1 line
2.17.1
author reviewer
#31 lore raw

RFC PATCH 5/6 perf auxtrace: Add auxtrace_cache__remove()

In reply to Adrian Hunter

Add auxtrace_cache__remove(). Intel PT uses an auxtrace_cache to store the results of code-walking, so that the same block of instructions does not have to be decoded repeatedly. However, when there are text poke events, the associated cache entries need to be removed.

---
 tools/perf/util/auxtrace.c | 28 ++++++++++++++++++++++++++++
 tools/perf/util/auxtrace.h |  1 +
 2 files changed, 29 insertions(+)
tools/perf/util/auxtrace.c +28 -0
index 8470dfe9fe97..c555c3ccd79d 100644
@@ -1457,6 +1457,34 @@ int auxtrace_cache__add(struct auxtrace_cache *c, u32 key,
 	return 0;
 }
 
+static struct auxtrace_cache_entry *auxtrace_cache__rm(struct auxtrace_cache *c,
+						       u32 key)
+{
+	struct auxtrace_cache_entry *entry;
+	struct hlist_head *hlist;
+	struct hlist_node *n;
+
+	if (!c)
+		return NULL;
+
+	hlist = &c->hashtable[hash_32(key, c->bits)];
+	hlist_for_each_entry_safe(entry, n, hlist, hash) {
+		if (entry->key == key) {
+			hlist_del(&entry->hash);
+			return entry;
+		}
+	}
+
+	return NULL;
+}
+
+void auxtrace_cache__remove(struct auxtrace_cache *c, u32 key)
+{
+	struct auxtrace_cache_entry *entry = auxtrace_cache__rm(c, key);
+
+	auxtrace_cache__free_entry(c, entry);
+}
+
 void *auxtrace_cache__lookup(struct auxtrace_cache *c, u32 key)
 {
 	struct auxtrace_cache_entry *entry;
tools/perf/util/auxtrace.h +1 -0
index f201f36bc35f..3f4aa5427d76 100644
@@ -489,6 +489,7 @@ void *auxtrace_cache__alloc_entry(struct auxtrace_cache *c);
 void auxtrace_cache__free_entry(struct auxtrace_cache *c, void *entry);
 int auxtrace_cache__add(struct auxtrace_cache *c, u32 key,
 			struct auxtrace_cache_entry *entry);
+void auxtrace_cache__remove(struct auxtrace_cache *c, u32 key);
 void *auxtrace_cache__lookup(struct auxtrace_cache *c, u32 key);
 
 struct auxtrace_record *auxtrace_record__init(struct evlist *evlist,
Signature · 1 line
2.17.1
#32 lore raw

Re: [PATCH RFC 5/6] perf auxtrace: Add auxtrace_cache__remove()

Em Fri, Oct 25, 2019 at 03:59:59PM +0300, Adrian Hunter escreveu:

Quoted 4 lines

Add auxtrace_cache__remove(). Intel PT uses an auxtrace_cache to store the results of code-walking, so that the same block of instructions does not have to be decoded repeatedly. However, when there are text poke events, the associated cache entries need to be removed.

Applied this one as it is just leg work for the rest, that I'll wait a bit for comments.

- Arnaldo

Quoted 59 lines
---
 tools/perf/util/auxtrace.c | 28 ++++++++++++++++++++++++++++
 tools/perf/util/auxtrace.h |  1 +
 2 files changed, 29 insertions(+)
tools/perf/util/auxtrace.c +28 -0
index 8470dfe9fe97..c555c3ccd79d 100644
@@ -1457,6 +1457,34 @@ int auxtrace_cache__add(struct auxtrace_cache *c, u32 key,
 	return 0;
 }
 
+static struct auxtrace_cache_entry *auxtrace_cache__rm(struct auxtrace_cache *c,
+						       u32 key)
+{
+	struct auxtrace_cache_entry *entry;
+	struct hlist_head *hlist;
+	struct hlist_node *n;
+
+	if (!c)
+		return NULL;
+
+	hlist = &c->hashtable[hash_32(key, c->bits)];
+	hlist_for_each_entry_safe(entry, n, hlist, hash) {
+		if (entry->key == key) {
+			hlist_del(&entry->hash);
+			return entry;
+		}
+	}
+
+	return NULL;
+}
+
+void auxtrace_cache__remove(struct auxtrace_cache *c, u32 key)
+{
+	struct auxtrace_cache_entry *entry = auxtrace_cache__rm(c, key);
+
+	auxtrace_cache__free_entry(c, entry);
+}
+
 void *auxtrace_cache__lookup(struct auxtrace_cache *c, u32 key)
 {
 	struct auxtrace_cache_entry *entry;
tools/perf/util/auxtrace.h +1 -0
index f201f36bc35f..3f4aa5427d76 100644
@@ -489,6 +489,7 @@ void *auxtrace_cache__alloc_entry(struct auxtrace_cache *c);
 void auxtrace_cache__free_entry(struct auxtrace_cache *c, void *entry);
 int auxtrace_cache__add(struct auxtrace_cache *c, u32 key,
 			struct auxtrace_cache_entry *entry);
+void auxtrace_cache__remove(struct auxtrace_cache *c, u32 key);
 void *auxtrace_cache__lookup(struct auxtrace_cache *c, u32 key);
 
 struct auxtrace_record *auxtrace_record__init(struct evlist *evlist,
Signature · 1 line
2.17.1
Signature · 1 line
- Arnaldo
merge bot Applied
#33 lore raw

[tip: perf/core] perf auxtrace: Add auxtrace_cache__remove()

In reply to Adrian Hunter
The following commit has been merged into the perf/core branch of tip:

The following commit has been merged into the perf/core branch of tip:

Commit-ID:     fd62c1097a0700484fc2cbc9a182f341f30890cd
Gitweb:        https://git.kernel.org/tip/fd62c1097a0700484fc2cbc9a182f341f30890cd
Author:        Adrian Hunter <[email protected]>
AuthorDate:    Fri, 25 Oct 2019 15:59:59 +03:00
Committer:     Arnaldo Carvalho de Melo <[email protected]>
CommitterDate: Wed, 06 Nov 2019 15:43:06 -03:00

perf auxtrace: Add auxtrace_cache__remove()

Add auxtrace_cache__remove(). Intel PT uses an auxtrace_cache to store the results of code-walking, so that the same block of instructions does not have to be decoded repeatedly. However, when there are text poke events, the associated cache entries need to be removed.

---
 tools/perf/util/auxtrace.c | 28 ++++++++++++++++++++++++++++
 tools/perf/util/auxtrace.h |  1 +
 2 files changed, 29 insertions(+)
tools/perf/util/auxtrace.c +28 -0
index 8470dfe..c555c3c 100644
@@ -1457,6 +1457,34 @@ int auxtrace_cache__add(struct auxtrace_cache *c, u32 key,
 	return 0;
 }
 
+static struct auxtrace_cache_entry *auxtrace_cache__rm(struct auxtrace_cache *c,
+						       u32 key)
+{
+	struct auxtrace_cache_entry *entry;
+	struct hlist_head *hlist;
+	struct hlist_node *n;
+
+	if (!c)
+		return NULL;
+
+	hlist = &c->hashtable[hash_32(key, c->bits)];
+	hlist_for_each_entry_safe(entry, n, hlist, hash) {
+		if (entry->key == key) {
+			hlist_del(&entry->hash);
+			return entry;
+		}
+	}
+
+	return NULL;
+}
+
+void auxtrace_cache__remove(struct auxtrace_cache *c, u32 key)
+{
+	struct auxtrace_cache_entry *entry = auxtrace_cache__rm(c, key);
+
+	auxtrace_cache__free_entry(c, entry);
+}
+
 void *auxtrace_cache__lookup(struct auxtrace_cache *c, u32 key)
 {
 	struct auxtrace_cache_entry *entry;
tools/perf/util/auxtrace.h +1 -0
index f201f36..3f4aa54 100644
@@ -489,6 +489,7 @@ void *auxtrace_cache__alloc_entry(struct auxtrace_cache *c);
 void auxtrace_cache__free_entry(struct auxtrace_cache *c, void *entry);
 int auxtrace_cache__add(struct auxtrace_cache *c, u32 key,
 			struct auxtrace_cache_entry *entry);
+void auxtrace_cache__remove(struct auxtrace_cache *c, u32 key);
 void *auxtrace_cache__lookup(struct auxtrace_cache *c, u32 key);
 
 struct auxtrace_record *auxtrace_record__init(struct evlist *evlist,
author reviewer
#34 lore raw

RFC PATCH 6/6 perf intel-pt: Add support for text poke events

In reply to Adrian Hunter

Select text poke events when available and the kernel is being traced. Process text poke events to invalidate entries in Intel PT's instruction cache.

---
 tools/perf/arch/x86/util/intel-pt.c |  4 ++
 tools/perf/util/intel-pt.c          | 71 +++++++++++++++++++++++++++++
 2 files changed, 75 insertions(+)
tools/perf/arch/x86/util/intel-pt.c +4 -0
index a81ce60876f6..b7d04698349c 100644
@@ -754,6 +754,10 @@ static int intel_pt_recording_options(struct auxtrace_record *itr,
 		}
 	}
 
+	if (have_timing_info && !intel_pt_evsel->core.attr.exclude_kernel &&
+	    perf_can_record_text_poke_events() && perf_can_record_cpu_wide())
+		opts->text_poke = true;
+
 	if (intel_pt_evsel) {
 		/*
 		 * To obtain the auxtrace buffer file descriptor, the auxtrace
tools/perf/util/intel-pt.c +71 -0

89 lines not on this page · raw message

Signature · 1 line
2.17.1