Skip to contentSkip to conversation

ftrace: Add modify_ftrace_direct()

RFC PATCH In review Steven Rostedt 2 participants 8 messages Nov 14, 2019 – Nov 19, 2019

Subsystems (5)
lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

cover ftrace: Add modify_ftrace_direct()

As discussed with Alexei, here's a simple implementation of modify_ftrace_direct() without the need for any architecture changes.

Steven Rostedt (VMware) (2):
ftrace: Add modify_ftrace_direct()
ftrace/samples: Add a sample module that implements modify_ftrace_direct()

----

 include/linux/ftrace.h                |  6 +++
 kernel/trace/ftrace.c                 | 78 +++++++++++++++++++++++++++++++
 samples/ftrace/Makefile               |  1 +
 samples/ftrace/ftrace-direct-modify.c | 88 +++++++++++++++++++++++++++++++++++
 4 files changed, 173 insertions(+)
 create mode 100644 samples/ftrace/ftrace-direct-modify.c
author maintainer
#1 lore raw

RFC PATCH 1/2 ftrace: Add modify_ftrace_direct()

From: "Steven Rostedt (VMware)" <[email protected]>

Add a new function modify_ftrace_direct() that will allow a user to update an existing direct caller to a new trampoline, without missing hits due to unregistering one and then adding another.

---
 include/linux/ftrace.h |  6 ++++
 kernel/trace/ftrace.c  | 78 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 84 insertions(+)
include/linux/ftrace.h +6 -0
index 55647e185141..73eb2e93593f 100644
@@ -250,6 +250,7 @@ static inline void ftrace_free_mem(struct module *mod, void *start, void *end) {
 extern int ftrace_direct_func_count;
 int register_ftrace_direct(unsigned long ip, unsigned long addr);
 int unregister_ftrace_direct(unsigned long ip, unsigned long addr);
+int modify_ftrace_direct(unsigned long ip, unsigned long old_addr, unsigned long new_addr);
 struct ftrace_direct_func *ftrace_find_direct_func(unsigned long addr);
 #else
 # define ftrace_direct_func_count 0
@@ -261,6 +262,11 @@ static inline int unregister_ftrace_direct(unsigned long ip, unsigned long addr)
 {
 	return -ENODEV;
 }
+static inline int modify_ftrace_direct(unsigned long ip,
+				       unsigned long old_addr, unsigned long new_addr)
+{
+	return -ENODEV;
+}
 static inline struct ftrace_direct_func *ftrace_find_direct_func(unsigned long addr)
 {
 	return NULL;
kernel/trace/ftrace.c +78 -0

84 lines not on this page · raw message

Signature · 1 line
2.23.0
author maintainer
#7 lore raw

RFC PATCH 2/2 ftrace/samples: Add a sample module that implements modify_ftrace_direct()

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

Add a sample module that tests modify_ftrace_direct(), and this can be used by the selftests as well.

---
 samples/ftrace/Makefile               |  1 +
 samples/ftrace/ftrace-direct-modify.c | 88 +++++++++++++++++++++++++++
 2 files changed, 89 insertions(+)
 create mode 100644 samples/ftrace/ftrace-direct-modify.c
samples/ftrace/Makefile +1 -0
index d8217c4e072e..fb0c3ae18295 100644
@@ -2,3 +2,4 @@
 
 obj-$(CONFIG_SAMPLE_FTRACE_DIRECT) += ftrace-direct.o
 obj-$(CONFIG_SAMPLE_FTRACE_DIRECT) += ftrace-direct-too.o
+obj-$(CONFIG_SAMPLE_FTRACE_DIRECT) += ftrace-direct-modify.o
samples/ftrace/ftrace-direct-modify.c new file +88 -0

88 lines not on this page · raw message

Signature · 1 line
2.23.0
#2 lore raw
In reply to Steven Rostedt
Quoted 107 lines · Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

Add a new function modify_ftrace_direct() that will allow a user to update an existing direct caller to a new trampoline, without missing hits due to unregistering one and then adding another.

---
 include/linux/ftrace.h |  6 ++++
 kernel/trace/ftrace.c  | 78 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 84 insertions(+)
include/linux/ftrace.h +6 -0
index 55647e185141..73eb2e93593f 100644
@@ -250,6 +250,7 @@ static inline void ftrace_free_mem(struct module *mod, void *start, void *end) {
 extern int ftrace_direct_func_count;
 int register_ftrace_direct(unsigned long ip, unsigned long addr);
 int unregister_ftrace_direct(unsigned long ip, unsigned long addr);
+int modify_ftrace_direct(unsigned long ip, unsigned long old_addr, unsigned long new_addr);
 struct ftrace_direct_func *ftrace_find_direct_func(unsigned long addr);
 #else
 # define ftrace_direct_func_count 0
@@ -261,6 +262,11 @@ static inline int unregister_ftrace_direct(unsigned long ip, unsigned long addr)
 {
 	return -ENODEV;
 }
+static inline int modify_ftrace_direct(unsigned long ip,
+				       unsigned long old_addr, unsigned long new_addr)
+{
+	return -ENODEV;
+}
 static inline struct ftrace_direct_func *ftrace_find_direct_func(unsigned long addr)
 {
 	return NULL;
kernel/trace/ftrace.c +60 -0
index 82ef8d60a42b..834f3556ea1e 100644
@@ -5160,6 +5160,84 @@ int unregister_ftrace_direct(unsigned long ip, unsigned long addr)
 	return ret;
 }
 EXPORT_SYMBOL_GPL(unregister_ftrace_direct);
+
+static struct ftrace_ops stub_ops = {
+	.func		= ftrace_stub,
+};
+
+/**
+ * modify_ftrace_direct - Modify an existing direct call to call something else
+ * @ip: The instruction pointer to modify
+ * @old_addr: The address that the current @ip calls directly
+ * @new_addr: The address that the @ip should call
+ *
+ * This modifies a ftrace direct caller at an instruction pointer without
+ * having to disable it first. The direct call will switch over to the
+ * @new_addr without missing anything.
+ *
+ * Returns: zero on success. Non zero on error, which includes:
+ *  -ENODEV : the @ip given has no direct caller attached
+ *  -EINVAL : the @old_addr does not match the current direct caller
+ */
+int modify_ftrace_direct(unsigned long ip,
+			 unsigned long old_addr, unsigned long new_addr)
+{
+	struct ftrace_func_entry *entry;
+	struct dyn_ftrace *rec;
+	int ret = -ENODEV;
+
+	mutex_lock(&direct_mutex);
+	entry = __ftrace_lookup_ip(direct_functions, ip);
+	if (!entry) {
+		/* OK if it is off by a little */
+		rec = lookup_rec(ip, ip);
+		if (!rec || rec->ip == ip)
+			goto out_unlock;
+
+		entry = __ftrace_lookup_ip(direct_functions, rec->ip);
+		if (!entry)
+			goto out_unlock;
+
+		ip = rec->ip;
+		WARN_ON(!(rec->flags & FTRACE_FL_DIRECT));
+	}
+
+	ret = -EINVAL;
+	if (entry->direct != old_addr)
+		goto out_unlock;
+
+	/*
+	 * By setting a stub function at the same address, we force
+	 * the code to call the iterator and the direct_ops helper.
+	 * This means that @ip does not call the direct call, and
+	 * we can simply modify it.
+	 */
+	ret = ftrace_set_filter_ip(&stub_ops, ip, 0, 0);
+	if (ret)
+		goto out_unlock;
+
+	ret = register_ftrace_function(&stub_ops);
+	if (ret) {
+		ftrace_set_filter_ip(&stub_ops, ip, 1, 0);
+		goto out_unlock;

kernel/trace/ftrace.c int unregister_ftrace_direct(unsigned long ip, unsigned long addr) · line 5232

Quoted 10 lines
kernel/trace/ftrace.c +10 -0
+	}
+
+	entry->direct = new_addr;
+
+	/*
+	 * By removing the stub, we put back the direct call, calling
+	 * the @new_addr.
+	 */
+	unregister_ftrace_function(&stub_ops);
+	ftrace_set_filter_ip(&stub_ops, ip, 1, 0);

Thanks a lot for implementing it.
Switching to iterator just to modify the call.. hmm.
So "call direct_bpf_A" gets replaced to "call ftrace_stub" to do the iterator only to patch "call direct_bpf_B" later. I'm struggling to see why do that when arch can provide call to call rewrite easily. x86 and others have such ability already. I don't understand why you want to sacrifice simplicity here. Anyway with all 3 apis (register, modify, unreg) it looks complete. I'll start playing with it on Monday.

author maintainer
#3 lore raw
Quoted 8 lines · On Fri, 15 Nov 2019 13:51:26 -0800 Alexei Starovoitov…

Thanks a lot for implementing it.
Switching to iterator just to modify the call.. hmm.
So "call direct_bpf_A" gets replaced to "call ftrace_stub" to do the iterator only to patch "call direct_bpf_B" later. I'm struggling to see why do that when arch can provide call to call rewrite easily. x86 and others have such ability already. I don't understand why you want to sacrifice simplicity here. Anyway with all 3 apis (register, modify, unreg) it looks complete. I'll start playing with it on Monday.

Now if you take my latest for-next branch, and add the patch below, you can then implement a ftrace_modify_direct_caller() in x86 (or whatever architecture you'd like), that does what you want. I encapsulated the code such that the stub trick is for those that don't implement their own modification.

-- Steve

From 6220daa748fcf9e7f6a12cad94ea50026571bda1 Mon Sep 17 00:00:00 2001
From: "Steven Rostedt (VMware)" <[email protected]>
Date: Sun, 17 Nov 2019 17:04:15 -0500
Subject: [PATCH] ftrace: Add a helper function to modify_ftrace_direct() to
allow arch optimization

If a direct ftrace callback is at a location that does not have any other ftrace helpers attached to it, it is possible to simply just change the text to call the new caller (if the architecture supports it). But this requires special architecture code. Currently, modify_ftrace_direct() uses a trick to add a stub ftrace callback to the location forcing it to call the ftrace iterator. Then it can change the direct helper to call the new function in C, and then remove the stub. Removing the stub will have the location now call the new location that the direct helper is using.

The new helper function does the registering the stub trick, but is a weak function, allowing an architecture to override it to do something a bit more direct.

---
 include/linux/ftrace.h |  18 +++++++
 kernel/trace/ftrace.c  | 120 ++++++++++++++++++++++++++++++-----------
 2 files changed, 106 insertions(+), 32 deletions(-)
include/linux/ftrace.h +18 -0
index 73eb2e93593f..ff89fd07a745 100644
@@ -246,12 +246,23 @@ static inline void ftrace_free_init_mem(void) { }
 static inline void ftrace_free_mem(struct module *mod, void *start, void *end) { }
 #endif /* CONFIG_FUNCTION_TRACER */
 
+struct ftrace_func_entry {
+	struct hlist_node hlist;
+	unsigned long ip;
+	unsigned long direct; /* for direct lookup only */
+};
+
 #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
+struct dyn_ftrace;
 extern int ftrace_direct_func_count;
 int register_ftrace_direct(unsigned long ip, unsigned long addr);
 int unregister_ftrace_direct(unsigned long ip, unsigned long addr);
 int modify_ftrace_direct(unsigned long ip, unsigned long old_addr, unsigned long new_addr);
 struct ftrace_direct_func *ftrace_find_direct_func(unsigned long addr);
+int ftrace_modify_direct_caller(struct ftrace_func_entry *entry,
+				struct dyn_ftrace *rec,
+				unsigned long old_addr,
+				unsigned long new_addr);
 #else
 # define ftrace_direct_func_count 0
 static inline int register_ftrace_direct(unsigned long ip, unsigned long addr)
@@ -271,6 +282,13 @@ static inline struct ftrace_direct_func *ftrace_find_direct_func(unsigned long a
 {
 	return NULL;
 }
+static inline int ftrace_modify_direct_caller(struct ftrace_func_entry *entry,
+					      struct dyn_ftrace *rec,
+					      unsigned long old_addr,
+					      unsigned long new_addr)
+{
+	return -ENODEV;
+}
 #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */
 
 #ifndef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
kernel/trace/ftrace.c +88 -32

170 lines not on this page · raw message

Signature · 1 line
2.20.1
#4 lore raw
Quoted 13 lines · Steven Rostedt
Quoted 8 lines · On Fri, 15 Nov 2019 13:51:26 -0800 Alexei Starovoitov…

Thanks a lot for implementing it.
Switching to iterator just to modify the call.. hmm.
So "call direct_bpf_A" gets replaced to "call ftrace_stub" to do the iterator only to patch "call direct_bpf_B" later. I'm struggling to see why do that when arch can provide call to call rewrite easily. x86 and others have such ability already. I don't understand why you want to sacrifice simplicity here. Anyway with all 3 apis (register, modify, unreg) it looks complete. I'll start playing with it on Monday.

Quoted 1 line

Now if you take my latest for-next branch, and add the patch below,

I took your for-next without the extra patch and used it from bpf trampoline. It's looking good so far. Passed basic testing. Will add more stress tests.

Do you mind doing:

include/linux/ftrace.h +0 -0
index 73eb2e93593f..6ddb203ca550 100644
@@ -256,16 +256,16 @@ struct ftrace_direct_func

*ftrace_find_direct_func(unsigned long addr);

 # define ftrace_direct_func_count 0
 static inline int register_ftrace_direct(unsigned long ip, unsigned long addr)
 {
-       return -ENODEV;
+       return -ENOTSUPP;
 }
 static inline int unregister_ftrace_direct(unsigned long ip, unsigned
long addr)
 {
-       return -ENODEV;
+       return -ENOTSUPP;
 }
 static inline int modify_ftrace_direct(unsigned long ip,
                                       unsigned long old_addr,
unsigned long new_addr)
 {
-       return -ENODEV;
+       return -ENOTSUPP;
 }

otherwise ENODEV is a valid error when ip is incorrect which is indistinguishable from ftrace not compiled in.

author maintainer
#5 lore raw
Quoted 30 lines · On Mon, 18 Nov 2019 22:04:28 -0800 Alexei Starovoitov…

I took your for-next without the extra patch and used it from bpf trampoline. It's looking good so far. Passed basic testing. Will add more stress tests.

Do you mind doing:

include/linux/ftrace.h +3 -3
index 73eb2e93593f..6ddb203ca550 100644
@@ -256,16 +256,16 @@ struct ftrace_direct_func
 *ftrace_find_direct_func(unsigned long addr);
 # define ftrace_direct_func_count 0
 static inline int register_ftrace_direct(unsigned long ip, unsigned long addr)
 {
-       return -ENODEV;
+       return -ENOTSUPP;
 }
 static inline int unregister_ftrace_direct(unsigned long ip, unsigned
 long addr)
 {
-       return -ENODEV;
+       return -ENOTSUPP;
 }
 static inline int modify_ftrace_direct(unsigned long ip,
                                       unsigned long old_addr,
 unsigned long new_addr)
 {
-       return -ENODEV;
+       return -ENOTSUPP;

}

Quoted 2 lines

otherwise ENODEV is a valid error when ip is incorrect which is indistinguishable from ftrace not compiled in.

Sure I can add this. Want to add a Signed-off-by to it, and I'll just pull it in directly? I can write up the change log.

-- Steve

#6 lore raw
Quoted 34 lines · Steven Rostedt
Quoted 30 lines · On Mon, 18 Nov 2019 22:04:28 -0800 Alexei Starovoitov…

I took your for-next without the extra patch and used it from bpf trampoline. It's looking good so far. Passed basic testing. Will add more stress tests.

Do you mind doing:

include/linux/ftrace.h +3 -3
index 73eb2e93593f..6ddb203ca550 100644
@@ -256,16 +256,16 @@ struct ftrace_direct_func
 *ftrace_find_direct_func(unsigned long addr);
 # define ftrace_direct_func_count 0
 static inline int register_ftrace_direct(unsigned long ip, unsigned long addr)
 {
-       return -ENODEV;
+       return -ENOTSUPP;
 }
 static inline int unregister_ftrace_direct(unsigned long ip, unsigned
 long addr)
 {
-       return -ENODEV;
+       return -ENOTSUPP;
 }
 static inline int modify_ftrace_direct(unsigned long ip,
                                       unsigned long old_addr,
 unsigned long new_addr)
 {
-       return -ENODEV;
+       return -ENOTSUPP;

}

Quoted 5 lines
Quoted 2 lines

otherwise ENODEV is a valid error when ip is incorrect which is indistinguishable from ftrace not compiled in.

Sure I can add this. Want to add a Signed-off-by to it, and I'll just pull it in directly? I can write up the change log.

Whichever way is easier for you.

Thanks!

Quoted 1 line

-- Steve