Skip to contentSkip to conversation

Fix kallsyms with CONFIG_LTO_CLANG

PATCH Superseded Song Liu 8 participants 37 messages Aug 2, 2024 – Aug 9, 2024

Subsystems (4)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Song Liu

author maintainer
#0 lore raw

v2 cover Fix kallsyms with CONFIG_LTO_CLANG

With CONFIG_LTO_CLANG, the compiler/linker adds .llvm.<hash> suffix to local symbols to avoid duplications. Existing scripts/kallsyms sorts symbols without .llvm.<hash> suffix. However, this causes quite some issues later on. Some users of kallsyms, such as livepatch, have to match symbols exactly; while other users, such as kprobe, would match symbols without the suffix.

Address this by sorting full symbols at build time, and split kallsyms APIs to explicitly match full symbols or without suffix. Specifically, exiting APIs will match symbols exactly. Two new APIs are added to match symbols with suffix. Use the new APIs in tracing/kprobes.

Changes v1 => v2:
1. Update the APIs to remove all .XXX suffixes (v1 only removes .llvm.*).
2. Rename the APIs as *_without_suffix. (Masami Hiramatsu)
3. Fix another user from kprobe. (Masami Hiramatsu)
4. Add tests for the new APIs in kallsyms_selftests.

v1: https://lore.kernel.org/live-patching/[email protected]/T/#u

Song Liu (3):
kallsyms: Do not cleanup .llvm.<hash> suffix before sorting symbols
kallsyms: Add APIs to match symbol without .XXXX suffix. tracing/kprobes: Use APIs that matches symbols without .XXX suffix

 include/linux/kallsyms.h    | 14 ++++++
 kernel/kallsyms.c           | 88 +++++++++++++++++++++++++------------
 kernel/kallsyms_selftest.c  | 75 ++++++++++++++++++++++---------
 kernel/kprobes.c            |  6 ++-
 kernel/trace/trace_kprobe.c | 11 ++++-
 scripts/kallsyms.c          | 31 +------------
 scripts/link-vmlinux.sh     |  4 --
 7 files changed, 145 insertions(+), 84 deletions(-)

--
2.43.0

author maintainer
#1 lore raw

PATCH v2 1/3 kallsyms: Do not cleanup .llvm.<hash> suffix before sorting symbols

Cleaning up the symbols causes various issues afterwards. Let's sort the list based on original name and handle suffix later during the symbol lookup.

---
 scripts/kallsyms.c      | 31 ++-----------------------------
 scripts/link-vmlinux.sh |  4 ----
 2 files changed, 2 insertions(+), 33 deletions(-)
scripts/kallsyms.c +2 -29
index 0ed873491bf5..123dab0572f8 100644
@@ -5,8 +5,7 @@
  * This software may be used and distributed according to the terms
  * of the GNU General Public License, incorporated herein by reference.
  *
- * Usage: kallsyms [--all-symbols] [--absolute-percpu]
- *                         [--lto-clang] in.map > out.S
+ * Usage: kallsyms [--all-symbols] [--absolute-percpu]  in.map > out.S
  *
  *      Table compression uses all the unused char codes on the symbols and
  *  maps these to the most used substrings (tokens). For instance, it might
@@ -62,7 +61,6 @@ static struct sym_entry **table;
 static unsigned int table_size, table_cnt;
 static int all_symbols;
 static int absolute_percpu;
-static int lto_clang;
 
 static int token_profit[0x10000];
 
@@ -73,8 +71,7 @@ static unsigned char best_table_len[256];
 
 static void usage(void)
 {
-	fprintf(stderr, "Usage: kallsyms [--all-symbols] [--absolute-percpu] "
-			"[--lto-clang] in.map > out.S\n");
+	fprintf(stderr, "Usage: kallsyms [--all-symbols] [--absolute-percpu] in.map > out.S\n");
 	exit(1);
 }
 
@@ -344,25 +341,6 @@ static bool symbol_absolute(const struct sym_entry *s)
 	return s->percpu_absolute;
 }
 
-static void cleanup_symbol_name(char *s)
-{
-	char *p;
-
-	/*
-	 * ASCII[.]   = 2e
-	 * ASCII[0-9] = 30,39
-	 * ASCII[A-Z] = 41,5a
-	 * ASCII[_]   = 5f
-	 * ASCII[a-z] = 61,7a
-	 *
-	 * As above, replacing the first '.' in ".llvm." with '\0' does not
-	 * affect the main sorting, but it helps us with subsorting.
-	 */
-	p = strstr(s, ".llvm.");
-	if (p)
-		*p = '\0';
-}
-
 static int compare_names(const void *a, const void *b)
 {
 	int ret;
@@ -526,10 +504,6 @@ static void write_src(void)
 	output_address(relative_base);
 	printf("\n");
 
-	if (lto_clang)
-		for (i = 0; i < table_cnt; i++)
-			cleanup_symbol_name((char *)table[i]->sym);
-
 	sort_symbols_by_name();
 	output_label("kallsyms_seqs_of_names");
 	for (i = 0; i < table_cnt; i++)
@@ -807,7 +781,6 @@ int main(int argc, char **argv)
 		static const struct option long_options[] = {
 			{"all-symbols",     no_argument, &all_symbols,     1},
 			{"absolute-percpu", no_argument, &absolute_percpu, 1},
-			{"lto-clang",       no_argument, &lto_clang,       1},
 			{},
 		};
 
scripts/link-vmlinux.sh +0 -4
index f7b2503cdba9..22d0bc843986 100755
@@ -156,10 +156,6 @@ kallsyms()
 		kallsymopt="${kallsymopt} --absolute-percpu"
 	fi
 
-	if is_enabled CONFIG_LTO_CLANG; then
-		kallsymopt="${kallsymopt} --lto-clang"
-	fi
-
 	info KSYMS "${2}.S"
 	scripts/kallsyms ${kallsymopt} "${1}" > "${2}.S"
 
Signature · 1 line
2.43.0
author maintainer
#2 lore raw

PATCH v2 2/3 kallsyms: Add APIs to match symbol without .XXXX suffix.

In reply to Song Liu

With CONFIG_LTO_CLANG=y, the compiler may add suffix to function names to avoid duplication. This causes confusion with users of kallsyms. On one hand, users like livepatch are required to match the symbols exactly. On the other hand, users like kprobe would like to match to original function names.

Solve this by splitting kallsyms APIs. Specifically, existing APIs now should match the symbols exactly. Add two APIs that match only the part without .XXXX suffix. Specifically, the following two APIs are added.

1. kallsyms_lookup_name_without_suffix()
2. kallsyms_on_each_match_symbol_without_suffix()

These APIs will be used by kprobe.

Also cleanup some code and update kallsyms_selftests accordingly.

---
 include/linux/kallsyms.h   | 14 ++++++
 kernel/kallsyms.c          | 88 ++++++++++++++++++++++++++------------
 kernel/kallsyms_selftest.c | 75 +++++++++++++++++++++++---------
 3 files changed, 128 insertions(+), 49 deletions(-)
include/linux/kallsyms.h +14 -0
index c3f075e8f60c..9ef2a944a993 100644
@@ -74,9 +74,12 @@ int kallsyms_on_each_symbol(int (*fn)(void *, const char *, unsigned long),
 			    void *data);
 int kallsyms_on_each_match_symbol(int (*fn)(void *, unsigned long),
 				  const char *name, void *data);
+int kallsyms_on_each_match_symbol_without_suffix(int (*fn)(void *, unsigned long),
+						 const char *name, void *data);
 
 /* Lookup the address for a symbol. Returns 0 if not found. */
 unsigned long kallsyms_lookup_name(const char *name);
+unsigned long kallsyms_lookup_name_without_suffix(const char *name);
 
 extern int kallsyms_lookup_size_offset(unsigned long addr,
 				  unsigned long *symbolsize,
@@ -104,6 +107,11 @@ static inline unsigned long kallsyms_lookup_name(const char *name)
 	return 0;
 }
 
+static inline unsigned long kallsyms_lookup_name_without_suffix(const char *name)
+{
+	return 0;
+}
+
 static inline int kallsyms_lookup_size_offset(unsigned long addr,
 					      unsigned long *symbolsize,
 					      unsigned long *offset)
@@ -165,6 +173,12 @@ static inline int kallsyms_on_each_match_symbol(int (*fn)(void *, unsigned long)
 {
 	return -EOPNOTSUPP;
 }
+
+static inline int kallsyms_on_each_match_symbol_without_suffix(int (*fn)(void *, unsigned long),
+							       const char *name, void *data)
+{
+	return -EOPNOTSUPP;
+}
 #endif /*CONFIG_KALLSYMS*/
 
 static inline void print_ip_sym(const char *loglvl, unsigned long ip)
kernel/kallsyms.c +60 -28

162 lines not on this page · raw message

kernel/kallsyms_selftest.c +54 -21

103 lines not on this page · raw message

Signature · 1 line
2.43.0
maintainer Reviewed-by on v2 2/3
#3 lore raw

Re: [PATCH v2 2/3] kallsyms: Add APIs to match symbol without .XXXX suffix.

Quoted 10 lines · On Fri, 2 Aug 2024 14:08:34 -0700 Song Liu <[email protected]> wrote:

With CONFIG_LTO_CLANG=y, the compiler may add suffix to function names to avoid duplication. This causes confusion with users of kallsyms. On one hand, users like livepatch are required to match the symbols exactly. On the other hand, users like kprobe would like to match to original function names.

Solve this by splitting kallsyms APIs. Specifically, existing APIs now should match the symbols exactly. Add two APIs that match only the part without .XXXX suffix. Specifically, the following two APIs are added.

Quoted 8 lines

1. kallsyms_lookup_name_without_suffix()
2. kallsyms_on_each_match_symbol_without_suffix()

These APIs will be used by kprobe.

Also cleanup some code and update kallsyms_selftests accordingly.

Looks good to me, but I have a nitpick.

Quoted 10 lines
kernel/kallsyms.c +0 -3
@@ -164,30 +164,27 @@ static void cleanup_symbol_name(char *s)
 {
 	char *res;
 
-	if (!IS_ENABLED(CONFIG_LTO_CLANG))
-		return;
-
 	/*

kernel/kallsyms.c static void cleanup_symbol_name(char *s) · line 174

Quoted 10 lines
kernel/kallsyms.c +3 -3
 	 * LLVM appends various suffixes for local functions and variables that
 	 * must be promoted to global scope as part of LTO.  This can break
 	 * hooking of static functions with kprobes. '.' is not a valid
-	 * character in an identifier in C. Suffixes only in LLVM LTO observed:
-	 * - foo.llvm.[0-9a-f]+
+	 * character in an identifier in C, so we can just remove the
+	 * suffix.
 	 */
-	res = strstr(s, ".llvm.");
+	res = strstr(s, ".");

nit: "strchr(s, '.')" ?

Anyway,

Thank you,

Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
author maintainer
#4 lore raw

Re: [PATCH v2 2/3] kallsyms: Add APIs to match symbol without .XXXX suffix.

Hi Masami,

Quoted 49 lines · Masami Hiramatsu
Quoted 10 lines · On Fri, 2 Aug 2024 14:08:34 -0700 Song Liu <[email protected]> wrote:

With CONFIG_LTO_CLANG=y, the compiler may add suffix to function names to avoid duplication. This causes confusion with users of kallsyms. On one hand, users like livepatch are required to match the symbols exactly. On the other hand, users like kprobe would like to match to original function names.

Solve this by splitting kallsyms APIs. Specifically, existing APIs now should match the symbols exactly. Add two APIs that match only the part without .XXXX suffix. Specifically, the following two APIs are added.

Quoted 8 lines

1. kallsyms_lookup_name_without_suffix()
2. kallsyms_on_each_match_symbol_without_suffix()

These APIs will be used by kprobe.

Also cleanup some code and update kallsyms_selftests accordingly.

Looks good to me, but I have a nitpick.

Quoted 10 lines
kernel/kallsyms.c +0 -3
@@ -164,30 +164,27 @@ static void cleanup_symbol_name(char *s)
 {
      char *res;
 
-     if (!IS_ENABLED(CONFIG_LTO_CLANG))
-             return;
-
      /*

kernel/kallsyms.c static void cleanup_symbol_name(char *s) · line 174

Quoted 10 lines
kernel/kallsyms.c +3 -3
       * LLVM appends various suffixes for local functions and variables that
       * must be promoted to global scope as part of LTO.  This can break
       * hooking of static functions with kprobes. '.' is not a valid
-      * character in an identifier in C. Suffixes only in LLVM LTO observed:
-      * - foo.llvm.[0-9a-f]+
+      * character in an identifier in C, so we can just remove the
+      * suffix.
       */
-     res = strstr(s, ".llvm.");
+     res = strstr(s, ".");

nit: "strchr(s, '.')" ?

Quoted 3 lines

Anyway,

Thanks for your kind review!

If we have another version, I will fold this change (and the one in kallsyms_selftest.c) in.

Song

maintainer
#5 lore raw

Re: [PATCH v2 2/3] kallsyms: Add APIs to match symbol without .XXXX suffix.

In reply to Song Liu
Quoted 27 lines · Song Liu

With CONFIG_LTO_CLANG=y, the compiler may add suffix to function names to avoid duplication. This causes confusion with users of kallsyms. On one hand, users like livepatch are required to match the symbols exactly. On the other hand, users like kprobe would like to match to original function names.

Solve this by splitting kallsyms APIs. Specifically, existing APIs now should match the symbols exactly. Add two APIs that match only the part without .XXXX suffix. Specifically, the following two APIs are added.

1. kallsyms_lookup_name_without_suffix()
2. kallsyms_on_each_match_symbol_without_suffix()

These APIs will be used by kprobe.

Also cleanup some code and update kallsyms_selftests accordingly.

kernel/kallsyms.c +0 -3
@@ -164,30 +164,27 @@ static void cleanup_symbol_name(char *s)
 {
 	char *res;
 
-	if (!IS_ENABLED(CONFIG_LTO_CLANG))
-		return;
-
 	/*

kernel/kallsyms.c static void cleanup_symbol_name(char *s) · line 174

Quoted 10 lines
kernel/kallsyms.c +3 -3
 	 * LLVM appends various suffixes for local functions and variables that
 	 * must be promoted to global scope as part of LTO.  This can break
 	 * hooking of static functions with kprobes. '.' is not a valid
-	 * character in an identifier in C. Suffixes only in LLVM LTO observed:
-	 * - foo.llvm.[0-9a-f]+
+	 * character in an identifier in C, so we can just remove the
+	 * suffix.
 	 */
-	res = strstr(s, ".llvm.");
+	res = strstr(s, ".");

IMHO, we should not remove the suffixes like .constprop*, .part*, .isra*. They implement a special optimized variant of the function. It is not longer the original full-featured one.

Quoted 27 lines
kernel/kallsyms.c +6 -7
 	if (res)
 		*res = '\0';
 
 	return;
 }
 
-static int compare_symbol_name(const char *name, char *namebuf)
+static int compare_symbol_name(const char *name, char *namebuf, bool exact_match)
 {
-	/* The kallsyms_seqs_of_names is sorted based on names after
-	 * cleanup_symbol_name() (see scripts/kallsyms.c) if clang lto is enabled.
-	 * To ensure correct bisection in kallsyms_lookup_names(), do
-	 * cleanup_symbol_name(namebuf) before comparing name and namebuf.
-	 */
+	int ret = strcmp(name, namebuf);
+
+	if (exact_match || !ret)
+		return ret;
+
 	cleanup_symbol_name(namebuf);
 	return strcmp(name, namebuf);
 }
@@ -204,13 +201,17 @@ static unsigned int get_symbol_seq(int index)
 
 static int kallsyms_lookup_names(const char *name,
 				 unsigned int *start,
-				 unsigned int *end)

kernel/kallsyms.c static unsigned int get_symbol_seq(int index) · line 213

Quoted 10 lines
kernel/kallsyms.c +4 -0
+				 unsigned int *end,
+				 bool exact_match)
 {
 	int ret;
 	int low, mid, high;
 	unsigned int seq, off;
 	char namebuf[KSYM_NAME_LEN];
 
+	if (!IS_ENABLED(CONFIG_LTO_CLANG))
+		exact_match = true;

IMHO, this is very a bad design. It causes that

     kallsyms_on_each_match_symbol_without_suffix(,,, false);

does not longer work as expected. It creates a hard to maintain code. The code does not do what it looks like.

The caller should decide whether it wants to ignore the suffix or no. And this function should always respect the @exact_match parameter value.

kernel/kallsyms.c static unsigned int get_symbol_seq(int index) · line 217

Quoted 4 lines
kernel/kallsyms.c +1 -0
+
 	low = 0;
 	high = kallsyms_num_syms - 1;
 

Best Regards,
Petr

#6 lore raw

Re: [PATCH v2 2/3] kallsyms: Add APIs to match symbol without .XXXX suffix.

Quoted 87 lines · Petr Mladek
Quoted 27 lines · Song Liu

With CONFIG_LTO_CLANG=y, the compiler may add suffix to function names to avoid duplication. This causes confusion with users of kallsyms. On one hand, users like livepatch are required to match the symbols exactly. On the other hand, users like kprobe would like to match to original function names.

Solve this by splitting kallsyms APIs. Specifically, existing APIs now should match the symbols exactly. Add two APIs that match only the part without .XXXX suffix. Specifically, the following two APIs are added.

1. kallsyms_lookup_name_without_suffix()
2. kallsyms_on_each_match_symbol_without_suffix()

These APIs will be used by kprobe.

Also cleanup some code and update kallsyms_selftests accordingly.

kernel/kallsyms.c +0 -3
@@ -164,30 +164,27 @@ static void cleanup_symbol_name(char *s)
 {
 char *res;
 
- if (!IS_ENABLED(CONFIG_LTO_CLANG))
- return;
-
 /*

kernel/kallsyms.c static void cleanup_symbol_name(char *s) · line 174

Quoted 10 lines
kernel/kallsyms.c +3 -3
 * LLVM appends various suffixes for local functions and variables that
 * must be promoted to global scope as part of LTO.  This can break
 * hooking of static functions with kprobes. '.' is not a valid
- * character in an identifier in C. Suffixes only in LLVM LTO observed:
- * - foo.llvm.[0-9a-f]+
+ * character in an identifier in C, so we can just remove the
+ * suffix.
 */
- res = strstr(s, ".llvm.");
+ res = strstr(s, ".");

IMHO, we should not remove the suffixes like .constprop*, .part*, .isra*. They implement a special optimized variant of the function. It is not longer the original full-featured one.

Quoted 27 lines
kernel/kallsyms.c +6 -7
 if (res)
 *res = '\0';
 
 return;
 }
 
-static int compare_symbol_name(const char *name, char *namebuf)
+static int compare_symbol_name(const char *name, char *namebuf, bool exact_match)
 {
- /* The kallsyms_seqs_of_names is sorted based on names after
- * cleanup_symbol_name() (see scripts/kallsyms.c) if clang lto is enabled.
- * To ensure correct bisection in kallsyms_lookup_names(), do
- * cleanup_symbol_name(namebuf) before comparing name and namebuf.
- */
+ int ret = strcmp(name, namebuf);
+
+ if (exact_match || !ret)
+ return ret;
+
 cleanup_symbol_name(namebuf);
 return strcmp(name, namebuf);
 }
@@ -204,13 +201,17 @@ static unsigned int get_symbol_seq(int index)
 
 static int kallsyms_lookup_names(const char *name,
 unsigned int *start,
- unsigned int *end)

kernel/kallsyms.c static unsigned int get_symbol_seq(int index) · line 213

Quoted 10 lines
kernel/kallsyms.c +4 -0
+ unsigned int *end,
+ bool exact_match)
 {
 int ret;
 int low, mid, high;
 unsigned int seq, off;
 char namebuf[KSYM_NAME_LEN];
 
+ if (!IS_ENABLED(CONFIG_LTO_CLANG))
+ exact_match = true;

IMHO, this is very a bad design. It causes that

Quoted 4 lines
    kallsyms_on_each_match_symbol_without_suffix(,,, false);

does not longer work as expected. It creates a hard to maintain code. The code does not do what it looks like.

Indeed. It actually caused issue with GCC-built kernel in my tests after submitting v2.

Thanks,
Song

author maintainer
#7 lore raw

PATCH v2 3/3 tracing/kprobes: Use APIs that matches symbols without .XXX suffix

In reply to Song Liu

Use the new kallsyms APIs that matches symbols name with .XXX suffix. This allows userspace tools to get kprobes on the expected function name, while the actual symbol has a .llvm.<hash> suffix.

This only effects kernel compile with CONFIG_LTO_CLANG.

---
 kernel/kprobes.c            |  6 +++++-
 kernel/trace/trace_kprobe.c | 11 ++++++++++-
 2 files changed, 15 insertions(+), 2 deletions(-)
kernel/kprobes.c +5 -1
index e85de37d9e1e..99102283b076 100644
@@ -70,7 +70,11 @@ static DEFINE_PER_CPU(struct kprobe *, kprobe_instance);
 kprobe_opcode_t * __weak kprobe_lookup_name(const char *name,
 					unsigned int __unused)
 {
-	return ((kprobe_opcode_t *)(kallsyms_lookup_name(name)));
+	unsigned long addr = kallsyms_lookup_name(name);
+
+	if (IS_ENABLED(CONFIG_LTO_CLANG) && !addr)
+		addr = kallsyms_lookup_name_without_suffix(name);
+	return ((kprobe_opcode_t *)(addr));
 }
 
 /*
kernel/trace/trace_kprobe.c +10 -1
index 61a6da808203..d2ad0c561c83 100644
@@ -203,6 +203,10 @@ unsigned long trace_kprobe_address(struct trace_kprobe *tk)
 	if (tk->symbol) {
 		addr = (unsigned long)
 			kallsyms_lookup_name(trace_kprobe_symbol(tk));
+
+		if (IS_ENABLED(CONFIG_LTO_CLANG) && !addr)
+			addr = kallsyms_lookup_name_without_suffix(trace_kprobe_symbol(tk));
+
 		if (addr)
 			addr += tk->rp.kp.offset;
 	} else {
@@ -766,8 +770,13 @@ static unsigned int number_of_same_symbols(const char *mod, const char *func_nam
 {
 	struct sym_count_ctx ctx = { .count = 0, .name = func_name };
 
-	if (!mod)
+	if (!mod) {
 		kallsyms_on_each_match_symbol(count_symbols, func_name, &ctx.count);
+		if (IS_ENABLED(CONFIG_LTO_CLANG) && !ctx.count) {
+			kallsyms_on_each_match_symbol_without_suffix(
+				count_symbols, func_name, &ctx.count);
+		}
+	}
 
 	module_kallsyms_on_each_symbol(mod, count_mod_symbols, &ctx);
 
Signature · 1 line
2.43.0
maintainer
#8 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

Quoted 34 lines · On Fri, 2 Aug 2024 14:08:35 -0700 Song Liu <[email protected]> wrote:

Use the new kallsyms APIs that matches symbols name with .XXX suffix. This allows userspace tools to get kprobes on the expected function name, while the actual symbol has a .llvm.<hash> suffix.

This only effects kernel compile with CONFIG_LTO_CLANG.

---
 kernel/kprobes.c            |  6 +++++-
 kernel/trace/trace_kprobe.c | 11 ++++++++++-
 2 files changed, 15 insertions(+), 2 deletions(-)
kernel/kprobes.c +5 -1
index e85de37d9e1e..99102283b076 100644
@@ -70,7 +70,11 @@ static DEFINE_PER_CPU(struct kprobe *, kprobe_instance);
 kprobe_opcode_t * __weak kprobe_lookup_name(const char *name,
 					unsigned int __unused)
 {
-	return ((kprobe_opcode_t *)(kallsyms_lookup_name(name)));
+	unsigned long addr = kallsyms_lookup_name(name);
+
+	if (IS_ENABLED(CONFIG_LTO_CLANG) && !addr)
+		addr = kallsyms_lookup_name_without_suffix(name);
+	return ((kprobe_opcode_t *)(addr));
 }
 
 /*
kernel/trace/trace_kprobe.c +0 -0
index 61a6da808203..d2ad0c561c83 100644

kernel/trace/trace_kprobe.c unsigned long trace_kprobe_address(struct trace_kprobe *tk) · line 209

Quoted 7 lines
kernel/trace/trace_kprobe.c +4 -0
@@ -203,6 +203,10 @@ unsigned long trace_kprobe_address(struct trace_kprobe *tk)
 	if (tk->symbol) {
 		addr = (unsigned long)
 			kallsyms_lookup_name(trace_kprobe_symbol(tk));
+
+		if (IS_ENABLED(CONFIG_LTO_CLANG) && !addr)
+			addr = kallsyms_lookup_name_without_suffix(trace_kprobe_symbol(tk));
+

So you do the lookup twice if this is enabled?

Why not just use "kallsyms_lookup_name_without_suffix()" the entire time, and it should work just the same as "kallsyms_lookup_name()" if it's not needed?

kernel/trace/trace_kprobe.c static unsigned int number_of_same_symbols(const char *mod, const char *func_nam · line 777

Quoted 13 lines
kernel/trace/trace_kprobe.c +4 -1
 		if (addr)
 			addr += tk->rp.kp.offset;
 	} else {
@@ -766,8 +770,13 @@ static unsigned int number_of_same_symbols(const char *mod, const char *func_nam
 {
 	struct sym_count_ctx ctx = { .count = 0, .name = func_name };
 
-	if (!mod)
+	if (!mod) {
 		kallsyms_on_each_match_symbol(count_symbols, func_name, &ctx.count);
+		if (IS_ENABLED(CONFIG_LTO_CLANG) && !ctx.count) {
+			kallsyms_on_each_match_symbol_without_suffix(
+				count_symbols, func_name, &ctx.count);

Same here.

-- Steve

Quoted 5 lines
kernel/trace/trace_kprobe.c +2 -0
+		}
+	}
 
 	module_kallsyms_on_each_symbol(mod, count_mod_symbols, &ctx);
 
#9 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

Hi Steven,

Quoted 49 lines · Steven Rostedt
Quoted 34 lines · On Fri, 2 Aug 2024 14:08:35 -0700 Song Liu <[email protected]> wrote:

Use the new kallsyms APIs that matches symbols name with .XXX suffix. This allows userspace tools to get kprobes on the expected function name, while the actual symbol has a .llvm.<hash> suffix.

This only effects kernel compile with CONFIG_LTO_CLANG.

---
kernel/kprobes.c | 6 +++++-
kernel/trace/trace_kprobe.c | 11 ++++++++++-

2 files changed, 15 insertions(+), 2 deletions(-)
kernel/kprobes.c +5 -1
index e85de37d9e1e..99102283b076 100644
@@ -70,7 +70,11 @@ static DEFINE_PER_CPU(struct kprobe *, kprobe_instance);
 kprobe_opcode_t * __weak kprobe_lookup_name(const char *name,
 unsigned int __unused)
 {
- return ((kprobe_opcode_t *)(kallsyms_lookup_name(name)));
+ unsigned long addr = kallsyms_lookup_name(name);
+
+ if (IS_ENABLED(CONFIG_LTO_CLANG) && !addr)
+ addr = kallsyms_lookup_name_without_suffix(name);
+ return ((kprobe_opcode_t *)(addr));
 }
 
 /*
kernel/trace/trace_kprobe.c +0 -0
index 61a6da808203..d2ad0c561c83 100644

kernel/trace/trace_kprobe.c unsigned long trace_kprobe_address(struct trace_kprobe *tk) · line 209

Quoted 7 lines
kernel/trace/trace_kprobe.c +4 -0
@@ -203,6 +203,10 @@ unsigned long trace_kprobe_address(struct trace_kprobe *tk)
 if (tk->symbol) {
 addr = (unsigned long)
 kallsyms_lookup_name(trace_kprobe_symbol(tk));
+
+ if (IS_ENABLED(CONFIG_LTO_CLANG) && !addr)
+ addr = kallsyms_lookup_name_without_suffix(trace_kprobe_symbol(tk));
+

So you do the lookup twice if this is enabled?

Quoted 3 lines

Why not just use "kallsyms_lookup_name_without_suffix()" the entire time, and it should work just the same as "kallsyms_lookup_name()" if it's not needed?

We still want to give priority to full match. For example, we have:

[root@~]# grep c_next /proc/kallsyms
ffffffff81419dc0 t c_next.llvm.7567888411731313343
ffffffff81680600 t c_next
ffffffff81854380 t c_next.llvm.14337844803752139461

If the goal is to explicitly trace c_next.llvm.7567888411731313343, the user can provide the full name. If we always match _without_suffix, all of the 3 will match to the first one.

Does this make sense?

Thanks,
Song

maintainer
#10 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

Quoted 17 lines · On Tue, 6 Aug 2024 19:35:07 +0000 Song Liu <[email protected]> wrote:
Quoted 9 lines
Quoted 3 lines

+ if (IS_ENABLED(CONFIG_LTO_CLANG) && !addr)
+ addr = kallsyms_lookup_name_without_suffix(trace_kprobe_symbol(tk)); +

So you do the lookup twice if this is enabled?

Why not just use "kallsyms_lookup_name_without_suffix()" the entire time, and it should work just the same as "kallsyms_lookup_name()" if it's not needed?

We still want to give priority to full match. For example, we have:

[root@~]# grep c_next /proc/kallsyms
ffffffff81419dc0 t c_next.llvm.7567888411731313343
ffffffff81680600 t c_next
ffffffff81854380 t c_next.llvm.14337844803752139461
Quoted 5 lines

If the goal is to explicitly trace c_next.llvm.7567888411731313343, the user can provide the full name. If we always match _without_suffix, all of the 3 will match to the first one.

Does this make sense?

Yes. Sorry, I missed the "&& !addr)" after the "IS_ENABLED()", which looked like you did the command twice.

I blame jetlag ;-)

-- Steve

maintainer
#11 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

Quoted 17 lines · On Tue, 6 Aug 2024 16:00:49 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 17 lines
Quoted 9 lines
Quoted 3 lines

+ if (IS_ENABLED(CONFIG_LTO_CLANG) && !addr)
+ addr = kallsyms_lookup_name_without_suffix(trace_kprobe_symbol(tk)); +

So you do the lookup twice if this is enabled?

Why not just use "kallsyms_lookup_name_without_suffix()" the entire time, and it should work just the same as "kallsyms_lookup_name()" if it's not needed?

We still want to give priority to full match. For example, we have:

[root@~]# grep c_next /proc/kallsyms
ffffffff81419dc0 t c_next.llvm.7567888411731313343
ffffffff81680600 t c_next
ffffffff81854380 t c_next.llvm.14337844803752139461
Quoted 8 lines
Quoted 5 lines

If the goal is to explicitly trace c_next.llvm.7567888411731313343, the user can provide the full name. If we always match _without_suffix, all of the 3 will match to the first one.

Does this make sense?

Yes. Sorry, I missed the "&& !addr)" after the "IS_ENABLED()", which looked like you did the command twice.

But that said, does this only have to be for llvm? Or should we do this for even gcc? As I believe gcc can give strange symbols too.

-- Steve

#12 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

Quoted 28 lines · Steven Rostedt
Quoted 21 lines · On Tue, 6 Aug 2024 16:00:49 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 21 lines
Quoted 9 lines
Quoted 3 lines

+ if (IS_ENABLED(CONFIG_LTO_CLANG) && !addr)
+ addr = kallsyms_lookup_name_without_suffix(trace_kprobe_symbol(tk)); +

So you do the lookup twice if this is enabled?

Why not just use "kallsyms_lookup_name_without_suffix()" the entire time, and it should work just the same as "kallsyms_lookup_name()" if it's not needed?

We still want to give priority to full match. For example, we have:

[root@~]# grep c_next /proc/kallsyms
ffffffff81419dc0 t c_next.llvm.7567888411731313343
ffffffff81680600 t c_next
ffffffff81854380 t c_next.llvm.14337844803752139461

If the goal is to explicitly trace c_next.llvm.7567888411731313343, the user can provide the full name. If we always match _without_suffix, all of the 3 will match to the first one.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Does this make sense?

Yes. Sorry, I missed the "&& !addr)" after the "IS_ENABLED()", which looked like you did the command twice.

But that said, does this only have to be for llvm? Or should we do this for even gcc? As I believe gcc can give strange symbols too.

I think most of the issue comes with LTO, as LTO promotes local static functions to global functions. IIUC, we don't have GCC built, LTO enabled kernel yet.

In my GCC built, we have suffixes like ".constprop.0", ".part.0", ".isra.0", and ".isra.0.cold". We didn't do anything about these before this set. So I think we are OK not handling them now. We sure can enable it for GCC built kernel in the future.

Thanks,
Song

#13 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

Quoted 36 lines · On Tue, 6 Aug 2024 20:12:55 +0000 Song Liu <[email protected]> wrote:
Quoted 28 lines · Steven Rostedt
Quoted 21 lines · On Tue, 6 Aug 2024 16:00:49 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 21 lines
Quoted 9 lines
Quoted 3 lines

+ if (IS_ENABLED(CONFIG_LTO_CLANG) && !addr)
+ addr = kallsyms_lookup_name_without_suffix(trace_kprobe_symbol(tk)); +

So you do the lookup twice if this is enabled?

Why not just use "kallsyms_lookup_name_without_suffix()" the entire time, and it should work just the same as "kallsyms_lookup_name()" if it's not needed?

We still want to give priority to full match. For example, we have:

[root@~]# grep c_next /proc/kallsyms
ffffffff81419dc0 t c_next.llvm.7567888411731313343
ffffffff81680600 t c_next
ffffffff81854380 t c_next.llvm.14337844803752139461

If the goal is to explicitly trace c_next.llvm.7567888411731313343, the user can provide the full name. If we always match _without_suffix, all of the 3 will match to the first one.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Does this make sense?

Yes. Sorry, I missed the "&& !addr)" after the "IS_ENABLED()", which looked like you did the command twice.

But that said, does this only have to be for llvm? Or should we do this for even gcc? As I believe gcc can give strange symbols too.

Quoted 8 lines

I think most of the issue comes with LTO, as LTO promotes local static functions to global functions. IIUC, we don't have GCC built, LTO enabled kernel yet.

In my GCC built, we have suffixes like ".constprop.0", ".part.0", ".isra.0", and ".isra.0.cold". We didn't do anything about these before this set. So I think we are OK not handling them now. We sure can enable it for GCC built kernel in the future.

Hmm, I think it should be handled as it is. This means it should do as livepatch does. Since I expected user will check kallsyms if gets error, we should keep this as it is. (if a symbol has suffix, it should accept symbol with suffix, or user will get confused because they can not find which symbol is kprobed.)

Sorry about the conclusion (so I NAK this), but this is a good discussion.

Thanks,

Quoted 6 lines

Thanks,
Song

Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
#14 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

Quoted 52 lines · Masami Hiramatsu
Quoted 36 lines · On Tue, 6 Aug 2024 20:12:55 +0000 Song Liu <[email protected]> wrote:
Quoted 28 lines · Steven Rostedt
Quoted 21 lines · On Tue, 6 Aug 2024 16:00:49 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 21 lines
Quoted 9 lines
Quoted 3 lines

+ if (IS_ENABLED(CONFIG_LTO_CLANG) && !addr)
+ addr = kallsyms_lookup_name_without_suffix(trace_kprobe_symbol(tk)); +

So you do the lookup twice if this is enabled?

Why not just use "kallsyms_lookup_name_without_suffix()" the entire time, and it should work just the same as "kallsyms_lookup_name()" if it's not needed?

We still want to give priority to full match. For example, we have:

[root@~]# grep c_next /proc/kallsyms
ffffffff81419dc0 t c_next.llvm.7567888411731313343
ffffffff81680600 t c_next
ffffffff81854380 t c_next.llvm.14337844803752139461

If the goal is to explicitly trace c_next.llvm.7567888411731313343, the user can provide the full name. If we always match _without_suffix, all of the 3 will match to the first one.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Does this make sense?

Yes. Sorry, I missed the "&& !addr)" after the "IS_ENABLED()", which looked like you did the command twice.

But that said, does this only have to be for llvm? Or should we do this for even gcc? As I believe gcc can give strange symbols too.

Quoted 8 lines

I think most of the issue comes with LTO, as LTO promotes local static functions to global functions. IIUC, we don't have GCC built, LTO enabled kernel yet.

In my GCC built, we have suffixes like ".constprop.0", ".part.0", ".isra.0", and ".isra.0.cold". We didn't do anything about these before this set. So I think we are OK not handling them now. We sure can enable it for GCC built kernel in the future.

Quoted 7 lines

Hmm, I think it should be handled as it is. This means it should do as livepatch does. Since I expected user will check kallsyms if gets error, we should keep this as it is. (if a symbol has suffix, it should accept symbol with suffix, or user will get confused because they can not find which symbol is kprobed.)

Sorry about the conclusion (so I NAK this), but this is a good discussion.

Do you mean we do not want patch 3/3, but would like to keep 1/3 and part of 2/3 (remove the _without_suffix APIs)? If this is the case, we are undoing the change by Sami in [1], and thus may break some tracing tools.

Sami, could you please share your thoughts on this?

If this works, I will send next version with 1/3 and part of 2/3.

Thanks,
Song

[1] https://lore.kernel.org/all/[email protected]/

#15 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

Quoted 58 lines · On Wed, 7 Aug 2024 00:19:20 +0000 Song Liu <[email protected]> wrote:
Quoted 58 lines · Masami Hiramatsu
Quoted 36 lines · On Tue, 6 Aug 2024 20:12:55 +0000 Song Liu <[email protected]> wrote:
Quoted 28 lines · Steven Rostedt
Quoted 21 lines · On Tue, 6 Aug 2024 16:00:49 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 21 lines
Quoted 9 lines
Quoted 3 lines

+ if (IS_ENABLED(CONFIG_LTO_CLANG) && !addr)
+ addr = kallsyms_lookup_name_without_suffix(trace_kprobe_symbol(tk)); +

So you do the lookup twice if this is enabled?

Why not just use "kallsyms_lookup_name_without_suffix()" the entire time, and it should work just the same as "kallsyms_lookup_name()" if it's not needed?

We still want to give priority to full match. For example, we have:

[root@~]# grep c_next /proc/kallsyms
ffffffff81419dc0 t c_next.llvm.7567888411731313343
ffffffff81680600 t c_next
ffffffff81854380 t c_next.llvm.14337844803752139461

If the goal is to explicitly trace c_next.llvm.7567888411731313343, the user can provide the full name. If we always match _without_suffix, all of the 3 will match to the first one.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Does this make sense?

Yes. Sorry, I missed the "&& !addr)" after the "IS_ENABLED()", which looked like you did the command twice.

But that said, does this only have to be for llvm? Or should we do this for even gcc? As I believe gcc can give strange symbols too.

Quoted 8 lines

I think most of the issue comes with LTO, as LTO promotes local static functions to global functions. IIUC, we don't have GCC built, LTO enabled kernel yet.

In my GCC built, we have suffixes like ".constprop.0", ".part.0", ".isra.0", and ".isra.0.cold". We didn't do anything about these before this set. So I think we are OK not handling them now. We sure can enable it for GCC built kernel in the future.

Hmm, I think it should be handled as it is. This means it should do as livepatch does. Since I expected user will check kallsyms if gets error, we should keep this as it is. (if a symbol has suffix, it should accept symbol with suffix, or user will get confused because they can not find which symbol is kprobed.)

Quoted 5 lines
Quoted 1 line

Sorry about the conclusion (so I NAK this), but this is a good discussion.

Do you mean we do not want patch 3/3, but would like to keep 1/3 and part of 2/3 (remove the _without_suffix APIs)? If this is the case, we are undoing the change by Sami in [1], and thus may break some tracing tools.

What tracing tools may be broke and why?

For this suffix problem, I would like to add another patch to allow probing on suffixed symbols. (It seems suffixed symbols are not available at this point)

The problem is that the suffixed symbols maybe a "part" of the original function, thus user has to carefully use it.

Quoted 2 lines

Sami, could you please share your thoughts on this?

Sami, I would like to know what problem you have on kprobes.

Thank you,

Quoted 8 lines

If this works, I will send next version with 1/3 and part of 2/3.

Thanks,
Song

[1] https://lore.kernel.org/all/[email protected]/

Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
#16 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

Hi,

Quoted 9 lines · Masami Hiramatsu
Quoted 3 lines · On Wed, 7 Aug 2024 00:19:20 +0000 Song Liu <[email protected]> wrote:

Do you mean we do not want patch 3/3, but would like to keep 1/3 and part of 2/3 (remove the _without_suffix APIs)? If this is the case, we are undoing the change by Sami in [1], and thus may break some tracing tools.

What tracing tools may be broke and why?

This was a few years ago when we were first adding LTO support, but the unexpected suffixes in tracing output broke systrace in Android, presumably because the tools expected to find specific function names without suffixes. I'm not sure if systrace would still be a problem today, but other tools might still make assumptions about the function name format. At the time, we decided to filter out the suffixes in all user space visible output to avoid these issues.

Quoted 10 lines

For this suffix problem, I would like to add another patch to allow probing on suffixed symbols. (It seems suffixed symbols are not available at this point)

The problem is that the suffixed symbols maybe a "part" of the original function, thus user has to carefully use it.

Quoted 2 lines

Sami, could you please share your thoughts on this?

Sami, I would like to know what problem you have on kprobes.

The reports we received back then were about registering kprobes for static functions, which obviously failed if the compiler added a suffix to the function name. This was more of a problem with ThinLTO and Clang CFI at the time because the compiler used to rename _all_ static functions, but one can obviously run into the same issue with just LTO.

Sami

#17 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

Quoted 34 lines · Sami Tolvanen

Hi,

Quoted 9 lines · Masami Hiramatsu
Quoted 3 lines · On Wed, 7 Aug 2024 00:19:20 +0000 Song Liu <[email protected]> wrote:

Do you mean we do not want patch 3/3, but would like to keep 1/3 and part of 2/3 (remove the _without_suffix APIs)? If this is the case, we are undoing the change by Sami in [1], and thus may break some tracing tools.

What tracing tools may be broke and why?

This was a few years ago when we were first adding LTO support, but the unexpected suffixes in tracing output broke systrace in Android, presumably because the tools expected to find specific function names without suffixes. I'm not sure if systrace would still be a problem today, but other tools might still make assumptions about the function name format. At the time, we decided to filter out the suffixes in all user space visible output to avoid these issues.

Quoted 9 lines

For this suffix problem, I would like to add another patch to allow probing on suffixed symbols. (It seems suffixed symbols are not available at this point)

The problem is that the suffixed symbols maybe a "part" of the original function, thus user has to carefully use it.

Quoted 2 lines

Sami, could you please share your thoughts on this?

Quoted 8 lines
Quoted 1 line

Sami, I would like to know what problem you have on kprobes.

The reports we received back then were about registering kprobes for static functions, which obviously failed if the compiler added a suffix to the function name. This was more of a problem with ThinLTO and Clang CFI at the time because the compiler used to rename _all_ static functions, but one can obviously run into the same issue with just LTO.

I think newer LLVM/clang no longer add suffixes to all static functions with LTO and CFI. So this may not be a real issue any more?

If we still need to allow tracing without suffix, I think the approach in this patch set is correct (sort syms based on full name, remove suffixes in special APIs during lookup).

Thanks,
Song

maintainer
#18 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

Quoted 43 lines · Song Liu
Quoted 34 lines · Sami Tolvanen

Hi,

Quoted 9 lines · Masami Hiramatsu
Quoted 3 lines · On Wed, 7 Aug 2024 00:19:20 +0000 Song Liu <[email protected]> wrote:

Do you mean we do not want patch 3/3, but would like to keep 1/3 and part of 2/3 (remove the _without_suffix APIs)? If this is the case, we are undoing the change by Sami in [1], and thus may break some tracing tools.

What tracing tools may be broke and why?

This was a few years ago when we were first adding LTO support, but the unexpected suffixes in tracing output broke systrace in Android, presumably because the tools expected to find specific function names without suffixes. I'm not sure if systrace would still be a problem today, but other tools might still make assumptions about the function name format. At the time, we decided to filter out the suffixes in all user space visible output to avoid these issues.

Quoted 9 lines

For this suffix problem, I would like to add another patch to allow probing on suffixed symbols. (It seems suffixed symbols are not available at this point)

The problem is that the suffixed symbols maybe a "part" of the original function, thus user has to carefully use it.

Quoted 2 lines

Sami, could you please share your thoughts on this?

Quoted 8 lines
Quoted 1 line

Sami, I would like to know what problem you have on kprobes.

The reports we received back then were about registering kprobes for static functions, which obviously failed if the compiler added a suffix to the function name. This was more of a problem with ThinLTO and Clang CFI at the time because the compiler used to rename _all_ static functions, but one can obviously run into the same issue with just LTO.

Quoted 5 lines

I think newer LLVM/clang no longer add suffixes to all static functions with LTO and CFI. So this may not be a real issue any more?

If we still need to allow tracing without suffix, I think the approach in this patch set is correct (sort syms based on full name,

Yes, we should allow to find the symbols via the full name, definitely.

Quoted 1 line

remove suffixes in special APIs during lookup).

Just an idea. Alternative solution would be to make make an alias without the suffix when there is only one symbol with the same name.

It would be complementary with the patch adding aliases for symbols with the same name, see
https://lore.kernel.org/r/[email protected]

I would allow to find the symbols with and without the suffix using a single API.

Best Regards,
Petr

#19 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

Quoted 58 lines · Petr Mladek
Quoted 43 lines · Song Liu
Quoted 34 lines · Sami Tolvanen

Hi,

Quoted 9 lines · Masami Hiramatsu
Quoted 3 lines · On Wed, 7 Aug 2024 00:19:20 +0000 Song Liu <[email protected]> wrote:

Do you mean we do not want patch 3/3, but would like to keep 1/3 and part of 2/3 (remove the _without_suffix APIs)? If this is the case, we are undoing the change by Sami in [1], and thus may break some tracing tools.

What tracing tools may be broke and why?

This was a few years ago when we were first adding LTO support, but the unexpected suffixes in tracing output broke systrace in Android, presumably because the tools expected to find specific function names without suffixes. I'm not sure if systrace would still be a problem today, but other tools might still make assumptions about the function name format. At the time, we decided to filter out the suffixes in all user space visible output to avoid these issues.

Quoted 9 lines

For this suffix problem, I would like to add another patch to allow probing on suffixed symbols. (It seems suffixed symbols are not available at this point)

The problem is that the suffixed symbols maybe a "part" of the original function, thus user has to carefully use it.

Quoted 2 lines

Sami, could you please share your thoughts on this?

Quoted 8 lines
Quoted 1 line

Sami, I would like to know what problem you have on kprobes.

The reports we received back then were about registering kprobes for static functions, which obviously failed if the compiler added a suffix to the function name. This was more of a problem with ThinLTO and Clang CFI at the time because the compiler used to rename _all_ static functions, but one can obviously run into the same issue with just LTO.

Quoted 5 lines

I think newer LLVM/clang no longer add suffixes to all static functions with LTO and CFI. So this may not be a real issue any more?

If we still need to allow tracing without suffix, I think the approach in this patch set is correct (sort syms based on full name,

Yes, we should allow to find the symbols via the full name, definitely.

Quoted 1 line

remove suffixes in special APIs during lookup).

Quoted 7 lines

Just an idea. Alternative solution would be to make make an alias without the suffix when there is only one symbol with the same name.

It would be complementary with the patch adding aliases for symbols with the same name, see
https://lore.kernel.org/r/[email protected]

I guess v3 plus this work may work well together.

Quoted 2 lines

I would allow to find the symbols with and without the suffix using a single API.

Could you please describe how this API would work? I tried some idea in v1, but it turned out to be quite confusing. So I decided to leave this logic to the users of kallsyms APIs in v2.

Thanks,
Song

maintainer
#20 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

Quoted 66 lines · Song Liu
Quoted 58 lines · Petr Mladek
Quoted 43 lines · Song Liu
Quoted 34 lines · Sami Tolvanen

Hi,

Quoted 9 lines · Masami Hiramatsu
Quoted 3 lines · On Wed, 7 Aug 2024 00:19:20 +0000 Song Liu <[email protected]> wrote:

Do you mean we do not want patch 3/3, but would like to keep 1/3 and part of 2/3 (remove the _without_suffix APIs)? If this is the case, we are undoing the change by Sami in [1], and thus may break some tracing tools.

What tracing tools may be broke and why?

This was a few years ago when we were first adding LTO support, but the unexpected suffixes in tracing output broke systrace in Android, presumably because the tools expected to find specific function names without suffixes. I'm not sure if systrace would still be a problem today, but other tools might still make assumptions about the function name format. At the time, we decided to filter out the suffixes in all user space visible output to avoid these issues.

Quoted 9 lines

For this suffix problem, I would like to add another patch to allow probing on suffixed symbols. (It seems suffixed symbols are not available at this point)

The problem is that the suffixed symbols maybe a "part" of the original function, thus user has to carefully use it.

Quoted 2 lines

Sami, could you please share your thoughts on this?

Quoted 8 lines
Quoted 1 line

Sami, I would like to know what problem you have on kprobes.

The reports we received back then were about registering kprobes for static functions, which obviously failed if the compiler added a suffix to the function name. This was more of a problem with ThinLTO and Clang CFI at the time because the compiler used to rename _all_ static functions, but one can obviously run into the same issue with just LTO.

Quoted 5 lines

I think newer LLVM/clang no longer add suffixes to all static functions with LTO and CFI. So this may not be a real issue any more?

If we still need to allow tracing without suffix, I think the approach in this patch set is correct (sort syms based on full name,

Yes, we should allow to find the symbols via the full name, definitely.

Quoted 1 line

remove suffixes in special APIs during lookup).

Quoted 7 lines

Just an idea. Alternative solution would be to make make an alias without the suffix when there is only one symbol with the same name.

It would be complementary with the patch adding aliases for symbols with the same name, see
https://lore.kernel.org/r/[email protected]

Quoted 8 lines

I guess v3 plus this work may work well together.

Quoted 2 lines

I would allow to find the symbols with and without the suffix using a single API.

Could you please describe how this API would work? I tried some idea in v1, but it turned out to be quite confusing. So I decided to leave this logic to the users of kallsyms APIs in v2.

If we create an alias without the suffix but only when there is only one symbol with such a name then we have, for example:

klp_complete_transition.lwn.123456
klp_complete_transition [alias]

init_once.lwn.2131221
init_once.lwn.3443243
init_once.lwn.4324322
init_once.lwn.5214121
init_once.lwn.2153121
init_once.lwn.4342343

This way, it will be possible to find the static symbol "klp_complete_transition" without the suffix via the alias. It will have the alias because it has an unique name.

While "init_once" symbol must always be searched with the suffix because it is not unique.

It looks like >99% of static symbols have unique name.

Best Regards,
Petr

#21 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

Quoted 93 lines · Petr Mladek
Quoted 66 lines · Song Liu
Quoted 58 lines · Petr Mladek
Quoted 43 lines · Song Liu
Quoted 34 lines · Sami Tolvanen

Hi,

Quoted 9 lines · Masami Hiramatsu
Quoted 3 lines · On Wed, 7 Aug 2024 00:19:20 +0000 Song Liu <[email protected]> wrote:

Do you mean we do not want patch 3/3, but would like to keep 1/3 and part of 2/3 (remove the _without_suffix APIs)? If this is the case, we are undoing the change by Sami in [1], and thus may break some tracing tools.

What tracing tools may be broke and why?

This was a few years ago when we were first adding LTO support, but the unexpected suffixes in tracing output broke systrace in Android, presumably because the tools expected to find specific function names without suffixes. I'm not sure if systrace would still be a problem today, but other tools might still make assumptions about the function name format. At the time, we decided to filter out the suffixes in all user space visible output to avoid these issues.

Quoted 9 lines

For this suffix problem, I would like to add another patch to allow probing on suffixed symbols. (It seems suffixed symbols are not available at this point)

The problem is that the suffixed symbols maybe a "part" of the original function, thus user has to carefully use it.

Quoted 2 lines

Sami, could you please share your thoughts on this?

Quoted 8 lines
Quoted 1 line

Sami, I would like to know what problem you have on kprobes.

The reports we received back then were about registering kprobes for static functions, which obviously failed if the compiler added a suffix to the function name. This was more of a problem with ThinLTO and Clang CFI at the time because the compiler used to rename _all_ static functions, but one can obviously run into the same issue with just LTO.

Quoted 5 lines

I think newer LLVM/clang no longer add suffixes to all static functions with LTO and CFI. So this may not be a real issue any more?

If we still need to allow tracing without suffix, I think the approach in this patch set is correct (sort syms based on full name,

Yes, we should allow to find the symbols via the full name, definitely.

Quoted 1 line

remove suffixes in special APIs during lookup).

Quoted 7 lines

Just an idea. Alternative solution would be to make make an alias without the suffix when there is only one symbol with the same name.

It would be complementary with the patch adding aliases for symbols with the same name, see
https://lore.kernel.org/r/[email protected]

Quoted 8 lines

I guess v3 plus this work may work well together.

Quoted 2 lines

I would allow to find the symbols with and without the suffix using a single API.

Could you please describe how this API would work? I tried some idea in v1, but it turned out to be quite confusing. So I decided to leave this logic to the users of kallsyms APIs in v2.

If we create an alias without the suffix but only when there is only one symbol with such a name then we have, for example:

klp_complete_transition.lwn.123456
klp_complete_transition [alias]

init_once.lwn.2131221
init_once.lwn.3443243
init_once.lwn.4324322
init_once.lwn.5214121
init_once.lwn.2153121
init_once.lwn.4342343

Quoted 8 lines

This way, it will be possible to find the static symbol "klp_complete_transition" without the suffix via the alias. It will have the alias because it has an unique name.

While "init_once" symbol must always be searched with the suffix because it is not unique.

It looks like >99% of static symbols have unique name.

Got it. The idea is to generate the alias at boot time. I think this will indeed work.

IIUC, v3 of this set with Alessandro's work (maybe with some variations) should do this.

Thanks,
Song

#22 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

In reply to Petr Mladek
Quoted 54 lines · On Thu, 8 Aug 2024 11:59:00 +0200 Petr Mladek <[email protected]> wrote:
Quoted 43 lines · Song Liu
Quoted 34 lines · Sami Tolvanen

Hi,

Quoted 9 lines · Masami Hiramatsu
Quoted 3 lines · On Wed, 7 Aug 2024 00:19:20 +0000 Song Liu <[email protected]> wrote:

Do you mean we do not want patch 3/3, but would like to keep 1/3 and part of 2/3 (remove the _without_suffix APIs)? If this is the case, we are undoing the change by Sami in [1], and thus may break some tracing tools.

What tracing tools may be broke and why?

This was a few years ago when we were first adding LTO support, but the unexpected suffixes in tracing output broke systrace in Android, presumably because the tools expected to find specific function names without suffixes. I'm not sure if systrace would still be a problem today, but other tools might still make assumptions about the function name format. At the time, we decided to filter out the suffixes in all user space visible output to avoid these issues.

Quoted 9 lines

For this suffix problem, I would like to add another patch to allow probing on suffixed symbols. (It seems suffixed symbols are not available at this point)

The problem is that the suffixed symbols maybe a "part" of the original function, thus user has to carefully use it.

Quoted 2 lines

Sami, could you please share your thoughts on this?

Quoted 8 lines
Quoted 1 line

Sami, I would like to know what problem you have on kprobes.

The reports we received back then were about registering kprobes for static functions, which obviously failed if the compiler added a suffix to the function name. This was more of a problem with ThinLTO and Clang CFI at the time because the compiler used to rename _all_ static functions, but one can obviously run into the same issue with just LTO.

Quoted 5 lines

I think newer LLVM/clang no longer add suffixes to all static functions with LTO and CFI. So this may not be a real issue any more?

If we still need to allow tracing without suffix, I think the approach in this patch set is correct (sort syms based on full name,

Yes, we should allow to find the symbols via the full name, definitely.

Quoted 1 line

remove suffixes in special APIs during lookup).

Quoted 7 lines

Just an idea. Alternative solution would be to make make an alias without the suffix when there is only one symbol with the same name.

It would be complementary with the patch adding aliases for symbols with the same name, see
https://lore.kernel.org/r/[email protected]

I think this is the best idea what we can do for tracing/stacktrace with same-name symbols. But the root cause here is a bit different, that's why I rejected the last patch.

With compiler suffixes, the source line aliases should remove those suffixes and add new suffix like below.

1234 t name_show.llvm.12345678
1234 t name_show@kernel_irq_irqdesc_c_264

Quoted 2 lines

I would allow to find the symbols with and without the suffix using a single API.

I wonder why we need it? for ftrace filter?

The same symbol name does not mean the same function prototype. For the kprobes (and fprobes, ftraces too) user must specify which actual symbol is what you want to probe.

Of course if user can say "hey kprobe, probe name_show", but that is unclear (not unique symbol) so kprobe will reject it. If there is .llvm suffix, user can specify one of them. But it is still unclear to user where in the source code the symbol is actually defined. So eventually, we need the debuginfo or this @suffix.

Thank you,

Quoted 3 lines

Best Regards,
Petr

Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
#23 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

In reply to Petr Mladek

Hello,
sorry to join late at the party.

Il giorno gio 8 ago 2024 alle ore 11:59 Petr Mladek <[email protected]> ha scritto:

Quoted 59 lines · Song Liu
Quoted 43 lines
Quoted 34 lines · Sami Tolvanen

Hi,

Quoted 9 lines · Masami Hiramatsu
Quoted 3 lines · On Wed, 7 Aug 2024 00:19:20 +0000 Song Liu <[email protected]> wrote:

Do you mean we do not want patch 3/3, but would like to keep 1/3 and part of 2/3 (remove the _without_suffix APIs)? If this is the case, we are undoing the change by Sami in [1], and thus may break some tracing tools.

What tracing tools may be broke and why?

This was a few years ago when we were first adding LTO support, but the unexpected suffixes in tracing output broke systrace in Android, presumably because the tools expected to find specific function names without suffixes. I'm not sure if systrace would still be a problem today, but other tools might still make assumptions about the function name format. At the time, we decided to filter out the suffixes in all user space visible output to avoid these issues.

Quoted 9 lines

For this suffix problem, I would like to add another patch to allow probing on suffixed symbols. (It seems suffixed symbols are not available at this point)

The problem is that the suffixed symbols maybe a "part" of the original function, thus user has to carefully use it.

Quoted 2 lines

Sami, could you please share your thoughts on this?

Quoted 8 lines
Quoted 1 line

Sami, I would like to know what problem you have on kprobes.

The reports we received back then were about registering kprobes for static functions, which obviously failed if the compiler added a suffix to the function name. This was more of a problem with ThinLTO and Clang CFI at the time because the compiler used to rename _all_ static functions, but one can obviously run into the same issue with just LTO.

Quoted 5 lines

I think newer LLVM/clang no longer add suffixes to all static functions with LTO and CFI. So this may not be a real issue any more?

If we still need to allow tracing without suffix, I think the approach in this patch set is correct (sort syms based on full name,

Yes, we should allow to find the symbols via the full name, definitely.

Quoted 1 line

remove suffixes in special APIs during lookup).

Just an idea. Alternative solution would be to make make an alias without the suffix when there is only one symbol with the same name.

Quoted 6 lines

It would be complementary with the patch adding aliases for symbols with the same name, see
https://lore.kernel.org/r/[email protected]

I would allow to find the symbols with and without the suffix using a single API.

kas_alias isn't handling LTO as effectively as it should.
This is something I plan to address in the next patch version. Introducing aliases is the best approach I found to preserve current tools behavior while adding this new feature.
While I believe it will deliver the promised benefits, there is a trade-off, particularly affecting features like live patching, which rely on handling duplicate symbols.
For instance, kallsyms_lookup_names typically returns the last occurrence of a symbol when the end argument is not NULL, but introducing aliases disrupts this behavior.
I'm working on a solution to manage duplicate symbols, ensuring compatibility with both LTO and kallsyms_lookup_names.

thanks,
Alessandro

Quoted 3 lines

Best Regards,
Petr

#24 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

Hi Alessandro,

Quoted 73 lines · Alessandro Carminati

Hello,
sorry to join late at the party.

Il giorno gio 8 ago 2024 alle ore 11:59 Petr Mladek <[email protected]> ha scritto:

Quoted 59 lines · Song Liu
Quoted 43 lines
Quoted 34 lines · Sami Tolvanen

Hi,

Quoted 9 lines · Masami Hiramatsu
Quoted 3 lines · On Wed, 7 Aug 2024 00:19:20 +0000 Song Liu <[email protected]> wrote:

Do you mean we do not want patch 3/3, but would like to keep 1/3 and part of 2/3 (remove the _without_suffix APIs)? If this is the case, we are undoing the change by Sami in [1], and thus may break some tracing tools.

What tracing tools may be broke and why?

This was a few years ago when we were first adding LTO support, but the unexpected suffixes in tracing output broke systrace in Android, presumably because the tools expected to find specific function names without suffixes. I'm not sure if systrace would still be a problem today, but other tools might still make assumptions about the function name format. At the time, we decided to filter out the suffixes in all user space visible output to avoid these issues.

Quoted 9 lines

For this suffix problem, I would like to add another patch to allow probing on suffixed symbols. (It seems suffixed symbols are not available at this point)

The problem is that the suffixed symbols maybe a "part" of the original function, thus user has to carefully use it.

Quoted 2 lines

Sami, could you please share your thoughts on this?

Quoted 8 lines
Quoted 1 line

Sami, I would like to know what problem you have on kprobes.

The reports we received back then were about registering kprobes for static functions, which obviously failed if the compiler added a suffix to the function name. This was more of a problem with ThinLTO and Clang CFI at the time because the compiler used to rename _all_ static functions, but one can obviously run into the same issue with just LTO.

Quoted 5 lines

I think newer LLVM/clang no longer add suffixes to all static functions with LTO and CFI. So this may not be a real issue any more?

If we still need to allow tracing without suffix, I think the approach in this patch set is correct (sort syms based on full name,

Yes, we should allow to find the symbols via the full name, definitely.

Quoted 1 line

remove suffixes in special APIs during lookup).

Just an idea. Alternative solution would be to make make an alias without the suffix when there is only one symbol with the same name.

Quoted 6 lines

It would be complementary with the patch adding aliases for symbols with the same name, see
https://lore.kernel.org/r/[email protected]

I would allow to find the symbols with and without the suffix using a single API.

Quoted 10 lines

kas_alias isn't handling LTO as effectively as it should.
This is something I plan to address in the next patch version. Introducing aliases is the best approach I found to preserve current tools behavior while adding this new feature.
While I believe it will deliver the promised benefits, there is a trade-off, particularly affecting features like live patching, which rely on handling duplicate symbols.
For instance, kallsyms_lookup_names typically returns the last occurrence of a symbol when the end argument is not NULL, but introducing aliases disrupts this behavior.

Do you think with v3 of this set [1], live patching should be fine? The idea is to let kallsyms_lookup_names() do full name match, then live patching can find the right symbol with symbol name + old_sympos. Did I miss some cases?

Quoted 2 lines

I'm working on a solution to manage duplicate symbols, ensuring compatibility with both LTO and kallsyms_lookup_names.

Thanks,
Song

[1] https://lore.kernel.org/live-patching/[email protected]/T/#u

#25 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

In reply to Sami Tolvanen
Quoted 13 lines · On Wed, 7 Aug 2024 08:33:07 -0700 Sami Tolvanen <[email protected]> wrote:

Hi,

Quoted 9 lines · Masami Hiramatsu
Quoted 3 lines · On Wed, 7 Aug 2024 00:19:20 +0000 Song Liu <[email protected]> wrote:

Do you mean we do not want patch 3/3, but would like to keep 1/3 and part of 2/3 (remove the _without_suffix APIs)? If this is the case, we are undoing the change by Sami in [1], and thus may break some tracing tools.

What tracing tools may be broke and why?

Quoted 7 lines

This was a few years ago when we were first adding LTO support, but the unexpected suffixes in tracing output broke systrace in Android, presumably because the tools expected to find specific function names without suffixes. I'm not sure if systrace would still be a problem today, but other tools might still make assumptions about the function name format. At the time, we decided to filter out the suffixes in all user space visible output to avoid these issues.

Thanks for explanation.
IMHO, those tools might change their assumptions and decide the policy that it drops suffixes (as you did) or keep the suffixes as it is.

Quoted 9 lines
Quoted 9 lines

For this suffix problem, I would like to add another patch to allow probing on suffixed symbols. (It seems suffixed symbols are not available at this point)

The problem is that the suffixed symbols maybe a "part" of the original function, thus user has to carefully use it.

Quoted 2 lines

Sami, could you please share your thoughts on this?

Quoted 8 lines
Quoted 1 line

Sami, I would like to know what problem you have on kprobes.

The reports we received back then were about registering kprobes for static functions, which obviously failed if the compiler added a suffix to the function name. This was more of a problem with ThinLTO and Clang CFI at the time because the compiler used to rename _all_ static functions, but one can obviously run into the same issue with just LTO.

Yeah, without 1/3 of this series, user can not specify llvm suffixed symbols on kprobe events. However, as perf-probe does, users can use the relative offset from a unique symbol too. (kprobe does not care the function boundary.)

Thank you,

Quoted 2 lines

Sami

Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
#26 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

In reply to Masami Hiramatsu (Google)
Quoted 71 lines · Masami Hiramatsu
Quoted 58 lines · On Wed, 7 Aug 2024 00:19:20 +0000 Song Liu <[email protected]> wrote:
Quoted 58 lines · Masami Hiramatsu
Quoted 36 lines · On Tue, 6 Aug 2024 20:12:55 +0000 Song Liu <[email protected]> wrote:
Quoted 28 lines · Steven Rostedt
Quoted 21 lines · On Tue, 6 Aug 2024 16:00:49 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 21 lines
Quoted 9 lines
Quoted 3 lines

+ if (IS_ENABLED(CONFIG_LTO_CLANG) && !addr)
+ addr = kallsyms_lookup_name_without_suffix(trace_kprobe_symbol(tk)); +

So you do the lookup twice if this is enabled?

Why not just use "kallsyms_lookup_name_without_suffix()" the entire time, and it should work just the same as "kallsyms_lookup_name()" if it's not needed?

We still want to give priority to full match. For example, we have:

[root@~]# grep c_next /proc/kallsyms
ffffffff81419dc0 t c_next.llvm.7567888411731313343
ffffffff81680600 t c_next
ffffffff81854380 t c_next.llvm.14337844803752139461

If the goal is to explicitly trace c_next.llvm.7567888411731313343, the user can provide the full name. If we always match _without_suffix, all of the 3 will match to the first one.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Does this make sense?

Yes. Sorry, I missed the "&& !addr)" after the "IS_ENABLED()", which looked like you did the command twice.

But that said, does this only have to be for llvm? Or should we do this for even gcc? As I believe gcc can give strange symbols too.

Quoted 8 lines

I think most of the issue comes with LTO, as LTO promotes local static functions to global functions. IIUC, we don't have GCC built, LTO enabled kernel yet.

In my GCC built, we have suffixes like ".constprop.0", ".part.0", ".isra.0", and ".isra.0.cold". We didn't do anything about these before this set. So I think we are OK not handling them now. We sure can enable it for GCC built kernel in the future.

Hmm, I think it should be handled as it is. This means it should do as livepatch does. Since I expected user will check kallsyms if gets error, we should keep this as it is. (if a symbol has suffix, it should accept symbol with suffix, or user will get confused because they can not find which symbol is kprobed.)

Quoted 5 lines
Quoted 1 line

Sorry about the conclusion (so I NAK this), but this is a good discussion.

Do you mean we do not want patch 3/3, but would like to keep 1/3 and part of 2/3 (remove the _without_suffix APIs)? If this is the case, we are undoing the change by Sami in [1], and thus may break some tracing tools.

Quoted 7 lines

What tracing tools may be broke and why?

For this suffix problem, I would like to add another patch to allow probing on suffixed symbols. (It seems suffixed symbols are not available at this point)

The problem is that the suffixed symbols maybe a "part" of the original function, thus user has to carefully use it.

It appears there are multiple APIs that may need change. For example, on gcc built kernel, /sys/kernel/tracing/available_filter_functions does not show the suffix:

  [root@(none)]# grep cmos_irq_enable /proc/kallsyms
  ffffffff81db5470 t __pfx_cmos_irq_enable.constprop.0
  ffffffff81db5480 t cmos_irq_enable.constprop.0
  ffffffff822dec6e t cmos_irq_enable.constprop.0.cold
  [root@(none)]# grep cmos_irq_enable /sys/kernel/tracing/available_filter_functions
  cmos_irq_enable

perf-probe uses _text+<offset> for such cases:

  [root@(none)]# cat /sys/kernel/tracing/kprobe_events
  p:probe/cmos_irq_enable _text+14374016 

I am not sure which APIs do we need to change here.

Thanks,
Song

maintainer
#27 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

Quoted 11 lines · On Wed, 7 Aug 2024 19:41:11 +0000 Song Liu <[email protected]> wrote:

It appears there are multiple APIs that may need change. For example, on gcc built kernel, /sys/kernel/tracing/available_filter_functions does not show the suffix:

  [root@(none)]# grep cmos_irq_enable /proc/kallsyms
  ffffffff81db5470 t __pfx_cmos_irq_enable.constprop.0
  ffffffff81db5480 t cmos_irq_enable.constprop.0
  ffffffff822dec6e t cmos_irq_enable.constprop.0.cold
  [root@(none)]# grep cmos_irq_enable /sys/kernel/tracing/available_filter_functions
  cmos_irq_enable

Strange, I don't see that:

~# grep cmos_irq_enable /proc/kallsyms ffffffff8f4b2500 t __pfx_cmos_irq_enable.constprop.0 ffffffff8f4b2510 t cmos_irq_enable.constprop.0

~# grep cmos_irq_enable /sys/kernel/tracing/available_filter_functions cmos_irq_enable.constprop.0

-- Steve

Quoted 12 lines

perf-probe uses _text+<offset> for such cases:

  [root@(none)]# cat /sys/kernel/tracing/kprobe_events
  p:probe/cmos_irq_enable _text+14374016 

I am not sure which APIs do we need to change here.

Thanks,
Song

#28 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

Quoted 20 lines · Steven Rostedt
Quoted 11 lines · On Wed, 7 Aug 2024 19:41:11 +0000 Song Liu <[email protected]> wrote:

It appears there are multiple APIs that may need change. For example, on gcc built kernel, /sys/kernel/tracing/available_filter_functions does not show the suffix:

 [root@(none)]# grep cmos_irq_enable /proc/kallsyms
 ffffffff81db5470 t __pfx_cmos_irq_enable.constprop.0
 ffffffff81db5480 t cmos_irq_enable.constprop.0
 ffffffff822dec6e t cmos_irq_enable.constprop.0.cold
 [root@(none)]# grep cmos_irq_enable /sys/kernel/tracing/available_filter_functions
 cmos_irq_enable
Quoted 8 lines

Strange, I don't see that:

~# grep cmos_irq_enable /proc/kallsyms ffffffff8f4b2500 t __pfx_cmos_irq_enable.constprop.0 ffffffff8f4b2510 t cmos_irq_enable.constprop.0

~# grep cmos_irq_enable /sys/kernel/tracing/available_filter_functions cmos_irq_enable.constprop.0

Ah, this is caused by my change. Let me fix that in the next version.

Thanks,
Song

#29 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

Quoted 26 lines · Song Liu
Quoted 23 lines · Steven Rostedt
Quoted 11 lines · On Wed, 7 Aug 2024 19:41:11 +0000 Song Liu <[email protected]> wrote:

It appears there are multiple APIs that may need change. For example, on gcc built kernel, /sys/kernel/tracing/available_filter_functions does not show the suffix:

[root@(none)]# grep cmos_irq_enable /proc/kallsyms
ffffffff81db5470 t __pfx_cmos_irq_enable.constprop.0
ffffffff81db5480 t cmos_irq_enable.constprop.0
ffffffff822dec6e t cmos_irq_enable.constprop.0.cold
[root@(none)]# grep cmos_irq_enable /sys/kernel/tracing/available_filter_functions
cmos_irq_enable

Strange, I don't see that:

Quoted 8 lines
Quoted 6 lines

~# grep cmos_irq_enable /proc/kallsyms ffffffff8f4b2500 t __pfx_cmos_irq_enable.constprop.0 ffffffff8f4b2510 t cmos_irq_enable.constprop.0

~# grep cmos_irq_enable /sys/kernel/tracing/available_filter_functions cmos_irq_enable.constprop.0

Ah, this is caused by my change. Let me fix that in the next version.

PS: Current code still remove .llvm.<hash> suffix from available_filter_functions.

I will change kallsyms_lookup_buildid() to not do the cleanup.

Thanks,
Song

#30 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

In reply to Song Liu
Quoted 58 lines · On Wed, 7 Aug 2024 00:19:20 +0000 Song Liu <[email protected]> wrote:
Quoted 58 lines · Masami Hiramatsu
Quoted 36 lines · On Tue, 6 Aug 2024 20:12:55 +0000 Song Liu <[email protected]> wrote:
Quoted 28 lines · Steven Rostedt
Quoted 21 lines · On Tue, 6 Aug 2024 16:00:49 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 21 lines
Quoted 9 lines
Quoted 3 lines

+ if (IS_ENABLED(CONFIG_LTO_CLANG) && !addr)
+ addr = kallsyms_lookup_name_without_suffix(trace_kprobe_symbol(tk)); +

So you do the lookup twice if this is enabled?

Why not just use "kallsyms_lookup_name_without_suffix()" the entire time, and it should work just the same as "kallsyms_lookup_name()" if it's not needed?

We still want to give priority to full match. For example, we have:

[root@~]# grep c_next /proc/kallsyms
ffffffff81419dc0 t c_next.llvm.7567888411731313343
ffffffff81680600 t c_next
ffffffff81854380 t c_next.llvm.14337844803752139461

If the goal is to explicitly trace c_next.llvm.7567888411731313343, the user can provide the full name. If we always match _without_suffix, all of the 3 will match to the first one.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Does this make sense?

Yes. Sorry, I missed the "&& !addr)" after the "IS_ENABLED()", which looked like you did the command twice.

But that said, does this only have to be for llvm? Or should we do this for even gcc? As I believe gcc can give strange symbols too.

Quoted 8 lines

I think most of the issue comes with LTO, as LTO promotes local static functions to global functions. IIUC, we don't have GCC built, LTO enabled kernel yet.

In my GCC built, we have suffixes like ".constprop.0", ".part.0", ".isra.0", and ".isra.0.cold". We didn't do anything about these before this set. So I think we are OK not handling them now. We sure can enable it for GCC built kernel in the future.

Hmm, I think it should be handled as it is. This means it should do as livepatch does. Since I expected user will check kallsyms if gets error, we should keep this as it is. (if a symbol has suffix, it should accept symbol with suffix, or user will get confused because they can not find which symbol is kprobed.)

Quoted 5 lines
Quoted 1 line

Sorry about the conclusion (so I NAK this), but this is a good discussion.

Do you mean we do not want patch 3/3, but would like to keep 1/3 and part of 2/3 (remove the _without_suffix APIs)? If this is the case, we are undoing the change by Sami in [1], and thus may break some tracing tools.

BTW, I confirmed that the PATCH 1/3 and 2/3 fixes kprobes to probe on suffixed symbols correctly. (because 1/3 allows to search suffixed symbols)

/sys/kernel/tracing # cat dynamic_events p:kprobes/p_c_stop_llvm_17132674095431275852_0 c_stop.llvm.17132674095431275852 p:kprobes/p_c_stop_llvm_8011538628216713357_0 c_stop.llvm.8011538628216713357 p:kprobes/p_c_stop_0 c_stop

Thank you,

Quoted 10 lines

Sami, could you please share your thoughts on this?

If this works, I will send next version with 1/3 and part of 2/3.

Thanks,
Song

[1] https://lore.kernel.org/all/[email protected]/

Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
#31 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

Quoted 71 lines · Masami Hiramatsu
Quoted 58 lines · On Wed, 7 Aug 2024 00:19:20 +0000 Song Liu <[email protected]> wrote:
Quoted 58 lines · Masami Hiramatsu
Quoted 36 lines · On Tue, 6 Aug 2024 20:12:55 +0000 Song Liu <[email protected]> wrote:
Quoted 28 lines · Steven Rostedt
Quoted 21 lines · On Tue, 6 Aug 2024 16:00:49 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 21 lines
Quoted 9 lines
Quoted 3 lines

+ if (IS_ENABLED(CONFIG_LTO_CLANG) && !addr)
+ addr = kallsyms_lookup_name_without_suffix(trace_kprobe_symbol(tk)); +

So you do the lookup twice if this is enabled?

Why not just use "kallsyms_lookup_name_without_suffix()" the entire time, and it should work just the same as "kallsyms_lookup_name()" if it's not needed?

We still want to give priority to full match. For example, we have:

[root@~]# grep c_next /proc/kallsyms
ffffffff81419dc0 t c_next.llvm.7567888411731313343
ffffffff81680600 t c_next
ffffffff81854380 t c_next.llvm.14337844803752139461

If the goal is to explicitly trace c_next.llvm.7567888411731313343, the user can provide the full name. If we always match _without_suffix, all of the 3 will match to the first one.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Does this make sense?

Yes. Sorry, I missed the "&& !addr)" after the "IS_ENABLED()", which looked like you did the command twice.

But that said, does this only have to be for llvm? Or should we do this for even gcc? As I believe gcc can give strange symbols too.

Quoted 8 lines

I think most of the issue comes with LTO, as LTO promotes local static functions to global functions. IIUC, we don't have GCC built, LTO enabled kernel yet.

In my GCC built, we have suffixes like ".constprop.0", ".part.0", ".isra.0", and ".isra.0.cold". We didn't do anything about these before this set. So I think we are OK not handling them now. We sure can enable it for GCC built kernel in the future.

Hmm, I think it should be handled as it is. This means it should do as livepatch does. Since I expected user will check kallsyms if gets error, we should keep this as it is. (if a symbol has suffix, it should accept symbol with suffix, or user will get confused because they can not find which symbol is kprobed.)

Quoted 5 lines
Quoted 1 line

Sorry about the conclusion (so I NAK this), but this is a good discussion.

Do you mean we do not want patch 3/3, but would like to keep 1/3 and part of 2/3 (remove the _without_suffix APIs)? If this is the case, we are undoing the change by Sami in [1], and thus may break some tracing tools.

Quoted 7 lines

BTW, I confirmed that the PATCH 1/3 and 2/3 fixes kprobes to probe on suffixed symbols correctly. (because 1/3 allows to search suffixed symbols)

/sys/kernel/tracing # cat dynamic_events p:kprobes/p_c_stop_llvm_17132674095431275852_0 c_stop.llvm.17132674095431275852 p:kprobes/p_c_stop_llvm_8011538628216713357_0 c_stop.llvm.8011538628216713357 p:kprobes/p_c_stop_0 c_stop

Thanks for confirming this works.

I will clean up the code and send v3. I plan to remove the _without_suffix APIs in v3. We can add them back if we need them to get systrace working.

Song

#32 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

In reply to Song Liu
Quoted 3 lines

In my GCC built, we have suffixes like ".constprop.0", ".part.0", ".isra.0", and ".isra.0.cold".

A fresher's eye, I met sometime when try to build a livepatch module and found some mistake caused by ".constprop.0" ".part.0" which is generated by GCC.

These section with such suffixes is special and sometime the symbol st_value is quite different. What is these kind of section (or symbol) use for?

Regards
Wardenjohn

#33 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

Quoted 11 lines · zhang warden
Quoted 4 lines

In my GCC built, we have suffixes like ".constprop.0", ".part.0", ".isra.0", and ".isra.0.cold".

A fresher's eye, I met sometime when try to build a livepatch module and found some mistake caused by ".constprop.0" ".part.0" which is generated by GCC.

These section with such suffixes is special and sometime the symbol st_value is quite different. What is these kind of section (or symbol) use for?

IIUC, constprop means const propagation. For example, function "foo(int a, int b)" that is called as "foo(a, 10)" will be come "foo(int a)" with a hard-coded b = 10 inside.

.part.0 is part of the function, as the other part is inlined in the caller.

With binary-diff based toolchain (kpatch-build), I think these will be handled automatically. However, if we write the livepatch manually, we need to understand these behavior with .constprop and .part.

Thanks,
Song

#34 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

Quoted 6 lines

IIUC, constprop means const propagation. For example, function "foo(int a, int b)" that is called as "foo(a, 10)" will be come "foo(int a)" with a hard-coded b = 10 inside.

Quoted 8 lines

.part.0 is part of the function, as the other part is inlined in the caller.

With binary-diff based toolchain (kpatch-build), I think these will be handled automatically.

Thanks,
Song

Yep, Thanks for your explanation!

I discuss here just for an interest.

IMO, more people may use tools like 'kpatch-build' to build a livepatch for it will be more easy. But I think such tools should also be attention to such changes because when it trying to do section-level or symbol-level operations. Otherwise it may cause problems :)

Quoted 2 lines

However, if we write the livepatch manually, we
need to understand these behavior with .constprop and .part.

Do you think it is wise to write a livepatch module manually? If we are trying to write a module manually, we should also take care of the klp_* function calling, while using a livepatch building tools will help handle it :)

Thanks.
Wardenjohn

maintainer
#35 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

In reply to Song Liu
Quoted 13 lines · Song Liu
Quoted 11 lines · zhang warden
Quoted 4 lines

In my GCC built, we have suffixes like ".constprop.0", ".part.0", ".isra.0", and ".isra.0.cold".

A fresher's eye, I met sometime when try to build a livepatch module and found some mistake caused by ".constprop.0" ".part.0" which is generated by GCC.

These section with such suffixes is special and sometime the symbol st_value is quite different. What is these kind of section (or symbol) use for?

Quoted 6 lines

IIUC, constprop means const propagation. For example, function "foo(int a, int b)" that is called as "foo(a, 10)" will be come "foo(int a)" with a hard-coded b = 10 inside.

.part.0 is part of the function, as the other part is inlined in the caller.

Hmm, we should not remove the suffixes like .constprop*, .part*, .isra*. They implement a special optimized variant of the function. It is not longer the original full-featured one.

This is a difference against adding a suffix for a static function. Such a symbol implements the original full-featured function.

Best Regards,
Petr

#36 lore raw

Re: [PATCH v2 3/3] tracing/kprobes: Use APIs that matches symbols without .XXX suffix

Quoted 25 lines · Petr Mladek
Quoted 13 lines · Song Liu
Quoted 11 lines · zhang warden
Quoted 4 lines

In my GCC built, we have suffixes like ".constprop.0", ".part.0", ".isra.0", and ".isra.0.cold".

A fresher's eye, I met sometime when try to build a livepatch module and found some mistake caused by ".constprop.0" ".part.0" which is generated by GCC.

These section with such suffixes is special and sometime the symbol st_value is quite different. What is these kind of section (or symbol) use for?

Quoted 6 lines

IIUC, constprop means const propagation. For example, function "foo(int a, int b)" that is called as "foo(a, 10)" will be come "foo(int a)" with a hard-coded b = 10 inside.

.part.0 is part of the function, as the other part is inlined in the caller.

Quoted 6 lines

Hmm, we should not remove the suffixes like .constprop*, .part*, .isra*. They implement a special optimized variant of the function. It is not longer the original full-featured one.

This is a difference against adding a suffix for a static function. Such a symbol implements the original full-featured function.

Allow tracing without .llvm.<hash> suffixes may target a different function with same name, i.e. func_a.llvm.1 vs. func_a.llvm.2. We can probably detect and report this in the kernel. However, I would rather we just disallow tracing without suffixes. I think Masami also agrees with this.

Thanks,
Song