perf tools: Fix /proc/kallsyms map split
Namhyung Kim
maintainerv2 cover perf tools: Fix /proc/kallsyms map split
Hello,
I found a weird bug in symbol handling for kallsyms. My system has a live patch which is a module and it has some symbols that conflict with the main kernel map.
* v2 changes)
- use map__kmaps() (Ian)
- use temporary root directory (Ian)
- Add Ian's Reviewed-by
For example, the symbols are common functions defined in the kernel like kmalloc and kfree. They have 'u' type which is a unique global symbol.
$ grep ' u ' /proc/kallsyms ffffffff98798dd0 u kmalloc_caches [livepatch] ffffffff97afb6c0 u kmalloc_trace_noprof [livepatch] ffffffff9739e7a0 u __list_add_valid_or_report [livepatch] ffffffff97afb240 u __kmalloc_noprof [livepatch] ffffffff970597f0 u klp_enable_patch [livepatch] ffffffff979939f0 u kfree [livepatch] ...
They are duplicate symbols and will be removed by the fixup routine. But if symbol_conf.allow_aliases is set, they remain. This is the case for perf lock contention, and it caused a trouble with the kallsyms split code.
$ grep ' kfree' /proc/kallsyms ffffffff97057a30 t kfree_rcu_shrink_scan ffffffff97394380 T kfree_strarray ffffffff9779b890 T kfree_skb_list_reason ffffffff9787be60 t kfree_pmc ffffffff979939f0 T kfree <<<--- here ffffffff979bbc50 T kfree_skb_partial ffffffff97a8f110 t kfree_rcu_work ffffffff97a8f2f0 t kfree_rcu_monitor ffffffff97a8f910 t kfree_rcu_shrink_count ffffffff97af67f0 T kfree_const ffffffff97afbbc0 T kfree_sensitive ffffffff97b5c4a0 T kfree_link ffffffff99255908 d kfree_rcu_shrinker ffffffff998beec0 T kfree_rcu_scheduler_running ffffffff979939f0 u kfree [livepatch] <<<--- duplicate
As the kfree function is in the livepatch module, any functions in the main kernel map that come later than 'kfree' will now to be splitted. This will create a lot of new kernel maps and loading them again will go to the routines to load kallsyms and split. So the process was in an infinite loop creating new maps and eventually gets killed.
I've added some defensive measures to prevent such situations and a test case to verify it. But maybe we need to do something for 'u' type symbols.
Thanks,
Namhyung
Namhyung Kim (5):
perf tools: Mark split kallsyms DSOs as loaded perf tools: Fix split kallsyms DSO counting perf tools: Fallback to initial kernel map properly perf tools: Use machine->root_dir to find /proc/kallsyms perf test: Add kallsyms split test
tools/perf/tests/Build | 1 + tools/perf/tests/builtin-test.c | 1 + tools/perf/tests/kallsyms-split.c | 156 ++++++++++++++++++++++++++++++ tools/perf/tests/tests.h | 1 + tools/perf/util/symbol.c | 16 ++- 5 files changed, 171 insertions(+), 4 deletions(-) create mode 100644 tools/perf/tests/kallsyms-split.c
Signature · 1 line
2.52.0.158.g65b55ccf14-goog
Namhyung Kim
maintainerPATCH v2 1/5 perf tools: Mark split kallsyms DSOs as loaded
The maps__split_kallsyms() will split symbols to module DSOs if it comes from a module. It also handled some unusual kernel symbols after modules by creating new kernel maps like "[kernel].0".
But they are pseudo DSOs to have those unexpected symbols. They should not be considered as unloaded kernel DSOs. Otherwise the dso__load() for them will end up calling dso__load_kallsyms() and then maps__split_kallsyms() again and again.
--- tools/perf/util/symbol.c | 1 + 1 file changed, 1 insertion(+)
tools/perf/util/symbol.c +1 -0
@@ -972,6 +972,7 @@ static int maps__split_kallsyms(struct maps *kmaps, struct dso *dso, u64 delta,return -1 ;dso__set_kernel (ndso,dso__kernel (dso)); +dso__set_loaded (ndso); curr_map =map__new2 (pos->start, ndso);if (curr_map ==NULL ) {
Signature · 1 line
2.52.0.158.g65b55ccf14-goog
Namhyung Kim
maintainerPATCH v2 2/5 perf tools: Fix split kallsyms DSO counting
In reply to Namhyung KimIt's counted twice as it's increased after calling maps__insert(). I guess we want to increase it only after it's added properly.
--- tools/perf/util/symbol.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
tools/perf/util/symbol.c +2 -2
@@ -960,11 +960,11 @@ static int maps__split_kallsyms(struct maps *kmaps, struct dso *dso, u64 delta,if (dso__kernel (dso) ==DSO_SPACE__KERNEL_GUEST )snprintf (dso_name,sizeof (dso_name),"[guest.kernel].%d" , - kernel_range++); + kernel_range);else snprintf (dso_name,sizeof (dso_name),"[kernel].%d" , - kernel_range++); + kernel_range); ndso =dso__new (dso_name);map__zput (curr_map);
Signature · 1 line
2.52.0.158.g65b55ccf14-goog
Namhyung Kim
maintainerPATCH v2 3/5 perf tools: Fallback to initial kernel map properly
In reply to Namhyung KimIn maps__split_kallsyms(), it assumes new kernel map when it finds a symbol without module after any module and the initial kernel map has some symbols. Because it expects modules are out of the kernel map so modules should not have symbols in the kernel map.
For example, the following memory map shows symbols and maps. Any symbols in the module 1 area will go to the module 1. The main kernel map starts at 0xffffffffbc200000. But if any symbol has a module between the symbols in that area, next symbols after 0xffffffffbd008000 will generate new kernel maps like [kernel].1.
kernel address | |
| |
0xffffffffc0000000 |---------------------|
| (symbols) |
| ... | <--- [kernel].N
0xffffffffbc400000 |---------------------|
| (symbols) |
| module 2 | <--- bad?
0xffffffffbc380000 |---------------------|
| ... |
| (symbols) |
| [kernel.kallsyms] | <--- initial map
0xffffffffbc200000 |---------------------|
| |
| |
0xffffffffabcde000 |---------------------|
| (symbols) |
| module 1 |
0xffffffffabcd0000 |---------------------|This is very fragile when the module has a symbol that falls into the main kernel map for some reason. My system has a livepatch module with such symbols. And it created a lot of new kernel maps after those symbols. But the symbol may have broken addresses and the later symbols can still be found in the initial kernel map.
Let's check the symbol address in the initial map and use it if found.
--- tools/perf/util/symbol.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
tools/perf/util/symbol.c +2 -1
@@ -951,7 +951,8 @@ static int maps__split_kallsyms(struct maps *kmaps, struct dso *dso, u64 delta, pos->end -= delta; } -if (count ==0 ) { +if (map__start (initial_map) <= (pos->start + delta) && + (pos->start + delta) <map__end (initial_map)) {map__zput (curr_map); curr_map =map__get (initial_map); goto add_symbol;
Signature · 1 line
2.52.0.158.g65b55ccf14-goog
Namhyung Kim
maintainerPATCH v2 4/5 perf tools: Use machine->root_dir to find /proc/kallsyms
In reply to Namhyung KimThis is for test functions to find the kallsyms correctly. It can find the machine from the kernel maps and use its root_dir. This is helpful to setup fake /proc directory for testing.
--- tools/perf/util/symbol.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
tools/perf/util/symbol.c +7 -1
@@ -2002,6 +2002,7 @@ static char *dso__find_kallsyms(struct dso *dso, struct map *map)char sbuild_id[SBUILD_ID_SIZE ];bool is_host = false;char path[PATH_MAX ]; +struct maps *kmaps =map__kmaps (map);if (!dso__has_build_id (dso)) {/*
@@ -2038,8 +2039,13 @@ static char *dso__find_kallsyms(struct dso *dso, struct map *map)return strdup (path);/* Use current /proc/kallsyms if possible */ -if (is_host) { proc_kallsyms: +if (kmaps) { +struct machine *machine =maps__machine (kmaps); + +scnprintf (path,sizeof (path),"%s/proc/kallsyms" , machine->root_dir); +return strdup (path); + }else if (is_host) {return strdup ("/proc/kallsyms" ); }
Signature · 1 line
2.52.0.158.g65b55ccf14-goog
Namhyung Kim
maintainerPATCH v2 5/5 perf test: Add kallsyms split test
In reply to Namhyung KimCreate a fake root directory for /proc/{version,modules,kallsyms} in /tmp for testing. The kallsyms has a bad symbol in the module and it causes the main map splitted. The test ensures it only has two maps - kernel and the module and it finds the initial map after the module without creating the split maps like [kernel].0 and so on.
$ perf test -vv "split kallsyms" 69: split kallsyms: --- start --- test child forked, pid 1016196 try to create fake root directory create kernel maps from the fake root directory maps__set_modules_path_dir: cannot open /tmp/perf-test.Zrv6Sy/lib/modules/X.Y.Z dir Problems setting modules path maps, continuing anyway... Failed to open /tmp/perf-test.Zrv6Sy/proc/kcore. Note /proc/kcore requires CAP_SYS_RAWIO capability to access. Using /tmp/perf-test.Zrv6Sy/proc/kallsyms for symbols kernel map loaded - check symbol and map ---- end(0) ---- 69: split kallsyms : Ok
--- tools/perf/tests/Build | 1 + tools/perf/tests/builtin-test.c | 1 + tools/perf/tests/kallsyms-split.c | 156 ++++++++++++++++++++++++++++++ tools/perf/tests/tests.h | 1 + 4 files changed, 159 insertions(+) create mode 100644 tools/perf/tests/kallsyms-split.c
tools/perf/tests/Build +1 -0
@@ -69,6 +69,7 @@ perf-test-y += util.o perf-test-y += hwmon_pmu.o perf-test-y += tool_pmu.o perf-test-y += subcmd-help.o +perf-test-y += kallsyms-split.o ifeq ($(SRCARCH),$(filter $(SRCARCH),x86 arm arm64 powerpc)) perf-test-$(CONFIG_DWARF_UNWIND) += dwarf-unwind.o
tools/perf/tests/builtin-test.c +1 -0
@@ -140,6 +140,7 @@ static struct test_suite *generic_tests[] = { &suite__symbols, &suite__util, &suite__subcmd_help, + &suite__kallsyms_split, NULL, };
tools/perf/tests/kallsyms-split.c new file +156 -0
156 lines not on this page · raw message
tools/perf/tests/tests.h +1 -0
@@ -178,6 +178,7 @@ DECLARE_SUITE(event_groups);DECLARE_SUITE (symbols);DECLARE_SUITE (util);DECLARE_SUITE (subcmd_help); +DECLARE_SUITE (kallsyms_split);/* * PowerPC and S390 do not support creation of instruction breakpoints using the
Signature · 1 line
2.52.0.158.g65b55ccf14-goog
Re: [PATCH v2 5/5] perf test: Add kallsyms split test
Quoted 21 lines · Namhyung Kim
Create a fake root directory for
/proc/{version,modules,kallsyms}in /tmp for testing. The kallsyms has a bad symbol in the module and it causes the main map splitted. The test ensures it only has two maps - kernel and the module and it finds the initial map after the module without creating the split maps like [kernel].0 and so on.$ perf test -vv "split kallsyms" 69: split kallsyms: --- start --- test child forked, pid 1016196 try to create fake root directory create kernel maps from the fake root directory maps__set_modules_path_dir: cannot open /tmp/perf-test.Zrv6Sy/lib/modules/X.Y.Z dir Problems setting modules path maps, continuing anyway... Failed to open /tmp/perf-test.Zrv6Sy/proc/kcore. Note /proc/kcore requires CAP_SYS_RAWIO capability to access. Using /tmp/perf-test.Zrv6Sy/proc/kallsyms for symbols kernel map loaded - check symbol and map ---- end(0) ---- 69: split kallsyms : Ok
Quoted 1 line
Signed-off-by Namhyung Kim <[email protected]>
Thanks,
Ian
Quoted 209 lines
--- tools/perf/tests/Build | 1 + tools/perf/tests/builtin-test.c | 1 + tools/perf/tests/kallsyms-split.c | 156 ++++++++++++++++++++++++++++++ tools/perf/tests/tests.h | 1 + 4 files changed, 159 insertions(+) create mode 100644 tools/perf/tests/kallsyms-split.ctools/perf/tests/Build +1
-0@@ -69,6 +69,7 @@ perf-test-y += util.o perf-test-y += hwmon_pmu.o perf-test-y += tool_pmu.o perf-test-y += subcmd-help.o +perf-test-y += kallsyms-split.o ifeq ($(SRCARCH),$(filter $(SRCARCH),x86 arm arm64 powerpc)) perf-test-$(CONFIG_DWARF_UNWIND) += dwarf-unwind.otools/perf/tests/builtin-test.c +1
-0@@ -140,6 +140,7 @@ static struct test_suite *generic_tests[] = { &suite__symbols, &suite__util, &suite__subcmd_help, + &suite__kallsyms_split, NULL, };tools/perf/tests/kallsyms-split.c new file +156
-0@@ -0,0 +1,156 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/compiler.h> +#include <fcntl.h> +#include <signal.h> +#include <unistd.h> +#include <sys/stat.h> +#include "util/dso.h" +#include "util/map.h" +#include "util/symbol.h" +#include "util/debug.h" +#include "util/machine.h" +#include "tests.h" + +/* +* This test is to check whether a bad symbol in a module won't split kallsyms maps. +* The main_symbol[1-3] should belong to the main [kernel.kallsyms] map even if the +* bad_symbol from the module is found in the middle. +*/ +static char root_template[] ="/tmp/perf-test.XXXXXX" ; +static char *root_dir; + +static const char proc_version[] ="Linux version X.Y.Z (just for perf test)\n" ; +static const char proc_modules[] ="module 4096 1 - Live 0xffffffffcd000000\n" ; +static const char proc_kallsyms[] = +"ffffffffab200000 T _stext\n" +"ffffffffab200010 T good_symbol\n" +"ffffffffab200020 t bad_symbol\n" +"ffffffffab200030 t main_symbol1\n" +"ffffffffab200040 t main_symbol2\n" +"ffffffffab200050 t main_symbol3\n" +"ffffffffab200060 T _etext\n" +"ffffffffcd000000 T start_module\t[module]\n" +"ffffffffab200020 u bad_symbol\t[module]\n" +"ffffffffcd000040 T end_module\t[module]\n" ; + +static struct { +const char *name; +const char *contents; +long len; +} proc_files[] = { + {"version" , proc_version,sizeof (proc_version) -1 }, + {"modules" , proc_modules,sizeof (proc_modules) -1 }, + {"kallsyms" , proc_kallsyms,sizeof (proc_kallsyms) -1 }, +}; + +static void remove_proc_dir (int sig__maybe_unused ) +{ +char buf[128 ]; + +if (root_dir ==NULL ) +return ; + +for (unsigned i =0 ; i <ARRAY_SIZE (proc_files); i++) { +scnprintf (buf,sizeof (buf),"%s/proc/%s" , root_dir, proc_files[i].name); +remove (buf); + } + +scnprintf (buf,sizeof (buf),"%s/proc" , root_dir); +rmdir (buf); + +rmdir (root_dir); + root_dir =NULL ; +} + +static int create_proc_dir (void ) +{ +char buf[128 ]; + + root_dir =mkdtemp (root_template); +if (root_dir ==NULL ) +return -1 ; + +scnprintf (buf,sizeof (buf),"%s/proc" , root_dir); +if (mkdir (buf,0700 ) <0 ) + goto err; + +for (unsigned i =0 ; i <ARRAY_SIZE (proc_files); i++) { +int fd, len; + +scnprintf (buf,sizeof (buf),"%s/proc/%s" , root_dir, proc_files[i].name); + fd =open (buf,O_RDWR |O_CREAT ,0600 ); +if (fd <0 ) + goto err; + + len =write (fd, proc_files[i].contents, proc_files[i].len); +close (fd); +if (len != proc_files[i].len) + goto err; + } +return 0 ; + +err: +remove_proc_dir (0 ); +return -1 ; +} + +static int test__kallsyms_split (struct test_suite *test__maybe_unused , +int subtest__maybe_unused ) +{ +struct machine m; +struct map *map =NULL ; +int ret =TEST_FAIL ; + +pr_debug ("try to create fake root directory\n" ); +if (create_proc_dir () <0 ) { +pr_debug ("SKIP: cannot create a fake root directory\n" ); +return TEST_SKIP ; + } + +signal (SIGINT , remove_proc_dir); +signal (SIGPIPE , remove_proc_dir); +signal (SIGSEGV , remove_proc_dir); +signal (SIGTERM , remove_proc_dir); + +pr_debug ("create kernel maps from the fake root directory\n" ); +machine__init (&m, root_dir,HOST_KERNEL_ID ); +if (machine__create_kernel_maps (&m) <0 ) { +pr_debug ("FAIL: failed to create kernel maps\n" ); + goto out; + } + +/* force to use /proc/kallsyms */ + symbol_conf.ignore_vmlinux = true; + symbol_conf.ignore_vmlinux_buildid = true; + symbol_conf.allow_aliases = true; + +if (map__load (machine__kernel_map (&m)) <0 ) { +pr_debug ("FAIL: failed to load kallsyms\n" ); + goto out; + } + +pr_debug ("kernel map loaded - check symbol and map\n" ); +if (maps__nr_maps (machine__kernel_maps (&m)) !=2 ) { +pr_debug ("FAIL: it should have the kernel and a module, but has %u maps\n" , +maps__nr_maps (machine__kernel_maps (&m))); + goto out; + } + +if (machine__find_kernel_symbol_by_name (&m,"main_symbol3" , &map) ==NULL ) { +pr_debug ("FAIL: failed to find a symbol\n" ); + goto out; + } + +if (!RC_CHK_EQUAL (map,machine__kernel_map (&m))) { +pr_debug ("FAIL: the symbol is not in the kernel map\n" ); + goto out; + } + ret =TEST_OK ; + +out: +remove_proc_dir (0 ); +machine__exit (&m); +return ret; +} + +DEFINE_SUITE ("split kallsyms" , kallsyms_split);tools/perf/tests/tests.h +1
-1@@ -178,6 +178,7 @@ DECLARE_SUITE(event_groups);DECLARE_SUITE (symbols);DECLARE_SUITE (util);DECLARE_SUITE (subcmd_help); +DECLARE_SUITE (kallsyms_split);/* * PowerPC and S390 do not support creation of instruction breakpoints using the -- 2.52.0.158.g65b55ccf14-goog
Quoted 10 lines · Namhyung Kim
I found a weird bug in symbol handling for kallsyms. My system has a live patch which is a module and it has some symbols that conflict with the main kernel map.
* v2 changes)
- usemap__kmaps()(Ian)
- use temporary root directory (Ian)
- Add Ian's Reviewed-by[...]
Applied to perf-tools-next, thanks!
Best regards,
Namhyung