perf/util: Symbol lookup can fail if multiple segmets match stext

PATCH Superseded Krister Johansen 3 participants 9 messages Jan 24, 2023 – Feb 20, 2023 perf events

Share & tools

Krister Johansen

author

2023-01-24 22:37 UTC #0 lore raw

PATCH 1/1 perf/util: Symbol lookup can fail if multiple segmets match stext

This problem was encountered on an arm64 system with a lot of memory. Without kernel debug symbols installed, and with both kcore and kallsyms available, perf managed to get confused and returned "unknown" for all of the kernel symbols that it tried to look up.

On this system, stext fell within the vmalloc segment. The kcore symbol matching code tries to find the first segment that contains stext and uses that to replace the segment generated from just the kallsyms information. In this case, however, there were two: a very large vmalloc segment, and the text segment. This caused perf to get confused because multiple overlapping segments were inserted into the RB tree that holds the discovered segments. However, that alone wasn't sufficient to cause the problem. Even when we could find the segment, the offsets were adjusted in such a way that the newly generated symbols didn't line up with the instruction addresses in the trace. The most obvious solution would be to consult which segment type is text from kcore, but this information is not exposed to users.

Instead, select the smallest matching segment that contains stext instead of the first matching segment. This allows us to match the text segment instead of vmalloc, if one is contained within the other.

Signed-off-by Krister Johansen <[email protected]>

---
 tools/perf/util/symbol.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

tools/perf/util/symbol.c +8 -2

index a3a165ae933a..14ac4189eaff 100644
@@ -1368,10 +1368,16 @@ static int dso__load_kcore(struct dso *dso, struct map *map,
 
 	/* Find the kernel map using the '_stext' symbol */
 	if (!kallsyms__get_function_start(kallsyms_filename, "_stext", &stext)) {
+		u64 replacement_size = 0;
 		list_for_each_entry(new_map, &md.maps, node) {
-			if (stext >= new_map->start && stext < new_map->end) {
+			u64 new_size = new_map->end - new_map->start;
+
+			if (!(stext >= new_map->start && stext < new_map->end))
+				continue;
+
+			if (!replacement_map || new_size < replacement_size) {
 				replacement_map = new_map;
-				break;
+				replacement_size = new_size;
 			}
 		}
 	}

Signature · 1 line

2.25.1

Adrian Hunter

reviewer

2023-01-25 07:29 UTC #1 lore raw

Quoted 32 lines · Krister Johansen

This problem was encountered on an arm64 system with a lot of memory. Without kernel debug symbols installed, and with both kcore and kallsyms available, perf managed to get confused and returned "unknown" for all of the kernel symbols that it tried to look up.
On this system, stext fell within the vmalloc segment. The kcore symbol matching code tries to find the first segment that contains stext and uses that to replace the segment generated from just the kallsyms information. In this case, however, there were two: a very large vmalloc segment, and the text segment. This caused perf to get confused because multiple overlapping segments were inserted into the RB tree that holds the discovered segments. However, that alone wasn't sufficient to cause the problem. Even when we could find the segment, the offsets were adjusted in such a way that the newly generated symbols didn't line up with the instruction addresses in the trace. The most obvious solution would be to consult which segment type is text from kcore, but this information is not exposed to users.
Instead, select the smallest matching segment that contains stext instead of the first matching segment. This allows us to match the text segment instead of vmalloc, if one is contained within the other.
Signed-off-by Krister Johansen <[email protected]>
---
 tools/perf/util/symbol.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
tools/perf/util/symbol.c +0 -0
index a3a165ae933a..14ac4189eaff 100644

tools/perf/util/symbol.c static int dso__load_kcore(struct dso *dso, struct map *map, · line 1371

Quoted 4 lines

tools/perf/util/symbol.c +1 -0

@@ -1368,10 +1368,16 @@ static int dso__load_kcore(struct dso *dso, struct map *map,
 
 	/* Find the kernel map using the '_stext' symbol */
 	if (!kallsyms__get_function_start(kallsyms_filename, "_stext", &stext)) {
+		u64 replacement_size = 0;

We'd usually put a blank line here

tools/perf/util/symbol.c static int dso__load_kcore(struct dso *dso, struct map *map, · line 1377

Quoted 7 lines

tools/perf/util/symbol.c +5 -1

 		list_for_each_entry(new_map, &md.maps, node) {
-			if (stext >= new_map->start && stext < new_map->end) {
+			u64 new_size = new_map->end - new_map->start;
+
+			if (!(stext >= new_map->start && stext < new_map->end))
+				continue;
+

Really needs a comment, and please be specific e.g.

ARM64 vmalloc segment overlaps the kernel text segment, so choosing the smaller segment will get the kernel text.

Quoted 7 lines

tools/perf/util/symbol.c +2 -1

+			if (!replacement_map || new_size < replacement_size) {
 				replacement_map = new_map;
-				break;
+				replacement_size = new_size;
 			}
 		}
 	}

Adrian Hunter

reviewer

2023-01-25 07:38 UTC #2 lore raw

On 25/01/23 09:29, Adrian Hunter wrote:

Also subject line has spelling mistake, and should identify kcore as the issue e.g.

perf symbol: Symbol lookup with kcore can fail if multiple segments match stext

Quoted 63 lines · Krister Johansen

Quoted 32 lines
This problem was encountered on an arm64 system with a lot of memory. Without kernel debug symbols installed, and with both kcore and kallsyms available, perf managed to get confused and returned "unknown" for all of the kernel symbols that it tried to look up.
On this system, stext fell within the vmalloc segment. The kcore symbol matching code tries to find the first segment that contains stext and uses that to replace the segment generated from just the kallsyms information. In this case, however, there were two: a very large vmalloc segment, and the text segment. This caused perf to get confused because multiple overlapping segments were inserted into the RB tree that holds the discovered segments. However, that alone wasn't sufficient to cause the problem. Even when we could find the segment, the offsets were adjusted in such a way that the newly generated symbols didn't line up with the instruction addresses in the trace. The most obvious solution would be to consult which segment type is text from kcore, but this information is not exposed to users.
Instead, select the smallest matching segment that contains stext instead of the first matching segment. This allows us to match the text segment instead of vmalloc, if one is contained within the other.
Signed-off-by Krister Johansen <[email protected]>
---
 tools/perf/util/symbol.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
tools/perf/util/symbol.c +0 -0
index a3a165ae933a..14ac4189eaff 100644
tools/perf/util/symbol.c static int dso__load_kcore(struct dso *dso, struct map *map, · line 1371
Quoted 4 lines
tools/perf/util/symbol.c +1 -0
@@ -1368,10 +1368,16 @@ static int dso__load_kcore(struct dso *dso, struct map *map,
 
 	/* Find the kernel map using the '_stext' symbol */
 	if (!kallsyms__get_function_start(kallsyms_filename, "_stext", &stext)) {
+		u64 replacement_size = 0;
We'd usually put a blank line here
tools/perf/util/symbol.c static int dso__load_kcore(struct dso *dso, struct map *map, · line 1377
Quoted 7 lines
tools/perf/util/symbol.c +5 -1
 		list_for_each_entry(new_map, &md.maps, node) {
-			if (stext >= new_map->start && stext < new_map->end) {
+			u64 new_size = new_map->end - new_map->start;
+
+			if (!(stext >= new_map->start && stext < new_map->end))
+				continue;
+
Really needs a comment, and please be specific e.g.
ARM64 vmalloc segment overlaps the kernel text segment, so choosing the smaller segment will get the kernel text.
Quoted 7 lines
tools/perf/util/symbol.c +2 -1
+			if (!replacement_map || new_size < replacement_size) {
 				replacement_map = new_map;
-				break;
+				replacement_size = new_size;
 			}
 		}
 	}

Krister Johansen

author

2023-01-25 18:32 UTC #3 lore raw

Hi Adrian,

Quoted 62 lines · Adrian Hunter

On 25/01/23 09:29, Adrian Hunter wrote:
Also subject line has spelling mistake, and should identify kcore as the issue e.g.
perf symbol: Symbol lookup with kcore can fail if multiple segments match stext
Quoted 55 lines · Krister Johansen
Quoted 32 lines
This problem was encountered on an arm64 system with a lot of memory. Without kernel debug symbols installed, and with both kcore and kallsyms available, perf managed to get confused and returned "unknown" for all of the kernel symbols that it tried to look up.
On this system, stext fell within the vmalloc segment. The kcore symbol matching code tries to find the first segment that contains stext and uses that to replace the segment generated from just the kallsyms information. In this case, however, there were two: a very large vmalloc segment, and the text segment. This caused perf to get confused because multiple overlapping segments were inserted into the RB tree that holds the discovered segments. However, that alone wasn't sufficient to cause the problem. Even when we could find the segment, the offsets were adjusted in such a way that the newly generated symbols didn't line up with the instruction addresses in the trace. The most obvious solution would be to consult which segment type is text from kcore, but this information is not exposed to users.
Instead, select the smallest matching segment that contains stext instead of the first matching segment. This allows us to match the text segment instead of vmalloc, if one is contained within the other.
Signed-off-by Krister Johansen <[email protected]>
---
 tools/perf/util/symbol.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
tools/perf/util/symbol.c +0 -0
index a3a165ae933a..14ac4189eaff 100644
tools/perf/util/symbol.c static int dso__load_kcore(struct dso *dso, struct map *map, · line 1371
Quoted 4 lines
tools/perf/util/symbol.c +1 -0
@@ -1368,10 +1368,16 @@ static int dso__load_kcore(struct dso *dso, struct map *map,
 
 	/* Find the kernel map using the '_stext' symbol */
 	if (!kallsyms__get_function_start(kallsyms_filename, "_stext", &stext)) {
+		u64 replacement_size = 0;
We'd usually put a blank line here
tools/perf/util/symbol.c static int dso__load_kcore(struct dso *dso, struct map *map, · line 1377
Quoted 7 lines
tools/perf/util/symbol.c +5 -1
 		list_for_each_entry(new_map, &md.maps, node) {
-			if (stext >= new_map->start && stext < new_map->end) {
+			u64 new_size = new_map->end - new_map->start;
+
+			if (!(stext >= new_map->start && stext < new_map->end))
+				continue;
+
Really needs a comment, and please be specific e.g.
ARM64 vmalloc segment overlaps the kernel text segment, so choosing the smaller segment will get the kernel text.

Quoted 7 lines

Quoted 7 lines
tools/perf/util/symbol.c static int dso__load_kcore(struct dso *dso, struct map *map, · line 1383
Quoted 7 lines
tools/perf/util/symbol.c +2 -1
+			if (!replacement_map || new_size < replacement_size) {
 				replacement_map = new_map;
-				break;
+				replacement_size = new_size;
 			}
 		}
 	}

Thanks for all the feedback. I'll incorporate these changes and send out a v2 shortly.

-K

Krister Johansen

author

2023-01-25 18:34 UTC #4 lore raw

PATCH v2 1/1 xen/x86: public: add TSC defines for cpuid leaf 4

In reply to Krister Johansen

Cpuid leaf 4 contains information about how the state of the tsc, its mode, and some additional information. A commit that is queued for linux would like to use this to determine whether the tsc mode has been set to 'no emulation' in order to make some decisions about which clocksource is more reliable.

Expose this information in the public API headers so that they can subsequently be imported into linux and used there.

Link https://lore.kernel.org/xen-devel/[email protected]/ open here

Signed-off-by Krister Johansen <[email protected]>

---
v2:
- Fix whitespace between comment and #defines (feedback from Jan Beulich)
- Add tsc mode 3: no emulate TSC_AUX (feedback from Jan Beulich)

---
 xen/include/public/arch-x86/cpuid.h | 8 ++++++++
 1 file changed, 8 insertions(+)

xen/include/public/arch-x86/cpuid.h +8 -0

index 7ecd16ae05..090f7f0034 100644
@@ -72,6 +72,14 @@
  * Sub-leaf 2: EAX: host tsc frequency in kHz
  */
 
+#define XEN_CPUID_TSC_EMULATED               (1u << 0)
+#define XEN_CPUID_HOST_TSC_RELIABLE          (1u << 1)
+#define XEN_CPUID_RDTSCP_INSTR_AVAIL         (1u << 2)
+#define XEN_CPUID_TSC_MODE_DEFAULT           (0)
+#define XEN_CPUID_TSC_MODE_EMULATE           (1u)
+#define XEN_CPUID_TSC_MODE_NOEMULATE         (2u)
+#define XEN_CPUID_TSC_MODE_NOEMULATE_TSC_AUX (3u)
+
 /*
  * Leaf 5 (0x40000x04)
  * HVM-specific features

Signature · 1 line

2.25.1

Krister Johansen

author

2023-01-25 18:42 UTC #5 lore raw

PATCH v2 1/1 perf/util: Symbol lookup with kcore can fail if multiple segments match stext

In reply to Krister Johansen

Signed-off-by Krister Johansen <[email protected]>

---
v2:
- Correct whitespace, add comments, and fix-up subject. (Feedback from Adrian Hunter)

---
 tools/perf/util/symbol.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

tools/perf/util/symbol.c +15 -2

index a3a165ae933a..98014f937568 100644
@@ -1368,10 +1368,23 @@ static int dso__load_kcore(struct dso *dso, struct map *map,
 
 	/* Find the kernel map using the '_stext' symbol */
 	if (!kallsyms__get_function_start(kallsyms_filename, "_stext", &stext)) {
+		u64 replacement_size = 0;
+
 		list_for_each_entry(new_map, &md.maps, node) {
-			if (stext >= new_map->start && stext < new_map->end) {
+			u64 new_size = new_map->end - new_map->start;
+
+			if (!(stext >= new_map->start && stext < new_map->end))
+				continue;
+
+			/*
+			 * On some architectures, ARM64 for example, the kernel
+			 * text can get allocated inside of the vmalloc segment.
+			 * Select the smallest matching segment, in case stext
+			 * falls within more than one in the list.
+			 */
+			if (!replacement_map || new_size < replacement_size) {
 				replacement_map = new_map;
-				break;
+				replacement_size = new_size;
 			}
 		}
 	}

Signature · 1 line

2.25.1

Adrian Hunter

reviewer Reviewed-by on v2 1/1

2023-01-30 13:30 UTC #6 lore raw

Re: [PATCH v2] perf/util: Symbol lookup with kcore can fail if multiple segments match stext

Quoted 18 lines · Krister Johansen

This problem was encountered on an arm64 system with a lot of memory. Without kernel debug symbols installed, and with both kcore and kallsyms available, perf managed to get confused and returned "unknown" for all of the kernel symbols that it tried to look up.
On this system, stext fell within the vmalloc segment. The kcore symbol matching code tries to find the first segment that contains stext and uses that to replace the segment generated from just the kallsyms information. In this case, however, there were two: a very large vmalloc segment, and the text segment. This caused perf to get confused because multiple overlapping segments were inserted into the RB tree that holds the discovered segments. However, that alone wasn't sufficient to cause the problem. Even when we could find the segment, the offsets were adjusted in such a way that the newly generated symbols didn't line up with the instruction addresses in the trace. The most obvious solution would be to consult which segment type is text from kcore, but this information is not exposed to users.

Quoted 5 lines

Instead, select the smallest matching segment that contains stext instead of the first matching segment. This allows us to match the text segment instead of vmalloc, if one is contained within the other.
Signed-off-by Krister Johansen <[email protected]>

Reviewed-by Adrian Hunter <[email protected]>

Quoted 38 lines

---
v2:
- Correct whitespace, add comments, and fix-up subject. (Feedback from Adrian Hunter)

---
 tools/perf/util/symbol.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

tools/perf/util/symbol.c +15 -2

index a3a165ae933a..98014f937568 100644
@@ -1368,10 +1368,23 @@ static int dso__load_kcore(struct dso *dso, struct map *map,
 
 	/* Find the kernel map using the '_stext' symbol */
 	if (!kallsyms__get_function_start(kallsyms_filename, "_stext", &stext)) {
+		u64 replacement_size = 0;
+
 		list_for_each_entry(new_map, &md.maps, node) {
-			if (stext >= new_map->start && stext < new_map->end) {
+			u64 new_size = new_map->end - new_map->start;
+
+			if (!(stext >= new_map->start && stext < new_map->end))
+				continue;
+
+			/*
+			 * On some architectures, ARM64 for example, the kernel
+			 * text can get allocated inside of the vmalloc segment.
+			 * Select the smallest matching segment, in case stext
+			 * falls within more than one in the list.
+			 */
+			if (!replacement_map || new_size < replacement_size) {
 				replacement_map = new_map;
-				break;
+				replacement_size = new_size;
 			}
 		}
 	}

Arnaldo Carvalho de Melo

maintainer Applied

2023-02-02 01:00 UTC #7 lore raw

Re: [PATCH v2] perf/util: Symbol lookup with kcore can fail if multiple segments match stext

Em Mon, Jan 30, 2023 at 03:29:54PM +0200, Adrian Hunter escreveu:

Quoted 19 lines · Krister Johansen

Quoted 18 lines
This problem was encountered on an arm64 system with a lot of memory. Without kernel debug symbols installed, and with both kcore and kallsyms available, perf managed to get confused and returned "unknown" for all of the kernel symbols that it tried to look up.
On this system, stext fell within the vmalloc segment. The kcore symbol matching code tries to find the first segment that contains stext and uses that to replace the segment generated from just the kallsyms information. In this case, however, there were two: a very large vmalloc segment, and the text segment. This caused perf to get confused because multiple overlapping segments were inserted into the RB tree that holds the discovered segments. However, that alone wasn't sufficient to cause the problem. Even when we could find the segment, the offsets were adjusted in such a way that the newly generated symbols didn't line up with the instruction addresses in the trace. The most obvious solution would be to consult which segment type is text from kcore, but this information is not exposed to users.

Quoted 7 lines

Quoted 5 lines
Instead, select the smallest matching segment that contains stext instead of the first matching segment. This allows us to match the text segment instead of vmalloc, if one is contained within the other.
Signed-off-by Krister Johansen <[email protected]>
Reviewed-by Adrian Hunter <[email protected]>

Thanks, applied.

- Arnaldo

Quoted 39 lines

Quoted 38 lines

---
v2:
- Correct whitespace, add comments, and fix-up subject. (Feedback from Adrian Hunter)

---
 tools/perf/util/symbol.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

tools/perf/util/symbol.c +15 -2

index a3a165ae933a..98014f937568 100644
@@ -1368,10 +1368,23 @@ static int dso__load_kcore(struct dso *dso, struct map *map,
 
 	/* Find the kernel map using the '_stext' symbol */
 	if (!kallsyms__get_function_start(kallsyms_filename, "_stext", &stext)) {
+		u64 replacement_size = 0;
+
 		list_for_each_entry(new_map, &md.maps, node) {
-			if (stext >= new_map->start && stext < new_map->end) {
+			u64 new_size = new_map->end - new_map->start;
+
+			if (!(stext >= new_map->start && stext < new_map->end))
+				continue;
+
+			/*
+			 * On some architectures, ARM64 for example, the kernel
+			 * text can get allocated inside of the vmalloc segment.
+			 * Select the smallest matching segment, in case stext
+			 * falls within more than one in the list.
+			 */
+			if (!replacement_map || new_size < replacement_size) {
 				replacement_map = new_map;
-				break;
+				replacement_size = new_size;
 			}
 		}
 	}

Signature · 1 line

- Arnaldo

Krister Johansen

author

2023-02-20 17:16 UTC #8 lore raw

Re: [PATCH v2] perf/util: Symbol lookup with kcore can fail if multiple segments match stext

Quoted 24 lines · Arnaldo Carvalho de Melo

Em Mon, Jan 30, 2023 at 03:29:54PM +0200, Adrian Hunter escreveu:
Quoted 23 lines · Krister Johansen
Quoted 22 lines
This problem was encountered on an arm64 system with a lot of memory. Without kernel debug symbols installed, and with both kcore and kallsyms available, perf managed to get confused and returned "unknown" for all of the kernel symbols that it tried to look up.
On this system, stext fell within the vmalloc segment. The kcore symbol matching code tries to find the first segment that contains stext and uses that to replace the segment generated from just the kallsyms information. In this case, however, there were two: a very large vmalloc segment, and the text segment. This caused perf to get confused because multiple overlapping segments were inserted into the RB tree that holds the discovered segments. However, that alone wasn't sufficient to cause the problem. Even when we could find the segment, the offsets were adjusted in such a way that the newly generated symbols didn't line up with the instruction addresses in the trace. The most obvious solution would be to consult which segment type is text from kcore, but this information is not exposed to users.
Instead, select the smallest matching segment that contains stext instead of the first matching segment. This allows us to match the text segment instead of vmalloc, if one is contained within the other.

Quoted 5 lines

Quoted 3 lines
Quoted 1 line
Signed-off-by Krister Johansen <[email protected]>
Reviewed-by Adrian Hunter <[email protected]>
Thanks, applied.

Thanks, Arnaldo. If it's not too late, would you be willing to tag this one as:

Fixes 5654997838c2 ("perf tools: Use the "_stest" symbol to identify the kernel map when loading kcore")

Cc <[email protected]>

So that it makes it to the right stable branches? I meant to do this prior to sending it along.

-K