Skip to contentSkip to conversation

: Removal of 128MB limit for BPF JIT programs broke perf symbolication on aarch64

External Russell King (Oracle) 1 participant 1 message Oct 5, 2022

lore
Share & tools
lore
#0 lore raw

----- Forwarded message from Ivan Babrou <[email protected]> -----
Date: Tue, 4 Oct 2022 16:50:01 -0700
From: Ivan Babrou <[email protected]>
To: "Russell King (Oracle)" <[email protected]>
Cc: Will Deacon <[email protected]>
Subject: Re: [External] : Removal of 128MB limit for BPF JIT programs broke
perf symbolication on aarch64

Hi Russell,

Thanks for looking closely into this. Could you send this as a bug report to the perf mailing list? Since it seems to be a bug in perf, I hope people there will be able to figure out how to address this properly.

Many thanks!

Quoted 112 lines · Russell King (Oracle)
Quoted 5 lines · Russell King (Oracle)

So yes, I can reproduce the problem given exactly the right kernel options, but I still don't see how it is caused by the commit you claim it is. I think it's indicative of a deeper problem, one where the kernel image is being placed in the vmalloc region, and /proc/kcore then ends up with conflicting LOAD segments in its PHDR.

It looks to be a bug in the perf tool:

working2 2022-08-18 12:11:06.716047129 +0100 renamed from broken2 2022-08-18 12:11:29.219489345 +0100 +14 -2
...
-replacement_map: ffffffc008000000-fffffffdf0000000 0000004008003000
+replacement_map: ffffffd793200000-ffffffd794f20000 0000005793203000
 New map:
 Start            End              pgoff              start-pgoff
 ffffff8000000000-ffffff807be00000 0000000000003000 = ffffff7fffffd000
 ffffffa000000000-ffffffa700000000 0000002000003000 = ffffff7fffffd000
 ffffffc000000000-ffffffc008000000 0000004000003000 = ffffff7fffffd000
-ffffffc008000000-fffffffdf0000000 0000004008003000 = ffffff7fffffd000
+ffffffc008000000-ffffffc0085a5890 0000004008003000 = ffffff7fffffd000
 ffffffc0085a5890-ffffffc0085a5a60 0000000000000000 = ffffffc0085a5890
+ffffffc0085a5a60-ffffffc0085b5b90 00000040085a8a60 = ffffff7fffffd000
 ffffffc0085b5b90-ffffffc0085b5bf0 0000000000000000 = ffffffc0085b5b90
+ffffffc0085b5bf0-ffffffc0085c58b0 00000040085b8bf0 = ffffff7fffffd000
 ffffffc0085c58b0-ffffffc0085c5910 0000000000000000 = ffffffc0085c58b0
+ffffffc0085c5910-ffffffc0086e5c64 00000040085c8910 = ffffff7fffffd000
 ffffffc0086e5c64-ffffffc0086e5e30 0000000000000000 = ffffffc0086e5c64
+ffffffc0086e5e30-ffffffc0086f5284 00000040086e8e30 = ffffff7fffffd000
 ffffffc0086f5284-ffffffc0086f52e4 0000000000000000 = ffffffc0086f5284
+ffffffc0086f52e4-ffffffc0087051c8 00000040086f82e4 = ffffff7fffffd000
 ffffffc0087051c8-ffffffc008705228 0000000000000000 = ffffffc0087051c8
+ffffffc008705228-ffffffc0093fdcbc 0000004008708228 = ffffff7fffffd000
 ffffffc0093fdcbc-ffffffc0093fdd1c 0000000000000000 = ffffffc0093fdcbc
+ffffffc0093fdd1c-ffffffc00940dc70 0000004009400d1c = ffffff7fffffd000
 ffffffc00940dc70-ffffffc00940dcd0 0000000000000000 = ffffffc00940dc70
+ffffffc00940dcd0-ffffffc0094257a0 0000004009410cd0 = ffffff7fffffd000
 ffffffc0094257a0-ffffffc009425800 0000000000000000 = ffffffc0094257a0
+ffffffc009425800-ffffffc00942f420 0000004009428800 = ffffff7fffffd000
 ffffffc00942f420-ffffffc00942f480 0000000000000000 = ffffffc00942f420
+ffffffc00942f480-ffffffc009435ae0 0000004009432480 = ffffff7fffffd000
 ffffffc009435ae0-ffffffc009435d64 0000000000000000 = ffffffc009435ae0
+ffffffc009435d64-ffffffd793200000 0000004009438d64 = ffffff7fffffd000
+ffffffd793200000-ffffffd794f20000 0000005793203000 = ffffff7fffffd000

The above is a red/black tree, which is ordered by start/end addresses. Overlaps are not allowed, yet we can see in the broken case that we end up with an overlap as this entry:

-ffffffc008000000-fffffffdf0000000 0000004008003000 = ffffff7fffffd000

overlaps all the following entries. This happens because the wrong entry is picked up from the sections in /proc/kcore, and is selected by:

        /* Find the kernel map using the '_stext' symbol */
        if (!kallsyms__get_function_start(kallsyms_filename, "_stext", &stext)) {
                list_for_each_entry(new_map, &md.maps, node) {
//                      if (new_map->start == 0xffffffc008000000)
//                              continue;
                        if (stext >= new_map->start && stext < new_map->end) {
                                replacement_map = new_map;
                                break;
                        }
                }
        }

The commented out code is my hack that makes perf work, by forcing it to use the proper entry in /proc/kcore for the kernel.

The side effect of this is that in thread__find_map(), maps__find() is unable to find the right map entry to use, so al->map is NULL. Consequently, the address can't be translated to a symbol. We can see this in my debug output:

rmk: ffffffd793e882e4: 0x7fc81bfd96 01 lvl k maps 0x55a3149bd0 map (nil)

ffffffd793e882e4 is the address being looked up. The maps being used were:

0x55a3bcd1f0: ffffff8000000000-ffffff807be00000 0000000000003000 0x55a3bcd4f0: ffffffa000000000-ffffffa700000000 0000002000003000 0x55a3bcd070: ffffffc000000000-ffffffc008000000 0000004000003000 0x55a3150000: ffffffc008000000-fffffffdf0000000 0000004008003000 0x55a31507c0: ffffffc0085a5890-ffffffc0085a5a60 0000000000000000 0x55a3150fb0: ffffffc0085b5b90-ffffffc0085b5bf0 0000000000000000 0x55a3151370: ffffffc0085c58b0-ffffffc0085c5910 0000000000000000 0x55a3151730: ffffffc0086e5c64-ffffffc0086e5e30 0000000000000000 0x55a3151af0: ffffffc0086f5284-ffffffc0086f52e4 0000000000000000 0x55a3151eb0: ffffffc0087051c8-ffffffc008705228 0000000000000000 0x55a3152630: ffffffc0093fdcbc-ffffffc0093fdd1c 0000000000000000 0x55a31529f0: ffffffc00940dc70-ffffffc00940dcd0 0000000000000000 0x55a3152db0: ffffffc0094257a0-ffffffc009425800 0000000000000000 0x55a3153170: ffffffc00942f420-ffffffc00942f480 0000000000000000 0x55a3152270: ffffffc009435ae0-ffffffc009435d64 0000000000000000

We can see this address would fall into the 0x55a3150000 map entry, but we can see "map (nil)" meaning it wasn't found.

I've no idea what the right fix for this is.

To be honest, I'm not sure I have a full enough understanding of the perf tool to really grasp what it's trying to do here, so I don't think I can progress this to a solution.

I also think that the commit you refer to as causing this is just exposing a bug that's been there for quite some time, and it is unfortunate that changing the kernel layout slightly has provoked it in the perf tool. I'm not sure if this is a perf tool bug or a kernel bug (maybe the kernel shouldn't be exposing overlapping LOAD sections in /proc/kcore).

Quoted 5 lines

Adding Will in case he has any ideas.

--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

----- End forwarded message -----

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!