: Removal of 128MB limit for BPF JIT programs broke perf symbolication on aarch64
----- Forwarded message from Ivan Babrou <[email protected]> -----
Date: Tue, 4 Oct 2022 16:50:01 -0700
From: Ivan Babrou <[email protected]>
To: "Russell King (Oracle)" <[email protected]>
Cc: Will Deacon <[email protected]>
Subject: Re: [External] : Removal of 128MB limit for BPF JIT programs broke
perf symbolication on aarch64
Hi Russell,
Thanks for looking closely into this. Could you send this as a bug report to the perf mailing list? Since it seems to be a bug in perf, I hope people there will be able to figure out how to address this properly.
Many thanks!
Quoted 112 lines · Russell King (Oracle)
Quoted 5 lines · Russell King (Oracle)
So yes, I can reproduce the problem given exactly the right kernel options, but I still don't see how it is caused by the commit you claim it is. I think it's indicative of a deeper problem, one where the kernel image is being placed in the vmalloc region, and
/proc/kcorethen ends up with conflicting LOAD segments in its PHDR.It looks to be a bug in the perf tool:
working2 2022-08-18 12:11:06.716047129 +0100 renamed from broken2 2022-08-18 12:11:29.219489345 +0100 +14
-2-replacement_map: ffffffc008000000-fffffffdf0000000 0000004008003000 +replacement_map: ffffffd793200000-ffffffd794f20000 0000005793203000 New map: Start End pgoff start-pgoff ffffff8000000000-ffffff807be00000 0000000000003000 = ffffff7fffffd000 ffffffa000000000-ffffffa700000000 0000002000003000 = ffffff7fffffd000 ffffffc000000000-ffffffc008000000 0000004000003000 = ffffff7fffffd000 -ffffffc008000000-fffffffdf0000000 0000004008003000 = ffffff7fffffd000 +ffffffc008000000-ffffffc0085a5890 0000004008003000 = ffffff7fffffd000 ffffffc0085a5890-ffffffc0085a5a60 0000000000000000 = ffffffc0085a5890 +ffffffc0085a5a60-ffffffc0085b5b90 00000040085a8a60 = ffffff7fffffd000 ffffffc0085b5b90-ffffffc0085b5bf0 0000000000000000 = ffffffc0085b5b90 +ffffffc0085b5bf0-ffffffc0085c58b0 00000040085b8bf0 = ffffff7fffffd000 ffffffc0085c58b0-ffffffc0085c5910 0000000000000000 = ffffffc0085c58b0 +ffffffc0085c5910-ffffffc0086e5c64 00000040085c8910 = ffffff7fffffd000 ffffffc0086e5c64-ffffffc0086e5e30 0000000000000000 = ffffffc0086e5c64 +ffffffc0086e5e30-ffffffc0086f5284 00000040086e8e30 = ffffff7fffffd000 ffffffc0086f5284-ffffffc0086f52e4 0000000000000000 = ffffffc0086f5284 +ffffffc0086f52e4-ffffffc0087051c8 00000040086f82e4 = ffffff7fffffd000 ffffffc0087051c8-ffffffc008705228 0000000000000000 = ffffffc0087051c8 +ffffffc008705228-ffffffc0093fdcbc 0000004008708228 = ffffff7fffffd000 ffffffc0093fdcbc-ffffffc0093fdd1c 0000000000000000 = ffffffc0093fdcbc +ffffffc0093fdd1c-ffffffc00940dc70 0000004009400d1c = ffffff7fffffd000 ffffffc00940dc70-ffffffc00940dcd0 0000000000000000 = ffffffc00940dc70 +ffffffc00940dcd0-ffffffc0094257a0 0000004009410cd0 = ffffff7fffffd000 ffffffc0094257a0-ffffffc009425800 0000000000000000 = ffffffc0094257a0 +ffffffc009425800-ffffffc00942f420 0000004009428800 = ffffff7fffffd000 ffffffc00942f420-ffffffc00942f480 0000000000000000 = ffffffc00942f420 +ffffffc00942f480-ffffffc009435ae0 0000004009432480 = ffffff7fffffd000 ffffffc009435ae0-ffffffc009435d64 0000000000000000 = ffffffc009435ae0 +ffffffc009435d64-ffffffd793200000 0000004009438d64 = ffffff7fffffd000 +ffffffd793200000-ffffffd794f20000 0000005793203000 = ffffff7fffffd000The above is a red/black tree, which is ordered by start/end addresses. Overlaps are not allowed, yet we can see in the broken case that we end up with an overlap as this entry:
-ffffffc008000000-fffffffdf0000000 0000004008003000 =
ffffff7fffffd000overlaps all the following entries. This happens because the wrong entry is picked up from the sections in
/proc/kcore, and is selected by:/* Find the kernel map using the '_stext' symbol */ if (!kallsyms__get_function_start(kallsyms_filename, "_stext", &stext)) { list_for_each_entry(new_map, &md.maps, node) { // if (new_map->start == 0xffffffc008000000) // continue; if (stext >= new_map->start && stext < new_map->end) { replacement_map = new_map; break; } } }The commented out code is my hack that makes perf work, by forcing it to use the proper entry in
/proc/kcorefor the kernel.The side effect of this is that in
thread__find_map(),maps__find()is unable to find the right map entry to use, soal->mapis NULL. Consequently, the address can't be translated to a symbol. We can see this in my debug output:rmk:
ffffffd793e882e4:0x7fc81bfd9601 lvl k maps0x55a3149bd0map (nil)
ffffffd793e882e4is the address being looked up. The maps being used were:
0x55a3bcd1f0: ffffff8000000000-ffffff807be00000 00000000000030000x55a3bcd4f0: ffffffa000000000-ffffffa700000000 00000020000030000x55a3bcd070: ffffffc000000000-ffffffc008000000 00000040000030000x55a3150000: ffffffc008000000-fffffffdf0000000 00000040080030000x55a31507c0: ffffffc0085a5890-ffffffc0085a5a60 00000000000000000x55a3150fb0: ffffffc0085b5b90-ffffffc0085b5bf0 00000000000000000x55a3151370: ffffffc0085c58b0-ffffffc0085c5910 00000000000000000x55a3151730: ffffffc0086e5c64-ffffffc0086e5e30 00000000000000000x55a3151af0: ffffffc0086f5284-ffffffc0086f52e4 00000000000000000x55a3151eb0: ffffffc0087051c8-ffffffc008705228 00000000000000000x55a3152630: ffffffc0093fdcbc-ffffffc0093fdd1c 00000000000000000x55a31529f0: ffffffc00940dc70-ffffffc00940dcd0 00000000000000000x55a3152db0: ffffffc0094257a0-ffffffc009425800 00000000000000000x55a3153170: ffffffc00942f420-ffffffc00942f480 00000000000000000x55a3152270: ffffffc009435ae0-ffffffc009435d64 0000000000000000We can see this address would fall into the
0x55a3150000map entry, but we can see "map (nil)" meaning it wasn't found.I've no idea what the right fix for this is.
To be honest, I'm not sure I have a full enough understanding of the perf tool to really grasp what it's trying to do here, so I don't think I can progress this to a solution.
I also think that the commit you refer to as causing this is just exposing a bug that's been there for quite some time, and it is unfortunate that changing the kernel layout slightly has provoked it in the perf tool. I'm not sure if this is a perf tool bug or a kernel bug (maybe the kernel shouldn't be exposing overlapping LOAD sections in
/proc/kcore).
Quoted 5 lines
Adding Will in case he has any ideas.
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
----- End forwarded message -----
Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!