perf jitdump: Add load_addr to build-ID generation
Namhyung Kim
maintainerPATCH 1/2 perf jitdump: Add load_addr to build-ID generation
It was reported that python backtrace with JIT dump was broken after the change to built-in SHA-1 implementation. It seems python generates the same JIT code for each function. They will become separate DSOs but the contents are the same. Only difference is in the symbol name.
But this caused a problem that every JIT'ed DSOs will have the same build-ID which makes perf confused. And it resulted in no python symbols (from JIT) in the output.
Looking back at the original code before the conversion, it used the load_addr as well as the code section to distinguish each DSO. I think we should do the same or use symbol table as an additional input for SHA-1.
This patch is a quick-and-dirty fix just to add each byte of the load_addr to the first 8 bytes of SHA-1 result. Probably we need to add sha1_update() or similar to update the existing hash value and use it here. I'd like something that can be backported to the stable trees easily.
--- tools/perf/util/genelf.c | 9 +++++++++ 1 file changed, 9 insertions(+)
tools/perf/util/genelf.c +9 -0
@@ -395,6 +395,15 @@ jit_write_elf(int fd, uint64_t load_addr __maybe_unused, const char *sym,* build-id generation */ sha1 (code, csize, bnote.build_id); +/* FIXME: update the SHA-1 hash using additional contents */ + bnote.build_id[0 ] += (load_addr >>0 ) &0xff ; + bnote.build_id[1 ] += (load_addr >>8 ) &0xff ; + bnote.build_id[2 ] += (load_addr >>16 ) &0xff ; + bnote.build_id[3 ] += (load_addr >>24 ) &0xff ; + bnote.build_id[4 ] += (load_addr >>32 ) &0xff ; + bnote.build_id[5 ] += (load_addr >>40 ) &0xff ; + bnote.build_id[6 ] += (load_addr >>48 ) &0xff ; + bnote.build_id[7 ] += (load_addr >>56 ) &0xff ; bnote.desc.namesz =sizeof (bnote.name);/* must include 0 termination */ bnote.desc.descsz =sizeof (bnote.build_id); bnote.desc.type =NT_GNU_BUILD_ID ;
Signature · 1 line
2.52.0.rc1.455.g30608eb744-goog
Namhyung Kim
maintainerPATCH 2/2 perf test: Add python JIT dump test
Add a test case for the python interpreter like below so that we can make sure it won't break again. To validate the effect of build-ID generation, it adds and removes the JIT'ed DSOs to/from the build-ID cache for the test.
$ perf test -vv jitdump 84: python profiling with jitdump: --- start --- test child forked, pid 214316 Run python with -Xperf_jit [ perf record: Woken up 5 times to write data ] [ perf record: Captured and wrote 1.180 MB /tmp/__perf_test.perf.data.XbqZNm (140 samples) ] Generate JIT-ed DSOs using perf inject Add JIT-ed DSOs to the build-ID cache Check the symbol containing the script name Found 108 matching lines Remove JIT-ed DSOs from the build-ID cache ---- end(0) ---- 84: python profiling with jitdump : Ok
--- tools/perf/tests/shell/jitdump-python.sh | 41 ++++++++++++++++++++++++ tools/perf/tests/shell/jitdump-test.py | 14 ++++++++ 2 files changed, 55 insertions(+) create mode 100755 tools/perf/tests/shell/jitdump-python.sh create mode 100644 tools/perf/tests/shell/jitdump-test.py
tools/perf/tests/shell/jitdump-python.sh new file +41 -0
@@ -0,0 +1,41 @@ +#!/bin/bash +# python profiling with jitdump +# SPDX-License-Identifier: GPL-2.0 + +if ! command -v python > /dev/null; then + echo "Skip: no python found" + exit 2 +fi + +SHELLDIR=$(dirname $0) +PERF_DATA=$(mktemp /tmp/__perf_test.perf.data.XXXXXX) + +echo "Run python with -Xperf_jit" +perf record -k 1 -g --call-graph dwarf -o "${PERF_DATA}" -- python -Xperf_jit ${SHELLDIR}/jitdump-test.py + +_PID=$(perf report -i "${PERF_DATA}" -F pid -q -g none | cut -d: -f1 -s) +PID=$(echo -n $_PID) # remove newlines + +echo "Generate JIT-ed DSOs using perf inject" +perf inject -i "${PERF_DATA}" -j -o "${PERF_DATA}.jit" + +echo "Add JIT-ed DSOs to the build-ID cache" +for F in /tmp/jitted-${PID}-*.so; do + perf buildid-cache -a "${F}" +done + +echo "Check the symbol containing the script name" +NUM=$(perf report -i "${PERF_DATA}.jit" -s sym | grep -c jitdump-test.py) + +echo "Found ${NUM} matching lines" + +echo "Remove JIT-ed DSOs from the build-ID cache" +for F in /tmp/jitted-${PID}-*.so; do + perf buildid-cache -r "${F}" +done + +rm -f ${PERF_DATA} ${PERF_DATA}.jit /tmp/jit-${PID}.dump /tmp/jitted-${PID}-*.so + +if [ "${NUM}" -eq 0 ]; then + exit 1 +fi
tools/perf/tests/shell/jitdump-test.py new file +14 -0
@@ -0,0 +1,14 @@ +def foo(n): + result = 0 + for _ in range(n): + result += 1 + return result + +def bar(n): + foo(n) + +def baz(n): + bar(n) + +if __name__ == "__main__": + baz(1000000)
Signature · 1 line
2.52.0.rc1.455.g30608eb744-goog
Ian Rogers
reviewerRe: [PATCH 2/2] perf test: Add python JIT dump test
Quoted 37 lines · Namhyung Kim
Add a test case for the python interpreter like below so that we can make sure it won't break again. To validate the effect of build-ID generation, it adds and removes the JIT'ed DSOs to/from the build-ID cache for the test.
$ perf test -vv jitdump 84: python profiling with jitdump: --- start --- test child forked, pid 214316 Run python with -Xperf_jit [ perf record: Woken up 5 times to write data ] [ perf record: Captured and wrote 1.180 MB /tmp/__perf_test.perf.data.XbqZNm (140 samples) ] Generate JIT-ed DSOs using perf inject Add JIT-ed DSOs to the build-ID cache Check the symbol containing the script name Found 108 matching lines Remove JIT-ed DSOs from the build-ID cache ---- end(0) ---- 84: python profiling with jitdump : OkCc Pablo Galindo <[email protected]>Signed-off-by Namhyung Kim <[email protected]>--- tools/perf/tests/shell/jitdump-python.sh | 41 ++++++++++++++++++++++++ tools/perf/tests/shell/jitdump-test.py | 14 ++++++++ 2 files changed, 55 insertions(+) create mode 100755 tools/perf/tests/shell/jitdump-python.sh create mode 100644 tools/perf/tests/shell/jitdump-test.pytools/perf/tests/shell/jitdump-python.sh new file +0
-0
tools/perf/tests/shell/jitdump-python.sh line 8
Quoted 8 lines
tools/perf/tests/shell/jitdump-python.sh new file +8
-0@@ -0,0 +1,41 @@ +#!/bin/bash +# python profiling with jitdump +# SPDX-License-Identifier: GPL-2.0 + +if ! command -v python > /dev/null; then + echo "Skip: no python found" + exit 2 +fi
Can this just reuse the python set up logic in:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/tests/shell/lib/setup_python.sh?h=perf-tools-next
tools/perf/tests/shell/jitdump-python.sh line 14
Quoted 6 lines
tools/perf/tests/shell/jitdump-python.sh new file +6
-0+ +SHELLDIR=$(dirname $0) +PERF_DATA=$(mktemp /tmp/__perf_test.perf.data.XXXXXX) + +echo "Run python with -Xperf_jit" +perf record -k 1 -g --call-graph dwarf -o "${PERF_DATA}" -- python -Xperf_jit ${SHELLDIR}/jitdump-test.py
We can avoid an additional file by using the REPL, this is why the java test is using jshell:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/tests/shell/test_java_symbol.sh?h=perf-tools-next#n41
tools/perf/tests/shell/jitdump-python.sh line 20
Quoted 6 lines
tools/perf/tests/shell/jitdump-python.sh new file +6
-0+ +_PID=$(perf report -i "${PERF_DATA}" -F pid -q -g none | cut -d: -f1 -s) +PID=$(echo -n $_PID) # remove newlines + +echo "Generate JIT-ed DSOs using perf inject" +perf inject -i "${PERF_DATA}" -j -o "${PERF_DATA}.jit"
Thomas Richter did a workaround in the Java version to avoid debuginfod queries:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/tests/shell/test_java_symbol.sh?h=perf-tools-next#n59
Quoted 7 lines
tools/perf/tests/shell/jitdump-python.sh new file +7
-0+ +echo "Add JIT-ed DSOs to the build-ID cache" +for F in /tmp/jitted-${PID}-*.so; do + perf buildid-cache -a "${F}" +done + +echo "Check the symbol containing the script name"
tools/perf/tests/shell/jitdump-python.sh line 37
Quoted 10 lines
tools/perf/tests/shell/jitdump-python.sh new file +10
-0+NUM=$(perf report -i "${PERF_DATA}.jit" -s sym | grep -c jitdump-test.py) + +echo "Found ${NUM} matching lines" + +echo "Remove JIT-ed DSOs from the build-ID cache" +for F in /tmp/jitted-${PID}-*.so; do + perf buildid-cache -r "${F}" +done + +rm -f ${PERF_DATA} ${PERF_DATA}.jit /tmp/jit-${PID}.dump /tmp/jitted-${PID}-*.so
It'd be nice to have this in a function that also gets run if there is a trap. It looks like the Java test is missing a call to cleanup_files on the success path.
Thanks,
Ian
Quoted 27 lines
tools/perf/tests/shell/jitdump-python.sh new file +4
-0+ +if [ "${NUM}" -eq 0 ]; then + exit 1 +fitools/perf/tests/shell/jitdump-test.py new file +14
-1@@ -0,0 +1,14 @@ +def foo(n): + result = 0 + for _ in range(n): + result += 1 + return result + +def bar(n): + foo(n) + +def baz(n): + bar(n) + +if __name__ == "__main__": + baz(1000000) --2.52.0.rc1.455.g30608eb744-goog
Namhyung Kim
maintainerRe: [PATCH 2/2] perf test: Add python JIT dump test
Quoted 41 lines · Ian Rogers
Quoted 40 lines · Namhyung Kim
Add a test case for the python interpreter like below so that we can make sure it won't break again. To validate the effect of build-ID generation, it adds and removes the JIT'ed DSOs to/from the build-ID cache for the test.
$ perf test -vv jitdump 84: python profiling with jitdump: --- start --- test child forked, pid 214316 Run python with -Xperf_jit [ perf record: Woken up 5 times to write data ] [ perf record: Captured and wrote 1.180 MB /tmp/__perf_test.perf.data.XbqZNm (140 samples) ] Generate JIT-ed DSOs using perf inject Add JIT-ed DSOs to the build-ID cache Check the symbol containing the script name Found 108 matching lines Remove JIT-ed DSOs from the build-ID cache ---- end(0) ---- 84: python profiling with jitdump : OkCc Pablo Galindo <[email protected]>Signed-off-by Namhyung Kim <[email protected]>--- tools/perf/tests/shell/jitdump-python.sh | 41 ++++++++++++++++++++++++ tools/perf/tests/shell/jitdump-test.py | 14 ++++++++ 2 files changed, 55 insertions(+) create mode 100755 tools/perf/tests/shell/jitdump-python.sh create mode 100644 tools/perf/tests/shell/jitdump-test.pytools/perf/tests/shell/jitdump-python.sh new file +3
-0@@ -0,0 +1,41 @@ +#!/bin/bash +# python profiling with jitdump +# SPDX-License-Identifier: GPL-2.0
Quoted 8 lines
tools/perf/tests/shell/jitdump-python.sh line 8
Quoted 5 lines
tools/perf/tests/shell/jitdump-python.sh new file +5
-0+ +if ! command -v python > /dev/null; then + echo "Skip: no python found" + exit 2 +fiCan this just reuse the python set up logic in:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/tests/shell/lib/setup_python.sh?h=perf-tools-next
Will do!
Quoted 11 lines
tools/perf/tests/shell/jitdump-python.sh line 14
Quoted 7 lines
tools/perf/tests/shell/jitdump-python.sh new file +6
-0+ +SHELLDIR=$(dirname $0) +PERF_DATA=$(mktemp /tmp/__perf_test.perf.data.XXXXXX) + +echo "Run python with -Xperf_jit" +perf record -k 1 -g --call-graph dwarf -o "${PERF_DATA}" -- python -Xperf_jit ${SHELLDIR}/jitdump-test.pyWe can avoid an additional file by using the REPL, this is why the java test is using jshell:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/tests/shell/test_java_symbol.sh?h=perf-tools-next#n41
I thought that too, but I wanted a filename to check the result. Hmm.. probably I can use a different pattern without it.
Quoted 10 lines
tools/perf/tests/shell/jitdump-python.sh line 20
Quoted 7 lines
tools/perf/tests/shell/jitdump-python.sh new file +6
-0+ +_PID=$(perf report -i "${PERF_DATA}" -F pid -q -g none | cut -d: -f1 -s) +PID=$(echo -n $_PID) # remove newlines + +echo "Generate JIT-ed DSOs using perf inject" +perf inject -i "${PERF_DATA}" -j -o "${PERF_DATA}.jit"Thomas Richter did a workaround in the Java version to avoid debuginfod queries:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/tests/shell/test_java_symbol.sh?h=perf-tools-next#n59
Yep, will add.
Quoted 14 lines
Quoted 14 lines
tools/perf/tests/shell/jitdump-python.sh new file +13
-0+ +echo "Add JIT-ed DSOs to the build-ID cache" +for F in /tmp/jitted-${PID}-*.so; do + perf buildid-cache -a "${F}" +done + +echo "Check the symbol containing the script name" +NUM=$(perf report -i "${PERF_DATA}.jit" -s sym | grep -c jitdump-test.py) + +echo "Found ${NUM} matching lines" + +echo "Remove JIT-ed DSOs from the build-ID cache" +for F in /tmp/jitted-${PID}-*.so; do
Quoted 8 lines
tools/perf/tests/shell/jitdump-python.sh line 37
Quoted 4 lines
tools/perf/tests/shell/jitdump-python.sh new file +4
-0+ perf buildid-cache -r "${F}" +done + +rm -f ${PERF_DATA} ${PERF_DATA}.jit /tmp/jit-${PID}.dump /tmp/jitted-${PID}-*.soIt'd be nice to have this in a function that also gets run if there is a trap. It looks like the Java test is missing a call to
cleanup_fileson the success path.
I thought that it will go the end of the script regardless of any command failures but we should protect it from Ctrl-C as well.. ok.
Thanks for your review!
Namhyung
Quoted 28 lines
Quoted 28 lines
tools/perf/tests/shell/jitdump-python.sh new file +4
-0+ +if [ "${NUM}" -eq 0 ]; then + exit 1 +fitools/perf/tests/shell/jitdump-test.py new file +14
-1@@ -0,0 +1,14 @@ +def foo(n): + result = 0 + for _ in range(n): + result += 1 + return result + +def bar(n): + foo(n) + +def baz(n): + bar(n) + +if __name__ == "__main__": + baz(1000000) --2.52.0.rc1.455.g30608eb744-goog
Re: [PATCH 2/2] perf test: Add python JIT dump test
In reply to Namhyung KimOne important aspect to have in mind here is that Python 3.13 was the first version to add Perf JIT support. In older versions -Xperf_jit is a noop. You may want to check for the version in your test and skip if the running version of Python is lower or alternatively ensure you always run with 3.13+.
Quoted 6 lines
tools/perf/tests/shell/jitdump-python.sh new file +6
-0+ +SHELLDIR=$(dirname $0) +PERF_DATA=$(mktemp /tmp/__perf_test.perf.data.XXXXXX) + +echo "Run python with -Xperf_jit" +perf record -k 1 -g --call-graph dwarf -o "${PERF_DATA}" -- python -Xperf_jit ${SHELLDIR}/jitdump-test.py
Quoted 101 lines · Namhyung Kim
Add a test case for the python interpreter like below so that we can make sure it won't break again. To validate the effect of build-ID generation, it adds and removes the JIT'ed DSOs to/from the build-ID cache for the test.
$ perf test -vv jitdump 84: python profiling with jitdump: --- start --- test child forked, pid 214316 Run python with -Xperf_jit [ perf record: Woken up 5 times to write data ] [ perf record: Captured and wrote 1.180 MB /tmp/__perf_test.perf.data.XbqZNm (140 samples) ] Generate JIT-ed DSOs using perf inject Add JIT-ed DSOs to the build-ID cache Check the symbol containing the script name Found 108 matching lines Remove JIT-ed DSOs from the build-ID cache ---- end(0) ---- 84: python profiling with jitdump : OkCc Pablo Galindo <[email protected]>Signed-off-by Namhyung Kim <[email protected]>--- tools/perf/tests/shell/jitdump-python.sh | 41 ++++++++++++++++++++++++ tools/perf/tests/shell/jitdump-test.py | 14 ++++++++ 2 files changed, 55 insertions(+) create mode 100755 tools/perf/tests/shell/jitdump-python.sh create mode 100644 tools/perf/tests/shell/jitdump-test.pytools/perf/tests/shell/jitdump-python.sh new file +41
-0@@ -0,0 +1,41 @@ +#!/bin/bash +# python profiling with jitdump +# SPDX-License-Identifier: GPL-2.0 + +if ! command -v python > /dev/null; then + echo "Skip: no python found" + exit 2 +fi + +SHELLDIR=$(dirname $0) +PERF_DATA=$(mktemp /tmp/__perf_test.perf.data.XXXXXX) + +echo "Run python with -Xperf_jit" +perf record -k 1 -g --call-graph dwarf -o "${PERF_DATA}" -- python -Xperf_jit ${SHELLDIR}/jitdump-test.py + +_PID=$(perf report -i "${PERF_DATA}" -F pid -q -g none | cut -d: -f1 -s) +PID=$(echo -n $_PID) # remove newlines + +echo "Generate JIT-ed DSOs using perf inject" +perf inject -i "${PERF_DATA}" -j -o "${PERF_DATA}.jit" + +echo "Add JIT-ed DSOs to the build-ID cache" +for F in /tmp/jitted-${PID}-*.so; do + perf buildid-cache -a "${F}" +done + +echo "Check the symbol containing the script name" +NUM=$(perf report -i "${PERF_DATA}.jit" -s sym | grep -c jitdump-test.py) + +echo "Found ${NUM} matching lines" + +echo "Remove JIT-ed DSOs from the build-ID cache" +for F in /tmp/jitted-${PID}-*.so; do + perf buildid-cache -r "${F}" +done + +rm -f ${PERF_DATA} ${PERF_DATA}.jit /tmp/jit-${PID}.dump /tmp/jitted-${PID}-*.so + +if [ "${NUM}" -eq 0 ]; then + exit 1 +fitools/perf/tests/shell/jitdump-test.py new file +14
-1@@ -0,0 +1,14 @@ +def foo(n): + result = 0 + for _ in range(n): + result += 1 + return result + +def bar(n): + foo(n) + +def baz(n): + bar(n) + +if __name__ == "__main__": + baz(1000000) --2.52.0.rc1.455.g30608eb744-goog
Namhyung Kim
maintainerRe: [PATCH 2/2] perf test: Add python JIT dump test
Hi Pablo,
Quoted 5 lines · Pablo Galindo Salgado
One important aspect to have in mind here is that Python 3.13 was the first version to add Perf JIT support. In older versions -Xperf_jit is a noop. You may want to check for the version in your test and skip if the running version of Python is lower or alternatively ensure you always run with 3.13+.
Good point. Is there a better way to check if -Xperf_jit is available or not other than reading the version number?
Thanks,
Namhyung
Re: [PATCH 2/2] perf test: Add python JIT dump test
The version is likely the more reliable method, but you could use sys.is_stack_trampoline_active()
in combination with the X option. For example:
$ python -c 'import sys; print(sys.is_stack_trampoline_active())' False $ python -Xperf_jit -c 'import sys; print(sys.is_stack_trampoline_active())' True
If you get False even when passing Xperf_jit, then it is not supported.
Quoted 2 lines
Good point. Is there a better way to check if -Xperf_jit is available or not other than reading the version number?
Namhyung Kim
maintainerRe: [PATCH 2/2] perf test: Add python JIT dump test
Quoted 3 lines · Pablo Galindo Salgado
The version is likely the more reliable method, but you could use
sys.is_stack_trampoline_active()
in combination with the X option. For example:
My concern is platform support. Is it available on ARM and others?
Quoted 7 lines
$ python -c 'import sys; print(sys.is_stack_trampoline_active())' False $ python -Xperf_jit -c 'import sys; print(sys.is_stack_trampoline_active())' TrueIf you get False even when passing
Xperf_jit, then it is not supported.
I like this. I guess it'll return False on unsupported archs.
Thanks,
Namhyung
Re: [PATCH 2/2] perf test: Add python JIT dump test
Quoted 1 line
My concern is platform support. Is it available on ARM and others?
Currently, it's available in x86-64, ARM64 (little endian, 64-bit ABI only) and RISC-V
Quoted 1 line
I like this. I guess it'll return False on unsupported archs.
Pablo
Ian Rogers
reviewerQuoted 9 lines · Namhyung Kim
It was reported that python backtrace with JIT dump was broken after the change to built-in SHA-1 implementation. It seems python generates the same JIT code for each function. They will become separate DSOs but the contents are the same. Only difference is in the symbol name.
But this caused a problem that every JIT'ed DSOs will have the same build-ID which makes perf confused. And it resulted in no python symbols (from JIT) in the output.
The lookup of a DSO involves the build ID and the filename. I'm confused as to why things weren't deduplicated and why no symbols rather than repeatedly the same symbol?
Quoted 4 lines
Looking back at the original code before the conversion, it used the
load_addras well as the code section to distinguish each DSO. I think we should do the same or use symbol table as an additional input for SHA-1.
Hmm.. the build ID for the contents of the code should be a constant. As the build ID is a note for the entire ELF file then something is wrong with the filename handling it seems.
Thanks,
Ian
Quoted 38 lines
This patch is a quick-and-dirty fix just to add each byte of the
load_addrto the first 8 bytes of SHA-1 result. Probably we need to addsha1_update()or similar to update the existing hash value and use it here. I'd like something that can be backported to the stable trees easily.Fixes e3f612c1d8f3945b ("perf genelf: Remove libcrypto dependency and use built-in sha1()")Cc Eric Biggers <[email protected]>Cc Pablo Galindo <[email protected]>Signed-off-by Namhyung Kim <[email protected]>--- tools/perf/util/genelf.c | 9 +++++++++ 1 file changed, 9 insertions(+)tools/perf/util/genelf.c +9
-1@@ -395,6 +395,15 @@ jit_write_elf(int fd, uint64_t load_addr __maybe_unused, const char *sym,* build-id generation */ sha1 (code, csize, bnote.build_id); +/* FIXME: update the SHA-1 hash using additional contents */ + bnote.build_id[0 ] += (load_addr >>0 ) &0xff ; + bnote.build_id[1 ] += (load_addr >>8 ) &0xff ; + bnote.build_id[2 ] += (load_addr >>16 ) &0xff ; + bnote.build_id[3 ] += (load_addr >>24 ) &0xff ; + bnote.build_id[4 ] += (load_addr >>32 ) &0xff ; + bnote.build_id[5 ] += (load_addr >>40 ) &0xff ; + bnote.build_id[6 ] += (load_addr >>48 ) &0xff ; + bnote.build_id[7 ] += (load_addr >>56 ) &0xff ; bnote.desc.namesz =sizeof (bnote.name);/* must include 0 termination */ bnote.desc.descsz =sizeof (bnote.build_id); bnote.desc.type =NT_GNU_BUILD_ID ; --2.52.0.rc1.455.g30608eb744-goog
Namhyung Kim
maintainerQuoted 7 lines · Ian Rogers
Quoted 6 lines · Namhyung Kim
It was reported that python backtrace with JIT dump was broken after the change to built-in SHA-1 implementation. It seems python generates the same JIT code for each function. They will become separate DSOs but the contents are the same. Only difference is in the symbol name.
Quoted 7 lines
Quoted 3 lines
But this caused a problem that every JIT'ed DSOs will have the same build-ID which makes perf confused. And it resulted in no python symbols (from JIT) in the output.
The lookup of a DSO involves the build ID and the filename. I'm confused as to why things weren't deduplicated and why no symbols rather than repeatedly the same symbol?
I don't know, but that's the symptom in the original bug report in the python github (see Links: below). I guess the behavior is non-deterministic.
Quoted 9 lines
Quoted 5 lines
Looking back at the original code before the conversion, it used the
load_addras well as the code section to distinguish each DSO. I think we should do the same or use symbol table as an additional input for SHA-1.Hmm.. the build ID for the contents of the code should be a constant. As the build ID is a note for the entire ELF file then something is wrong with the filename handling it seems.
When it tries to load symbols from a DSO, it prefer reading from the build-ID cache than the file system since it trusts build-IDs more than the path name. See dso__load() and binary_type_symtab[].
So having multiple DSO's with the same build-ID can be a problem if they are in the build-ID cache. Normally perf inject -j won't add the new JIT-ed DSOs to the build-ID cache but it's still possible.
Thanks,
Namhyung
Quoted 39 lines
Quoted 39 lines
This patch is a quick-and-dirty fix just to add each byte of the
load_addrto the first 8 bytes of SHA-1 result. Probably we need to addsha1_update()or similar to update the existing hash value and use it here. I'd like something that can be backported to the stable trees easily.Fixes e3f612c1d8f3945b ("perf genelf: Remove libcrypto dependency and use built-in sha1()")Cc Eric Biggers <[email protected]>Cc Pablo Galindo <[email protected]>Signed-off-by Namhyung Kim <[email protected]>--- tools/perf/util/genelf.c | 9 +++++++++ 1 file changed, 9 insertions(+)tools/perf/util/genelf.c +9
-1@@ -395,6 +395,15 @@ jit_write_elf(int fd, uint64_t load_addr __maybe_unused, const char *sym,* build-id generation */ sha1 (code, csize, bnote.build_id); +/* FIXME: update the SHA-1 hash using additional contents */ + bnote.build_id[0 ] += (load_addr >>0 ) &0xff ; + bnote.build_id[1 ] += (load_addr >>8 ) &0xff ; + bnote.build_id[2 ] += (load_addr >>16 ) &0xff ; + bnote.build_id[3 ] += (load_addr >>24 ) &0xff ; + bnote.build_id[4 ] += (load_addr >>32 ) &0xff ; + bnote.build_id[5 ] += (load_addr >>40 ) &0xff ; + bnote.build_id[6 ] += (load_addr >>48 ) &0xff ; + bnote.build_id[7 ] += (load_addr >>56 ) &0xff ; bnote.desc.namesz =sizeof (bnote.name);/* must include 0 termination */ bnote.desc.descsz =sizeof (bnote.build_id); bnote.desc.type =NT_GNU_BUILD_ID ; --2.52.0.rc1.455.g30608eb744-goog
Ian Rogers
reviewerQuoted 31 lines · Namhyung Kim
Quoted 7 lines · Ian Rogers
Quoted 6 lines · Namhyung Kim
It was reported that python backtrace with JIT dump was broken after the change to built-in SHA-1 implementation. It seems python generates the same JIT code for each function. They will become separate DSOs but the contents are the same. Only difference is in the symbol name.
Quoted 7 lines
Quoted 3 lines
But this caused a problem that every JIT'ed DSOs will have the same build-ID which makes perf confused. And it resulted in no python symbols (from JIT) in the output.
The lookup of a DSO involves the build ID and the filename. I'm confused as to why things weren't deduplicated and why no symbols rather than repeatedly the same symbol?
I don't know, but that's the symptom in the original bug report in the python github (see Links: below). I guess the behavior is non-deterministic.
Quoted 9 lines
Quoted 5 lines
Looking back at the original code before the conversion, it used the
load_addras well as the code section to distinguish each DSO. I think we should do the same or use symbol table as an additional input for SHA-1.Hmm.. the build ID for the contents of the code should be a constant. As the build ID is a note for the entire ELF file then something is wrong with the filename handling it seems.
Quoted 7 lines
When it tries to load symbols from a DSO, it prefer reading from the build-ID cache than the file system since it trusts build-IDs more than the path name. See
dso__load()and binary_type_symtab[].So having multiple DSO's with the same build-ID can be a problem if they are in the build-ID cache. Normally
perf inject -jwon't add the new JIT-ed DSOs to the build-ID cache but it's still possible.
+Fangrui
I'm surprised that build IDs don't include symbol names but:
```
$ cat a.s
.text
.global main
.global foo
main:
foo:
ret
$ cat b.s
.text
.global main
.global bar
main:
bar:
ret
$ gcc -Wl,--build-id a.s -o a.out
$ gcc -Wl,--build-id b.s -o b.out
$ readelf -n a.out
...
Build ID: 9dd0371b953db5d72929af5d98552e4ee1043616
...
$ readelf -n b.out
...
Build ID: 9dd0371b953db5d72929af5d98552e4ee1043616
...
```so ugh. Perhaps we need to have jitdump make a single object file (and so 1 build ID) but with multiple unique symbols.
Thanks,
Ian
Quoted 42 lines
Thanks,
NamhyungQuoted 39 lines
Quoted 39 lines
This patch is a quick-and-dirty fix just to add each byte of the
load_addrto the first 8 bytes of SHA-1 result. Probably we need to addsha1_update()or similar to update the existing hash value and use it here. I'd like something that can be backported to the stable trees easily.Fixes e3f612c1d8f3945b ("perf genelf: Remove libcrypto dependency and use built-in sha1()")Cc Eric Biggers <[email protected]>Cc Pablo Galindo <[email protected]>Signed-off-by Namhyung Kim <[email protected]>--- tools/perf/util/genelf.c | 9 +++++++++ 1 file changed, 9 insertions(+)tools/perf/util/genelf.c +9
-1@@ -395,6 +395,15 @@ jit_write_elf(int fd, uint64_t load_addr __maybe_unused, const char *sym,* build-id generation */ sha1 (code, csize, bnote.build_id); +/* FIXME: update the SHA-1 hash using additional contents */ + bnote.build_id[0 ] += (load_addr >>0 ) &0xff ; + bnote.build_id[1 ] += (load_addr >>8 ) &0xff ; + bnote.build_id[2 ] += (load_addr >>16 ) &0xff ; + bnote.build_id[3 ] += (load_addr >>24 ) &0xff ; + bnote.build_id[4 ] += (load_addr >>32 ) &0xff ; + bnote.build_id[5 ] += (load_addr >>40 ) &0xff ; + bnote.build_id[6 ] += (load_addr >>48 ) &0xff ; + bnote.build_id[7 ] += (load_addr >>56 ) &0xff ; bnote.desc.namesz =sizeof (bnote.name);/* must include 0 termination */ bnote.desc.descsz =sizeof (bnote.build_id); bnote.desc.type =NT_GNU_BUILD_ID ; --2.52.0.rc1.455.g30608eb744-goog
Namhyung Kim
maintainerQuoted 69 lines · Ian Rogers
Quoted 31 lines · Namhyung Kim
Quoted 7 lines · Ian Rogers
Quoted 6 lines · Namhyung Kim
It was reported that python backtrace with JIT dump was broken after the change to built-in SHA-1 implementation. It seems python generates the same JIT code for each function. They will become separate DSOs but the contents are the same. Only difference is in the symbol name.
Quoted 7 lines
Quoted 3 lines
But this caused a problem that every JIT'ed DSOs will have the same build-ID which makes perf confused. And it resulted in no python symbols (from JIT) in the output.
The lookup of a DSO involves the build ID and the filename. I'm confused as to why things weren't deduplicated and why no symbols rather than repeatedly the same symbol?
I don't know, but that's the symptom in the original bug report in the python github (see Links: below). I guess the behavior is non-deterministic.
Quoted 9 lines
Quoted 5 lines
Looking back at the original code before the conversion, it used the
load_addras well as the code section to distinguish each DSO. I think we should do the same or use symbol table as an additional input for SHA-1.Hmm.. the build ID for the contents of the code should be a constant. As the build ID is a note for the entire ELF file then something is wrong with the filename handling it seems.
Quoted 7 lines
When it tries to load symbols from a DSO, it prefer reading from the build-ID cache than the file system since it trusts build-IDs more than the path name. See
dso__load()and binary_type_symtab[].So having multiple DSO's with the same build-ID can be a problem if they are in the build-ID cache. Normally
perf inject -jwon't add the new JIT-ed DSOs to the build-ID cache but it's still possible.+Fangrui
I'm surprised that build IDs don't include symbol names but:
``` $ cat a.s .text .global main .global foo main: foo: ret $ cat b.s .text .global main .global bar main: bar: ret $ gcc -Wl,--build-id a.s -o a.out $ gcc -Wl,--build-id b.s -o b.out $ readelf -n a.out ... Build ID: 9dd0371b953db5d72929af5d98552e4ee1043616 ... $ readelf -n b.out ... Build ID: 9dd0371b953db5d72929af5d98552e4ee1043616 ... ```
Quoted 2 lines
so ugh. Perhaps we need to have jitdump make a single object file (and so 1 build ID) but with multiple unique symbols.
Right, that'd be better. But I'm afraid some JIT code could spread to many segments so it's not possible to create a map to cover all areas due to conflicts with other libraries.
Thanks,
Namhyung
Ian Rogers
reviewerQuoted 74 lines · Namhyung Kim
Quoted 69 lines · Ian Rogers
Quoted 31 lines · Namhyung Kim
Quoted 7 lines · Ian Rogers
Quoted 6 lines · Namhyung Kim
It was reported that python backtrace with JIT dump was broken after the change to built-in SHA-1 implementation. It seems python generates the same JIT code for each function. They will become separate DSOs but the contents are the same. Only difference is in the symbol name.
Quoted 7 lines
Quoted 3 lines
But this caused a problem that every JIT'ed DSOs will have the same build-ID which makes perf confused. And it resulted in no python symbols (from JIT) in the output.
The lookup of a DSO involves the build ID and the filename. I'm confused as to why things weren't deduplicated and why no symbols rather than repeatedly the same symbol?
I don't know, but that's the symptom in the original bug report in the python github (see Links: below). I guess the behavior is non-deterministic.
Quoted 9 lines
Quoted 5 lines
Looking back at the original code before the conversion, it used the
load_addras well as the code section to distinguish each DSO. I think we should do the same or use symbol table as an additional input for SHA-1.Hmm.. the build ID for the contents of the code should be a constant. As the build ID is a note for the entire ELF file then something is wrong with the filename handling it seems.
Quoted 7 lines
When it tries to load symbols from a DSO, it prefer reading from the build-ID cache than the file system since it trusts build-IDs more than the path name. See
dso__load()and binary_type_symtab[].So having multiple DSO's with the same build-ID can be a problem if they are in the build-ID cache. Normally
perf inject -jwon't add the new JIT-ed DSOs to the build-ID cache but it's still possible.+Fangrui
I'm surprised that build IDs don't include symbol names but:
``` $ cat a.s .text .global main .global foo main: foo: ret $ cat b.s .text .global main .global bar main: bar: ret $ gcc -Wl,--build-id a.s -o a.out $ gcc -Wl,--build-id b.s -o b.out $ readelf -n a.out ... Build ID: 9dd0371b953db5d72929af5d98552e4ee1043616 ... $ readelf -n b.out ... Build ID: 9dd0371b953db5d72929af5d98552e4ee1043616 ... ```Quoted 2 lines
so ugh. Perhaps we need to have jitdump make a single object file (and so 1 build ID) but with multiple unique symbols.
Quoted 3 lines
Right, that'd be better. But I'm afraid some JIT code could spread to many segments so it's not possible to create a map to cover all areas due to conflicts with other libraries.
I'm not familiar with any JITs doing this. JITs often have to run on Windows where there is a reserve for the code cache and then commits to actually use individual pages. On Linux you can do a large PROT_NONE mmap and then mprotect pages within this code cache region (we should have perf events on this). Apple has some optimizations with the MAP_JIT mmap flag. SBCL has a code cache within the executable itself. On x86 you could spread code around within +/-2GB, on RISC it'd be a pain due to offset limitations.
Anyway, it sounds like something is off when we are writing the executable and likely having too many of them. There is also something off with dsos__find as it should only give a dso if the name and build ID are matching. The sample should have an IP we turn into a map within maps, the dso of the map should differ for each of the call chain IPs (even though they are < page_size()) and the symbol lookup should work.
If we're not going to maintain that jitdump build IDs behave like build IDs then there is little point using a SHA-1 hash and patch series like:
https://lore.kernel.org/lkml/[email protected]/
are even more tedious to land :-)
Thanks,
Ian
Quoted 70 lines · Ian Rogers
Quoted 31 lines · Namhyung Kim
Quoted 7 lines · Ian Rogers
Quoted 6 lines · Namhyung Kim
It was reported that python backtrace with JIT dump was broken after the change to built-in SHA-1 implementation. It seems python generates the same JIT code for each function. They will become separate DSOs but the contents are the same. Only difference is in the symbol name.
Quoted 7 lines
Quoted 3 lines
But this caused a problem that every JIT'ed DSOs will have the same build-ID which makes perf confused. And it resulted in no python symbols (from JIT) in the output.
The lookup of a DSO involves the build ID and the filename. I'm confused as to why things weren't deduplicated and why no symbols rather than repeatedly the same symbol?
I don't know, but that's the symptom in the original bug report in the python github (see Links: below). I guess the behavior is non-deterministic.
Quoted 9 lines
Quoted 5 lines
Looking back at the original code before the conversion, it used the
load_addras well as the code section to distinguish each DSO. I think we should do the same or use symbol table as an additional input for SHA-1.Hmm.. the build ID for the contents of the code should be a constant. As the build ID is a note for the entire ELF file then something is wrong with the filename handling it seems.
Quoted 7 lines
When it tries to load symbols from a DSO, it prefer reading from the build-ID cache than the file system since it trusts build-IDs more than the path name. See
dso__load()and binary_type_symtab[].So having multiple DSO's with the same build-ID can be a problem if they are in the build-ID cache. Normally
perf inject -jwon't add the new JIT-ed DSOs to the build-ID cache but it's still possible.+Fangrui
I'm surprised that build IDs don't include symbol names but:
``` $ cat a.s .text .global main .global foo main: foo: ret $ cat b.s .text .global main .global bar main: bar: ret $ gcc -Wl,--build-id a.s -o a.out $ gcc -Wl,--build-id b.s -o b.out $ readelf -n a.out ... Build ID: 9dd0371b953db5d72929af5d98552e4ee1043616 ... $ readelf -n b.out ... Build ID: 9dd0371b953db5d72929af5d98552e4ee1043616 ... ```
Quoted 5 lines
so ugh. Perhaps we need to have jitdump make a single object file (and so 1 build ID) but with multiple unique symbols.
Thanks,
Ian
Looks like a GNU ld issue. Other linkers should give different build IDs.
.symtab/.strtab/.shstrtab are special sections and BFD seem to skip
their content for the build ID.
Raised https://sourceware.org/bugzilla/show_bug.cgi?id=33633 ("ld --build-id does not use symtab/strtab content")
Quoted 45 lines
Quoted 45 lines
Thanks,
NamhyungQuoted 39 lines
Quoted 39 lines
This patch is a quick-and-dirty fix just to add each byte of the
load_addrto the first 8 bytes of SHA-1 result. Probably we need to addsha1_update()or similar to update the existing hash value and use it here. I'd like something that can be backported to the stable trees easily.Fixes e3f612c1d8f3945b ("perf genelf: Remove libcrypto dependency and use built-in sha1()")Cc Eric Biggers <[email protected]>Cc Pablo Galindo <[email protected]>Signed-off-by Namhyung Kim <[email protected]>--- tools/perf/util/genelf.c | 9 +++++++++ 1 file changed, 9 insertions(+)tools/perf/util/genelf.c +9
-1@@ -395,6 +395,15 @@ jit_write_elf(int fd, uint64_t load_addr __maybe_unused, const char *sym,* build-id generation */ sha1 (code, csize, bnote.build_id); +/* FIXME: update the SHA-1 hash using additional contents */ + bnote.build_id[0 ] += (load_addr >>0 ) &0xff ; + bnote.build_id[1 ] += (load_addr >>8 ) &0xff ; + bnote.build_id[2 ] += (load_addr >>16 ) &0xff ; + bnote.build_id[3 ] += (load_addr >>24 ) &0xff ; + bnote.build_id[4 ] += (load_addr >>32 ) &0xff ; + bnote.build_id[5 ] += (load_addr >>40 ) &0xff ; + bnote.build_id[6 ] += (load_addr >>48 ) &0xff ; + bnote.build_id[7 ] += (load_addr >>56 ) &0xff ; bnote.desc.namesz =sizeof (bnote.name);/* must include 0 termination */ bnote.desc.descsz =sizeof (bnote.build_id); bnote.desc.type =NT_GNU_BUILD_ID ; --2.52.0.rc1.455.g30608eb744-goog
Ian Rogers
reviewerQuoted 78 lines · Fangrui Song
Quoted 70 lines · Ian Rogers
Quoted 31 lines · Namhyung Kim
Quoted 7 lines · Ian Rogers
Quoted 6 lines · Namhyung Kim
It was reported that python backtrace with JIT dump was broken after the change to built-in SHA-1 implementation. It seems python generates the same JIT code for each function. They will become separate DSOs but the contents are the same. Only difference is in the symbol name.
Quoted 7 lines
Quoted 3 lines
But this caused a problem that every JIT'ed DSOs will have the same build-ID which makes perf confused. And it resulted in no python symbols (from JIT) in the output.
The lookup of a DSO involves the build ID and the filename. I'm confused as to why things weren't deduplicated and why no symbols rather than repeatedly the same symbol?
I don't know, but that's the symptom in the original bug report in the python github (see Links: below). I guess the behavior is non-deterministic.
Quoted 9 lines
Quoted 5 lines
Looking back at the original code before the conversion, it used the
load_addras well as the code section to distinguish each DSO. I think we should do the same or use symbol table as an additional input for SHA-1.Hmm.. the build ID for the contents of the code should be a constant. As the build ID is a note for the entire ELF file then something is wrong with the filename handling it seems.
Quoted 7 lines
When it tries to load symbols from a DSO, it prefer reading from the build-ID cache than the file system since it trusts build-IDs more than the path name. See
dso__load()and binary_type_symtab[].So having multiple DSO's with the same build-ID can be a problem if they are in the build-ID cache. Normally
perf inject -jwon't add the new JIT-ed DSOs to the build-ID cache but it's still possible.+Fangrui
I'm surprised that build IDs don't include symbol names but:
``` $ cat a.s .text .global main .global foo main: foo: ret $ cat b.s .text .global main .global bar main: bar: ret $ gcc -Wl,--build-id a.s -o a.out $ gcc -Wl,--build-id b.s -o b.out $ readelf -n a.out ... Build ID: 9dd0371b953db5d72929af5d98552e4ee1043616 ... $ readelf -n b.out ... Build ID: 9dd0371b953db5d72929af5d98552e4ee1043616 ... ```Quoted 5 lines
so ugh. Perhaps we need to have jitdump make a single object file (and so 1 build ID) but with multiple unique symbols.
Thanks,
Ian
Quoted 6 lines
Looks like a GNU ld issue. Other linkers should give different build IDs.
.symtab/.strtab/.shstrtab are special sections and BFD seem to skip
their content for the build ID.
Raised https://sourceware.org/bugzilla/show_bug.cgi?id=33633 ("ld --build-id does not use symtab/strtab content")
Thanks Fangrui!
I repeated my test above and found for GNU ld and bfd the build IDs didn't vary, but for gold and lld they do vary.
So it looks like in perf we need to update the sha1 calculation:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/genelf.c?h=perf-tools-next#n397
to include at least the symbol information.
I don't know if it is specified how the build ID should be calculated. It seems like it should be a specified thing to stop tampering with files, but I also see each linker produce a different value for my test. I'm wondering if the sha1 calculation should really be including other sections in the jitdump generated ELF files? For example, the .eh_frame could be useful as that has unwinding implications.
Thanks,
Ian
Quoted 45 lines
Quoted 45 lines
Quoted 45 lines
Thanks,
NamhyungQuoted 39 lines
Quoted 39 lines
This patch is a quick-and-dirty fix just to add each byte of the
load_addrto the first 8 bytes of SHA-1 result. Probably we need to addsha1_update()or similar to update the existing hash value and use it here. I'd like something that can be backported to the stable trees easily.Fixes e3f612c1d8f3945b ("perf genelf: Remove libcrypto dependency and use built-in sha1()")Cc Eric Biggers <[email protected]>Cc Pablo Galindo <[email protected]>Signed-off-by Namhyung Kim <[email protected]>--- tools/perf/util/genelf.c | 9 +++++++++ 1 file changed, 9 insertions(+)tools/perf/util/genelf.c +9
-1@@ -395,6 +395,15 @@ jit_write_elf(int fd, uint64_t load_addr __maybe_unused, const char *sym,* build-id generation */ sha1 (code, csize, bnote.build_id); +/* FIXME: update the SHA-1 hash using additional contents */ + bnote.build_id[0 ] += (load_addr >>0 ) &0xff ; + bnote.build_id[1 ] += (load_addr >>8 ) &0xff ; + bnote.build_id[2 ] += (load_addr >>16 ) &0xff ; + bnote.build_id[3 ] += (load_addr >>24 ) &0xff ; + bnote.build_id[4 ] += (load_addr >>32 ) &0xff ; + bnote.build_id[5 ] += (load_addr >>40 ) &0xff ; + bnote.build_id[6 ] += (load_addr >>48 ) &0xff ; + bnote.build_id[7 ] += (load_addr >>56 ) &0xff ; bnote.desc.namesz =sizeof (bnote.name);/* must include 0 termination */ bnote.desc.descsz =sizeof (bnote.build_id); bnote.desc.type =NT_GNU_BUILD_ID ; --2.52.0.rc1.455.g30608eb744-goog