perf test for inet_pton probe fails to match stacktrace
Hi Arnaldo, Namhyung,
I've had this test fail for me since a while, so I'm reporting below in case we can find a good solution to fix it on my system without breaking other OSs. The test that fails is this:
gentoo ~ # perf test -vF inet_pton
--- start ---
ping 303105 [029] 47754.974977: probe_libc:inet_pton: (7f70ca48feb7)7f70ca48feb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)7f70ca4a2517 gaih_inet+0x112 (/usr/lib64/libc.so.6)
FAIL: expected backtrace entry ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" got "7f70ca4a2517 gaih_inet+0x112 (/usr/lib64/libc.so.6)" ---- end ----
92: probe libc's inet_pton & backtrace it with ping : FAILED!
gentoo ~ # perf version
perf version 6.12
Here's why it fails. My stack traces has more than the test expects:
gentoo ~ # perf script | tac | sed -e /^ping/q | tac
ping 267392 [010] 42482.886216: probe_libc:inet_pton: (7f168d9b4eb7)
7f168d9b4eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
7f168d9c7517 gaih_inet+0x112 (/usr/lib64/libc.so.6)
7f168d9c8b73 getaddrinfo+0x133 (/usr/lib64/libc.so.6)
55947171a518 [unknown] (/usr/bin/ping)
7f168d8d6a72 __libc_start_call_main+0x7a (/usr/lib64/libc.so.6)
7f168d8d6b29 __libc_start_main@@GLIBC_2.34+0x84 (/usr/lib64/libc.so.6)
55947171ac55 [unknown] (/usr/bin/ping)Likely this is due to how I compile glibc:
gentoo ~ # cat /var/db/pkg/sys-libs/glibc-2.40-r5/CFLAGS
-Og -march=native -Wl,--build-id -gdwarf-4 -fno-builtin-strlen -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer
If you could suggest a way to fix this, I'd appreciate it.
Best regards,
-Guilherme
Hi Guilherme,
On Fri, Nov 22, 2024 at 10:35:16PM +0100, Guilherme Amadio wrote:
[...]
Quoted 29 lines
Hi Arnaldo, Namhyung,
I've had this test fail for me since a while, so I'm reporting below in case we can find a good solution to fix it on my system without breaking other OSs. The test that fails is this:
gentoo ~ # perf test -vF
inet_pton
--- start ---
ping 303105 [029] 47754.974977: probe_libc:inet_pton: (7f70ca48feb7)7f70ca48feb7__GI___inet_pton+0x0(/usr/lib64/libc.so.6)7f70ca4a2517gaih_inet+0x112(/usr/lib64/libc.so.6)
FAIL: expected backtrace entry ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" got "7f70ca4a2517gaih_inet+0x112(/usr/lib64/libc.so.6)" ---- end ----
92: probe libc'sinet_pton& backtrace it with ping : FAILED!
gentoo ~ # perf version
perf version 6.12Here's why it fails. My stack traces has more than the test expects:
gentoo ~ # perf script | tac | sed -e /^ping/q | tac
ping 267392 [010] 42482.886216: probe_libc:inet_pton: (7f168d9b4eb7)7f168d9b4eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6) 7f168d9c7517 gaih_inet+0x112 (/usr/lib64/libc.so.6) 7f168d9c8b73 getaddrinfo+0x133 (/usr/lib64/libc.so.6) 55947171a518 [unknown] (/usr/bin/ping) 7f168d8d6a72 __libc_start_call_main+0x7a (/usr/lib64/libc.so.6) 7f168d8d6b29 __libc_start_main@@GLIBC_2.34+0x84 (/usr/lib64/libc.so.6) 55947171ac55 [unknown] (/usr/bin/ping)
Quoted 3 lines
Likely this is due to how I compile glibc:
gentoo ~ # cat/var/db/pkg/sys-libs/glibc-2.40-r5/CFLAGS
-Og -march=native -Wl,--build-id -gdwarf-4 -fno-builtin-strlen -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer
This is not the first reported failure on this case, see [1] for previous one.
I assume you are working on x86_64 machine. Could you confirm if the change below can fix your issue? If yes, I will send a formal patch.
Thanks,
Leo
---8<--
tools/perf/tests/shell/record+probe_libc_inet_pton.sh +9 -0
@@ -54,6 +54,15 @@ trace_libc_inet_pton_backtrace() { ;; *) eventattr='max-stack=3' + # Add gaih_inet to expected backtrace only if it is part of libc. + if nm -D $libc | grep -F -q gaih_inet; then + echo "gaih_inet\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected + fi + # The new version libc (e.g. libc 2.36) contains symbol getaddrinfo@@GLIBC_2.17 + # which is not used. Do strict checking for 'getaddrinfo' without any suffix. + if nm -D $libc | grep -q 'getaddrinfo$'; then + echo "getaddrinfo\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected + fi echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected ;; esac
[1] https://lore.kernel.org/linux-perf-users/[email protected]/
Quoted 6 lines
If you could suggest a way to fix this, I'd appreciate it.
Best regards,
-Guilherme
Dear Leo,
Quoted 4 lines · Leo Yan
Hi Guilherme,
On Fri, Nov 22, 2024 at 10:35:16PM +0100, Guilherme Amadio wrote:
Quoted 8 lines
[...]
Quoted 1 line
[...]
This is not the first reported failure on this case, see [1] for previous one.
I assume you are working on
x86_64machine. Could you confirm if the change below can fix your issue? If yes, I will send a formal patch.
Indeed, I am working on an AMD 3950X, and have glibc 2.40 installed. The patch you sent didn't work out of the box, but with minor modifications I could fix it. I just needed to replace nm -D $libc with nm $libc for it work. The reason is that gaih_inet doesn't show with nm -D for my glibc:
$ nm -D /usr/lib64/libc.so.6 | grep getaddrinfo 0000000000116a40 T getaddrinfo@@GLIBC_2.2.5 000000000010c140 T getaddrinfo_a@GLIBC_2.2.5 000000000010c140 T getaddrinfo_a@@GLIBC_2.34 $ nm /usr/lib64/libc.so.6 | grep getaddrinfo 0000000000116a40 t __GI_getaddrinfo 000000000010c140 t __getaddrinfo_a 0000000000115e53 t __libc_getaddrinfo_freemem 0000000000116a40 T getaddrinfo 000000000010c140 T getaddrinfo_a@@GLIBC_2.34 000000000010c140 T getaddrinfo_a@GLIBC_2.2.5
$ nm -D /usr/lib64/libc.so.6 | grep gaih_inet $ nm /usr/lib64/libc.so.6 | grep gaih_inet 0000000000115405 t gaih_inet 00000000001139b2 t gaih_inet_serv 0000000000177780 r gaih_inet_typeproto
Using nm $libc in both cases, and increasing max-stack to 4 allows the test to pass:
$ sudo ./perf test -vF pton --- start --- ping 1237113 [000] 18561.204983: probe_libc:inet_pton: (7f7c34d38eb7) 7f7c34d38eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6) 7f7c34d4b517 gaih_inet+0x112 (/usr/lib64/libc.so.6) 7f7c34d4cb73 getaddrinfo+0x133 (/usr/lib64/libc.so.6) 55680ac46518 [unknown] (/usr/bin/ping) ---- end ---- 86: probe libc's inet_pton & backtrace it with ping : Ok
The final patch I had to apply is below:
perf $ git diff
tools/perf/tests/shell/record+probe_libc_inet_pton.sh +10 -1
@@ -53,7 +53,16 @@ trace_libc_inet_pton_backtrace() { echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected ;; *) - eventattr='max-stack=3' + eventattr='max-stack=4' + # Add gaih_inet to expected backtrace only if it is part of libc. + if nm $libc | grep -F -q gaih_inet; then + echo "gaih_inet\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected + fi + # The new version libc (e.g. libc 2.36) contains symbol getaddrinfo@@GLIBC_2.17 + # which is not used. Do strict checking for 'getaddrinfo' without any suffix. + if nm $libc | grep -q 'getaddrinfo$'; then + echo "getaddrinfo\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected + fi echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected ;; esac
Please feel free to add Tested-by: and/or Reviewed-by: Guilherme Amadio <[email protected]> to the final patch as you see fit. Thank you!
Best regards,
-Guilherme
Quoted 35 lines
Thanks,
Leo---8<--
tools/perf/tests/shell/record+probe_libc_inet_pton.sh +9
-0@@ -54,6 +54,15 @@ trace_libc_inet_pton_backtrace() { ;; *) eventattr='max-stack=3' + # Add gaih_inet to expected backtrace only if it is part of libc. + if nm -D $libc | grep -F -q gaih_inet; then + echo "gaih_inet\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected + fi + # The new version libc (e.g. libc 2.36) contains symbol getaddrinfo@@GLIBC_2.17 + # which is not used. Do strict checking for 'getaddrinfo' without any suffix. + if nm -D $libc | grep -q 'getaddrinfo$'; then + echo "getaddrinfo\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected + fi echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected ;; esac[1] https://lore.kernel.org/linux-perf-users/[email protected]/
Quoted 6 lines
If you could suggest a way to fix this, I'd appreciate it.
Best regards,
-Guilherme
Namhyung Kim
maintainerHello guys,
Quoted 12 lines · Guilherme Amadio
Dear Leo,
Quoted 9 lines · Leo Yan
Hi Guilherme,
On Fri, Nov 22, 2024 at 10:35:16PM +0100, Guilherme Amadio wrote:
[...]
Quoted 1 line
[...]
This is not the first reported failure on this case, see [1] for previous one.
Right, and I'm also seeing the failure now.
Quoted 29 lines
Quoted 3 lines
I assume you are working on
x86_64machine. Could you confirm if the change below can fix your issue? If yes, I will send a formal patch.Indeed, I am working on an AMD 3950X, and have glibc 2.40 installed. The patch you sent didn't work out of the box, but with minor modifications I could fix it. I just needed to replace nm -D $libc with nm $libc for it work. The reason is that
gaih_inetdoesn't show with nm -D for my glibc:$ nm -D /usr/lib64/libc.so.6 | grep getaddrinfo 0000000000116a40 T getaddrinfo@@GLIBC_2.2.5 000000000010c140 T getaddrinfo_a@GLIBC_2.2.5 000000000010c140 T getaddrinfo_a@@GLIBC_2.34 $ nm /usr/lib64/libc.so.6 | grep getaddrinfo 0000000000116a40 t __GI_getaddrinfo 000000000010c140 t __getaddrinfo_a 0000000000115e53 t __libc_getaddrinfo_freemem 0000000000116a40 T getaddrinfo 000000000010c140 T getaddrinfo_a@@GLIBC_2.34 000000000010c140 T getaddrinfo_a@GLIBC_2.2.5$ nm -D /usr/lib64/libc.so.6 | grep gaih_inet $ nm /usr/lib64/libc.so.6 | grep gaih_inet 0000000000115405 t gaih_inet 00000000001139b2 t gaih_inet_serv 0000000000177780 r gaih_inet_typeprotoUsing nm $libc in both cases, and increasing max-stack to 4 allows the test to pass:
Quoted 9 lines
$ sudo ./perf test -vF pton --- start --- ping 1237113 [000] 18561.204983: probe_libc:inet_pton: (7f7c34d38eb7) 7f7c34d38eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6) 7f7c34d4b517 gaih_inet+0x112 (/usr/lib64/libc.so.6) 7f7c34d4cb73 getaddrinfo+0x133 (/usr/lib64/libc.so.6) 55680ac46518 [unknown] (/usr/bin/ping) ---- end ---- 86: probe libc's inet_pton & backtrace it with ping : Ok
I think it's very fragile as it depends on specific entries in the call stack. As we want to test if perf probe (for uprobes) and perf record working correctly, maybe it can just check the number of samples only.
Arnaldo, what do you think?
Thanks,
Namhyung
Quoted 28 lines
The final patch I had to apply is below:
perf $ git diff
tools/perf/tests/shell/record+probe_libc_inet_pton.sh +10
-1@@ -53,7 +53,16 @@ trace_libc_inet_pton_backtrace() { echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected ;; *) - eventattr='max-stack=3' + eventattr='max-stack=4' + # Add gaih_inet to expected backtrace only if it is part of libc. + if nm $libc | grep -F -q gaih_inet; then + echo "gaih_inet\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected + fi + # The new version libc (e.g. libc 2.36) contains symbol getaddrinfo@@GLIBC_2.17 + # which is not used. Do strict checking for 'getaddrinfo' without any suffix. + if nm $libc | grep -q 'getaddrinfo$'; then + echo "getaddrinfo\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected + fi echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected ;; esac
Quoted 2 lines
Please feel free to add Tested-by: and/or Reviewed-by: Guilherme Amadio <[email protected]> to the final patch as you see fit. Thank you!
Thanks for the report and the fix.
Namhyung
On Mon, Nov 25, 2024 at 09:08:15PM -0800, Namhyung Kim wrote:
[...]
Quoted 12 lines
Quoted 11 lines
Using nm $libc in both cases, and increasing max-stack to 4 allows the test to pass:
$ sudo ./perf test -vF pton --- start --- ping 1237113 [000] 18561.204983: probe_libc:inet_pton: (7f7c34d38eb7) 7f7c34d38eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6) 7f7c34d4b517 gaih_inet+0x112 (/usr/lib64/libc.so.6) 7f7c34d4cb73 getaddrinfo+0x133 (/usr/lib64/libc.so.6) 55680ac46518 [unknown] (/usr/bin/ping) ---- end ---- 86: probe libc's inet_pton & backtrace it with ping : Ok
Quoted 5 lines
I think it's very fragile as it depends on specific entries in the call stack. As we want to test if perf probe (for uprobes) and perf record working correctly, maybe it can just check the number of samples only.
Arnaldo, what do you think?
Though this question is asking Arnaldo, I have a different idea for fixing the test ;)
The middle entries in the callchain are varied on different archs and distro. E.g. Guilherme reported back the '-D' option is not needed but at my side it is required on Arm64 machine with Ubuntu distro.
However, the three entries in the callchain are existed consistently:
ping 1237113 [000] 18561.204983: probe_libc:inet_pton: (7f7c34d38eb7) 7f7c34d38eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
...
55680ac46518 [unknown] (/usr/bin/ping)
We can change the code to only matching these three entries. The change is as below. We can consolidate the code for all archs (I am not 100% sure for x390x so I leave it).
Please let me know if this makes sense for you?
Thanks,
Leo
---8<---
tools/perf/tests/shell/record+probe_libc_inet_pton.sh +16 -16
@@ -43,17 +43,8 @@ trace_libc_inet_pton_backtrace() { echo "((__GI_)?getaddrinfo|text_to_binary_address)\+0x[[:xdigit:]]+[[:space:]]\($libc|inlined\)$" >> $expected echo "(gaih_inet|main)\+0x[[:xdigit:]]+[[:space:]]\(inlined|.*/bin/ping.*\)$" >> $expected ;; - ppc64|ppc64le) - eventattr='max-stack=4' - # Add gaih_inet to expected backtrace only if it is part of libc. - if nm $libc | grep -F -q gaih_inet.; then - echo "gaih_inet.*\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected - fi - echo "getaddrinfo\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected - echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected - ;; *) - eventattr='max-stack=3' + eventattr='max-stack=4' echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected ;; esac
@@ -76,14 +67,23 @@ trace_libc_inet_pton_backtrace() { fi perf script -i $perf_data | tac | grep -m1 ^ping -B9 | tac > $perf_script - exec 3<$perf_script exec 4<$expected - while read line <&3 && read -r pattern <&4; do + while read -r pattern <&4; do + echo "pattern: $pattern" [ -z "$pattern" ] && break - echo $line - echo "$line" | grep -E -q "$pattern" - if [ $? -ne 0 ] ; then - printf "FAIL: expected backtrace entry \"%s\" got \"%s\"\n" "$pattern" "$line" + + exec 3<$perf_script + while read line <&3; do + [ -z "$line" ] && break + echo "line: $line" + echo "$line" | grep -E -q "$pattern" + found=$? + echo "found=$found" + [ $found -eq 0 ] && break + done + + if [ $found -ne 0 ] ; then + printf "FAIL: unfound the expected backtrace entry \"%s\"\n" "$pattern" return 1 fi done
Arnaldo Carvalho de Melo
maintainerQuoted 36 lines · Leo Yan
On Mon, Nov 25, 2024 at 09:08:15PM -0800, Namhyung Kim wrote:
[...]
Quoted 12 lines
Quoted 11 lines
Using nm $libc in both cases, and increasing max-stack to 4 allows the test to pass:
$ sudo ./perf test -vF pton --- start --- ping 1237113 [000] 18561.204983: probe_libc:inet_pton: (7f7c34d38eb7) 7f7c34d38eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6) 7f7c34d4b517 gaih_inet+0x112 (/usr/lib64/libc.so.6) 7f7c34d4cb73 getaddrinfo+0x133 (/usr/lib64/libc.so.6) 55680ac46518 [unknown] (/usr/bin/ping) ---- end ---- 86: probe libc's inet_pton & backtrace it with ping : OkQuoted 5 lines
I think it's very fragile as it depends on specific entries in the call stack. As we want to test if perf probe (for uprobes) and perf record working correctly, maybe it can just check the number of samples only.
Arnaldo, what do you think?
Though this question is asking Arnaldo, I have a different idea for fixing the test ;)
The middle entries in the callchain are varied on different archs and distro. E.g. Guilherme reported back the '-D' option is not needed but at my side it is required on Arm64 machine with Ubuntu distro.
However, the three entries in the callchain are existed consistently:
ping 1237113 [000] 18561.204983: probe_libc:inet_pton: (
7f7c34d38eb7)7f7c34d38eb7__GI___inet_pton+0x0(/usr/lib64/libc.so.6)
...
55680ac46518[unknown] (/usr/bin/ping)
Quoted 8 lines
We can change the code to only matching these three entries. The change is as below. We can consolidate the code for all archs (I am not 100% sure for x390x so I leave it).
Please let me know if this makes sense for you?
Thanks,
Leo
Seems to work:
Before, on a:
root@number:~# grep -m1 'model name' /proc/cpuinfo model name : Intel(R) Core(TM) i7-14700K root@number:~#
root@number:~# perf test -vF pton --- start --- ping 229025 [017] 34027.968348: probe_libc:inet_pton: (7fc4542c10d0) 7fc4542c10d0 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6) 7fc4542d5a55 getaddrinfo+0xcf5 (/usr/lib64/libc.so.6) FAIL: expected backtrace entry ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" got "7fc4542d5a55 getaddrinfo+0xcf5 (/usr/lib64/libc.so.6)" ---- end ---- 86: probe libc's inet_pton & backtrace it with ping : FAILED! root@number:~#
With your patch:
root@number:~# perf test -vF pton --- start --- pattern: ping[][0-9 \.:]+probe_libc:inet_pton: \([[:xdigit:]]+\) line: ping 231118 [023] 34089.948542: probe_libc:inet_pton: (7f635297f0d0) found=0 pattern: .*inet_pton\+0x[[:xdigit:]]+[[:space:]]\(/usr/lib64/libc.so.6|inlined\)$ line: ping 231118 [023] 34089.948542: probe_libc:inet_pton: (7f635297f0d0) found=1 line: 7f635297f0d0 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6) found=0 pattern: .*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$ line: ping 231118 [023] 34089.948542: probe_libc:inet_pton: (7f635297f0d0) found=1 line: 7f635297f0d0 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6) found=1 line: 7f6352993a55 getaddrinfo+0xcf5 (/usr/lib64/libc.so.6) found=1 line: 55ae71919969 [unknown] (/usr/bin/ping) found=0 ---- end ---- 86: probe libc's inet_pton & backtrace it with ping : Ok root@number:~#
Replace that "unfound" with "Didn't find the"
And using 'found' as a variable name that when zero means "Ok, found" and when 1 (grep failed to find the regexp) means "Didn't find" looks confusing.
General idea seems to make the test resilient to the differences while still testing the backtrace in uprobes events for expected entries.
- Arnaldo
Quoted 55 lines
---8<---
tools/perf/tests/shell/record+probe_libc_inet_pton.sh +16
-16@@ -43,17 +43,8 @@ trace_libc_inet_pton_backtrace() { echo "((__GI_)?getaddrinfo|text_to_binary_address)\+0x[[:xdigit:]]+[[:space:]]\($libc|inlined\)$" >> $expected echo "(gaih_inet|main)\+0x[[:xdigit:]]+[[:space:]]\(inlined|.*/bin/ping.*\)$" >> $expected ;; - ppc64|ppc64le) - eventattr='max-stack=4' - # Add gaih_inet to expected backtrace only if it is part of libc. - if nm $libc | grep -F -q gaih_inet.; then - echo "gaih_inet.*\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected - fi - echo "getaddrinfo\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected - echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected - ;; *) - eventattr='max-stack=3' + eventattr='max-stack=4' echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected ;; esac@@ -76,14 +67,23 @@ trace_libc_inet_pton_backtrace() { fi perf script -i $perf_data | tac | grep -m1 ^ping -B9 | tac > $perf_script - exec 3<$perf_script exec 4<$expected - while read line <&3 && read -r pattern <&4; do + while read -r pattern <&4; do + echo "pattern: $pattern" [ -z "$pattern" ] && break - echo $line - echo "$line" | grep -E -q "$pattern" - if [ $? -ne 0 ] ; then - printf "FAIL: expected backtrace entry \"%s\" got \"%s\"\n" "$pattern" "$line" + + exec 3<$perf_script + while read line <&3; do + [ -z "$line" ] && break + echo "line: $line" + echo "$line" | grep -E -q "$pattern" + found=$? + echo "found=$found" + [ $found -eq 0 ] && break + done + + if [ $found -ne 0 ] ; then + printf "FAIL: unfound the expected backtrace entry \"%s\"\n" "$pattern" return 1 fi done
Hi Arnaldo,
On Tue, Nov 26, 2024 at 01:39:39PM -0300, Arnaldo Carvalho de Melo wrote:
[...]
Quoted 43 lines
Seems to work:
Before, on a:
root@number:~# grep -m1 'model name' /proc/cpuinfo model name : Intel(R) Core(TM) i7-14700K root@number:~#root@number:~# perf test -vF pton --- start --- ping 229025 [017] 34027.968348: probe_libc:inet_pton: (7fc4542c10d0) 7fc4542c10d0 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6) 7fc4542d5a55 getaddrinfo+0xcf5 (/usr/lib64/libc.so.6) FAIL: expected backtrace entry ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" got "7fc4542d5a55 getaddrinfo+0xcf5 (/usr/lib64/libc.so.6)" ---- end ---- 86: probe libc's inet_pton & backtrace it with ping : FAILED! root@number:~#With your patch:
root@number:~# perf test -vF pton --- start --- pattern: ping[][0-9 \.:]+probe_libc:inet_pton: \([[:xdigit:]]+\) line: ping 231118 [023] 34089.948542: probe_libc:inet_pton: (7f635297f0d0) found=0 pattern: .*inet_pton\+0x[[:xdigit:]]+[[:space:]]\(/usr/lib64/libc.so.6|inlined\)$ line: ping 231118 [023] 34089.948542: probe_libc:inet_pton: (7f635297f0d0) found=1 line: 7f635297f0d0 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6) found=0 pattern: .*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$ line: ping 231118 [023] 34089.948542: probe_libc:inet_pton: (7f635297f0d0) found=1 line: 7f635297f0d0 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6) found=1 line: 7f6352993a55 getaddrinfo+0xcf5 (/usr/lib64/libc.so.6) found=1 line: 55ae71919969 [unknown] (/usr/bin/ping) found=0 ---- end ---- 86: probe libc's inet_pton & backtrace it with ping : Ok root@number:~#
Quoted 1 line
Replace that "unfound" with "Didn't find the"
Will do.
Quoted 3 lines
And using 'found' as a variable name that when zero means "Ok, found" and when 1 (grep failed to find the regexp) means "Didn't find" looks confusing.
Indeed. Will refine this.
Quoted 2 lines
General idea seems to make the test resilient to the differences while still testing the backtrace in uprobes events for expected entries.
Will send formal patch.
Thanks a lot for the testing,
Leo
Arnaldo Carvalho de Melo
maintainerQuoted 57 lines · Namhyung Kim
Hello guys,
Quoted 12 lines · Guilherme Amadio
Dear Leo,
Quoted 9 lines · Leo Yan
Hi Guilherme,
On Fri, Nov 22, 2024 at 10:35:16PM +0100, Guilherme Amadio wrote:
[...]
Quoted 1 line
[...]
This is not the first reported failure on this case, see [1] for previous one.
Right, and I'm also seeing the failure now.
Quoted 29 lines
Quoted 3 lines
I assume you are working on
x86_64machine. Could you confirm if the change below can fix your issue? If yes, I will send a formal patch.Indeed, I am working on an AMD 3950X, and have glibc 2.40 installed. The patch you sent didn't work out of the box, but with minor modifications I could fix it. I just needed to replace nm -D $libc with nm $libc for it work. The reason is that
gaih_inetdoesn't show with nm -D for my glibc:$ nm -D /usr/lib64/libc.so.6 | grep getaddrinfo 0000000000116a40 T getaddrinfo@@GLIBC_2.2.5 000000000010c140 T getaddrinfo_a@GLIBC_2.2.5 000000000010c140 T getaddrinfo_a@@GLIBC_2.34 $ nm /usr/lib64/libc.so.6 | grep getaddrinfo 0000000000116a40 t __GI_getaddrinfo 000000000010c140 t __getaddrinfo_a 0000000000115e53 t __libc_getaddrinfo_freemem 0000000000116a40 T getaddrinfo 000000000010c140 T getaddrinfo_a@@GLIBC_2.34 000000000010c140 T getaddrinfo_a@GLIBC_2.2.5$ nm -D /usr/lib64/libc.so.6 | grep gaih_inet $ nm /usr/lib64/libc.so.6 | grep gaih_inet 0000000000115405 t gaih_inet 00000000001139b2 t gaih_inet_serv 0000000000177780 r gaih_inet_typeprotoUsing nm $libc in both cases, and increasing max-stack to 4 allows the test to pass:
Quoted 9 lines
$ sudo ./perf test -vF pton --- start --- ping 1237113 [000] 18561.204983: probe_libc:inet_pton: (7f7c34d38eb7) 7f7c34d38eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6) 7f7c34d4b517 gaih_inet+0x112 (/usr/lib64/libc.so.6) 7f7c34d4cb73 getaddrinfo+0x133 (/usr/lib64/libc.so.6) 55680ac46518 [unknown] (/usr/bin/ping) ---- end ---- 86: probe libc's inet_pton & backtrace it with ping : Ok
Quoted 5 lines
I think it's very fragile as it depends on specific entries in the call stack. As we want to test if perf probe (for uprobes) and perf record working correctly, maybe it can just check the number of samples only.
Arnaldo, what do you think?
That would reduce the usefulness of the test as it would be testing fewer aspects of perf's functionality, including backtraces from uprobes in addition to backtraces in general.
So I think Leo has a point in a followup message to this one where he seems to have identified a subset of functions that are present in the architectures we test?
- Arnaldo