Skip to contentSkip to conversation

perf test for inet_pton probe fails to match stacktrace

DISCUSSION Guilherme Amadio 4 participants 8 messages Nov 22, 2024 – Nov 26, 2024 perf events

lore
Share & tools
lore
#0 lore raw

Hi Arnaldo, Namhyung,

I've had this test fail for me since a while, so I'm reporting below in case we can find a good solution to fix it on my system without breaking other OSs. The test that fails is this:

gentoo ~ # perf test -vF inet_pton
--- start ---
ping 303105 [029] 47754.974977: probe_libc:inet_pton: (7f70ca48feb7)
7f70ca48feb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
7f70ca4a2517 gaih_inet+0x112 (/usr/lib64/libc.so.6)
FAIL: expected backtrace entry ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" got "7f70ca4a2517 gaih_inet+0x112 (/usr/lib64/libc.so.6)" ---- end ----
92: probe libc's inet_pton & backtrace it with ping : FAILED!
gentoo ~ # perf version
perf version 6.12

Here's why it fails. My stack traces has more than the test expects:

gentoo ~ # perf script | tac | sed -e /^ping/q | tac
ping 267392 [010] 42482.886216: probe_libc:inet_pton: (7f168d9b4eb7)

            7f168d9b4eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
            7f168d9c7517 gaih_inet+0x112 (/usr/lib64/libc.so.6)
            7f168d9c8b73 getaddrinfo+0x133 (/usr/lib64/libc.so.6)
            55947171a518 [unknown] (/usr/bin/ping)
            7f168d8d6a72 __libc_start_call_main+0x7a (/usr/lib64/libc.so.6)
            7f168d8d6b29 __libc_start_main@@GLIBC_2.34+0x84 (/usr/lib64/libc.so.6)
            55947171ac55 [unknown] (/usr/bin/ping)

Likely this is due to how I compile glibc:
gentoo ~ # cat /var/db/pkg/sys-libs/glibc-2.40-r5/CFLAGS
-Og -march=native -Wl,--build-id -gdwarf-4 -fno-builtin-strlen -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer

If you could suggest a way to fix this, I'd appreciate it.

Best regards,
-Guilherme

#1 lore raw

Hi Guilherme,

On Fri, Nov 22, 2024 at 10:35:16PM +0100, Guilherme Amadio wrote:

[...]

Quoted 29 lines

Hi Arnaldo, Namhyung,

I've had this test fail for me since a while, so I'm reporting below in case we can find a good solution to fix it on my system without breaking other OSs. The test that fails is this:

gentoo ~ # perf test -vF inet_pton
--- start ---
ping 303105 [029] 47754.974977: probe_libc:inet_pton: (7f70ca48feb7)
7f70ca48feb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
7f70ca4a2517 gaih_inet+0x112 (/usr/lib64/libc.so.6)
FAIL: expected backtrace entry ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" got "7f70ca4a2517 gaih_inet+0x112 (/usr/lib64/libc.so.6)" ---- end ----
92: probe libc's inet_pton & backtrace it with ping : FAILED!
gentoo ~ # perf version
perf version 6.12

Here's why it fails. My stack traces has more than the test expects:

gentoo ~ # perf script | tac | sed -e /^ping/q | tac
ping 267392 [010] 42482.886216: probe_libc:inet_pton: (7f168d9b4eb7)

            7f168d9b4eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
            7f168d9c7517 gaih_inet+0x112 (/usr/lib64/libc.so.6)
            7f168d9c8b73 getaddrinfo+0x133 (/usr/lib64/libc.so.6)
            55947171a518 [unknown] (/usr/bin/ping)
            7f168d8d6a72 __libc_start_call_main+0x7a (/usr/lib64/libc.so.6)
            7f168d8d6b29 __libc_start_main@@GLIBC_2.34+0x84 (/usr/lib64/libc.so.6)
            55947171ac55 [unknown] (/usr/bin/ping)
Quoted 3 lines

Likely this is due to how I compile glibc:
gentoo ~ # cat /var/db/pkg/sys-libs/glibc-2.40-r5/CFLAGS
-Og -march=native -Wl,--build-id -gdwarf-4 -fno-builtin-strlen -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer

This is not the first reported failure on this case, see [1] for previous one.

I assume you are working on x86_64 machine. Could you confirm if the change below can fix your issue? If yes, I will send a formal patch.

Thanks,
Leo

---8<--

tools/perf/tests/shell/record+probe_libc_inet_pton.sh +9 -0
index 47a26f25db9f..5a2d641bf166 100755
@@ -54,6 +54,15 @@ trace_libc_inet_pton_backtrace() {
                ;;
        *)
                eventattr='max-stack=3'
+               # Add gaih_inet to expected backtrace only if it is part of libc.
+               if nm -D $libc | grep -F -q gaih_inet; then
+                       echo "gaih_inet\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected
+               fi
+               # The new version libc (e.g. libc 2.36) contains symbol getaddrinfo@@GLIBC_2.17
+               # which is not used. Do strict checking for 'getaddrinfo' without any suffix.
+               if nm -D $libc | grep -q 'getaddrinfo$'; then
+                       echo "getaddrinfo\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected
+               fi
                echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected
                ;;
        esac

[1] https://lore.kernel.org/linux-perf-users/[email protected]/

Quoted 6 lines

If you could suggest a way to fix this, I'd appreciate it.

Best regards,
-Guilherme

#2 lore raw

Dear Leo,

Quoted 4 lines · Leo Yan

Hi Guilherme,

On Fri, Nov 22, 2024 at 10:35:16PM +0100, Guilherme Amadio wrote:

Quoted 8 lines

[...]

Quoted 1 line

[...]

This is not the first reported failure on this case, see [1] for previous one.

I assume you are working on x86_64 machine. Could you confirm if the change below can fix your issue? If yes, I will send a formal patch.

Indeed, I am working on an AMD 3950X, and have glibc 2.40 installed. The patch you sent didn't work out of the box, but with minor modifications I could fix it. I just needed to replace nm -D $libc with nm $libc for it work. The reason is that gaih_inet doesn't show with nm -D for my glibc:

$ nm -D /usr/lib64/libc.so.6 | grep getaddrinfo
0000000000116a40 T getaddrinfo@@GLIBC_2.2.5
000000000010c140 T getaddrinfo_a@GLIBC_2.2.5
000000000010c140 T getaddrinfo_a@@GLIBC_2.34
$ nm /usr/lib64/libc.so.6 | grep getaddrinfo
0000000000116a40 t __GI_getaddrinfo
000000000010c140 t __getaddrinfo_a
0000000000115e53 t __libc_getaddrinfo_freemem
0000000000116a40 T getaddrinfo
000000000010c140 T getaddrinfo_a@@GLIBC_2.34
000000000010c140 T getaddrinfo_a@GLIBC_2.2.5
$ nm -D /usr/lib64/libc.so.6 | grep gaih_inet
$ nm /usr/lib64/libc.so.6 | grep gaih_inet
0000000000115405 t gaih_inet
00000000001139b2 t gaih_inet_serv
0000000000177780 r gaih_inet_typeproto

Using nm $libc in both cases, and increasing max-stack to 4 allows the test to pass:

$ sudo ./perf test -vF pton
--- start ---
ping 1237113 [000] 18561.204983: probe_libc:inet_pton: (7f7c34d38eb7)
7f7c34d38eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
7f7c34d4b517 gaih_inet+0x112 (/usr/lib64/libc.so.6)
7f7c34d4cb73 getaddrinfo+0x133 (/usr/lib64/libc.so.6)
55680ac46518 [unknown] (/usr/bin/ping)
---- end ----
 86: probe libc's inet_pton & backtrace it with ping                 : Ok

The final patch I had to apply is below:

perf $ git diff

tools/perf/tests/shell/record+probe_libc_inet_pton.sh +10 -1
index 47a26f25db9f..18274e1c315c 100755
@@ -53,7 +53,16 @@ trace_libc_inet_pton_backtrace() {
                echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected
                ;;
        *)
-               eventattr='max-stack=3'
+               eventattr='max-stack=4'
+               # Add gaih_inet to expected backtrace only if it is part of libc.
+               if nm $libc | grep -F -q gaih_inet; then
+                       echo "gaih_inet\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected
+               fi
+               # The new version libc (e.g. libc 2.36) contains symbol getaddrinfo@@GLIBC_2.17
+               # which is not used. Do strict checking for 'getaddrinfo' without any suffix.
+               if nm $libc | grep -q 'getaddrinfo$'; then
+                       echo "getaddrinfo\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected
+               fi
                echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected
                ;;
        esac

Please feel free to add Tested-by: and/or Reviewed-by: Guilherme Amadio <[email protected]> to the final patch as you see fit. Thank you!

Best regards,
-Guilherme

Quoted 35 lines

Thanks,
Leo

---8<--

tools/perf/tests/shell/record+probe_libc_inet_pton.sh +9 -0
index 47a26f25db9f..5a2d641bf166 100755
@@ -54,6 +54,15 @@ trace_libc_inet_pton_backtrace() {
                ;;
        *)
                eventattr='max-stack=3'
+               # Add gaih_inet to expected backtrace only if it is part of libc.
+               if nm -D $libc | grep -F -q gaih_inet; then
+                       echo "gaih_inet\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected
+               fi
+               # The new version libc (e.g. libc 2.36) contains symbol getaddrinfo@@GLIBC_2.17
+               # which is not used. Do strict checking for 'getaddrinfo' without any suffix.
+               if nm -D $libc | grep -q 'getaddrinfo$'; then
+                       echo "getaddrinfo\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected
+               fi
                echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected
                ;;
        esac

[1] https://lore.kernel.org/linux-perf-users/[email protected]/

Quoted 6 lines

If you could suggest a way to fix this, I'd appreciate it.

Best regards,
-Guilherme

maintainer
#3 lore raw

Hello guys,

Quoted 12 lines · Guilherme Amadio

Dear Leo,

Quoted 9 lines · Leo Yan

Hi Guilherme,

On Fri, Nov 22, 2024 at 10:35:16PM +0100, Guilherme Amadio wrote:

[...]

Quoted 1 line

[...]

This is not the first reported failure on this case, see [1] for previous one.

Right, and I'm also seeing the failure now.

Quoted 29 lines
Quoted 3 lines

I assume you are working on x86_64 machine. Could you confirm if the change below can fix your issue? If yes, I will send a formal patch.

Indeed, I am working on an AMD 3950X, and have glibc 2.40 installed. The patch you sent didn't work out of the box, but with minor modifications I could fix it. I just needed to replace nm -D $libc with nm $libc for it work. The reason is that gaih_inet doesn't show with nm -D for my glibc:

$ nm -D /usr/lib64/libc.so.6 | grep getaddrinfo
0000000000116a40 T getaddrinfo@@GLIBC_2.2.5
000000000010c140 T getaddrinfo_a@GLIBC_2.2.5
000000000010c140 T getaddrinfo_a@@GLIBC_2.34
$ nm /usr/lib64/libc.so.6 | grep getaddrinfo
0000000000116a40 t __GI_getaddrinfo
000000000010c140 t __getaddrinfo_a
0000000000115e53 t __libc_getaddrinfo_freemem
0000000000116a40 T getaddrinfo
000000000010c140 T getaddrinfo_a@@GLIBC_2.34
000000000010c140 T getaddrinfo_a@GLIBC_2.2.5
$ nm -D /usr/lib64/libc.so.6 | grep gaih_inet
$ nm /usr/lib64/libc.so.6 | grep gaih_inet
0000000000115405 t gaih_inet
00000000001139b2 t gaih_inet_serv
0000000000177780 r gaih_inet_typeproto

Using nm $libc in both cases, and increasing max-stack to 4 allows the test to pass:

Quoted 9 lines
$ sudo ./perf test -vF pton
--- start ---
ping 1237113 [000] 18561.204983: probe_libc:inet_pton: (7f7c34d38eb7)
7f7c34d38eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
7f7c34d4b517 gaih_inet+0x112 (/usr/lib64/libc.so.6)
7f7c34d4cb73 getaddrinfo+0x133 (/usr/lib64/libc.so.6)
55680ac46518 [unknown] (/usr/bin/ping)
---- end ----
 86: probe libc's inet_pton & backtrace it with ping                 : Ok

I think it's very fragile as it depends on specific entries in the call stack. As we want to test if perf probe (for uprobes) and perf record working correctly, maybe it can just check the number of samples only.

Arnaldo, what do you think?

Thanks,
Namhyung

Quoted 28 lines

The final patch I had to apply is below:

perf $ git diff

tools/perf/tests/shell/record+probe_libc_inet_pton.sh +10 -1
index 47a26f25db9f..18274e1c315c 100755
@@ -53,7 +53,16 @@ trace_libc_inet_pton_backtrace() {
                echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected
                ;;
        *)
-               eventattr='max-stack=3'
+               eventattr='max-stack=4'
+               # Add gaih_inet to expected backtrace only if it is part of libc.
+               if nm $libc | grep -F -q gaih_inet; then
+                       echo "gaih_inet\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected
+               fi
+               # The new version libc (e.g. libc 2.36) contains symbol getaddrinfo@@GLIBC_2.17
+               # which is not used. Do strict checking for 'getaddrinfo' without any suffix.
+               if nm $libc | grep -q 'getaddrinfo$'; then
+                       echo "getaddrinfo\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected
+               fi
                echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected
                ;;
        esac
Quoted 2 lines

Please feel free to add Tested-by: and/or Reviewed-by: Guilherme Amadio <[email protected]> to the final patch as you see fit. Thank you!

Thanks for the report and the fix.
Namhyung

#4 lore raw

On Mon, Nov 25, 2024 at 09:08:15PM -0800, Namhyung Kim wrote:

[...]

Quoted 12 lines
Quoted 11 lines

Using nm $libc in both cases, and increasing max-stack to 4 allows the test to pass:

$ sudo ./perf test -vF pton
--- start ---
ping 1237113 [000] 18561.204983: probe_libc:inet_pton: (7f7c34d38eb7)
7f7c34d38eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
7f7c34d4b517 gaih_inet+0x112 (/usr/lib64/libc.so.6)
7f7c34d4cb73 getaddrinfo+0x133 (/usr/lib64/libc.so.6)
55680ac46518 [unknown] (/usr/bin/ping)
---- end ----
 86: probe libc's inet_pton & backtrace it with ping                 : Ok
Quoted 5 lines

I think it's very fragile as it depends on specific entries in the call stack. As we want to test if perf probe (for uprobes) and perf record working correctly, maybe it can just check the number of samples only.

Arnaldo, what do you think?

Though this question is asking Arnaldo, I have a different idea for fixing the test ;)

The middle entries in the callchain are varied on different archs and distro. E.g. Guilherme reported back the '-D' option is not needed but at my side it is required on Arm64 machine with Ubuntu distro.

However, the three entries in the callchain are existed consistently:

ping 1237113 [000] 18561.204983: probe_libc:inet_pton: (7f7c34d38eb7) 7f7c34d38eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
...
55680ac46518 [unknown] (/usr/bin/ping)

We can change the code to only matching these three entries. The change is as below. We can consolidate the code for all archs (I am not 100% sure for x390x so I leave it).

Please let me know if this makes sense for you?

Thanks,
Leo

---8<---

tools/perf/tests/shell/record+probe_libc_inet_pton.sh +16 -16
index 47a26f25db9f..6ef58aedf2d8 100755
@@ -43,17 +43,8 @@ trace_libc_inet_pton_backtrace() {
 		echo "((__GI_)?getaddrinfo|text_to_binary_address)\+0x[[:xdigit:]]+[[:space:]]\($libc|inlined\)$" >> $expected
 		echo "(gaih_inet|main)\+0x[[:xdigit:]]+[[:space:]]\(inlined|.*/bin/ping.*\)$" >> $expected
 		;;
-	ppc64|ppc64le)
-		eventattr='max-stack=4'
-		# Add gaih_inet to expected backtrace only if it is part of libc.
-		if nm $libc | grep -F -q gaih_inet.; then
-			echo "gaih_inet.*\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected
-		fi
-		echo "getaddrinfo\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected
-		echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected
-		;;
 	*)
-		eventattr='max-stack=3'
+		eventattr='max-stack=4'
 		echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected
 		;;
 	esac
@@ -76,14 +67,23 @@ trace_libc_inet_pton_backtrace() {
 	fi
 	perf script -i $perf_data | tac | grep -m1 ^ping -B9 | tac > $perf_script
 
-	exec 3<$perf_script
 	exec 4<$expected
-	while read line <&3 && read -r pattern <&4; do
+	while read -r pattern <&4; do
+		echo "pattern: $pattern"
 		[ -z "$pattern" ] && break
-		echo $line
-		echo "$line" | grep -E -q "$pattern"
-		if [ $? -ne 0 ] ; then
-			printf "FAIL: expected backtrace entry \"%s\" got \"%s\"\n" "$pattern" "$line"
+
+		exec 3<$perf_script
+		while read line <&3; do
+			[ -z "$line" ] && break
+			echo "line: $line"
+			echo "$line" | grep -E -q "$pattern"
+			found=$?
+			echo "found=$found"
+			[ $found -eq 0 ] && break
+		done
+
+		if [ $found -ne 0 ] ; then
+			printf "FAIL: unfound the expected backtrace entry \"%s\"\n" "$pattern"
 			return 1
 		fi
 	done
#5 lore raw
Quoted 36 lines · Leo Yan

On Mon, Nov 25, 2024 at 09:08:15PM -0800, Namhyung Kim wrote:

[...]

Quoted 12 lines
Quoted 11 lines

Using nm $libc in both cases, and increasing max-stack to 4 allows the test to pass:

$ sudo ./perf test -vF pton
--- start ---
ping 1237113 [000] 18561.204983: probe_libc:inet_pton: (7f7c34d38eb7)
7f7c34d38eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
7f7c34d4b517 gaih_inet+0x112 (/usr/lib64/libc.so.6)
7f7c34d4cb73 getaddrinfo+0x133 (/usr/lib64/libc.so.6)
55680ac46518 [unknown] (/usr/bin/ping)
---- end ----
 86: probe libc's inet_pton & backtrace it with ping                 : Ok
Quoted 5 lines

I think it's very fragile as it depends on specific entries in the call stack. As we want to test if perf probe (for uprobes) and perf record working correctly, maybe it can just check the number of samples only.

Arnaldo, what do you think?

Though this question is asking Arnaldo, I have a different idea for fixing the test ;)

The middle entries in the callchain are varied on different archs and distro. E.g. Guilherme reported back the '-D' option is not needed but at my side it is required on Arm64 machine with Ubuntu distro.

However, the three entries in the callchain are existed consistently:

ping 1237113 [000] 18561.204983: probe_libc:inet_pton: (7f7c34d38eb7) 7f7c34d38eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
...
55680ac46518 [unknown] (/usr/bin/ping)

Quoted 8 lines

We can change the code to only matching these three entries. The change is as below. We can consolidate the code for all archs (I am not 100% sure for x390x so I leave it).

Please let me know if this makes sense for you?

Thanks,
Leo

Seems to work:

Before, on a:

root@number:~# grep -m1 'model name' /proc/cpuinfo 
model name	: Intel(R) Core(TM) i7-14700K
root@number:~#
root@number:~# perf test -vF pton
--- start ---
ping 229025 [017] 34027.968348: probe_libc:inet_pton: (7fc4542c10d0)
7fc4542c10d0 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
7fc4542d5a55 getaddrinfo+0xcf5 (/usr/lib64/libc.so.6)
FAIL: expected backtrace entry ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" got "7fc4542d5a55 getaddrinfo+0xcf5 (/usr/lib64/libc.so.6)"
---- end ----
 86: probe libc's inet_pton & backtrace it with ping                 : FAILED!
root@number:~# 

With your patch:

root@number:~# perf test -vF pton
--- start ---
pattern: ping[][0-9 \.:]+probe_libc:inet_pton: \([[:xdigit:]]+\)
line: ping  231118 [023] 34089.948542: probe_libc:inet_pton: (7f635297f0d0)
found=0
pattern: .*inet_pton\+0x[[:xdigit:]]+[[:space:]]\(/usr/lib64/libc.so.6|inlined\)$
line: ping  231118 [023] 34089.948542: probe_libc:inet_pton: (7f635297f0d0)
found=1
line: 7f635297f0d0 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
found=0
pattern: .*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$
line: ping  231118 [023] 34089.948542: probe_libc:inet_pton: (7f635297f0d0)
found=1
line: 7f635297f0d0 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
found=1
line: 7f6352993a55 getaddrinfo+0xcf5 (/usr/lib64/libc.so.6)
found=1
line: 55ae71919969 [unknown] (/usr/bin/ping)
found=0
---- end ----
 86: probe libc's inet_pton & backtrace it with ping                 : Ok
root@number:~#

Replace that "unfound" with "Didn't find the"

And using 'found' as a variable name that when zero means "Ok, found" and when 1 (grep failed to find the regexp) means "Didn't find" looks confusing.

General idea seems to make the test resilient to the differences while still testing the backtrace in uprobes events for expected entries.

- Arnaldo

Quoted 55 lines

---8<---

tools/perf/tests/shell/record+probe_libc_inet_pton.sh +16 -16
index 47a26f25db9f..6ef58aedf2d8 100755
@@ -43,17 +43,8 @@ trace_libc_inet_pton_backtrace() {
 		echo "((__GI_)?getaddrinfo|text_to_binary_address)\+0x[[:xdigit:]]+[[:space:]]\($libc|inlined\)$" >> $expected
 		echo "(gaih_inet|main)\+0x[[:xdigit:]]+[[:space:]]\(inlined|.*/bin/ping.*\)$" >> $expected
 		;;
-	ppc64|ppc64le)
-		eventattr='max-stack=4'
-		# Add gaih_inet to expected backtrace only if it is part of libc.
-		if nm $libc | grep -F -q gaih_inet.; then
-			echo "gaih_inet.*\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected
-		fi
-		echo "getaddrinfo\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected
-		echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected
-		;;
 	*)
-		eventattr='max-stack=3'
+		eventattr='max-stack=4'
 		echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected
 		;;
 	esac
@@ -76,14 +67,23 @@ trace_libc_inet_pton_backtrace() {
 	fi
 	perf script -i $perf_data | tac | grep -m1 ^ping -B9 | tac > $perf_script
 
-	exec 3<$perf_script
 	exec 4<$expected
-	while read line <&3 && read -r pattern <&4; do
+	while read -r pattern <&4; do
+		echo "pattern: $pattern"
 		[ -z "$pattern" ] && break
-		echo $line
-		echo "$line" | grep -E -q "$pattern"
-		if [ $? -ne 0 ] ; then
-			printf "FAIL: expected backtrace entry \"%s\" got \"%s\"\n" "$pattern" "$line"
+
+		exec 3<$perf_script
+		while read line <&3; do
+			[ -z "$line" ] && break
+			echo "line: $line"
+			echo "$line" | grep -E -q "$pattern"
+			found=$?
+			echo "found=$found"
+			[ $found -eq 0 ] && break
+		done
+
+		if [ $found -ne 0 ] ; then
+			printf "FAIL: unfound the expected backtrace entry \"%s\"\n" "$pattern"
 			return 1
 		fi
 	done
#6 lore raw

Hi Arnaldo,

On Tue, Nov 26, 2024 at 01:39:39PM -0300, Arnaldo Carvalho de Melo wrote:

[...]

Quoted 43 lines

Seems to work:

Before, on a:

root@number:~# grep -m1 'model name' /proc/cpuinfo
model name      : Intel(R) Core(TM) i7-14700K
root@number:~#
root@number:~# perf test -vF pton
--- start ---
ping 229025 [017] 34027.968348: probe_libc:inet_pton: (7fc4542c10d0)
7fc4542c10d0 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
7fc4542d5a55 getaddrinfo+0xcf5 (/usr/lib64/libc.so.6)
FAIL: expected backtrace entry ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" got "7fc4542d5a55 getaddrinfo+0xcf5 (/usr/lib64/libc.so.6)"
---- end ----
 86: probe libc's inet_pton & backtrace it with ping                 : FAILED!
root@number:~#

With your patch:

root@number:~# perf test -vF pton
--- start ---
pattern: ping[][0-9 \.:]+probe_libc:inet_pton: \([[:xdigit:]]+\)
line: ping  231118 [023] 34089.948542: probe_libc:inet_pton: (7f635297f0d0)
found=0
pattern: .*inet_pton\+0x[[:xdigit:]]+[[:space:]]\(/usr/lib64/libc.so.6|inlined\)$
line: ping  231118 [023] 34089.948542: probe_libc:inet_pton: (7f635297f0d0)
found=1
line: 7f635297f0d0 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
found=0
pattern: .*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$
line: ping  231118 [023] 34089.948542: probe_libc:inet_pton: (7f635297f0d0)
found=1
line: 7f635297f0d0 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
found=1
line: 7f6352993a55 getaddrinfo+0xcf5 (/usr/lib64/libc.so.6)
found=1
line: 55ae71919969 [unknown] (/usr/bin/ping)
found=0
---- end ----
 86: probe libc's inet_pton & backtrace it with ping                 : Ok
root@number:~#
Quoted 1 line

Replace that "unfound" with "Didn't find the"

Will do.

Quoted 3 lines

And using 'found' as a variable name that when zero means "Ok, found" and when 1 (grep failed to find the regexp) means "Didn't find" looks confusing.

Indeed. Will refine this.

Quoted 2 lines

General idea seems to make the test resilient to the differences while still testing the backtrace in uprobes events for expected entries.

Will send formal patch.

Thanks a lot for the testing,
Leo

#7 lore raw
In reply to Namhyung Kim
Quoted 57 lines · Namhyung Kim

Hello guys,

Quoted 12 lines · Guilherme Amadio

Dear Leo,

Quoted 9 lines · Leo Yan

Hi Guilherme,

On Fri, Nov 22, 2024 at 10:35:16PM +0100, Guilherme Amadio wrote:

[...]

Quoted 1 line

[...]

This is not the first reported failure on this case, see [1] for previous one.

Right, and I'm also seeing the failure now.

Quoted 29 lines
Quoted 3 lines

I assume you are working on x86_64 machine. Could you confirm if the change below can fix your issue? If yes, I will send a formal patch.

Indeed, I am working on an AMD 3950X, and have glibc 2.40 installed. The patch you sent didn't work out of the box, but with minor modifications I could fix it. I just needed to replace nm -D $libc with nm $libc for it work. The reason is that gaih_inet doesn't show with nm -D for my glibc:

$ nm -D /usr/lib64/libc.so.6 | grep getaddrinfo
0000000000116a40 T getaddrinfo@@GLIBC_2.2.5
000000000010c140 T getaddrinfo_a@GLIBC_2.2.5
000000000010c140 T getaddrinfo_a@@GLIBC_2.34
$ nm /usr/lib64/libc.so.6 | grep getaddrinfo
0000000000116a40 t __GI_getaddrinfo
000000000010c140 t __getaddrinfo_a
0000000000115e53 t __libc_getaddrinfo_freemem
0000000000116a40 T getaddrinfo
000000000010c140 T getaddrinfo_a@@GLIBC_2.34
000000000010c140 T getaddrinfo_a@GLIBC_2.2.5
$ nm -D /usr/lib64/libc.so.6 | grep gaih_inet
$ nm /usr/lib64/libc.so.6 | grep gaih_inet
0000000000115405 t gaih_inet
00000000001139b2 t gaih_inet_serv
0000000000177780 r gaih_inet_typeproto

Using nm $libc in both cases, and increasing max-stack to 4 allows the test to pass:

Quoted 9 lines
$ sudo ./perf test -vF pton
--- start ---
ping 1237113 [000] 18561.204983: probe_libc:inet_pton: (7f7c34d38eb7)
7f7c34d38eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
7f7c34d4b517 gaih_inet+0x112 (/usr/lib64/libc.so.6)
7f7c34d4cb73 getaddrinfo+0x133 (/usr/lib64/libc.so.6)
55680ac46518 [unknown] (/usr/bin/ping)
---- end ----
 86: probe libc's inet_pton & backtrace it with ping                 : Ok
Quoted 5 lines

I think it's very fragile as it depends on specific entries in the call stack. As we want to test if perf probe (for uprobes) and perf record working correctly, maybe it can just check the number of samples only.

Arnaldo, what do you think?

That would reduce the usefulness of the test as it would be testing fewer aspects of perf's functionality, including backtraces from uprobes in addition to backtraces in general.

So I think Leo has a point in a followup message to this one where he seems to have identified a subset of functions that are present in the architectures we test?

- Arnaldo