Skip to contentSkip to conversation

bpf: s390: missed/kprobe_recursion

TEST FAILURE Martin KaFai Lau 4 participants 7 messages Jan 23, 2025 – Jan 28, 2025

Subsystems (3)
lore
Share & tools
lore
#1 lore raw
Quoted 5 lines · Martin KaFai Lau

Hi Jiri,

The "missed/kprobe_recursion" fails consistently on s390. It seems to start failing after the recent bpf and bpf-next tree ffwd.

Quoted 7 lines

An example:
https://github.com/kernel-patches/bpf/actions/runs/12934431612/job/36076956920

Can you help to take a look?

afaict, it only happens on s390 so far, so cc IIya if there is any recent change that may ring the bell.

hi,
I need to check more but I wonder it's the:
7495e179b478 s390/tracing: Enable HAVE_FTRACE_GRAPH_FUNC

which seems to add recursion check and bail out before we have a chance to trigger it in bpf code

jirka

#2 lore raw
Quoted 14 lines · Jiri Olsa
Quoted 5 lines · Martin KaFai Lau

Hi Jiri,

The "missed/kprobe_recursion" fails consistently on s390. It seems to start failing after the recent bpf and bpf-next tree ffwd.

Quoted 7 lines

An example:
https://github.com/kernel-patches/bpf/actions/runs/12934431612/job/36076956920

Can you help to take a look?

afaict, it only happens on s390 so far, so cc IIya if there is any recent change that may ring the bell.

Quoted 6 lines

hi,
I need to check more but I wonder it's the:
7495e179b478 s390/tracing: Enable HAVE_FTRACE_GRAPH_FUNC

which seems to add recursion check and bail out before we have a chance to trigger it in bpf code

so the test attaches bpf program test1 to bpf_fentry_test1 via kprobe.multi

	SEC("kprobe.multi/bpf_fentry_test1")
	int test1(struct pt_regs *ctx)
	{
		bpf_kfunc_common_test();
		return 0;
	}

and several other programs are attached to bpf_kfunc_common_test function

I can't test this on s390, but looks like following is happening:

kprobe.multi uses fprobe, so the test kernel path goes:

bpf_fentry_test1
ftrace_graph_func
function_graph_enter_regs
fprobe_entry
kprobe_multi_link_prog_run
test1 (bpf program)
bpf_kfunc_common_test
kprobe_ftrace_handler
kprobe_perf_func
trace_call_bpf
-> bpf_prog_active check fails, missed count is incremented

kprobe_ftrace_handler calls/takes ftrace_test_recursion_trylock (ftrace recursion lock)

but s390 now calls/takes ftrace_test_recursion_trylock already in ftrace_graph_func, so s390 stops at kprobe_ftrace_handler and does not get to trace_call_bpf to increment prog->missed counters

adding Sven, Masami, any idea?

if the ftrace_test_recursion_trylock is needed ftrace_graph_func on s390, then I think we will need to fix our test to skip s390 arch

thanks,
jirka

#3 lore raw
Quoted 51 lines · On Fri, 24 Jan 2025 16:41:38 +0100 Jiri Olsa <[email protected]> wrote:
Quoted 14 lines · Jiri Olsa
Quoted 5 lines · Martin KaFai Lau

Hi Jiri,

The "missed/kprobe_recursion" fails consistently on s390. It seems to start failing after the recent bpf and bpf-next tree ffwd.

Quoted 7 lines

An example:
https://github.com/kernel-patches/bpf/actions/runs/12934431612/job/36076956920

Can you help to take a look?

afaict, it only happens on s390 so far, so cc IIya if there is any recent change that may ring the bell.

Quoted 6 lines

hi,
I need to check more but I wonder it's the:
7495e179b478 s390/tracing: Enable HAVE_FTRACE_GRAPH_FUNC

which seems to add recursion check and bail out before we have a chance to trigger it in bpf code

so the test attaches bpf program test1 to bpf_fentry_test1 via kprobe.multi

	SEC("kprobe.multi/bpf_fentry_test1")
	int test1(struct pt_regs *ctx)
	{
		bpf_kfunc_common_test();
		return 0;
	}

and several other programs are attached to bpf_kfunc_common_test function

I can't test this on s390, but looks like following is happening:

kprobe.multi uses fprobe, so the test kernel path goes:

bpf_fentry_test1
ftrace_graph_func
function_graph_enter_regs
fprobe_entry
kprobe_multi_link_prog_run
test1 (bpf program)
bpf_kfunc_common_test
kprobe_ftrace_handler
kprobe_perf_func
trace_call_bpf
-> bpf_prog_active check fails, missed count is incremented

Quoted 5 lines

kprobe_ftrace_handler calls/takes ftrace_test_recursion_trylock (ftrace recursion lock)

but s390 now calls/takes ftrace_test_recursion_trylock already in ftrace_graph_func, so s390 stops at kprobe_ftrace_handler and does not get to trace_call_bpf to increment prog->missed counters

Oops, good catch! I missed to remove it from s390. We've already moved it in function_graph_enter_regs().

Quoted 5 lines

adding Sven, Masami, any idea?

if the ftrace_test_recursion_trylock is needed ftrace_graph_func on s390, then I think we will need to fix our test to skip s390 arch

Yes. Please try this patch;

From 12fcda79d0b1082449d5f7cfb8039b0237cf246d Mon Sep 17 00:00:00 2001
From: "Masami Hiramatsu (Google)" <[email protected]>
Date: Sun, 26 Jan 2025 23:38:59 +0900
Subject: [PATCH] s390: fgraph: Fix to remove ftrace_test_recursion_trylock()

Fix to remove ftrace_test_recursion_trylock() from ftrace_graph_func() because commit d576aec24df9 ("fgraph: Get ftrace recursion lock in function_graph_enter") has been moved it to function_graph_enter_regs() already.

Fixes d576aec24df9 ("fgraph: Get ftrace recursion lock in function_graph_enter")
---
 arch/s390/kernel/ftrace.c | 5 -----
 1 file changed, 5 deletions(-)
arch/s390/kernel/ftrace.c +0 -5
index c0b2c97efefb..63ba6306632e 100644
@@ -266,18 +266,13 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip,
 		       struct ftrace_ops *op, struct ftrace_regs *fregs)
 {
 	unsigned long *parent = &arch_ftrace_regs(fregs)->regs.gprs[14];
-	int bit;
 
 	if (unlikely(ftrace_graph_is_dead()))
 		return;
 	if (unlikely(atomic_read(&current->tracing_graph_pause)))
 		return;
-	bit = ftrace_test_recursion_trylock(ip, *parent);
-	if (bit < 0)
-		return;
 	if (!function_graph_enter_regs(*parent, ip, 0, parent, fregs))
 		*parent = (unsigned long)&return_to_handler;
-	ftrace_test_recursion_unlock(bit);
 }
 
 #endif /* CONFIG_FUNCTION_GRAPH_TRACER */
Signature · 6 lines
2.43.0

Thank you,

-- 
Masami Hiramatsu (Google) <[email protected]>
Tested-by
#4 lore raw
Quoted 78 lines · Masami Hiramatsu
Quoted 51 lines · On Fri, 24 Jan 2025 16:41:38 +0100 Jiri Olsa <[email protected]> wrote:
Quoted 14 lines · Jiri Olsa
Quoted 5 lines · Martin KaFai Lau

Hi Jiri,

The "missed/kprobe_recursion" fails consistently on s390. It seems to start failing after the recent bpf and bpf-next tree ffwd.

Quoted 7 lines

An example:
https://github.com/kernel-patches/bpf/actions/runs/12934431612/job/36076956920

Can you help to take a look?

afaict, it only happens on s390 so far, so cc IIya if there is any recent change that may ring the bell.

Quoted 6 lines

hi,
I need to check more but I wonder it's the:
7495e179b478 s390/tracing: Enable HAVE_FTRACE_GRAPH_FUNC

which seems to add recursion check and bail out before we have a chance to trigger it in bpf code

so the test attaches bpf program test1 to bpf_fentry_test1 via kprobe.multi

	SEC("kprobe.multi/bpf_fentry_test1")
	int test1(struct pt_regs *ctx)
	{
		bpf_kfunc_common_test();
		return 0;
	}

and several other programs are attached to bpf_kfunc_common_test function

I can't test this on s390, but looks like following is happening:

kprobe.multi uses fprobe, so the test kernel path goes:

bpf_fentry_test1
ftrace_graph_func
function_graph_enter_regs
fprobe_entry
kprobe_multi_link_prog_run
test1 (bpf program)
bpf_kfunc_common_test
kprobe_ftrace_handler
kprobe_perf_func
trace_call_bpf
-> bpf_prog_active check fails, missed count is incremented

Quoted 5 lines

kprobe_ftrace_handler calls/takes ftrace_test_recursion_trylock (ftrace recursion lock)

but s390 now calls/takes ftrace_test_recursion_trylock already in ftrace_graph_func, so s390 stops at kprobe_ftrace_handler and does not get to trace_call_bpf to increment prog->missed counters

Oops, good catch! I missed to remove it from s390. We've already moved it in function_graph_enter_regs().

Quoted 5 lines

adding Sven, Masami, any idea?

if the ftrace_test_recursion_trylock is needed ftrace_graph_func on s390, then I think we will need to fix our test to skip s390 arch

Yes. Please try this patch;

From 12fcda79d0b1082449d5f7cfb8039b0237cf246d Mon Sep 17 00:00:00 2001
From: "Masami Hiramatsu (Google)" <[email protected]>
Date: Sun, 26 Jan 2025 23:38:59 +0900
Subject: [PATCH] s390: fgraph: Fix to remove ftrace_test_recursion_trylock()

Quoted 8 lines

Fix to remove ftrace_test_recursion_trylock() from ftrace_graph_func() because commit d576aec24df9 ("fgraph: Get ftrace recursion lock in function_graph_enter") has been moved it to function_graph_enter_regs() already.

Fixes d576aec24df9 ("fgraph: Get ftrace recursion lock in function_graph_enter")

great, ci is passing with this fix

thanks,
jirka

Quoted 34 lines
---
 arch/s390/kernel/ftrace.c | 5 -----
 1 file changed, 5 deletions(-)
arch/s390/kernel/ftrace.c +0 -5
index c0b2c97efefb..63ba6306632e 100644
@@ -266,18 +266,13 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip,
 		       struct ftrace_ops *op, struct ftrace_regs *fregs)
 {
 	unsigned long *parent = &arch_ftrace_regs(fregs)->regs.gprs[14];
-	int bit;
 
 	if (unlikely(ftrace_graph_is_dead()))
 		return;
 	if (unlikely(atomic_read(&current->tracing_graph_pause)))
 		return;
-	bit = ftrace_test_recursion_trylock(ip, *parent);
-	if (bit < 0)
-		return;
 	if (!function_graph_enter_regs(*parent, ip, 0, parent, fregs))
 		*parent = (unsigned long)&return_to_handler;
-	ftrace_test_recursion_unlock(bit);
 }
 
 #endif /* CONFIG_FUNCTION_GRAPH_TRACER */
Signature · 6 lines
2.43.0

Thank you,

-- 
Masami Hiramatsu (Google) <[email protected]>
#5 lore raw
Quoted 85 lines · Jiri Olsa
Quoted 83 lines · Masami Hiramatsu
Quoted 51 lines · On Fri, 24 Jan 2025 16:41:38 +0100 Jiri Olsa <[email protected]> wrote:
Quoted 14 lines · Jiri Olsa
Quoted 5 lines · Martin KaFai Lau

Hi Jiri,

The "missed/kprobe_recursion" fails consistently on s390. It seems to start failing after the recent bpf and bpf-next tree ffwd.

Quoted 7 lines

An example:
https://github.com/kernel-patches/bpf/actions/runs/12934431612/job/36076956920

Can you help to take a look?

afaict, it only happens on s390 so far, so cc IIya if there is any recent change that may ring the bell.

Quoted 6 lines

hi,
I need to check more but I wonder it's the:
7495e179b478 s390/tracing: Enable HAVE_FTRACE_GRAPH_FUNC

which seems to add recursion check and bail out before we have a chance to trigger it in bpf code

so the test attaches bpf program test1 to bpf_fentry_test1 via kprobe.multi

    SEC("kprobe.multi/bpf_fentry_test1")
    int test1(struct pt_regs *ctx)
    {
            bpf_kfunc_common_test();
            return 0;
    }

and several other programs are attached to bpf_kfunc_common_test function

I can't test this on s390, but looks like following is happening:

kprobe.multi uses fprobe, so the test kernel path goes:

bpf_fentry_test1
ftrace_graph_func
function_graph_enter_regs
fprobe_entry
kprobe_multi_link_prog_run
test1 (bpf program)
bpf_kfunc_common_test
kprobe_ftrace_handler
kprobe_perf_func
trace_call_bpf
-> bpf_prog_active check fails, missed count is incremented

Quoted 5 lines

kprobe_ftrace_handler calls/takes ftrace_test_recursion_trylock (ftrace recursion lock)

but s390 now calls/takes ftrace_test_recursion_trylock already in ftrace_graph_func, so s390 stops at kprobe_ftrace_handler and does not get to trace_call_bpf to increment prog->missed counters

Oops, good catch! I missed to remove it from s390. We've already moved it in function_graph_enter_regs().

Quoted 5 lines

adding Sven, Masami, any idea?

if the ftrace_test_recursion_trylock is needed ftrace_graph_func on s390, then I think we will need to fix our test to skip s390 arch

Yes. Please try this patch;

From 12fcda79d0b1082449d5f7cfb8039b0237cf246d Mon Sep 17 00:00:00 2001
From: "Masami Hiramatsu (Google)" <[email protected]>
Date: Sun, 26 Jan 2025 23:38:59 +0900
Subject: [PATCH] s390: fgraph: Fix to remove ftrace_test_recursion_trylock()

Fix to remove ftrace_test_recursion_trylock() from ftrace_graph_func() because commit d576aec24df9 ("fgraph: Get ftrace recursion lock in function_graph_enter") has been moved it to function_graph_enter_regs() already.

Quoted 7 lines
Quoted 3 lines
Fixes d576aec24df9 ("fgraph: Get ftrace recursion lock in function_graph_enter")

great, ci is passing with this fix

Masami,

Are you going to land this fix in your tree? We can create a temporary patch for BPF CI once you have the commit in the tree.

Quoted 40 lines

thanks,
jirka

Quoted 34 lines
---
 arch/s390/kernel/ftrace.c | 5 -----
 1 file changed, 5 deletions(-)
arch/s390/kernel/ftrace.c +0 -6
index c0b2c97efefb..63ba6306632e 100644
@@ -266,18 +266,13 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip,
                     struct ftrace_ops *op, struct ftrace_regs *fregs)
 {
      unsigned long *parent = &arch_ftrace_regs(fregs)->regs.gprs[14];
-     int bit;
 
      if (unlikely(ftrace_graph_is_dead()))
              return;
      if (unlikely(atomic_read(&current->tracing_graph_pause)))
              return;
-     bit = ftrace_test_recursion_trylock(ip, *parent);
-     if (bit < 0)
-             return;
      if (!function_graph_enter_regs(*parent, ip, 0, parent, fregs))
              *parent = (unsigned long)&return_to_handler;
-     ftrace_test_recursion_unlock(bit);
 }
 
 #endif /* CONFIG_FUNCTION_GRAPH_TRACER */
--

2.43.0

Thank you,

--
Masami Hiramatsu (Google) <[email protected]>

#6 lore raw
Quoted 86 lines · On Mon, 27 Jan 2025 11:09:27 -0800 Andrii Nakryiko <[email protected]>…
Quoted 85 lines · Jiri Olsa
Quoted 83 lines · Masami Hiramatsu
Quoted 51 lines · On Fri, 24 Jan 2025 16:41:38 +0100 Jiri Olsa <[email protected]> wrote:
Quoted 14 lines · Jiri Olsa
Quoted 5 lines · Martin KaFai Lau

Hi Jiri,

The "missed/kprobe_recursion" fails consistently on s390. It seems to start failing after the recent bpf and bpf-next tree ffwd.

Quoted 7 lines

An example:
https://github.com/kernel-patches/bpf/actions/runs/12934431612/job/36076956920

Can you help to take a look?

afaict, it only happens on s390 so far, so cc IIya if there is any recent change that may ring the bell.

Quoted 6 lines

hi,
I need to check more but I wonder it's the:
7495e179b478 s390/tracing: Enable HAVE_FTRACE_GRAPH_FUNC

which seems to add recursion check and bail out before we have a chance to trigger it in bpf code

so the test attaches bpf program test1 to bpf_fentry_test1 via kprobe.multi

    SEC("kprobe.multi/bpf_fentry_test1")
    int test1(struct pt_regs *ctx)
    {
            bpf_kfunc_common_test();
            return 0;
    }

and several other programs are attached to bpf_kfunc_common_test function

I can't test this on s390, but looks like following is happening:

kprobe.multi uses fprobe, so the test kernel path goes:

bpf_fentry_test1
ftrace_graph_func
function_graph_enter_regs
fprobe_entry
kprobe_multi_link_prog_run
test1 (bpf program)
bpf_kfunc_common_test
kprobe_ftrace_handler
kprobe_perf_func
trace_call_bpf
-> bpf_prog_active check fails, missed count is incremented

Quoted 5 lines

kprobe_ftrace_handler calls/takes ftrace_test_recursion_trylock (ftrace recursion lock)

but s390 now calls/takes ftrace_test_recursion_trylock already in ftrace_graph_func, so s390 stops at kprobe_ftrace_handler and does not get to trace_call_bpf to increment prog->missed counters

Oops, good catch! I missed to remove it from s390. We've already moved it in function_graph_enter_regs().

Quoted 5 lines

adding Sven, Masami, any idea?

if the ftrace_test_recursion_trylock is needed ftrace_graph_func on s390, then I think we will need to fix our test to skip s390 arch

Yes. Please try this patch;

From 12fcda79d0b1082449d5f7cfb8039b0237cf246d Mon Sep 17 00:00:00 2001
From: "Masami Hiramatsu (Google)" <[email protected]>
Date: Sun, 26 Jan 2025 23:38:59 +0900
Subject: [PATCH] s390: fgraph: Fix to remove ftrace_test_recursion_trylock()

Fix to remove ftrace_test_recursion_trylock() from ftrace_graph_func() because commit d576aec24df9 ("fgraph: Get ftrace recursion lock in function_graph_enter") has been moved it to function_graph_enter_regs() already.

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Fixes d576aec24df9 ("fgraph: Get ftrace recursion lock in function_graph_enter")

great, ci is passing with this fix

Thanks for testing!

Quoted 5 lines

Masami,

Are you going to land this fix in your tree? We can create a temporary patch for BPF CI once you have the commit in the tree.

I think this should be a fix from linux-trace tree. I also found another issue on s390. (s390 does not implemented ) Let me resend it because I missed to Cc to linux-trace ML.

Thank you,

Quoted 41 lines
Quoted 41 lines

thanks,
jirka

Quoted 34 lines
---
 arch/s390/kernel/ftrace.c | 5 -----
 1 file changed, 5 deletions(-)
arch/s390/kernel/ftrace.c +0 -6
index c0b2c97efefb..63ba6306632e 100644
@@ -266,18 +266,13 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip,
                     struct ftrace_ops *op, struct ftrace_regs *fregs)
 {
      unsigned long *parent = &arch_ftrace_regs(fregs)->regs.gprs[14];
-     int bit;
 
      if (unlikely(ftrace_graph_is_dead()))
              return;
      if (unlikely(atomic_read(&current->tracing_graph_pause)))
              return;
-     bit = ftrace_test_recursion_trylock(ip, *parent);
-     if (bit < 0)
-             return;
      if (!function_graph_enter_regs(*parent, ip, 0, parent, fregs))
              *parent = (unsigned long)&return_to_handler;
-     ftrace_test_recursion_unlock(bit);
 }
 
 #endif /* CONFIG_FUNCTION_GRAPH_TRACER */
--

2.43.0

Thank you,

--
Masami Hiramatsu (Google) <[email protected]>

Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>