BPF trampolines break because of hang in synchronize_rcu_tasks() on PREEMPT kernels
A message that is not in the archive (sent off-list or not yet received): [email protected]
Paul E. McKenney
maintainerQuoted 54 lines · Toke Høiland-Jørgensen
Hi Paul
Magnus and I have been debugging an issue where
close()on abpf_linkfile descriptor would hang indefinitely when the system was under load on a kernel compiled withCONFIG_PREEMPT=y, and it seems to be related tosynchronize_rcu_tasks(), so I'm hoping you can help us with it.The issue is triggered reliably by loading up a system with network traffic (causing 100% softirq CPU load on one or more cores), and then attaching an freplace
bpf_linkand closing it again. Theclose()will hang until the network traffic load is lowered.Digging further, it appears that the hang happens in
synchronize_rcu_tasks(), as seen by running a bpftrace script like:bpftrace -e 'kprobe:synchronize_rcu_tasks { @start = nsecs;
printf("enter\n"); } kretprobe:synchronize_rcu_tasks {printf("exit after %d ms\n", (nsecs - @start) / 1000000); }' Attaching 2 probes...
enter
exit after 54 ms
enter
exit after 3249 ms(the two enter/exit pairs are, respectively, from an unloaded system, and from a loaded system where I stopped the network traffic after a couple of seconds).
The call to
synchronize_rcu_tasks()happens inbpf_trampoline_put():https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L376
And because it does this while holding
trampoline_mutex, even deferring the put to a worker (as a previously applied-then-reverted patch did[0]) doesn't help: that'll fix the initial hang onclose(), but any subsequent use of BPF trampolines will then be blocked because of the mutex.Also, if I just keep the network traffic running I will eventually get a kernel panic with:
kernel:[44348.426312] Kernel panic - not syncing:
hung_task: blocked tasksI've created a reproducer for the issue here:
https://github.com/xdp-project/bpf-examples/tree/master/bpf-link-hangTo compile simply do this (needs a recent llvm/clang for compiling the BPF program):
$ git clone --recurse-submodules https://github.com/xdp-project/bpf-examples $ cd bpf-examples/bpf-link-hang $ make $ ./sudo bpf-link-hangyou'll need to load up the system to trigger the hang; I'm using pktgen from a separate machine to do this.
Quoted 6 lines
My question is, of course, as ever, What Is To Be Done? Is it expected that
synchronize_rcu_tasks()can hang indefinitely on a PREEMPT system, or can this be fixed? And if it is expected, how can the BPF code be fixed so it doesn't deadlock because of this?Hoping you can help us with this - many thanks in advance! :)
Let me start with the usual question... Is the network traffic intense enough that one of the CPUs might remain in a loop handling softirqs indefinitely?
If so, does the (untested, probably does not build) patch below help?
Please note that this is only a diagnostic patch. It has the serious side effect of making __do_softirq() and anything that calls it implicitly noinstr. But it might at least be a decent starting point for a real fix. Or might be part of the real fix, who knows?
Thanx, Paul
------------------------------------------------------------------------
kernel/rcu/tree.c +1 -0
@@ -242,6 +242,7 @@ void rcu_softirq_qs(void) {rcu_qs ();rcu_preempt_deferred_qs (current); +rcu_tasks_qs (current, true); }/*
Quoted 60 lines · Paul E. McKenney
Quoted 59 lines · Toke Høiland-Jørgensen
Hi Paul
Magnus and I have been debugging an issue where
close()on abpf_linkfile descriptor would hang indefinitely when the system was under load on a kernel compiled withCONFIG_PREEMPT=y, and it seems to be related tosynchronize_rcu_tasks(), so I'm hoping you can help us with it.The issue is triggered reliably by loading up a system with network traffic (causing 100% softirq CPU load on one or more cores), and then attaching an freplace
bpf_linkand closing it again. Theclose()will hang until the network traffic load is lowered.Digging further, it appears that the hang happens in
synchronize_rcu_tasks(), as seen by running a bpftrace script like:bpftrace -e 'kprobe:synchronize_rcu_tasks { @start = nsecs;
printf("enter\n"); } kretprobe:synchronize_rcu_tasks {printf("exit after %d ms\n", (nsecs - @start) / 1000000); }' Attaching 2 probes...
enter
exit after 54 ms
enter
exit after 3249 ms(the two enter/exit pairs are, respectively, from an unloaded system, and from a loaded system where I stopped the network traffic after a couple of seconds).
The call to
synchronize_rcu_tasks()happens inbpf_trampoline_put():https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L376
And because it does this while holding
trampoline_mutex, even deferring the put to a worker (as a previously applied-then-reverted patch did[0]) doesn't help: that'll fix the initial hang onclose(), but any subsequent use of BPF trampolines will then be blocked because of the mutex.Also, if I just keep the network traffic running I will eventually get a kernel panic with:
kernel:[44348.426312] Kernel panic - not syncing:
hung_task: blocked tasksI've created a reproducer for the issue here:
https://github.com/xdp-project/bpf-examples/tree/master/bpf-link-hangTo compile simply do this (needs a recent llvm/clang for compiling the BPF program):
$ git clone --recurse-submodules https://github.com/xdp-project/bpf-examples $ cd bpf-examples/bpf-link-hang $ make $ ./sudo bpf-link-hangyou'll need to load up the system to trigger the hang; I'm using pktgen from a separate machine to do this.
My question is, of course, as ever, What Is To Be Done? Is it expected that
synchronize_rcu_tasks()can hang indefinitely on a PREEMPT system, or can this be fixed? And if it is expected, how can the BPF code be fixed so it doesn't deadlock because of this?
Quoted 5 lines
Quoted 1 line
Hoping you can help us with this - many thanks in advance! :)
Let me start with the usual question... Is the network traffic intense enough that one of the CPUs might remain in a loop handling softirqs indefinitely?
Yup, I'm pegging all CPUs in softirq:
$ mpstat -P ALL 1 [...] 18:26:52 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 18:26:53 all 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 1 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 2 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 3 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 4 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 5 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00
Quoted 1 line
If so, does the (untested, probably does not build) patch below help?
Doesn't appear to, no. It builds fine, but I still get:
Attaching 2 probes...
enter
exit after 8480 ms
(that was me interrupting the network traffic again)
-Toke
Paul E. McKenney
maintainerQuoted 83 lines · Toke Høiland-Jørgensen
Quoted 60 lines · Paul E. McKenney
Quoted 59 lines · Toke Høiland-Jørgensen
Hi Paul
Magnus and I have been debugging an issue where
close()on abpf_linkfile descriptor would hang indefinitely when the system was under load on a kernel compiled withCONFIG_PREEMPT=y, and it seems to be related tosynchronize_rcu_tasks(), so I'm hoping you can help us with it.The issue is triggered reliably by loading up a system with network traffic (causing 100% softirq CPU load on one or more cores), and then attaching an freplace
bpf_linkand closing it again. Theclose()will hang until the network traffic load is lowered.Digging further, it appears that the hang happens in
synchronize_rcu_tasks(), as seen by running a bpftrace script like:bpftrace -e 'kprobe:synchronize_rcu_tasks { @start = nsecs;
printf("enter\n"); } kretprobe:synchronize_rcu_tasks {printf("exit after %d ms\n", (nsecs - @start) / 1000000); }' Attaching 2 probes...
enter
exit after 54 ms
enter
exit after 3249 ms(the two enter/exit pairs are, respectively, from an unloaded system, and from a loaded system where I stopped the network traffic after a couple of seconds).
The call to
synchronize_rcu_tasks()happens inbpf_trampoline_put():https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L376
And because it does this while holding
trampoline_mutex, even deferring the put to a worker (as a previously applied-then-reverted patch did[0]) doesn't help: that'll fix the initial hang onclose(), but any subsequent use of BPF trampolines will then be blocked because of the mutex.Also, if I just keep the network traffic running I will eventually get a kernel panic with:
kernel:[44348.426312] Kernel panic - not syncing:
hung_task: blocked tasksI've created a reproducer for the issue here:
https://github.com/xdp-project/bpf-examples/tree/master/bpf-link-hangTo compile simply do this (needs a recent llvm/clang for compiling the BPF program):
$ git clone --recurse-submodules https://github.com/xdp-project/bpf-examples $ cd bpf-examples/bpf-link-hang $ make $ ./sudo bpf-link-hangyou'll need to load up the system to trigger the hang; I'm using pktgen from a separate machine to do this.
My question is, of course, as ever, What Is To Be Done? Is it expected that
synchronize_rcu_tasks()can hang indefinitely on a PREEMPT system, or can this be fixed? And if it is expected, how can the BPF code be fixed so it doesn't deadlock because of this?Quoted 5 lines
Quoted 1 line
Hoping you can help us with this - many thanks in advance! :)
Let me start with the usual question... Is the network traffic intense enough that one of the CPUs might remain in a loop handling softirqs indefinitely?
Yup, I'm pegging all CPUs in softirq:
$ mpstat -P ALL 1 [...] 18:26:52 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 18:26:53 all 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 1 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 2 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 3 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 4 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 5 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00Quoted 1 line
If so, does the (untested, probably does not build) patch below help?
Quoted 7 lines
Doesn't appear to, no. It builds fine, but I still get:
Attaching 2 probes...
enter
exit after 8480 ms(that was me interrupting the network traffic again)
Is your kernel properly shifting from back-of-interrupt softirq processing to ksoftirqd under heavy load? If not, my patch will not have any effect.
Thanx, Paul
Quoted 90 lines · Paul E. McKenney
Quoted 89 lines · Toke Høiland-Jørgensen
Quoted 60 lines · Paul E. McKenney
Quoted 59 lines · Toke Høiland-Jørgensen
Hi Paul
Magnus and I have been debugging an issue where
close()on abpf_linkfile descriptor would hang indefinitely when the system was under load on a kernel compiled withCONFIG_PREEMPT=y, and it seems to be related tosynchronize_rcu_tasks(), so I'm hoping you can help us with it.The issue is triggered reliably by loading up a system with network traffic (causing 100% softirq CPU load on one or more cores), and then attaching an freplace
bpf_linkand closing it again. Theclose()will hang until the network traffic load is lowered.Digging further, it appears that the hang happens in
synchronize_rcu_tasks(), as seen by running a bpftrace script like:bpftrace -e 'kprobe:synchronize_rcu_tasks { @start = nsecs;
printf("enter\n"); } kretprobe:synchronize_rcu_tasks {printf("exit after %d ms\n", (nsecs - @start) / 1000000); }' Attaching 2 probes...
enter
exit after 54 ms
enter
exit after 3249 ms(the two enter/exit pairs are, respectively, from an unloaded system, and from a loaded system where I stopped the network traffic after a couple of seconds).
The call to
synchronize_rcu_tasks()happens inbpf_trampoline_put():https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L376
And because it does this while holding
trampoline_mutex, even deferring the put to a worker (as a previously applied-then-reverted patch did[0]) doesn't help: that'll fix the initial hang onclose(), but any subsequent use of BPF trampolines will then be blocked because of the mutex.Also, if I just keep the network traffic running I will eventually get a kernel panic with:
kernel:[44348.426312] Kernel panic - not syncing:
hung_task: blocked tasksI've created a reproducer for the issue here:
https://github.com/xdp-project/bpf-examples/tree/master/bpf-link-hangTo compile simply do this (needs a recent llvm/clang for compiling the BPF program):
$ git clone --recurse-submodules https://github.com/xdp-project/bpf-examples $ cd bpf-examples/bpf-link-hang $ make $ ./sudo bpf-link-hangyou'll need to load up the system to trigger the hang; I'm using pktgen from a separate machine to do this.
My question is, of course, as ever, What Is To Be Done? Is it expected that
synchronize_rcu_tasks()can hang indefinitely on a PREEMPT system, or can this be fixed? And if it is expected, how can the BPF code be fixed so it doesn't deadlock because of this?Quoted 5 lines
Quoted 1 line
Hoping you can help us with this - many thanks in advance! :)
Let me start with the usual question... Is the network traffic intense enough that one of the CPUs might remain in a loop handling softirqs indefinitely?
Yup, I'm pegging all CPUs in softirq:
$ mpstat -P ALL 1 [...] 18:26:52 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 18:26:53 all 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 1 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 2 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 3 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 4 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 5 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00Quoted 1 line
If so, does the (untested, probably does not build) patch below help?
Doesn't appear to, no. It builds fine, but I still get:
Attaching 2 probes...
enter
exit after 8480 ms
Quoted 5 lines
Quoted 1 line
(that was me interrupting the network traffic again)
Is your kernel properly shifting from back-of-interrupt softirq processing to ksoftirqd under heavy load? If not, my patch will not have any effect.
Seems to be - this is from top:
12 root 20 0 0 0 0 R 99.3 0.0 0:43.64 ksoftirqd/0
24 root 20 0 0 0 0 R 99.3 0.0 0:43.62 ksoftirqd/2
34 root 20 0 0 0 0 R 99.3 0.0 0:43.64 ksoftirqd/4
39 root 20 0 0 0 0 R 99.3 0.0 0:43.65 ksoftirqd/5
19 root 20 0 0 0 0 R 99.0 0.0 0:43.63 ksoftirqd/1
29 root 20 0 0 0 0 R 99.0 0.0 0:43.63 ksoftirqd/3 Any other ideas? :)
(And thanks for taking a look, BTW!)
-Toke
Quoted 101 lines · Toke Høiland-Jørgensen
Quoted 90 lines · Paul E. McKenney
Quoted 89 lines · Toke Høiland-Jørgensen
Quoted 60 lines · Paul E. McKenney
Quoted 59 lines · Toke Høiland-Jørgensen
Hi Paul
Magnus and I have been debugging an issue where
close()on abpf_linkfile descriptor would hang indefinitely when the system was under load on a kernel compiled withCONFIG_PREEMPT=y, and it seems to be related tosynchronize_rcu_tasks(), so I'm hoping you can help us with it.The issue is triggered reliably by loading up a system with network traffic (causing 100% softirq CPU load on one or more cores), and then attaching an freplace
bpf_linkand closing it again. Theclose()will hang until the network traffic load is lowered.Digging further, it appears that the hang happens in
synchronize_rcu_tasks(), as seen by running a bpftrace script like:bpftrace -e 'kprobe:synchronize_rcu_tasks { @start = nsecs;
printf("enter\n"); } kretprobe:synchronize_rcu_tasks {printf("exit after %d ms\n", (nsecs - @start) / 1000000); }' Attaching 2 probes...
enter
exit after 54 ms
enter
exit after 3249 ms(the two enter/exit pairs are, respectively, from an unloaded system, and from a loaded system where I stopped the network traffic after a couple of seconds).
The call to
synchronize_rcu_tasks()happens inbpf_trampoline_put():https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L376
And because it does this while holding
trampoline_mutex, even deferring the put to a worker (as a previously applied-then-reverted patch did[0]) doesn't help: that'll fix the initial hang onclose(), but any subsequent use of BPF trampolines will then be blocked because of the mutex.Also, if I just keep the network traffic running I will eventually get a kernel panic with:
kernel:[44348.426312] Kernel panic - not syncing:
hung_task: blocked tasksI've created a reproducer for the issue here:
https://github.com/xdp-project/bpf-examples/tree/master/bpf-link-hangTo compile simply do this (needs a recent llvm/clang for compiling the BPF program):
$ git clone --recurse-submodules https://github.com/xdp-project/bpf-examples $ cd bpf-examples/bpf-link-hang $ make $ ./sudo bpf-link-hangyou'll need to load up the system to trigger the hang; I'm using pktgen from a separate machine to do this.
My question is, of course, as ever, What Is To Be Done? Is it expected that
synchronize_rcu_tasks()can hang indefinitely on a PREEMPT system, or can this be fixed? And if it is expected, how can the BPF code be fixed so it doesn't deadlock because of this?Quoted 5 lines
Quoted 1 line
Hoping you can help us with this - many thanks in advance! :)
Let me start with the usual question... Is the network traffic intense enough that one of the CPUs might remain in a loop handling softirqs indefinitely?
Yup, I'm pegging all CPUs in softirq:
$ mpstat -P ALL 1 [...] 18:26:52 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 18:26:53 all 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 1 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 2 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 3 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 4 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 5 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00Quoted 1 line
If so, does the (untested, probably does not build) patch below help?
Doesn't appear to, no. It builds fine, but I still get:
Attaching 2 probes...
enter
exit after 8480 msQuoted 5 lines
Quoted 1 line
(that was me interrupting the network traffic again)
Is your kernel properly shifting from back-of-interrupt softirq processing to ksoftirqd under heavy load? If not, my patch will not have any effect.
Seems to be - this is from top:
Quoted 8 lines
12 root 20 0 0 0 0 R 99.3 0.0 0:43.64 ksoftirqd/0 24 root 20 0 0 0 0 R 99.3 0.0 0:43.62 ksoftirqd/2 34 root 20 0 0 0 0 R 99.3 0.0 0:43.64 ksoftirqd/4 39 root 20 0 0 0 0 R 99.3 0.0 0:43.65 ksoftirqd/5 19 root 20 0 0 0 0 R 99.0 0.0 0:43.63 ksoftirqd/1 29 root 20 0 0 0 0 R 99.0 0.0 0:43.63 ksoftirqd/3Any other ideas? :)
bpf_trampoline_put() got significantly changed by e21aa341785c ("bpf:
Fix fexit trampoline. "), it doesn't do synchronize_rcu_tasks() anymore. Please give it a try. It's in bpf tree.
Quoted 5 lines
(And thanks for taking a look, BTW!)
-Toke
Quoted 109 lines · Andrii Nakryiko
Quoted 108 lines · Toke Høiland-Jørgensen
Quoted 90 lines · Paul E. McKenney
Quoted 89 lines · Toke Høiland-Jørgensen
Quoted 60 lines · Paul E. McKenney
Quoted 59 lines · Toke Høiland-Jørgensen
Hi Paul
Magnus and I have been debugging an issue where
close()on abpf_linkfile descriptor would hang indefinitely when the system was under load on a kernel compiled withCONFIG_PREEMPT=y, and it seems to be related tosynchronize_rcu_tasks(), so I'm hoping you can help us with it.The issue is triggered reliably by loading up a system with network traffic (causing 100% softirq CPU load on one or more cores), and then attaching an freplace
bpf_linkand closing it again. Theclose()will hang until the network traffic load is lowered.Digging further, it appears that the hang happens in
synchronize_rcu_tasks(), as seen by running a bpftrace script like:bpftrace -e 'kprobe:synchronize_rcu_tasks { @start = nsecs;
printf("enter\n"); } kretprobe:synchronize_rcu_tasks {printf("exit after %d ms\n", (nsecs - @start) / 1000000); }' Attaching 2 probes...
enter
exit after 54 ms
enter
exit after 3249 ms(the two enter/exit pairs are, respectively, from an unloaded system, and from a loaded system where I stopped the network traffic after a couple of seconds).
The call to
synchronize_rcu_tasks()happens inbpf_trampoline_put():https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L376
And because it does this while holding
trampoline_mutex, even deferring the put to a worker (as a previously applied-then-reverted patch did[0]) doesn't help: that'll fix the initial hang onclose(), but any subsequent use of BPF trampolines will then be blocked because of the mutex.Also, if I just keep the network traffic running I will eventually get a kernel panic with:
kernel:[44348.426312] Kernel panic - not syncing:
hung_task: blocked tasksI've created a reproducer for the issue here:
https://github.com/xdp-project/bpf-examples/tree/master/bpf-link-hangTo compile simply do this (needs a recent llvm/clang for compiling the BPF program):
$ git clone --recurse-submodules https://github.com/xdp-project/bpf-examples $ cd bpf-examples/bpf-link-hang $ make $ ./sudo bpf-link-hangyou'll need to load up the system to trigger the hang; I'm using pktgen from a separate machine to do this.
My question is, of course, as ever, What Is To Be Done? Is it expected that
synchronize_rcu_tasks()can hang indefinitely on a PREEMPT system, or can this be fixed? And if it is expected, how can the BPF code be fixed so it doesn't deadlock because of this?Quoted 5 lines
Quoted 1 line
Hoping you can help us with this - many thanks in advance! :)
Let me start with the usual question... Is the network traffic intense enough that one of the CPUs might remain in a loop handling softirqs indefinitely?
Yup, I'm pegging all CPUs in softirq:
$ mpstat -P ALL 1 [...] 18:26:52 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 18:26:53 all 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 1 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 2 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 3 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 4 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 5 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00Quoted 1 line
If so, does the (untested, probably does not build) patch below help?
Doesn't appear to, no. It builds fine, but I still get:
Attaching 2 probes...
enter
exit after 8480 msQuoted 5 lines
Quoted 1 line
(that was me interrupting the network traffic again)
Is your kernel properly shifting from back-of-interrupt softirq processing to ksoftirqd under heavy load? If not, my patch will not have any effect.
Seems to be - this is from top:
12 root 20 0 0 0 0 R 99.3 0.0 0:43.64 ksoftirqd/0 24 root 20 0 0 0 0 R 99.3 0.0 0:43.62 ksoftirqd/2 34 root 20 0 0 0 0 R 99.3 0.0 0:43.64 ksoftirqd/4 39 root 20 0 0 0 0 R 99.3 0.0 0:43.65 ksoftirqd/5 19 root 20 0 0 0 0 R 99.0 0.0 0:43.63 ksoftirqd/1 29 root 20 0 0 0 0 R 99.0 0.0 0:43.63 ksoftirqd/3
Quoted 5 lines
Quoted 1 line
Any other ideas? :)
bpf_trampoline_put()got significantly changed bye21aa341785c("bpf:
Fix fexit trampoline. "), it doesn't dosynchronize_rcu_tasks()anymore. Please give it a try. It's in bpf tree.
Ah! I had missed that patch, and only tested this on bpf-next. Yes, that indeed works better; awesome!
And sorry for bothering you with this, Paul; guess I should have looked harder for fixes first... :/
-Toke
Paul E. McKenney
maintainerQuoted 117 lines · Toke Høiland-Jørgensen
Quoted 109 lines · Andrii Nakryiko
Quoted 108 lines · Toke Høiland-Jørgensen
Quoted 90 lines · Paul E. McKenney
Quoted 89 lines · Toke Høiland-Jørgensen
Quoted 60 lines · Paul E. McKenney
Quoted 59 lines · Toke Høiland-Jørgensen
Hi Paul
Magnus and I have been debugging an issue where
close()on abpf_linkfile descriptor would hang indefinitely when the system was under load on a kernel compiled withCONFIG_PREEMPT=y, and it seems to be related tosynchronize_rcu_tasks(), so I'm hoping you can help us with it.The issue is triggered reliably by loading up a system with network traffic (causing 100% softirq CPU load on one or more cores), and then attaching an freplace
bpf_linkand closing it again. Theclose()will hang until the network traffic load is lowered.Digging further, it appears that the hang happens in
synchronize_rcu_tasks(), as seen by running a bpftrace script like:bpftrace -e 'kprobe:synchronize_rcu_tasks { @start = nsecs;
printf("enter\n"); } kretprobe:synchronize_rcu_tasks {printf("exit after %d ms\n", (nsecs - @start) / 1000000); }' Attaching 2 probes...
enter
exit after 54 ms
enter
exit after 3249 ms(the two enter/exit pairs are, respectively, from an unloaded system, and from a loaded system where I stopped the network traffic after a couple of seconds).
The call to
synchronize_rcu_tasks()happens inbpf_trampoline_put():https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L376
And because it does this while holding
trampoline_mutex, even deferring the put to a worker (as a previously applied-then-reverted patch did[0]) doesn't help: that'll fix the initial hang onclose(), but any subsequent use of BPF trampolines will then be blocked because of the mutex.Also, if I just keep the network traffic running I will eventually get a kernel panic with:
kernel:[44348.426312] Kernel panic - not syncing:
hung_task: blocked tasksI've created a reproducer for the issue here:
https://github.com/xdp-project/bpf-examples/tree/master/bpf-link-hangTo compile simply do this (needs a recent llvm/clang for compiling the BPF program):
$ git clone --recurse-submodules https://github.com/xdp-project/bpf-examples $ cd bpf-examples/bpf-link-hang $ make $ ./sudo bpf-link-hangyou'll need to load up the system to trigger the hang; I'm using pktgen from a separate machine to do this.
My question is, of course, as ever, What Is To Be Done? Is it expected that
synchronize_rcu_tasks()can hang indefinitely on a PREEMPT system, or can this be fixed? And if it is expected, how can the BPF code be fixed so it doesn't deadlock because of this?Quoted 5 lines
Quoted 1 line
Hoping you can help us with this - many thanks in advance! :)
Let me start with the usual question... Is the network traffic intense enough that one of the CPUs might remain in a loop handling softirqs indefinitely?
Yup, I'm pegging all CPUs in softirq:
$ mpstat -P ALL 1 [...] 18:26:52 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 18:26:53 all 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 1 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 2 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 3 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 4 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 5 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00Quoted 1 line
If so, does the (untested, probably does not build) patch below help?
Doesn't appear to, no. It builds fine, but I still get:
Attaching 2 probes...
enter
exit after 8480 msQuoted 5 lines
Quoted 1 line
(that was me interrupting the network traffic again)
Is your kernel properly shifting from back-of-interrupt softirq processing to ksoftirqd under heavy load? If not, my patch will not have any effect.
Seems to be - this is from top:
12 root 20 0 0 0 0 R 99.3 0.0 0:43.64 ksoftirqd/0 24 root 20 0 0 0 0 R 99.3 0.0 0:43.62 ksoftirqd/2 34 root 20 0 0 0 0 R 99.3 0.0 0:43.64 ksoftirqd/4 39 root 20 0 0 0 0 R 99.3 0.0 0:43.65 ksoftirqd/5 19 root 20 0 0 0 0 R 99.0 0.0 0:43.63 ksoftirqd/1 29 root 20 0 0 0 0 R 99.0 0.0 0:43.63 ksoftirqd/3Quoted 5 lines
Quoted 1 line
Any other ideas? :)
bpf_trampoline_put()got significantly changed bye21aa341785c("bpf:
Fix fexit trampoline. "), it doesn't dosynchronize_rcu_tasks()anymore. Please give it a try. It's in bpf tree.
Quoted 5 lines
Ah! I had missed that patch, and only tested this on bpf-next. Yes, that indeed works better; awesome!
And sorry for bothering you with this, Paul; guess I should have looked harder for fixes first... :/
Glad it is now working!
And in any case, my patch needed an s/true/false/. :-/
Hey, I did say "untested"! ;-)
Thanx, Paul
Quoted 121 lines · Paul E. McKenney
Quoted 120 lines · Toke Høiland-Jørgensen
Quoted 109 lines · Andrii Nakryiko
Quoted 108 lines · Toke Høiland-Jørgensen
Quoted 90 lines · Paul E. McKenney
Quoted 89 lines · Toke Høiland-Jørgensen
Quoted 60 lines · Paul E. McKenney
Quoted 59 lines · Toke Høiland-Jørgensen
Hi Paul
Magnus and I have been debugging an issue where
close()on abpf_linkfile descriptor would hang indefinitely when the system was under load on a kernel compiled withCONFIG_PREEMPT=y, and it seems to be related tosynchronize_rcu_tasks(), so I'm hoping you can help us with it.The issue is triggered reliably by loading up a system with network traffic (causing 100% softirq CPU load on one or more cores), and then attaching an freplace
bpf_linkand closing it again. Theclose()will hang until the network traffic load is lowered.Digging further, it appears that the hang happens in
synchronize_rcu_tasks(), as seen by running a bpftrace script like:bpftrace -e 'kprobe:synchronize_rcu_tasks { @start = nsecs;
printf("enter\n"); } kretprobe:synchronize_rcu_tasks {printf("exit after %d ms\n", (nsecs - @start) / 1000000); }' Attaching 2 probes...
enter
exit after 54 ms
enter
exit after 3249 ms(the two enter/exit pairs are, respectively, from an unloaded system, and from a loaded system where I stopped the network traffic after a couple of seconds).
The call to
synchronize_rcu_tasks()happens inbpf_trampoline_put():https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L376
And because it does this while holding
trampoline_mutex, even deferring the put to a worker (as a previously applied-then-reverted patch did[0]) doesn't help: that'll fix the initial hang onclose(), but any subsequent use of BPF trampolines will then be blocked because of the mutex.Also, if I just keep the network traffic running I will eventually get a kernel panic with:
kernel:[44348.426312] Kernel panic - not syncing:
hung_task: blocked tasksI've created a reproducer for the issue here:
https://github.com/xdp-project/bpf-examples/tree/master/bpf-link-hangTo compile simply do this (needs a recent llvm/clang for compiling the BPF program):
$ git clone --recurse-submodules https://github.com/xdp-project/bpf-examples $ cd bpf-examples/bpf-link-hang $ make $ ./sudo bpf-link-hangyou'll need to load up the system to trigger the hang; I'm using pktgen from a separate machine to do this.
My question is, of course, as ever, What Is To Be Done? Is it expected that
synchronize_rcu_tasks()can hang indefinitely on a PREEMPT system, or can this be fixed? And if it is expected, how can the BPF code be fixed so it doesn't deadlock because of this?Quoted 5 lines
Quoted 1 line
Hoping you can help us with this - many thanks in advance! :)
Let me start with the usual question... Is the network traffic intense enough that one of the CPUs might remain in a loop handling softirqs indefinitely?
Yup, I'm pegging all CPUs in softirq:
$ mpstat -P ALL 1 [...] 18:26:52 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 18:26:53 all 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 1 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 2 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 3 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 4 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 5 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00Quoted 1 line
If so, does the (untested, probably does not build) patch below help?
Doesn't appear to, no. It builds fine, but I still get:
Attaching 2 probes...
enter
exit after 8480 msQuoted 5 lines
Quoted 1 line
(that was me interrupting the network traffic again)
Is your kernel properly shifting from back-of-interrupt softirq processing to ksoftirqd under heavy load? If not, my patch will not have any effect.
Seems to be - this is from top:
12 root 20 0 0 0 0 R 99.3 0.0 0:43.64 ksoftirqd/0 24 root 20 0 0 0 0 R 99.3 0.0 0:43.62 ksoftirqd/2 34 root 20 0 0 0 0 R 99.3 0.0 0:43.64 ksoftirqd/4 39 root 20 0 0 0 0 R 99.3 0.0 0:43.65 ksoftirqd/5 19 root 20 0 0 0 0 R 99.0 0.0 0:43.63 ksoftirqd/1 29 root 20 0 0 0 0 R 99.0 0.0 0:43.63 ksoftirqd/3Quoted 5 lines
Quoted 1 line
Any other ideas? :)
bpf_trampoline_put()got significantly changed bye21aa341785c("bpf:
Fix fexit trampoline. "), it doesn't dosynchronize_rcu_tasks()anymore. Please give it a try. It's in bpf tree.Ah! I had missed that patch, and only tested this on bpf-next. Yes, that indeed works better; awesome!
Quoted 8 lines
Quoted 2 lines
And sorry for bothering you with this, Paul; guess I should have looked harder for fixes first... :/
Glad it is now working!
And in any case, my patch needed an s/true/false/. :-/
Hey, I did say "untested"! ;-)
Haha, right, well at least you run afoul of the 'truth in advertising' committee ;)
-Toke
Paul E. McKenney
maintainerQuoted 126 lines · Toke Høiland-Jørgensen
Quoted 124 lines · Paul E. McKenney
Quoted 122 lines · Toke Høiland-Jørgensen
Quoted 109 lines · Andrii Nakryiko
Quoted 108 lines · Toke Høiland-Jørgensen
Quoted 90 lines · Paul E. McKenney
Quoted 89 lines · Toke Høiland-Jørgensen
Quoted 60 lines · Paul E. McKenney
Quoted 59 lines · Toke Høiland-Jørgensen
Hi Paul
Magnus and I have been debugging an issue where
close()on abpf_linkfile descriptor would hang indefinitely when the system was under load on a kernel compiled withCONFIG_PREEMPT=y, and it seems to be related tosynchronize_rcu_tasks(), so I'm hoping you can help us with it.The issue is triggered reliably by loading up a system with network traffic (causing 100% softirq CPU load on one or more cores), and then attaching an freplace
bpf_linkand closing it again. Theclose()will hang until the network traffic load is lowered.Digging further, it appears that the hang happens in
synchronize_rcu_tasks(), as seen by running a bpftrace script like:bpftrace -e 'kprobe:synchronize_rcu_tasks { @start = nsecs;
printf("enter\n"); } kretprobe:synchronize_rcu_tasks {printf("exit after %d ms\n", (nsecs - @start) / 1000000); }' Attaching 2 probes...
enter
exit after 54 ms
enter
exit after 3249 ms(the two enter/exit pairs are, respectively, from an unloaded system, and from a loaded system where I stopped the network traffic after a couple of seconds).
The call to
synchronize_rcu_tasks()happens inbpf_trampoline_put():https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L376
And because it does this while holding
trampoline_mutex, even deferring the put to a worker (as a previously applied-then-reverted patch did[0]) doesn't help: that'll fix the initial hang onclose(), but any subsequent use of BPF trampolines will then be blocked because of the mutex.Also, if I just keep the network traffic running I will eventually get a kernel panic with:
kernel:[44348.426312] Kernel panic - not syncing:
hung_task: blocked tasksI've created a reproducer for the issue here:
https://github.com/xdp-project/bpf-examples/tree/master/bpf-link-hangTo compile simply do this (needs a recent llvm/clang for compiling the BPF program):
$ git clone --recurse-submodules https://github.com/xdp-project/bpf-examples $ cd bpf-examples/bpf-link-hang $ make $ ./sudo bpf-link-hangyou'll need to load up the system to trigger the hang; I'm using pktgen from a separate machine to do this.
My question is, of course, as ever, What Is To Be Done? Is it expected that
synchronize_rcu_tasks()can hang indefinitely on a PREEMPT system, or can this be fixed? And if it is expected, how can the BPF code be fixed so it doesn't deadlock because of this?Quoted 5 lines
Quoted 1 line
Hoping you can help us with this - many thanks in advance! :)
Let me start with the usual question... Is the network traffic intense enough that one of the CPUs might remain in a loop handling softirqs indefinitely?
Yup, I'm pegging all CPUs in softirq:
$ mpstat -P ALL 1 [...] 18:26:52 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 18:26:53 all 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 1 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 2 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 3 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 4 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 5 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00Quoted 1 line
If so, does the (untested, probably does not build) patch below help?
Doesn't appear to, no. It builds fine, but I still get:
Attaching 2 probes...
enter
exit after 8480 msQuoted 5 lines
Quoted 1 line
(that was me interrupting the network traffic again)
Is your kernel properly shifting from back-of-interrupt softirq processing to ksoftirqd under heavy load? If not, my patch will not have any effect.
Seems to be - this is from top:
12 root 20 0 0 0 0 R 99.3 0.0 0:43.64 ksoftirqd/0 24 root 20 0 0 0 0 R 99.3 0.0 0:43.62 ksoftirqd/2 34 root 20 0 0 0 0 R 99.3 0.0 0:43.64 ksoftirqd/4 39 root 20 0 0 0 0 R 99.3 0.0 0:43.65 ksoftirqd/5 19 root 20 0 0 0 0 R 99.0 0.0 0:43.63 ksoftirqd/1 29 root 20 0 0 0 0 R 99.0 0.0 0:43.63 ksoftirqd/3Quoted 5 lines
Quoted 1 line
Any other ideas? :)
bpf_trampoline_put()got significantly changed bye21aa341785c("bpf:
Fix fexit trampoline. "), it doesn't dosynchronize_rcu_tasks()anymore. Please give it a try. It's in bpf tree.Ah! I had missed that patch, and only tested this on bpf-next. Yes, that indeed works better; awesome!
And sorry for bothering you with this, Paul; guess I should have looked harder for fixes first... :/
Quoted 8 lines
Quoted 5 lines
Glad it is now working!
And in any case, my patch needed an s/true/false/. :-/
Hey, I did say "untested"! ;-)
Haha, right, well at least you run afoul of the 'truth in advertising' committee ;)
If you get a chance, could you please test the (hopefully) corrected patch shown below? This issue might affect other use cases.
Thanx, Paul
------------------------------------------------------------------------
kernel/rcu/tree.c +1 -0
@@ -242,6 +242,7 @@ void rcu_softirq_qs(void) {rcu_qs ();rcu_preempt_deferred_qs (current); +rcu_tasks_qs (current, false); }/*
Quoted 131 lines · Paul E. McKenney
Quoted 130 lines · Toke Høiland-Jørgensen
Quoted 128 lines · Paul E. McKenney
Quoted 117 lines · Toke Høiland-Jørgensen
Quoted 109 lines · Andrii Nakryiko
Quoted 108 lines · Toke Høiland-Jørgensen
Quoted 90 lines · Paul E. McKenney
Quoted 89 lines · Toke Høiland-Jørgensen
Quoted 60 lines · Paul E. McKenney
Quoted 59 lines · Toke Høiland-Jørgensen
Hi Paul
Magnus and I have been debugging an issue where
close()on abpf_linkfile descriptor would hang indefinitely when the system was under load on a kernel compiled withCONFIG_PREEMPT=y, and it seems to be related tosynchronize_rcu_tasks(), so I'm hoping you can help us with it.The issue is triggered reliably by loading up a system with network traffic (causing 100% softirq CPU load on one or more cores), and then attaching an freplace
bpf_linkand closing it again. Theclose()will hang until the network traffic load is lowered.Digging further, it appears that the hang happens in
synchronize_rcu_tasks(), as seen by running a bpftrace script like:bpftrace -e 'kprobe:synchronize_rcu_tasks { @start = nsecs;
printf("enter\n"); } kretprobe:synchronize_rcu_tasks {printf("exit after %d ms\n", (nsecs - @start) / 1000000); }' Attaching 2 probes...
enter
exit after 54 ms
enter
exit after 3249 ms(the two enter/exit pairs are, respectively, from an unloaded system, and from a loaded system where I stopped the network traffic after a couple of seconds).
The call to
synchronize_rcu_tasks()happens inbpf_trampoline_put():https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L376
And because it does this while holding
trampoline_mutex, even deferring the put to a worker (as a previously applied-then-reverted patch did[0]) doesn't help: that'll fix the initial hang onclose(), but any subsequent use of BPF trampolines will then be blocked because of the mutex.Also, if I just keep the network traffic running I will eventually get a kernel panic with:
kernel:[44348.426312] Kernel panic - not syncing:
hung_task: blocked tasksI've created a reproducer for the issue here:
https://github.com/xdp-project/bpf-examples/tree/master/bpf-link-hangTo compile simply do this (needs a recent llvm/clang for compiling the BPF program):
$ git clone --recurse-submodules https://github.com/xdp-project/bpf-examples $ cd bpf-examples/bpf-link-hang $ make $ ./sudo bpf-link-hangyou'll need to load up the system to trigger the hang; I'm using pktgen from a separate machine to do this.
My question is, of course, as ever, What Is To Be Done? Is it expected that
synchronize_rcu_tasks()can hang indefinitely on a PREEMPT system, or can this be fixed? And if it is expected, how can the BPF code be fixed so it doesn't deadlock because of this?Quoted 5 lines
Quoted 1 line
Hoping you can help us with this - many thanks in advance! :)
Let me start with the usual question... Is the network traffic intense enough that one of the CPUs might remain in a loop handling softirqs indefinitely?
Yup, I'm pegging all CPUs in softirq:
$ mpstat -P ALL 1 [...] 18:26:52 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 18:26:53 all 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 1 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 2 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 3 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 4 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 5 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00Quoted 1 line
If so, does the (untested, probably does not build) patch below help?
Doesn't appear to, no. It builds fine, but I still get:
Attaching 2 probes...
enter
exit after 8480 msQuoted 5 lines
Quoted 1 line
(that was me interrupting the network traffic again)
Is your kernel properly shifting from back-of-interrupt softirq processing to ksoftirqd under heavy load? If not, my patch will not have any effect.
Seems to be - this is from top:
12 root 20 0 0 0 0 R 99.3 0.0 0:43.64 ksoftirqd/0 24 root 20 0 0 0 0 R 99.3 0.0 0:43.62 ksoftirqd/2 34 root 20 0 0 0 0 R 99.3 0.0 0:43.64 ksoftirqd/4 39 root 20 0 0 0 0 R 99.3 0.0 0:43.65 ksoftirqd/5 19 root 20 0 0 0 0 R 99.0 0.0 0:43.63 ksoftirqd/1 29 root 20 0 0 0 0 R 99.0 0.0 0:43.63 ksoftirqd/3Quoted 5 lines
Quoted 1 line
Any other ideas? :)
bpf_trampoline_put()got significantly changed bye21aa341785c("bpf:
Fix fexit trampoline. "), it doesn't dosynchronize_rcu_tasks()anymore. Please give it a try. It's in bpf tree.Quoted 5 lines
Ah! I had missed that patch, and only tested this on bpf-next. Yes, that indeed works better; awesome!
And sorry for bothering you with this, Paul; guess I should have looked harder for fixes first... :/
Glad it is now working!
And in any case, my patch needed an s/true/false/. :-/
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Hey, I did say "untested"! ;-)
Haha, right, well at least you run afoul of the 'truth in advertising' committee ;)
If you get a chance, could you please test the (hopefully) corrected patch shown below? This issue might affect other use cases.
Yup, that does seem to help:
Attaching 2 probes...
enter
exit after 136 ms
-Toke
Paul E. McKenney
maintainerQuoted 138 lines · Toke Høiland-Jørgensen
Quoted 136 lines · Paul E. McKenney
Quoted 134 lines · Toke Høiland-Jørgensen
Quoted 121 lines · Paul E. McKenney
Quoted 120 lines · Toke Høiland-Jørgensen
Quoted 109 lines · Andrii Nakryiko
Quoted 108 lines · Toke Høiland-Jørgensen
Quoted 90 lines · Paul E. McKenney
Quoted 89 lines · Toke Høiland-Jørgensen
Quoted 60 lines · Paul E. McKenney
Quoted 59 lines · Toke Høiland-Jørgensen
Hi Paul
Magnus and I have been debugging an issue where
close()on abpf_linkfile descriptor would hang indefinitely when the system was under load on a kernel compiled withCONFIG_PREEMPT=y, and it seems to be related tosynchronize_rcu_tasks(), so I'm hoping you can help us with it.The issue is triggered reliably by loading up a system with network traffic (causing 100% softirq CPU load on one or more cores), and then attaching an freplace
bpf_linkand closing it again. Theclose()will hang until the network traffic load is lowered.Digging further, it appears that the hang happens in
synchronize_rcu_tasks(), as seen by running a bpftrace script like:bpftrace -e 'kprobe:synchronize_rcu_tasks { @start = nsecs;
printf("enter\n"); } kretprobe:synchronize_rcu_tasks {printf("exit after %d ms\n", (nsecs - @start) / 1000000); }' Attaching 2 probes...
enter
exit after 54 ms
enter
exit after 3249 ms(the two enter/exit pairs are, respectively, from an unloaded system, and from a loaded system where I stopped the network traffic after a couple of seconds).
The call to
synchronize_rcu_tasks()happens inbpf_trampoline_put():https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L376
And because it does this while holding
trampoline_mutex, even deferring the put to a worker (as a previously applied-then-reverted patch did[0]) doesn't help: that'll fix the initial hang onclose(), but any subsequent use of BPF trampolines will then be blocked because of the mutex.Also, if I just keep the network traffic running I will eventually get a kernel panic with:
kernel:[44348.426312] Kernel panic - not syncing:
hung_task: blocked tasksI've created a reproducer for the issue here:
https://github.com/xdp-project/bpf-examples/tree/master/bpf-link-hangTo compile simply do this (needs a recent llvm/clang for compiling the BPF program):
$ git clone --recurse-submodules https://github.com/xdp-project/bpf-examples $ cd bpf-examples/bpf-link-hang $ make $ ./sudo bpf-link-hangyou'll need to load up the system to trigger the hang; I'm using pktgen from a separate machine to do this.
My question is, of course, as ever, What Is To Be Done? Is it expected that
synchronize_rcu_tasks()can hang indefinitely on a PREEMPT system, or can this be fixed? And if it is expected, how can the BPF code be fixed so it doesn't deadlock because of this?Quoted 5 lines
Quoted 1 line
Hoping you can help us with this - many thanks in advance! :)
Let me start with the usual question... Is the network traffic intense enough that one of the CPUs might remain in a loop handling softirqs indefinitely?
Yup, I'm pegging all CPUs in softirq:
$ mpstat -P ALL 1 [...] 18:26:52 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 18:26:53 all 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 1 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 2 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 3 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 4 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 5 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00Quoted 1 line
If so, does the (untested, probably does not build) patch below help?
Doesn't appear to, no. It builds fine, but I still get:
Attaching 2 probes...
enter
exit after 8480 msQuoted 5 lines
Quoted 1 line
(that was me interrupting the network traffic again)
Is your kernel properly shifting from back-of-interrupt softirq processing to ksoftirqd under heavy load? If not, my patch will not have any effect.
Seems to be - this is from top:
12 root 20 0 0 0 0 R 99.3 0.0 0:43.64 ksoftirqd/0 24 root 20 0 0 0 0 R 99.3 0.0 0:43.62 ksoftirqd/2 34 root 20 0 0 0 0 R 99.3 0.0 0:43.64 ksoftirqd/4 39 root 20 0 0 0 0 R 99.3 0.0 0:43.65 ksoftirqd/5 19 root 20 0 0 0 0 R 99.0 0.0 0:43.63 ksoftirqd/1 29 root 20 0 0 0 0 R 99.0 0.0 0:43.63 ksoftirqd/3Quoted 5 lines
Quoted 1 line
Any other ideas? :)
bpf_trampoline_put()got significantly changed bye21aa341785c("bpf:
Fix fexit trampoline. "), it doesn't dosynchronize_rcu_tasks()anymore. Please give it a try. It's in bpf tree.Ah! I had missed that patch, and only tested this on bpf-next. Yes, that indeed works better; awesome!
Quoted 8 lines
Quoted 2 lines
And sorry for bothering you with this, Paul; guess I should have looked harder for fixes first... :/
Glad it is now working!
And in any case, my patch needed an s/true/false/. :-/
Hey, I did say "untested"! ;-)
Haha, right, well at least you run afoul of the 'truth in advertising' committee ;)
Quoted 8 lines
Quoted 2 lines
If you get a chance, could you please test the (hopefully) corrected patch shown below? This issue might affect other use cases.
Yup, that does seem to help:
Attaching 2 probes...
enter
exit after 136 ms
Thank you very much! May I please apply your Tested-by?
Thanx, Paul
Toke Høiland-Jørgensen
Tested-byQuoted 142 lines · Paul E. McKenney
Quoted 141 lines · Toke Høiland-Jørgensen
Quoted 138 lines · Paul E. McKenney
Quoted 126 lines · Toke Høiland-Jørgensen
Quoted 124 lines · Paul E. McKenney
Quoted 122 lines · Toke Høiland-Jørgensen
Quoted 109 lines · Andrii Nakryiko
Quoted 108 lines · Toke Høiland-Jørgensen
Quoted 90 lines · Paul E. McKenney
Quoted 89 lines · Toke Høiland-Jørgensen
Quoted 60 lines · Paul E. McKenney
Quoted 59 lines · Toke Høiland-Jørgensen
Hi Paul
Magnus and I have been debugging an issue where
close()on abpf_linkfile descriptor would hang indefinitely when the system was under load on a kernel compiled withCONFIG_PREEMPT=y, and it seems to be related tosynchronize_rcu_tasks(), so I'm hoping you can help us with it.The issue is triggered reliably by loading up a system with network traffic (causing 100% softirq CPU load on one or more cores), and then attaching an freplace
bpf_linkand closing it again. Theclose()will hang until the network traffic load is lowered.Digging further, it appears that the hang happens in
synchronize_rcu_tasks(), as seen by running a bpftrace script like:bpftrace -e 'kprobe:synchronize_rcu_tasks { @start = nsecs;
printf("enter\n"); } kretprobe:synchronize_rcu_tasks {printf("exit after %d ms\n", (nsecs - @start) / 1000000); }' Attaching 2 probes...
enter
exit after 54 ms
enter
exit after 3249 ms(the two enter/exit pairs are, respectively, from an unloaded system, and from a loaded system where I stopped the network traffic after a couple of seconds).
The call to
synchronize_rcu_tasks()happens inbpf_trampoline_put():https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L376
And because it does this while holding
trampoline_mutex, even deferring the put to a worker (as a previously applied-then-reverted patch did[0]) doesn't help: that'll fix the initial hang onclose(), but any subsequent use of BPF trampolines will then be blocked because of the mutex.Also, if I just keep the network traffic running I will eventually get a kernel panic with:
kernel:[44348.426312] Kernel panic - not syncing:
hung_task: blocked tasksI've created a reproducer for the issue here:
https://github.com/xdp-project/bpf-examples/tree/master/bpf-link-hangTo compile simply do this (needs a recent llvm/clang for compiling the BPF program):
$ git clone --recurse-submodules https://github.com/xdp-project/bpf-examples $ cd bpf-examples/bpf-link-hang $ make $ ./sudo bpf-link-hangyou'll need to load up the system to trigger the hang; I'm using pktgen from a separate machine to do this.
My question is, of course, as ever, What Is To Be Done? Is it expected that
synchronize_rcu_tasks()can hang indefinitely on a PREEMPT system, or can this be fixed? And if it is expected, how can the BPF code be fixed so it doesn't deadlock because of this?Quoted 5 lines
Quoted 1 line
Hoping you can help us with this - many thanks in advance! :)
Let me start with the usual question... Is the network traffic intense enough that one of the CPUs might remain in a loop handling softirqs indefinitely?
Yup, I'm pegging all CPUs in softirq:
$ mpstat -P ALL 1 [...] 18:26:52 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 18:26:53 all 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 1 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 2 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 3 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 4 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 5 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00Quoted 1 line
If so, does the (untested, probably does not build) patch below help?
Doesn't appear to, no. It builds fine, but I still get:
Attaching 2 probes...
enter
exit after 8480 msQuoted 5 lines
Quoted 1 line
(that was me interrupting the network traffic again)
Is your kernel properly shifting from back-of-interrupt softirq processing to ksoftirqd under heavy load? If not, my patch will not have any effect.
Seems to be - this is from top:
12 root 20 0 0 0 0 R 99.3 0.0 0:43.64 ksoftirqd/0 24 root 20 0 0 0 0 R 99.3 0.0 0:43.62 ksoftirqd/2 34 root 20 0 0 0 0 R 99.3 0.0 0:43.64 ksoftirqd/4 39 root 20 0 0 0 0 R 99.3 0.0 0:43.65 ksoftirqd/5 19 root 20 0 0 0 0 R 99.0 0.0 0:43.63 ksoftirqd/1 29 root 20 0 0 0 0 R 99.0 0.0 0:43.63 ksoftirqd/3Quoted 5 lines
Quoted 1 line
Any other ideas? :)
bpf_trampoline_put()got significantly changed bye21aa341785c("bpf:
Fix fexit trampoline. "), it doesn't dosynchronize_rcu_tasks()anymore. Please give it a try. It's in bpf tree.Ah! I had missed that patch, and only tested this on bpf-next. Yes, that indeed works better; awesome!
And sorry for bothering you with this, Paul; guess I should have looked harder for fixes first... :/
Quoted 8 lines
Quoted 5 lines
Glad it is now working!
And in any case, my patch needed an s/true/false/. :-/
Hey, I did say "untested"! ;-)
Haha, right, well at least you run afoul of the 'truth in advertising' committee ;)
If you get a chance, could you please test the (hopefully) corrected patch shown below? This issue might affect other use cases.
Quoted 7 lines
Quoted 5 lines
Yup, that does seem to help:
Attaching 2 probes...
enter
exit after 136 msThank you very much! May I please apply your Tested-by?
Sure!
Quoted 150 lines · Toke Høiland-Jørgensen
Quoted 148 lines · Paul E. McKenney
Quoted 146 lines · Toke Høiland-Jørgensen
Quoted 131 lines · Paul E. McKenney
Quoted 130 lines · Toke Høiland-Jørgensen
Quoted 128 lines · Paul E. McKenney
Quoted 117 lines · Toke Høiland-Jørgensen
Quoted 109 lines · Andrii Nakryiko
Quoted 108 lines · Toke Høiland-Jørgensen
Quoted 90 lines · Paul E. McKenney
Quoted 89 lines · Toke Høiland-Jørgensen
Quoted 60 lines · Paul E. McKenney
Quoted 59 lines · Toke Høiland-Jørgensen
Hi Paul
Magnus and I have been debugging an issue where close() on a bpf_link
file descriptor would hang indefinitely when the system was under load
on a kernel compiled with CONFIG_PREEMPT=y, and it seems to be related
to synchronize_rcu_tasks(), so I'm hoping you can help us with it.The issue is triggered reliably by loading up a system with network
traffic (causing 100% softirq CPU load on one or more cores), and then
attaching an freplace bpf_link and closing it again. The close() will
hang until the network traffic load is lowered.Digging further, it appears that the hang happens in
synchronize_rcu_tasks(), as seen by running a bpftrace script like:bpftrace -e 'kprobe:synchronize_rcu_tasks { @start = nsecs; printf("enter\n"); } kretprobe:synchronize_rcu_tasks { printf("exit after %d ms\n", (nsecs - @start) / 1000000); }'
Attaching 2 probes...
enter
exit after 54 ms
enter
exit after 3249 ms(the two enter/exit pairs are, respectively, from an unloaded system,
and from a loaded system where I stopped the network traffic after a
couple of seconds).The call to synchronize_rcu_tasks() happens in bpf_trampoline_put():
https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L376
And because it does this while holding trampoline_mutex, even deferring
the put to a worker (as a previously applied-then-reverted patch did[0])
doesn't help: that'll fix the initial hang on close(), but any
subsequent use of BPF trampolines will then be blocked because of the
mutex.Also, if I just keep the network traffic running I will eventually get a
kernel panic with:kernel:[44348.426312] Kernel panic - not syncing: hung_task: blocked tasks
I've created a reproducer for the issue here:
https://github.com/xdp-project/bpf-examples/tree/master/bpf-link-hangTo compile simply do this (needs a recent llvm/clang for compiling the BPF program):
$ git clone --recurse-submodules https://github.com/xdp-project/bpf-examples
$ cd bpf-examples/bpf-link-hang
$ make
$ ./sudo bpf-link-hangyou'll need to load up the system to trigger the hang; I'm using pktgen
from a separate machine to do this.My question is, of course, as ever, What Is To Be Done? Is it expected
that synchronize_rcu_tasks() can hang indefinitely on a PREEMPT system,
or can this be fixed? And if it is expected, how can the BPF code be
fixed so it doesn't deadlock because of this?Quoted 5 lines
Quoted 1 line
Hoping you can help us with this - many thanks in advance! :)
Let me start with the usual question... Is the network traffic intense enough that one of the CPUs might remain in a loop handling softirqs indefinitely?
Yup, I'm pegging all CPUs in softirq:
$ mpstat -P ALL 1 [...] 18:26:52 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 18:26:53 all 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 1 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 2 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 3 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 4 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 5 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00Quoted 1 line
If so, does the (untested, probably does not build) patch below help?
Doesn't appear to, no. It builds fine, but I still get:
Attaching 2 probes...
enter
exit after 8480 msQuoted 5 lines
Quoted 1 line
(that was me interrupting the network traffic again)
Is your kernel properly shifting from back-of-interrupt softirq processing to ksoftirqd under heavy load? If not, my patch will not have any effect.
Seems to be - this is from top:
12 root 20 0 0 0 0 R 99.3 0.0 0:43.64 ksoftirqd/0 24 root 20 0 0 0 0 R 99.3 0.0 0:43.62 ksoftirqd/2 34 root 20 0 0 0 0 R 99.3 0.0 0:43.64 ksoftirqd/4 39 root 20 0 0 0 0 R 99.3 0.0 0:43.65 ksoftirqd/5 19 root 20 0 0 0 0 R 99.0 0.0 0:43.63 ksoftirqd/1 29 root 20 0 0 0 0 R 99.0 0.0 0:43.63 ksoftirqd/3Quoted 5 lines
Quoted 1 line
Any other ideas? :)
bpf_trampoline_put()got significantly changed bye21aa341785c("bpf:
Fix fexit trampoline. "), it doesn't dosynchronize_rcu_tasks()anymore. Please give it a try. It's in bpf tree.Quoted 5 lines
Ah! I had missed that patch, and only tested this on bpf-next. Yes, that indeed works better; awesome!
And sorry for bothering you with this, Paul; guess I should have looked harder for fixes first... :/
Glad it is now working!
And in any case, my patch needed an s/true/false/. :-/
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Hey, I did say "untested"! ;-)
Haha, right, well at least you run afoul of the 'truth in advertising' committee ;)
If you get a chance, could you please test the (hopefully) corrected patch shown below? This issue might affect other use cases.
Yup, that does seem to help:
Attaching 2 probes...
enter
exit after 136 ms
Quoted 5 lines
Quoted 1 line
Thank you very much! May I please apply your Tested-by?
Sure!
Applied, and thank you!
Thanx, Paul
------------------------------------------------------------------------
commit 1a0dfc099c1e61e22045705265a1323ac294bea6
Author: Paul E. McKenney <[email protected]> Date: Wed Mar 24 17:08:48 2021 -0700
rcu-tasks: Make ksoftirqd provide RCU Tasks quiescent states
Heavy networking load can cause a CPU to execute continuously and indefinitely within ksoftirqd, in which case there will be no voluntary task switches and thus no RCU-tasks quiescent states. This commit therefore causes the exiting rcu_softirq_qs() to provide an RCU-tasks quiescent state.
This of course means that __do_softirq() and its callers cannot be invoked from within a tracing trampoline.
kernel/rcu/tree.c +1 -0
@@ -242,6 +242,7 @@ void rcu_softirq_qs(void) {rcu_qs ();rcu_preempt_deferred_qs (current); +rcu_tasks_qs (current, false); }/*
Quoted 151 lines · Paul E. McKenney
Quoted 150 lines · Toke Høiland-Jørgensen
Quoted 148 lines · Paul E. McKenney
Quoted 146 lines · Toke Høiland-Jørgensen
Quoted 131 lines · Paul E. McKenney
Quoted 130 lines · Toke Høiland-Jørgensen
Quoted 128 lines · Paul E. McKenney
Quoted 117 lines · Toke Høiland-Jørgensen
Quoted 109 lines · Andrii Nakryiko
Quoted 108 lines · Toke Høiland-Jørgensen
Quoted 90 lines · Paul E. McKenney
Quoted 89 lines · Toke Høiland-Jørgensen
Quoted 65 lines · Paul E. McKenney
On Tue, Mar 23, 2021 at 01:26:36PM +0100, Toke Høiland-Jørgensen wrote:
Hi PaulMagnus and I have been debugging an issue where close() on a bpf_link
file descriptor would hang indefinitely when the system was under load
on a kernel compiled with CONFIG_PREEMPT=y, and it seems to be related
to synchronize_rcu_tasks(), so I'm hoping you can help us with it.The issue is triggered reliably by loading up a system with network
traffic (causing 100% softirq CPU load on one or more cores), and then
attaching an freplace bpf_link and closing it again. The close() will
hang until the network traffic load is lowered.Digging further, it appears that the hang happens in
synchronize_rcu_tasks(), as seen by running a bpftrace script like:bpftrace -e 'kprobe:synchronize_rcu_tasks { @start = nsecs; printf("enter\n"); } kretprobe:synchronize_rcu_tasks { printf("exit after %d ms\n", (nsecs - @start) / 1000000); }'
Attaching 2 probes...
enter
exit after 54 ms
enter
exit after 3249 ms(the two enter/exit pairs are, respectively, from an unloaded system,
and from a loaded system where I stopped the network traffic after a
couple of seconds).The call to synchronize_rcu_tasks() happens in bpf_trampoline_put():
https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L376
And because it does this while holding trampoline_mutex, even deferring
the put to a worker (as a previously applied-then-reverted patch did[0])
doesn't help: that'll fix the initial hang on close(), but any
subsequent use of BPF trampolines will then be blocked because of the
mutex.Also, if I just keep the network traffic running I will eventually get a
kernel panic with:kernel:[44348.426312] Kernel panic - not syncing: hung_task: blocked tasks
I've created a reproducer for the issue here:
https://github.com/xdp-project/bpf-examples/tree/master/bpf-link-hangTo compile simply do this (needs a recent llvm/clang for compiling the BPF program):
$ git clone --recurse-submodules https://github.com/xdp-project/bpf-examples
$ cd bpf-examples/bpf-link-hang
$ make
$ ./sudo bpf-link-hangyou'll need to load up the system to trigger the hang; I'm using pktgen
from a separate machine to do this.My question is, of course, as ever, What Is To Be Done? Is it expected
that synchronize_rcu_tasks() can hang indefinitely on a PREEMPT system,
or can this be fixed? And if it is expected, how can the BPF code be
fixed so it doesn't deadlock because of this?Hoping you can help us with this - many thanks in advance! :)
Let me start with the usual question... Is the network traffic intense
enough that one of the CPUs might remain in a loop handling softirqs
indefinitely?Yup, I'm pegging all CPUs in softirq:
$ mpstat -P ALL 1 [...] 18:26:52 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 18:26:53 all 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 1 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 2 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 3 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 4 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 18:26:53 5 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00Quoted 1 line
If so, does the (untested, probably does not build) patch below help?
Doesn't appear to, no. It builds fine, but I still get:
Attaching 2 probes...
enter
exit after 8480 msQuoted 5 lines
Quoted 1 line
(that was me interrupting the network traffic again)
Is your kernel properly shifting from back-of-interrupt softirq processing to ksoftirqd under heavy load? If not, my patch will not have any effect.
Seems to be - this is from top:
12 root 20 0 0 0 0 R 99.3 0.0 0:43.64 ksoftirqd/0 24 root 20 0 0 0 0 R 99.3 0.0 0:43.62 ksoftirqd/2 34 root 20 0 0 0 0 R 99.3 0.0 0:43.64 ksoftirqd/4 39 root 20 0 0 0 0 R 99.3 0.0 0:43.65 ksoftirqd/5 19 root 20 0 0 0 0 R 99.0 0.0 0:43.63 ksoftirqd/1 29 root 20 0 0 0 0 R 99.0 0.0 0:43.63 ksoftirqd/3Quoted 5 lines
Quoted 1 line
Any other ideas? :)
bpf_trampoline_put()got significantly changed bye21aa341785c("bpf:
Fix fexit trampoline. "), it doesn't dosynchronize_rcu_tasks()anymore. Please give it a try. It's in bpf tree.Quoted 5 lines
Ah! I had missed that patch, and only tested this on bpf-next. Yes, that indeed works better; awesome!
And sorry for bothering you with this, Paul; guess I should have looked harder for fixes first... :/
Glad it is now working!
And in any case, my patch needed an s/true/false/. :-/
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Hey, I did say "untested"! ;-)
Haha, right, well at least you run afoul of the 'truth in advertising' committee ;)
If you get a chance, could you please test the (hopefully) corrected patch shown below? This issue might affect other use cases.
Yup, that does seem to help:
Attaching 2 probes...
enter
exit after 136 ms
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Thank you very much! May I please apply your Tested-by?
Sure!
Applied, and thank you!
Awesome! You're welcome, and thank you for the fix (and the quick turnaround)! :)
-Toke