Skip to contentSkip to conversation

BPF trampolines break because of hang in synchronize_rcu_tasks() on PREEMPT kernels

DISCUSSION Paul E. McKenney 3 participants 14 messages Mar 23, 2021 – Mar 25, 2021 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore

Paul E. McKenney

author maintainer
#1 lore raw
In reply to a message not in the archive
Quoted 54 lines · Toke Høiland-Jørgensen

Hi Paul

Magnus and I have been debugging an issue where close() on a bpf_link file descriptor would hang indefinitely when the system was under load on a kernel compiled with CONFIG_PREEMPT=y, and it seems to be related to synchronize_rcu_tasks(), so I'm hoping you can help us with it.

The issue is triggered reliably by loading up a system with network traffic (causing 100% softirq CPU load on one or more cores), and then attaching an freplace bpf_link and closing it again. The close() will hang until the network traffic load is lowered.

Digging further, it appears that the hang happens in synchronize_rcu_tasks(), as seen by running a bpftrace script like:

bpftrace -e 'kprobe:synchronize_rcu_tasks { @start = nsecs; printf("enter\n"); } kretprobe:synchronize_rcu_tasks { printf("exit after %d ms\n", (nsecs - @start) / 1000000); }' Attaching 2 probes...
enter
exit after 54 ms
enter
exit after 3249 ms

(the two enter/exit pairs are, respectively, from an unloaded system, and from a loaded system where I stopped the network traffic after a couple of seconds).

The call to synchronize_rcu_tasks() happens in bpf_trampoline_put():

https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L376

And because it does this while holding trampoline_mutex, even deferring the put to a worker (as a previously applied-then-reverted patch did[0]) doesn't help: that'll fix the initial hang on close(), but any subsequent use of BPF trampolines will then be blocked because of the mutex.

Also, if I just keep the network traffic running I will eventually get a kernel panic with:

kernel:[44348.426312] Kernel panic - not syncing: hung_task: blocked tasks

I've created a reproducer for the issue here:
https://github.com/xdp-project/bpf-examples/tree/master/bpf-link-hang

To compile simply do this (needs a recent llvm/clang for compiling the BPF program):

$ git clone --recurse-submodules https://github.com/xdp-project/bpf-examples
$ cd bpf-examples/bpf-link-hang
$ make
$ ./sudo bpf-link-hang

you'll need to load up the system to trigger the hang; I'm using pktgen from a separate machine to do this.

Quoted 6 lines

My question is, of course, as ever, What Is To Be Done? Is it expected that synchronize_rcu_tasks() can hang indefinitely on a PREEMPT system, or can this be fixed? And if it is expected, how can the BPF code be fixed so it doesn't deadlock because of this?

Hoping you can help us with this - many thanks in advance! :)

Let me start with the usual question... Is the network traffic intense enough that one of the CPUs might remain in a loop handling softirqs indefinitely?

If so, does the (untested, probably does not build) patch below help?

Please note that this is only a diagnostic patch. It has the serious side effect of making __do_softirq() and anything that calls it implicitly noinstr. But it might at least be a decent starting point for a real fix. Or might be part of the real fix, who knows?

Thanx, Paul

------------------------------------------------------------------------

kernel/rcu/tree.c +1 -0
index 0b06be5..e21e7b0 100644
@@ -242,6 +242,7 @@ void rcu_softirq_qs(void)
 {
 	rcu_qs();
 	rcu_preempt_deferred_qs(current);
+	rcu_tasks_qs(current, true);
 }
 
 /*
#2 lore raw
Quoted 60 lines · Paul E. McKenney
Quoted 59 lines · Toke Høiland-Jørgensen

Hi Paul

Magnus and I have been debugging an issue where close() on a bpf_link file descriptor would hang indefinitely when the system was under load on a kernel compiled with CONFIG_PREEMPT=y, and it seems to be related to synchronize_rcu_tasks(), so I'm hoping you can help us with it.

The issue is triggered reliably by loading up a system with network traffic (causing 100% softirq CPU load on one or more cores), and then attaching an freplace bpf_link and closing it again. The close() will hang until the network traffic load is lowered.

Digging further, it appears that the hang happens in synchronize_rcu_tasks(), as seen by running a bpftrace script like:

bpftrace -e 'kprobe:synchronize_rcu_tasks { @start = nsecs; printf("enter\n"); } kretprobe:synchronize_rcu_tasks { printf("exit after %d ms\n", (nsecs - @start) / 1000000); }' Attaching 2 probes...
enter
exit after 54 ms
enter
exit after 3249 ms

(the two enter/exit pairs are, respectively, from an unloaded system, and from a loaded system where I stopped the network traffic after a couple of seconds).

The call to synchronize_rcu_tasks() happens in bpf_trampoline_put():

https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L376

And because it does this while holding trampoline_mutex, even deferring the put to a worker (as a previously applied-then-reverted patch did[0]) doesn't help: that'll fix the initial hang on close(), but any subsequent use of BPF trampolines will then be blocked because of the mutex.

Also, if I just keep the network traffic running I will eventually get a kernel panic with:

kernel:[44348.426312] Kernel panic - not syncing: hung_task: blocked tasks

I've created a reproducer for the issue here:
https://github.com/xdp-project/bpf-examples/tree/master/bpf-link-hang

To compile simply do this (needs a recent llvm/clang for compiling the BPF program):

$ git clone --recurse-submodules https://github.com/xdp-project/bpf-examples
$ cd bpf-examples/bpf-link-hang
$ make
$ ./sudo bpf-link-hang

you'll need to load up the system to trigger the hang; I'm using pktgen from a separate machine to do this.

My question is, of course, as ever, What Is To Be Done? Is it expected that synchronize_rcu_tasks() can hang indefinitely on a PREEMPT system, or can this be fixed? And if it is expected, how can the BPF code be fixed so it doesn't deadlock because of this?

Quoted 5 lines
Quoted 1 line

Hoping you can help us with this - many thanks in advance! :)

Let me start with the usual question... Is the network traffic intense enough that one of the CPUs might remain in a loop handling softirqs indefinitely?

Yup, I'm pegging all CPUs in softirq:

$ mpstat -P ALL 1
[...]
18:26:52     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
18:26:53     all    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       0    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       1    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       2    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       3    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       4    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       5    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
Quoted 1 line

If so, does the (untested, probably does not build) patch below help?

Doesn't appear to, no. It builds fine, but I still get:

Attaching 2 probes...
enter
exit after 8480 ms

(that was me interrupting the network traffic again)

-Toke

author maintainer
#3 lore raw
Quoted 83 lines · Toke Høiland-Jørgensen
Quoted 60 lines · Paul E. McKenney
Quoted 59 lines · Toke Høiland-Jørgensen

Hi Paul

Magnus and I have been debugging an issue where close() on a bpf_link file descriptor would hang indefinitely when the system was under load on a kernel compiled with CONFIG_PREEMPT=y, and it seems to be related to synchronize_rcu_tasks(), so I'm hoping you can help us with it.

The issue is triggered reliably by loading up a system with network traffic (causing 100% softirq CPU load on one or more cores), and then attaching an freplace bpf_link and closing it again. The close() will hang until the network traffic load is lowered.

Digging further, it appears that the hang happens in synchronize_rcu_tasks(), as seen by running a bpftrace script like:

bpftrace -e 'kprobe:synchronize_rcu_tasks { @start = nsecs; printf("enter\n"); } kretprobe:synchronize_rcu_tasks { printf("exit after %d ms\n", (nsecs - @start) / 1000000); }' Attaching 2 probes...
enter
exit after 54 ms
enter
exit after 3249 ms

(the two enter/exit pairs are, respectively, from an unloaded system, and from a loaded system where I stopped the network traffic after a couple of seconds).

The call to synchronize_rcu_tasks() happens in bpf_trampoline_put():

https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L376

And because it does this while holding trampoline_mutex, even deferring the put to a worker (as a previously applied-then-reverted patch did[0]) doesn't help: that'll fix the initial hang on close(), but any subsequent use of BPF trampolines will then be blocked because of the mutex.

Also, if I just keep the network traffic running I will eventually get a kernel panic with:

kernel:[44348.426312] Kernel panic - not syncing: hung_task: blocked tasks

I've created a reproducer for the issue here:
https://github.com/xdp-project/bpf-examples/tree/master/bpf-link-hang

To compile simply do this (needs a recent llvm/clang for compiling the BPF program):

$ git clone --recurse-submodules https://github.com/xdp-project/bpf-examples
$ cd bpf-examples/bpf-link-hang
$ make
$ ./sudo bpf-link-hang

you'll need to load up the system to trigger the hang; I'm using pktgen from a separate machine to do this.

My question is, of course, as ever, What Is To Be Done? Is it expected that synchronize_rcu_tasks() can hang indefinitely on a PREEMPT system, or can this be fixed? And if it is expected, how can the BPF code be fixed so it doesn't deadlock because of this?

Quoted 5 lines
Quoted 1 line

Hoping you can help us with this - many thanks in advance! :)

Let me start with the usual question... Is the network traffic intense enough that one of the CPUs might remain in a loop handling softirqs indefinitely?

Yup, I'm pegging all CPUs in softirq:

$ mpstat -P ALL 1
[...]
18:26:52     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
18:26:53     all    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       0    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       1    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       2    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       3    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       4    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       5    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
Quoted 1 line

If so, does the (untested, probably does not build) patch below help?

Quoted 7 lines

Doesn't appear to, no. It builds fine, but I still get:

Attaching 2 probes...
enter
exit after 8480 ms

(that was me interrupting the network traffic again)

Is your kernel properly shifting from back-of-interrupt softirq processing to ksoftirqd under heavy load? If not, my patch will not have any effect.

Thanx, Paul

#4 lore raw
Quoted 90 lines · Paul E. McKenney
Quoted 89 lines · Toke Høiland-Jørgensen
Quoted 60 lines · Paul E. McKenney
Quoted 59 lines · Toke Høiland-Jørgensen

Hi Paul

Magnus and I have been debugging an issue where close() on a bpf_link file descriptor would hang indefinitely when the system was under load on a kernel compiled with CONFIG_PREEMPT=y, and it seems to be related to synchronize_rcu_tasks(), so I'm hoping you can help us with it.

The issue is triggered reliably by loading up a system with network traffic (causing 100% softirq CPU load on one or more cores), and then attaching an freplace bpf_link and closing it again. The close() will hang until the network traffic load is lowered.

Digging further, it appears that the hang happens in synchronize_rcu_tasks(), as seen by running a bpftrace script like:

bpftrace -e 'kprobe:synchronize_rcu_tasks { @start = nsecs; printf("enter\n"); } kretprobe:synchronize_rcu_tasks { printf("exit after %d ms\n", (nsecs - @start) / 1000000); }' Attaching 2 probes...
enter
exit after 54 ms
enter
exit after 3249 ms

(the two enter/exit pairs are, respectively, from an unloaded system, and from a loaded system where I stopped the network traffic after a couple of seconds).

The call to synchronize_rcu_tasks() happens in bpf_trampoline_put():

https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L376

And because it does this while holding trampoline_mutex, even deferring the put to a worker (as a previously applied-then-reverted patch did[0]) doesn't help: that'll fix the initial hang on close(), but any subsequent use of BPF trampolines will then be blocked because of the mutex.

Also, if I just keep the network traffic running I will eventually get a kernel panic with:

kernel:[44348.426312] Kernel panic - not syncing: hung_task: blocked tasks

I've created a reproducer for the issue here:
https://github.com/xdp-project/bpf-examples/tree/master/bpf-link-hang

To compile simply do this (needs a recent llvm/clang for compiling the BPF program):

$ git clone --recurse-submodules https://github.com/xdp-project/bpf-examples
$ cd bpf-examples/bpf-link-hang
$ make
$ ./sudo bpf-link-hang

you'll need to load up the system to trigger the hang; I'm using pktgen from a separate machine to do this.

My question is, of course, as ever, What Is To Be Done? Is it expected that synchronize_rcu_tasks() can hang indefinitely on a PREEMPT system, or can this be fixed? And if it is expected, how can the BPF code be fixed so it doesn't deadlock because of this?

Quoted 5 lines
Quoted 1 line

Hoping you can help us with this - many thanks in advance! :)

Let me start with the usual question... Is the network traffic intense enough that one of the CPUs might remain in a loop handling softirqs indefinitely?

Yup, I'm pegging all CPUs in softirq:

$ mpstat -P ALL 1
[...]
18:26:52     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
18:26:53     all    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       0    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       1    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       2    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       3    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       4    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       5    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
Quoted 1 line

If so, does the (untested, probably does not build) patch below help?

Doesn't appear to, no. It builds fine, but I still get:

Attaching 2 probes...
enter
exit after 8480 ms

Quoted 5 lines
Quoted 1 line

(that was me interrupting the network traffic again)

Is your kernel properly shifting from back-of-interrupt softirq processing to ksoftirqd under heavy load? If not, my patch will not have any effect.

Seems to be - this is from top:

     12 root      20   0       0      0      0 R  99.3   0.0   0:43.64 ksoftirqd/0                                                           
     24 root      20   0       0      0      0 R  99.3   0.0   0:43.62 ksoftirqd/2                                                           
     34 root      20   0       0      0      0 R  99.3   0.0   0:43.64 ksoftirqd/4                                                           
     39 root      20   0       0      0      0 R  99.3   0.0   0:43.65 ksoftirqd/5                                                           
     19 root      20   0       0      0      0 R  99.0   0.0   0:43.63 ksoftirqd/1                                                           
     29 root      20   0       0      0      0 R  99.0   0.0   0:43.63 ksoftirqd/3     

Any other ideas? :)

(And thanks for taking a look, BTW!)

-Toke

#5 lore raw
Quoted 101 lines · Toke Høiland-Jørgensen
Quoted 90 lines · Paul E. McKenney
Quoted 89 lines · Toke Høiland-Jørgensen
Quoted 60 lines · Paul E. McKenney
Quoted 59 lines · Toke Høiland-Jørgensen

Hi Paul

Magnus and I have been debugging an issue where close() on a bpf_link file descriptor would hang indefinitely when the system was under load on a kernel compiled with CONFIG_PREEMPT=y, and it seems to be related to synchronize_rcu_tasks(), so I'm hoping you can help us with it.

The issue is triggered reliably by loading up a system with network traffic (causing 100% softirq CPU load on one or more cores), and then attaching an freplace bpf_link and closing it again. The close() will hang until the network traffic load is lowered.

Digging further, it appears that the hang happens in synchronize_rcu_tasks(), as seen by running a bpftrace script like:

bpftrace -e 'kprobe:synchronize_rcu_tasks { @start = nsecs; printf("enter\n"); } kretprobe:synchronize_rcu_tasks { printf("exit after %d ms\n", (nsecs - @start) / 1000000); }' Attaching 2 probes...
enter
exit after 54 ms
enter
exit after 3249 ms

(the two enter/exit pairs are, respectively, from an unloaded system, and from a loaded system where I stopped the network traffic after a couple of seconds).

The call to synchronize_rcu_tasks() happens in bpf_trampoline_put():

https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L376

And because it does this while holding trampoline_mutex, even deferring the put to a worker (as a previously applied-then-reverted patch did[0]) doesn't help: that'll fix the initial hang on close(), but any subsequent use of BPF trampolines will then be blocked because of the mutex.

Also, if I just keep the network traffic running I will eventually get a kernel panic with:

kernel:[44348.426312] Kernel panic - not syncing: hung_task: blocked tasks

I've created a reproducer for the issue here:
https://github.com/xdp-project/bpf-examples/tree/master/bpf-link-hang

To compile simply do this (needs a recent llvm/clang for compiling the BPF program):

$ git clone --recurse-submodules https://github.com/xdp-project/bpf-examples
$ cd bpf-examples/bpf-link-hang
$ make
$ ./sudo bpf-link-hang

you'll need to load up the system to trigger the hang; I'm using pktgen from a separate machine to do this.

My question is, of course, as ever, What Is To Be Done? Is it expected that synchronize_rcu_tasks() can hang indefinitely on a PREEMPT system, or can this be fixed? And if it is expected, how can the BPF code be fixed so it doesn't deadlock because of this?

Quoted 5 lines
Quoted 1 line

Hoping you can help us with this - many thanks in advance! :)

Let me start with the usual question... Is the network traffic intense enough that one of the CPUs might remain in a loop handling softirqs indefinitely?

Yup, I'm pegging all CPUs in softirq:

$ mpstat -P ALL 1
[...]
18:26:52     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
18:26:53     all    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       0    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       1    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       2    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       3    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       4    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       5    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
Quoted 1 line

If so, does the (untested, probably does not build) patch below help?

Doesn't appear to, no. It builds fine, but I still get:

Attaching 2 probes...
enter
exit after 8480 ms

Quoted 5 lines
Quoted 1 line

(that was me interrupting the network traffic again)

Is your kernel properly shifting from back-of-interrupt softirq processing to ksoftirqd under heavy load? If not, my patch will not have any effect.

Seems to be - this is from top:

Quoted 8 lines
     12 root      20   0       0      0      0 R  99.3   0.0   0:43.64 ksoftirqd/0
     24 root      20   0       0      0      0 R  99.3   0.0   0:43.62 ksoftirqd/2
     34 root      20   0       0      0      0 R  99.3   0.0   0:43.64 ksoftirqd/4
     39 root      20   0       0      0      0 R  99.3   0.0   0:43.65 ksoftirqd/5
     19 root      20   0       0      0      0 R  99.0   0.0   0:43.63 ksoftirqd/1
     29 root      20   0       0      0      0 R  99.0   0.0   0:43.63 ksoftirqd/3

Any other ideas? :)

bpf_trampoline_put() got significantly changed by e21aa341785c ("bpf:
Fix fexit trampoline. "), it doesn't do synchronize_rcu_tasks() anymore. Please give it a try. It's in bpf tree.

Quoted 5 lines

(And thanks for taking a look, BTW!)

-Toke

#6 lore raw
Quoted 109 lines · Andrii Nakryiko
Quoted 108 lines · Toke Høiland-Jørgensen
Quoted 90 lines · Paul E. McKenney
Quoted 89 lines · Toke Høiland-Jørgensen
Quoted 60 lines · Paul E. McKenney
Quoted 59 lines · Toke Høiland-Jørgensen

Hi Paul

Magnus and I have been debugging an issue where close() on a bpf_link file descriptor would hang indefinitely when the system was under load on a kernel compiled with CONFIG_PREEMPT=y, and it seems to be related to synchronize_rcu_tasks(), so I'm hoping you can help us with it.

The issue is triggered reliably by loading up a system with network traffic (causing 100% softirq CPU load on one or more cores), and then attaching an freplace bpf_link and closing it again. The close() will hang until the network traffic load is lowered.

Digging further, it appears that the hang happens in synchronize_rcu_tasks(), as seen by running a bpftrace script like:

bpftrace -e 'kprobe:synchronize_rcu_tasks { @start = nsecs; printf("enter\n"); } kretprobe:synchronize_rcu_tasks { printf("exit after %d ms\n", (nsecs - @start) / 1000000); }' Attaching 2 probes...
enter
exit after 54 ms
enter
exit after 3249 ms

(the two enter/exit pairs are, respectively, from an unloaded system, and from a loaded system where I stopped the network traffic after a couple of seconds).

The call to synchronize_rcu_tasks() happens in bpf_trampoline_put():

https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L376

And because it does this while holding trampoline_mutex, even deferring the put to a worker (as a previously applied-then-reverted patch did[0]) doesn't help: that'll fix the initial hang on close(), but any subsequent use of BPF trampolines will then be blocked because of the mutex.

Also, if I just keep the network traffic running I will eventually get a kernel panic with:

kernel:[44348.426312] Kernel panic - not syncing: hung_task: blocked tasks

I've created a reproducer for the issue here:
https://github.com/xdp-project/bpf-examples/tree/master/bpf-link-hang

To compile simply do this (needs a recent llvm/clang for compiling the BPF program):

$ git clone --recurse-submodules https://github.com/xdp-project/bpf-examples
$ cd bpf-examples/bpf-link-hang
$ make
$ ./sudo bpf-link-hang

you'll need to load up the system to trigger the hang; I'm using pktgen from a separate machine to do this.

My question is, of course, as ever, What Is To Be Done? Is it expected that synchronize_rcu_tasks() can hang indefinitely on a PREEMPT system, or can this be fixed? And if it is expected, how can the BPF code be fixed so it doesn't deadlock because of this?

Quoted 5 lines
Quoted 1 line

Hoping you can help us with this - many thanks in advance! :)

Let me start with the usual question... Is the network traffic intense enough that one of the CPUs might remain in a loop handling softirqs indefinitely?

Yup, I'm pegging all CPUs in softirq:

$ mpstat -P ALL 1
[...]
18:26:52     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
18:26:53     all    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       0    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       1    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       2    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       3    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       4    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       5    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
Quoted 1 line

If so, does the (untested, probably does not build) patch below help?

Doesn't appear to, no. It builds fine, but I still get:

Attaching 2 probes...
enter
exit after 8480 ms

Quoted 5 lines
Quoted 1 line

(that was me interrupting the network traffic again)

Is your kernel properly shifting from back-of-interrupt softirq processing to ksoftirqd under heavy load? If not, my patch will not have any effect.

Seems to be - this is from top:

     12 root      20   0       0      0      0 R  99.3   0.0   0:43.64 ksoftirqd/0
     24 root      20   0       0      0      0 R  99.3   0.0   0:43.62 ksoftirqd/2
     34 root      20   0       0      0      0 R  99.3   0.0   0:43.64 ksoftirqd/4
     39 root      20   0       0      0      0 R  99.3   0.0   0:43.65 ksoftirqd/5
     19 root      20   0       0      0      0 R  99.0   0.0   0:43.63 ksoftirqd/1
     29 root      20   0       0      0      0 R  99.0   0.0   0:43.63 ksoftirqd/3
Quoted 5 lines
Quoted 1 line

Any other ideas? :)

bpf_trampoline_put() got significantly changed by e21aa341785c ("bpf:
Fix fexit trampoline. "), it doesn't do synchronize_rcu_tasks() anymore. Please give it a try. It's in bpf tree.

Ah! I had missed that patch, and only tested this on bpf-next. Yes, that indeed works better; awesome!

And sorry for bothering you with this, Paul; guess I should have looked harder for fixes first... :/

-Toke

author maintainer
#7 lore raw
Quoted 117 lines · Toke Høiland-Jørgensen
Quoted 109 lines · Andrii Nakryiko
Quoted 108 lines · Toke Høiland-Jørgensen
Quoted 90 lines · Paul E. McKenney
Quoted 89 lines · Toke Høiland-Jørgensen
Quoted 60 lines · Paul E. McKenney
Quoted 59 lines · Toke Høiland-Jørgensen

Hi Paul

Magnus and I have been debugging an issue where close() on a bpf_link file descriptor would hang indefinitely when the system was under load on a kernel compiled with CONFIG_PREEMPT=y, and it seems to be related to synchronize_rcu_tasks(), so I'm hoping you can help us with it.

The issue is triggered reliably by loading up a system with network traffic (causing 100% softirq CPU load on one or more cores), and then attaching an freplace bpf_link and closing it again. The close() will hang until the network traffic load is lowered.

Digging further, it appears that the hang happens in synchronize_rcu_tasks(), as seen by running a bpftrace script like:

bpftrace -e 'kprobe:synchronize_rcu_tasks { @start = nsecs; printf("enter\n"); } kretprobe:synchronize_rcu_tasks { printf("exit after %d ms\n", (nsecs - @start) / 1000000); }' Attaching 2 probes...
enter
exit after 54 ms
enter
exit after 3249 ms

(the two enter/exit pairs are, respectively, from an unloaded system, and from a loaded system where I stopped the network traffic after a couple of seconds).

The call to synchronize_rcu_tasks() happens in bpf_trampoline_put():

https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L376

And because it does this while holding trampoline_mutex, even deferring the put to a worker (as a previously applied-then-reverted patch did[0]) doesn't help: that'll fix the initial hang on close(), but any subsequent use of BPF trampolines will then be blocked because of the mutex.

Also, if I just keep the network traffic running I will eventually get a kernel panic with:

kernel:[44348.426312] Kernel panic - not syncing: hung_task: blocked tasks

I've created a reproducer for the issue here:
https://github.com/xdp-project/bpf-examples/tree/master/bpf-link-hang

To compile simply do this (needs a recent llvm/clang for compiling the BPF program):

$ git clone --recurse-submodules https://github.com/xdp-project/bpf-examples
$ cd bpf-examples/bpf-link-hang
$ make
$ ./sudo bpf-link-hang

you'll need to load up the system to trigger the hang; I'm using pktgen from a separate machine to do this.

My question is, of course, as ever, What Is To Be Done? Is it expected that synchronize_rcu_tasks() can hang indefinitely on a PREEMPT system, or can this be fixed? And if it is expected, how can the BPF code be fixed so it doesn't deadlock because of this?

Quoted 5 lines
Quoted 1 line

Hoping you can help us with this - many thanks in advance! :)

Let me start with the usual question... Is the network traffic intense enough that one of the CPUs might remain in a loop handling softirqs indefinitely?

Yup, I'm pegging all CPUs in softirq:

$ mpstat -P ALL 1
[...]
18:26:52     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
18:26:53     all    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       0    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       1    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       2    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       3    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       4    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       5    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
Quoted 1 line

If so, does the (untested, probably does not build) patch below help?

Doesn't appear to, no. It builds fine, but I still get:

Attaching 2 probes...
enter
exit after 8480 ms

Quoted 5 lines
Quoted 1 line

(that was me interrupting the network traffic again)

Is your kernel properly shifting from back-of-interrupt softirq processing to ksoftirqd under heavy load? If not, my patch will not have any effect.

Seems to be - this is from top:

     12 root      20   0       0      0      0 R  99.3   0.0   0:43.64 ksoftirqd/0
     24 root      20   0       0      0      0 R  99.3   0.0   0:43.62 ksoftirqd/2
     34 root      20   0       0      0      0 R  99.3   0.0   0:43.64 ksoftirqd/4
     39 root      20   0       0      0      0 R  99.3   0.0   0:43.65 ksoftirqd/5
     19 root      20   0       0      0      0 R  99.0   0.0   0:43.63 ksoftirqd/1
     29 root      20   0       0      0      0 R  99.0   0.0   0:43.63 ksoftirqd/3
Quoted 5 lines
Quoted 1 line

Any other ideas? :)

bpf_trampoline_put() got significantly changed by e21aa341785c ("bpf:
Fix fexit trampoline. "), it doesn't do synchronize_rcu_tasks() anymore. Please give it a try. It's in bpf tree.

Quoted 5 lines

Ah! I had missed that patch, and only tested this on bpf-next. Yes, that indeed works better; awesome!

And sorry for bothering you with this, Paul; guess I should have looked harder for fixes first... :/

Glad it is now working!

And in any case, my patch needed an s/true/false/. :-/

Hey, I did say "untested"! ;-)

Thanx, Paul

#8 lore raw
Quoted 121 lines · Paul E. McKenney
Quoted 120 lines · Toke Høiland-Jørgensen
Quoted 109 lines · Andrii Nakryiko
Quoted 108 lines · Toke Høiland-Jørgensen
Quoted 90 lines · Paul E. McKenney
Quoted 89 lines · Toke Høiland-Jørgensen
Quoted 60 lines · Paul E. McKenney
Quoted 59 lines · Toke Høiland-Jørgensen

Hi Paul

Magnus and I have been debugging an issue where close() on a bpf_link file descriptor would hang indefinitely when the system was under load on a kernel compiled with CONFIG_PREEMPT=y, and it seems to be related to synchronize_rcu_tasks(), so I'm hoping you can help us with it.

The issue is triggered reliably by loading up a system with network traffic (causing 100% softirq CPU load on one or more cores), and then attaching an freplace bpf_link and closing it again. The close() will hang until the network traffic load is lowered.

Digging further, it appears that the hang happens in synchronize_rcu_tasks(), as seen by running a bpftrace script like:

bpftrace -e 'kprobe:synchronize_rcu_tasks { @start = nsecs; printf("enter\n"); } kretprobe:synchronize_rcu_tasks { printf("exit after %d ms\n", (nsecs - @start) / 1000000); }' Attaching 2 probes...
enter
exit after 54 ms
enter
exit after 3249 ms

(the two enter/exit pairs are, respectively, from an unloaded system, and from a loaded system where I stopped the network traffic after a couple of seconds).

The call to synchronize_rcu_tasks() happens in bpf_trampoline_put():

https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L376

And because it does this while holding trampoline_mutex, even deferring the put to a worker (as a previously applied-then-reverted patch did[0]) doesn't help: that'll fix the initial hang on close(), but any subsequent use of BPF trampolines will then be blocked because of the mutex.

Also, if I just keep the network traffic running I will eventually get a kernel panic with:

kernel:[44348.426312] Kernel panic - not syncing: hung_task: blocked tasks

I've created a reproducer for the issue here:
https://github.com/xdp-project/bpf-examples/tree/master/bpf-link-hang

To compile simply do this (needs a recent llvm/clang for compiling the BPF program):

$ git clone --recurse-submodules https://github.com/xdp-project/bpf-examples
$ cd bpf-examples/bpf-link-hang
$ make
$ ./sudo bpf-link-hang

you'll need to load up the system to trigger the hang; I'm using pktgen from a separate machine to do this.

My question is, of course, as ever, What Is To Be Done? Is it expected that synchronize_rcu_tasks() can hang indefinitely on a PREEMPT system, or can this be fixed? And if it is expected, how can the BPF code be fixed so it doesn't deadlock because of this?

Quoted 5 lines
Quoted 1 line

Hoping you can help us with this - many thanks in advance! :)

Let me start with the usual question... Is the network traffic intense enough that one of the CPUs might remain in a loop handling softirqs indefinitely?

Yup, I'm pegging all CPUs in softirq:

$ mpstat -P ALL 1
[...]
18:26:52     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
18:26:53     all    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       0    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       1    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       2    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       3    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       4    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       5    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
Quoted 1 line

If so, does the (untested, probably does not build) patch below help?

Doesn't appear to, no. It builds fine, but I still get:

Attaching 2 probes...
enter
exit after 8480 ms

Quoted 5 lines
Quoted 1 line

(that was me interrupting the network traffic again)

Is your kernel properly shifting from back-of-interrupt softirq processing to ksoftirqd under heavy load? If not, my patch will not have any effect.

Seems to be - this is from top:

     12 root      20   0       0      0      0 R  99.3   0.0   0:43.64 ksoftirqd/0
     24 root      20   0       0      0      0 R  99.3   0.0   0:43.62 ksoftirqd/2
     34 root      20   0       0      0      0 R  99.3   0.0   0:43.64 ksoftirqd/4
     39 root      20   0       0      0      0 R  99.3   0.0   0:43.65 ksoftirqd/5
     19 root      20   0       0      0      0 R  99.0   0.0   0:43.63 ksoftirqd/1
     29 root      20   0       0      0      0 R  99.0   0.0   0:43.63 ksoftirqd/3
Quoted 5 lines
Quoted 1 line

Any other ideas? :)

bpf_trampoline_put() got significantly changed by e21aa341785c ("bpf:
Fix fexit trampoline. "), it doesn't do synchronize_rcu_tasks() anymore. Please give it a try. It's in bpf tree.

Ah! I had missed that patch, and only tested this on bpf-next. Yes, that indeed works better; awesome!

Quoted 8 lines
Quoted 2 lines

And sorry for bothering you with this, Paul; guess I should have looked harder for fixes first... :/

Glad it is now working!

And in any case, my patch needed an s/true/false/. :-/

Hey, I did say "untested"! ;-)

Haha, right, well at least you run afoul of the 'truth in advertising' committee ;)

-Toke

author maintainer
#9 lore raw
Quoted 126 lines · Toke Høiland-Jørgensen
Quoted 124 lines · Paul E. McKenney
Quoted 122 lines · Toke Høiland-Jørgensen
Quoted 109 lines · Andrii Nakryiko
Quoted 108 lines · Toke Høiland-Jørgensen
Quoted 90 lines · Paul E. McKenney
Quoted 89 lines · Toke Høiland-Jørgensen
Quoted 60 lines · Paul E. McKenney
Quoted 59 lines · Toke Høiland-Jørgensen

Hi Paul

Magnus and I have been debugging an issue where close() on a bpf_link file descriptor would hang indefinitely when the system was under load on a kernel compiled with CONFIG_PREEMPT=y, and it seems to be related to synchronize_rcu_tasks(), so I'm hoping you can help us with it.

The issue is triggered reliably by loading up a system with network traffic (causing 100% softirq CPU load on one or more cores), and then attaching an freplace bpf_link and closing it again. The close() will hang until the network traffic load is lowered.

Digging further, it appears that the hang happens in synchronize_rcu_tasks(), as seen by running a bpftrace script like:

bpftrace -e 'kprobe:synchronize_rcu_tasks { @start = nsecs; printf("enter\n"); } kretprobe:synchronize_rcu_tasks { printf("exit after %d ms\n", (nsecs - @start) / 1000000); }' Attaching 2 probes...
enter
exit after 54 ms
enter
exit after 3249 ms

(the two enter/exit pairs are, respectively, from an unloaded system, and from a loaded system where I stopped the network traffic after a couple of seconds).

The call to synchronize_rcu_tasks() happens in bpf_trampoline_put():

https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L376

And because it does this while holding trampoline_mutex, even deferring the put to a worker (as a previously applied-then-reverted patch did[0]) doesn't help: that'll fix the initial hang on close(), but any subsequent use of BPF trampolines will then be blocked because of the mutex.

Also, if I just keep the network traffic running I will eventually get a kernel panic with:

kernel:[44348.426312] Kernel panic - not syncing: hung_task: blocked tasks

I've created a reproducer for the issue here:
https://github.com/xdp-project/bpf-examples/tree/master/bpf-link-hang

To compile simply do this (needs a recent llvm/clang for compiling the BPF program):

$ git clone --recurse-submodules https://github.com/xdp-project/bpf-examples
$ cd bpf-examples/bpf-link-hang
$ make
$ ./sudo bpf-link-hang

you'll need to load up the system to trigger the hang; I'm using pktgen from a separate machine to do this.

My question is, of course, as ever, What Is To Be Done? Is it expected that synchronize_rcu_tasks() can hang indefinitely on a PREEMPT system, or can this be fixed? And if it is expected, how can the BPF code be fixed so it doesn't deadlock because of this?

Quoted 5 lines
Quoted 1 line

Hoping you can help us with this - many thanks in advance! :)

Let me start with the usual question... Is the network traffic intense enough that one of the CPUs might remain in a loop handling softirqs indefinitely?

Yup, I'm pegging all CPUs in softirq:

$ mpstat -P ALL 1
[...]
18:26:52     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
18:26:53     all    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       0    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       1    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       2    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       3    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       4    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       5    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
Quoted 1 line

If so, does the (untested, probably does not build) patch below help?

Doesn't appear to, no. It builds fine, but I still get:

Attaching 2 probes...
enter
exit after 8480 ms

Quoted 5 lines
Quoted 1 line

(that was me interrupting the network traffic again)

Is your kernel properly shifting from back-of-interrupt softirq processing to ksoftirqd under heavy load? If not, my patch will not have any effect.

Seems to be - this is from top:

     12 root      20   0       0      0      0 R  99.3   0.0   0:43.64 ksoftirqd/0
     24 root      20   0       0      0      0 R  99.3   0.0   0:43.62 ksoftirqd/2
     34 root      20   0       0      0      0 R  99.3   0.0   0:43.64 ksoftirqd/4
     39 root      20   0       0      0      0 R  99.3   0.0   0:43.65 ksoftirqd/5
     19 root      20   0       0      0      0 R  99.0   0.0   0:43.63 ksoftirqd/1
     29 root      20   0       0      0      0 R  99.0   0.0   0:43.63 ksoftirqd/3
Quoted 5 lines
Quoted 1 line

Any other ideas? :)

bpf_trampoline_put() got significantly changed by e21aa341785c ("bpf:
Fix fexit trampoline. "), it doesn't do synchronize_rcu_tasks() anymore. Please give it a try. It's in bpf tree.

Ah! I had missed that patch, and only tested this on bpf-next. Yes, that indeed works better; awesome!

And sorry for bothering you with this, Paul; guess I should have looked harder for fixes first... :/

Quoted 8 lines
Quoted 5 lines

Glad it is now working!

And in any case, my patch needed an s/true/false/. :-/

Hey, I did say "untested"! ;-)

Haha, right, well at least you run afoul of the 'truth in advertising' committee ;)

If you get a chance, could you please test the (hopefully) corrected patch shown below? This issue might affect other use cases.

Thanx, Paul

------------------------------------------------------------------------

kernel/rcu/tree.c +1 -0
index 0b06be5..e21e7b0 100644
@@ -242,6 +242,7 @@ void rcu_softirq_qs(void)
 {
 	rcu_qs();
 	rcu_preempt_deferred_qs(current);
+	rcu_tasks_qs(current, false);
 }
 
 /*
#10 lore raw
Quoted 131 lines · Paul E. McKenney
Quoted 130 lines · Toke Høiland-Jørgensen
Quoted 128 lines · Paul E. McKenney
Quoted 117 lines · Toke Høiland-Jørgensen
Quoted 109 lines · Andrii Nakryiko
Quoted 108 lines · Toke Høiland-Jørgensen
Quoted 90 lines · Paul E. McKenney
Quoted 89 lines · Toke Høiland-Jørgensen
Quoted 60 lines · Paul E. McKenney
Quoted 59 lines · Toke Høiland-Jørgensen

Hi Paul

Magnus and I have been debugging an issue where close() on a bpf_link file descriptor would hang indefinitely when the system was under load on a kernel compiled with CONFIG_PREEMPT=y, and it seems to be related to synchronize_rcu_tasks(), so I'm hoping you can help us with it.

The issue is triggered reliably by loading up a system with network traffic (causing 100% softirq CPU load on one or more cores), and then attaching an freplace bpf_link and closing it again. The close() will hang until the network traffic load is lowered.

Digging further, it appears that the hang happens in synchronize_rcu_tasks(), as seen by running a bpftrace script like:

bpftrace -e 'kprobe:synchronize_rcu_tasks { @start = nsecs; printf("enter\n"); } kretprobe:synchronize_rcu_tasks { printf("exit after %d ms\n", (nsecs - @start) / 1000000); }' Attaching 2 probes...
enter
exit after 54 ms
enter
exit after 3249 ms

(the two enter/exit pairs are, respectively, from an unloaded system, and from a loaded system where I stopped the network traffic after a couple of seconds).

The call to synchronize_rcu_tasks() happens in bpf_trampoline_put():

https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L376

And because it does this while holding trampoline_mutex, even deferring the put to a worker (as a previously applied-then-reverted patch did[0]) doesn't help: that'll fix the initial hang on close(), but any subsequent use of BPF trampolines will then be blocked because of the mutex.

Also, if I just keep the network traffic running I will eventually get a kernel panic with:

kernel:[44348.426312] Kernel panic - not syncing: hung_task: blocked tasks

I've created a reproducer for the issue here:
https://github.com/xdp-project/bpf-examples/tree/master/bpf-link-hang

To compile simply do this (needs a recent llvm/clang for compiling the BPF program):

$ git clone --recurse-submodules https://github.com/xdp-project/bpf-examples
$ cd bpf-examples/bpf-link-hang
$ make
$ ./sudo bpf-link-hang

you'll need to load up the system to trigger the hang; I'm using pktgen from a separate machine to do this.

My question is, of course, as ever, What Is To Be Done? Is it expected that synchronize_rcu_tasks() can hang indefinitely on a PREEMPT system, or can this be fixed? And if it is expected, how can the BPF code be fixed so it doesn't deadlock because of this?

Quoted 5 lines
Quoted 1 line

Hoping you can help us with this - many thanks in advance! :)

Let me start with the usual question... Is the network traffic intense enough that one of the CPUs might remain in a loop handling softirqs indefinitely?

Yup, I'm pegging all CPUs in softirq:

$ mpstat -P ALL 1
[...]
18:26:52     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
18:26:53     all    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       0    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       1    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       2    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       3    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       4    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       5    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
Quoted 1 line

If so, does the (untested, probably does not build) patch below help?

Doesn't appear to, no. It builds fine, but I still get:

Attaching 2 probes...
enter
exit after 8480 ms

Quoted 5 lines
Quoted 1 line

(that was me interrupting the network traffic again)

Is your kernel properly shifting from back-of-interrupt softirq processing to ksoftirqd under heavy load? If not, my patch will not have any effect.

Seems to be - this is from top:

     12 root      20   0       0      0      0 R  99.3   0.0   0:43.64 ksoftirqd/0
     24 root      20   0       0      0      0 R  99.3   0.0   0:43.62 ksoftirqd/2
     34 root      20   0       0      0      0 R  99.3   0.0   0:43.64 ksoftirqd/4
     39 root      20   0       0      0      0 R  99.3   0.0   0:43.65 ksoftirqd/5
     19 root      20   0       0      0      0 R  99.0   0.0   0:43.63 ksoftirqd/1
     29 root      20   0       0      0      0 R  99.0   0.0   0:43.63 ksoftirqd/3
Quoted 5 lines
Quoted 1 line

Any other ideas? :)

bpf_trampoline_put() got significantly changed by e21aa341785c ("bpf:
Fix fexit trampoline. "), it doesn't do synchronize_rcu_tasks() anymore. Please give it a try. It's in bpf tree.

Quoted 5 lines

Ah! I had missed that patch, and only tested this on bpf-next. Yes, that indeed works better; awesome!

And sorry for bothering you with this, Paul; guess I should have looked harder for fixes first... :/

Glad it is now working!

And in any case, my patch needed an s/true/false/. :-/

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Hey, I did say "untested"! ;-)

Haha, right, well at least you run afoul of the 'truth in advertising' committee ;)

If you get a chance, could you please test the (hopefully) corrected patch shown below? This issue might affect other use cases.

Yup, that does seem to help:

Attaching 2 probes...
enter
exit after 136 ms

-Toke

author maintainer
#11 lore raw
Quoted 138 lines · Toke Høiland-Jørgensen
Quoted 136 lines · Paul E. McKenney
Quoted 134 lines · Toke Høiland-Jørgensen
Quoted 121 lines · Paul E. McKenney
Quoted 120 lines · Toke Høiland-Jørgensen
Quoted 109 lines · Andrii Nakryiko
Quoted 108 lines · Toke Høiland-Jørgensen
Quoted 90 lines · Paul E. McKenney
Quoted 89 lines · Toke Høiland-Jørgensen
Quoted 60 lines · Paul E. McKenney
Quoted 59 lines · Toke Høiland-Jørgensen

Hi Paul

Magnus and I have been debugging an issue where close() on a bpf_link file descriptor would hang indefinitely when the system was under load on a kernel compiled with CONFIG_PREEMPT=y, and it seems to be related to synchronize_rcu_tasks(), so I'm hoping you can help us with it.

The issue is triggered reliably by loading up a system with network traffic (causing 100% softirq CPU load on one or more cores), and then attaching an freplace bpf_link and closing it again. The close() will hang until the network traffic load is lowered.

Digging further, it appears that the hang happens in synchronize_rcu_tasks(), as seen by running a bpftrace script like:

bpftrace -e 'kprobe:synchronize_rcu_tasks { @start = nsecs; printf("enter\n"); } kretprobe:synchronize_rcu_tasks { printf("exit after %d ms\n", (nsecs - @start) / 1000000); }' Attaching 2 probes...
enter
exit after 54 ms
enter
exit after 3249 ms

(the two enter/exit pairs are, respectively, from an unloaded system, and from a loaded system where I stopped the network traffic after a couple of seconds).

The call to synchronize_rcu_tasks() happens in bpf_trampoline_put():

https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L376

And because it does this while holding trampoline_mutex, even deferring the put to a worker (as a previously applied-then-reverted patch did[0]) doesn't help: that'll fix the initial hang on close(), but any subsequent use of BPF trampolines will then be blocked because of the mutex.

Also, if I just keep the network traffic running I will eventually get a kernel panic with:

kernel:[44348.426312] Kernel panic - not syncing: hung_task: blocked tasks

I've created a reproducer for the issue here:
https://github.com/xdp-project/bpf-examples/tree/master/bpf-link-hang

To compile simply do this (needs a recent llvm/clang for compiling the BPF program):

$ git clone --recurse-submodules https://github.com/xdp-project/bpf-examples
$ cd bpf-examples/bpf-link-hang
$ make
$ ./sudo bpf-link-hang

you'll need to load up the system to trigger the hang; I'm using pktgen from a separate machine to do this.

My question is, of course, as ever, What Is To Be Done? Is it expected that synchronize_rcu_tasks() can hang indefinitely on a PREEMPT system, or can this be fixed? And if it is expected, how can the BPF code be fixed so it doesn't deadlock because of this?

Quoted 5 lines
Quoted 1 line

Hoping you can help us with this - many thanks in advance! :)

Let me start with the usual question... Is the network traffic intense enough that one of the CPUs might remain in a loop handling softirqs indefinitely?

Yup, I'm pegging all CPUs in softirq:

$ mpstat -P ALL 1
[...]
18:26:52     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
18:26:53     all    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       0    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       1    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       2    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       3    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       4    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       5    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
Quoted 1 line

If so, does the (untested, probably does not build) patch below help?

Doesn't appear to, no. It builds fine, but I still get:

Attaching 2 probes...
enter
exit after 8480 ms

Quoted 5 lines
Quoted 1 line

(that was me interrupting the network traffic again)

Is your kernel properly shifting from back-of-interrupt softirq processing to ksoftirqd under heavy load? If not, my patch will not have any effect.

Seems to be - this is from top:

     12 root      20   0       0      0      0 R  99.3   0.0   0:43.64 ksoftirqd/0
     24 root      20   0       0      0      0 R  99.3   0.0   0:43.62 ksoftirqd/2
     34 root      20   0       0      0      0 R  99.3   0.0   0:43.64 ksoftirqd/4
     39 root      20   0       0      0      0 R  99.3   0.0   0:43.65 ksoftirqd/5
     19 root      20   0       0      0      0 R  99.0   0.0   0:43.63 ksoftirqd/1
     29 root      20   0       0      0      0 R  99.0   0.0   0:43.63 ksoftirqd/3
Quoted 5 lines
Quoted 1 line

Any other ideas? :)

bpf_trampoline_put() got significantly changed by e21aa341785c ("bpf:
Fix fexit trampoline. "), it doesn't do synchronize_rcu_tasks() anymore. Please give it a try. It's in bpf tree.

Ah! I had missed that patch, and only tested this on bpf-next. Yes, that indeed works better; awesome!

Quoted 8 lines
Quoted 2 lines

And sorry for bothering you with this, Paul; guess I should have looked harder for fixes first... :/

Glad it is now working!

And in any case, my patch needed an s/true/false/. :-/

Hey, I did say "untested"! ;-)

Haha, right, well at least you run afoul of the 'truth in advertising' committee ;)

Quoted 8 lines
Quoted 2 lines

If you get a chance, could you please test the (hopefully) corrected patch shown below? This issue might affect other use cases.

Yup, that does seem to help:

Attaching 2 probes...
enter
exit after 136 ms

Thank you very much! May I please apply your Tested-by?

Thanx, Paul

#12 lore raw
Quoted 142 lines · Paul E. McKenney
Quoted 141 lines · Toke Høiland-Jørgensen
Quoted 138 lines · Paul E. McKenney
Quoted 126 lines · Toke Høiland-Jørgensen
Quoted 124 lines · Paul E. McKenney
Quoted 122 lines · Toke Høiland-Jørgensen
Quoted 109 lines · Andrii Nakryiko
Quoted 108 lines · Toke Høiland-Jørgensen
Quoted 90 lines · Paul E. McKenney
Quoted 89 lines · Toke Høiland-Jørgensen
Quoted 60 lines · Paul E. McKenney
Quoted 59 lines · Toke Høiland-Jørgensen

Hi Paul

Magnus and I have been debugging an issue where close() on a bpf_link file descriptor would hang indefinitely when the system was under load on a kernel compiled with CONFIG_PREEMPT=y, and it seems to be related to synchronize_rcu_tasks(), so I'm hoping you can help us with it.

The issue is triggered reliably by loading up a system with network traffic (causing 100% softirq CPU load on one or more cores), and then attaching an freplace bpf_link and closing it again. The close() will hang until the network traffic load is lowered.

Digging further, it appears that the hang happens in synchronize_rcu_tasks(), as seen by running a bpftrace script like:

bpftrace -e 'kprobe:synchronize_rcu_tasks { @start = nsecs; printf("enter\n"); } kretprobe:synchronize_rcu_tasks { printf("exit after %d ms\n", (nsecs - @start) / 1000000); }' Attaching 2 probes...
enter
exit after 54 ms
enter
exit after 3249 ms

(the two enter/exit pairs are, respectively, from an unloaded system, and from a loaded system where I stopped the network traffic after a couple of seconds).

The call to synchronize_rcu_tasks() happens in bpf_trampoline_put():

https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L376

And because it does this while holding trampoline_mutex, even deferring the put to a worker (as a previously applied-then-reverted patch did[0]) doesn't help: that'll fix the initial hang on close(), but any subsequent use of BPF trampolines will then be blocked because of the mutex.

Also, if I just keep the network traffic running I will eventually get a kernel panic with:

kernel:[44348.426312] Kernel panic - not syncing: hung_task: blocked tasks

I've created a reproducer for the issue here:
https://github.com/xdp-project/bpf-examples/tree/master/bpf-link-hang

To compile simply do this (needs a recent llvm/clang for compiling the BPF program):

$ git clone --recurse-submodules https://github.com/xdp-project/bpf-examples
$ cd bpf-examples/bpf-link-hang
$ make
$ ./sudo bpf-link-hang

you'll need to load up the system to trigger the hang; I'm using pktgen from a separate machine to do this.

My question is, of course, as ever, What Is To Be Done? Is it expected that synchronize_rcu_tasks() can hang indefinitely on a PREEMPT system, or can this be fixed? And if it is expected, how can the BPF code be fixed so it doesn't deadlock because of this?

Quoted 5 lines
Quoted 1 line

Hoping you can help us with this - many thanks in advance! :)

Let me start with the usual question... Is the network traffic intense enough that one of the CPUs might remain in a loop handling softirqs indefinitely?

Yup, I'm pegging all CPUs in softirq:

$ mpstat -P ALL 1
[...]
18:26:52     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
18:26:53     all    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       0    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       1    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       2    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       3    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       4    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       5    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
Quoted 1 line

If so, does the (untested, probably does not build) patch below help?

Doesn't appear to, no. It builds fine, but I still get:

Attaching 2 probes...
enter
exit after 8480 ms

Quoted 5 lines
Quoted 1 line

(that was me interrupting the network traffic again)

Is your kernel properly shifting from back-of-interrupt softirq processing to ksoftirqd under heavy load? If not, my patch will not have any effect.

Seems to be - this is from top:

     12 root      20   0       0      0      0 R  99.3   0.0   0:43.64 ksoftirqd/0
     24 root      20   0       0      0      0 R  99.3   0.0   0:43.62 ksoftirqd/2
     34 root      20   0       0      0      0 R  99.3   0.0   0:43.64 ksoftirqd/4
     39 root      20   0       0      0      0 R  99.3   0.0   0:43.65 ksoftirqd/5
     19 root      20   0       0      0      0 R  99.0   0.0   0:43.63 ksoftirqd/1
     29 root      20   0       0      0      0 R  99.0   0.0   0:43.63 ksoftirqd/3
Quoted 5 lines
Quoted 1 line

Any other ideas? :)

bpf_trampoline_put() got significantly changed by e21aa341785c ("bpf:
Fix fexit trampoline. "), it doesn't do synchronize_rcu_tasks() anymore. Please give it a try. It's in bpf tree.

Ah! I had missed that patch, and only tested this on bpf-next. Yes, that indeed works better; awesome!

And sorry for bothering you with this, Paul; guess I should have looked harder for fixes first... :/

Quoted 8 lines
Quoted 5 lines

Glad it is now working!

And in any case, my patch needed an s/true/false/. :-/

Hey, I did say "untested"! ;-)

Haha, right, well at least you run afoul of the 'truth in advertising' committee ;)

If you get a chance, could you please test the (hopefully) corrected patch shown below? This issue might affect other use cases.

Quoted 7 lines
Quoted 5 lines

Yup, that does seem to help:

Attaching 2 probes...
enter
exit after 136 ms

Thank you very much! May I please apply your Tested-by?

Sure!

author maintainer Tested-by Toke Høiland-Jørgensen Applied
#13 lore raw
Quoted 150 lines · Toke Høiland-Jørgensen
Quoted 148 lines · Paul E. McKenney
Quoted 146 lines · Toke Høiland-Jørgensen
Quoted 131 lines · Paul E. McKenney
Quoted 130 lines · Toke Høiland-Jørgensen
Quoted 128 lines · Paul E. McKenney
Quoted 117 lines · Toke Høiland-Jørgensen
Quoted 109 lines · Andrii Nakryiko
Quoted 108 lines · Toke Høiland-Jørgensen
Quoted 90 lines · Paul E. McKenney
Quoted 89 lines · Toke Høiland-Jørgensen
Quoted 60 lines · Paul E. McKenney
Quoted 59 lines · Toke Høiland-Jørgensen

Hi Paul

Magnus and I have been debugging an issue where close() on a bpf_link
file descriptor would hang indefinitely when the system was under load
on a kernel compiled with CONFIG_PREEMPT=y, and it seems to be related
to synchronize_rcu_tasks(), so I'm hoping you can help us with it.

The issue is triggered reliably by loading up a system with network
traffic (causing 100% softirq CPU load on one or more cores), and then
attaching an freplace bpf_link and closing it again. The close() will
hang until the network traffic load is lowered.

Digging further, it appears that the hang happens in
synchronize_rcu_tasks(), as seen by running a bpftrace script like:

bpftrace -e 'kprobe:synchronize_rcu_tasks { @start = nsecs; printf("enter\n"); } kretprobe:synchronize_rcu_tasks { printf("exit after %d ms\n", (nsecs - @start) / 1000000); }'
Attaching 2 probes...
enter
exit after 54 ms
enter
exit after 3249 ms

(the two enter/exit pairs are, respectively, from an unloaded system,
and from a loaded system where I stopped the network traffic after a
couple of seconds).

The call to synchronize_rcu_tasks() happens in bpf_trampoline_put():

https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L376

And because it does this while holding trampoline_mutex, even deferring
the put to a worker (as a previously applied-then-reverted patch did[0])
doesn't help: that'll fix the initial hang on close(), but any
subsequent use of BPF trampolines will then be blocked because of the
mutex.

Also, if I just keep the network traffic running I will eventually get a
kernel panic with:

kernel:[44348.426312] Kernel panic - not syncing: hung_task: blocked tasks

I've created a reproducer for the issue here:
https://github.com/xdp-project/bpf-examples/tree/master/bpf-link-hang

To compile simply do this (needs a recent llvm/clang for compiling the BPF program):

$ git clone --recurse-submodules https://github.com/xdp-project/bpf-examples
$ cd bpf-examples/bpf-link-hang
$ make
$ ./sudo bpf-link-hang

you'll need to load up the system to trigger the hang; I'm using pktgen
from a separate machine to do this.

My question is, of course, as ever, What Is To Be Done? Is it expected
that synchronize_rcu_tasks() can hang indefinitely on a PREEMPT system,
or can this be fixed? And if it is expected, how can the BPF code be
fixed so it doesn't deadlock because of this?

Quoted 5 lines
Quoted 1 line

Hoping you can help us with this - many thanks in advance! :)

Let me start with the usual question... Is the network traffic intense enough that one of the CPUs might remain in a loop handling softirqs indefinitely?

Yup, I'm pegging all CPUs in softirq:

$ mpstat -P ALL 1
[...]
18:26:52     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
18:26:53     all    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       0    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       1    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       2    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       3    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       4    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       5    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
Quoted 1 line

If so, does the (untested, probably does not build) patch below help?

Doesn't appear to, no. It builds fine, but I still get:

Attaching 2 probes...
enter
exit after 8480 ms

Quoted 5 lines
Quoted 1 line

(that was me interrupting the network traffic again)

Is your kernel properly shifting from back-of-interrupt softirq processing to ksoftirqd under heavy load? If not, my patch will not have any effect.

Seems to be - this is from top:

     12 root      20   0       0      0      0 R  99.3   0.0   0:43.64 ksoftirqd/0
     24 root      20   0       0      0      0 R  99.3   0.0   0:43.62 ksoftirqd/2
     34 root      20   0       0      0      0 R  99.3   0.0   0:43.64 ksoftirqd/4
     39 root      20   0       0      0      0 R  99.3   0.0   0:43.65 ksoftirqd/5
     19 root      20   0       0      0      0 R  99.0   0.0   0:43.63 ksoftirqd/1
     29 root      20   0       0      0      0 R  99.0   0.0   0:43.63 ksoftirqd/3
Quoted 5 lines
Quoted 1 line

Any other ideas? :)

bpf_trampoline_put() got significantly changed by e21aa341785c ("bpf:
Fix fexit trampoline. "), it doesn't do synchronize_rcu_tasks() anymore. Please give it a try. It's in bpf tree.

Quoted 5 lines

Ah! I had missed that patch, and only tested this on bpf-next. Yes, that indeed works better; awesome!

And sorry for bothering you with this, Paul; guess I should have looked harder for fixes first... :/

Glad it is now working!

And in any case, my patch needed an s/true/false/. :-/

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Hey, I did say "untested"! ;-)

Haha, right, well at least you run afoul of the 'truth in advertising' committee ;)

If you get a chance, could you please test the (hopefully) corrected patch shown below? This issue might affect other use cases.

Yup, that does seem to help:

Attaching 2 probes...
enter
exit after 136 ms

Quoted 5 lines
Quoted 1 line

Thank you very much! May I please apply your Tested-by?

Sure!

Applied, and thank you!

Thanx, Paul

------------------------------------------------------------------------

commit 1a0dfc099c1e61e22045705265a1323ac294bea6

Author: Paul E. McKenney <[email protected]>
Date:   Wed Mar 24 17:08:48 2021 -0700

rcu-tasks: Make ksoftirqd provide RCU Tasks quiescent states

Heavy networking load can cause a CPU to execute continuously and indefinitely within ksoftirqd, in which case there will be no voluntary task switches and thus no RCU-tasks quiescent states. This commit therefore causes the exiting rcu_softirq_qs() to provide an RCU-tasks quiescent state.

This of course means that __do_softirq() and its callers cannot be invoked from within a tracing trampoline.

kernel/rcu/tree.c +1 -0
index 0b06be5..9d7cb74 100644
@@ -242,6 +242,7 @@ void rcu_softirq_qs(void)
 {
 	rcu_qs();
 	rcu_preempt_deferred_qs(current);
+	rcu_tasks_qs(current, false);
 }
 
 /*
#14 lore raw
Quoted 151 lines · Paul E. McKenney
Quoted 150 lines · Toke Høiland-Jørgensen
Quoted 148 lines · Paul E. McKenney
Quoted 146 lines · Toke Høiland-Jørgensen
Quoted 131 lines · Paul E. McKenney
Quoted 130 lines · Toke Høiland-Jørgensen
Quoted 128 lines · Paul E. McKenney
Quoted 117 lines · Toke Høiland-Jørgensen
Quoted 109 lines · Andrii Nakryiko
Quoted 108 lines · Toke Høiland-Jørgensen
Quoted 90 lines · Paul E. McKenney
Quoted 89 lines · Toke Høiland-Jørgensen
Quoted 65 lines · Paul E. McKenney

On Tue, Mar 23, 2021 at 01:26:36PM +0100, Toke Høiland-Jørgensen wrote:
Hi Paul

Magnus and I have been debugging an issue where close() on a bpf_link
file descriptor would hang indefinitely when the system was under load
on a kernel compiled with CONFIG_PREEMPT=y, and it seems to be related
to synchronize_rcu_tasks(), so I'm hoping you can help us with it.

The issue is triggered reliably by loading up a system with network
traffic (causing 100% softirq CPU load on one or more cores), and then
attaching an freplace bpf_link and closing it again. The close() will
hang until the network traffic load is lowered.

Digging further, it appears that the hang happens in
synchronize_rcu_tasks(), as seen by running a bpftrace script like:

bpftrace -e 'kprobe:synchronize_rcu_tasks { @start = nsecs; printf("enter\n"); } kretprobe:synchronize_rcu_tasks { printf("exit after %d ms\n", (nsecs - @start) / 1000000); }'
Attaching 2 probes...
enter
exit after 54 ms
enter
exit after 3249 ms

(the two enter/exit pairs are, respectively, from an unloaded system,
and from a loaded system where I stopped the network traffic after a
couple of seconds).

The call to synchronize_rcu_tasks() happens in bpf_trampoline_put():

https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L376

And because it does this while holding trampoline_mutex, even deferring
the put to a worker (as a previously applied-then-reverted patch did[0])
doesn't help: that'll fix the initial hang on close(), but any
subsequent use of BPF trampolines will then be blocked because of the
mutex.

Also, if I just keep the network traffic running I will eventually get a
kernel panic with:

kernel:[44348.426312] Kernel panic - not syncing: hung_task: blocked tasks

I've created a reproducer for the issue here:
https://github.com/xdp-project/bpf-examples/tree/master/bpf-link-hang

To compile simply do this (needs a recent llvm/clang for compiling the BPF program):

$ git clone --recurse-submodules https://github.com/xdp-project/bpf-examples
$ cd bpf-examples/bpf-link-hang
$ make
$ ./sudo bpf-link-hang

you'll need to load up the system to trigger the hang; I'm using pktgen
from a separate machine to do this.

My question is, of course, as ever, What Is To Be Done? Is it expected
that synchronize_rcu_tasks() can hang indefinitely on a PREEMPT system,
or can this be fixed? And if it is expected, how can the BPF code be
fixed so it doesn't deadlock because of this?

Hoping you can help us with this - many thanks in advance! :)

Let me start with the usual question... Is the network traffic intense
enough that one of the CPUs might remain in a loop handling softirqs
indefinitely?

Yup, I'm pegging all CPUs in softirq:

$ mpstat -P ALL 1
[...]
18:26:52     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
18:26:53     all    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       0    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       1    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       2    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       3    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       4    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
18:26:53       5    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00
Quoted 1 line

If so, does the (untested, probably does not build) patch below help?

Doesn't appear to, no. It builds fine, but I still get:

Attaching 2 probes...
enter
exit after 8480 ms

Quoted 5 lines
Quoted 1 line

(that was me interrupting the network traffic again)

Is your kernel properly shifting from back-of-interrupt softirq processing to ksoftirqd under heavy load? If not, my patch will not have any effect.

Seems to be - this is from top:

     12 root      20   0       0      0      0 R  99.3   0.0   0:43.64 ksoftirqd/0
     24 root      20   0       0      0      0 R  99.3   0.0   0:43.62 ksoftirqd/2
     34 root      20   0       0      0      0 R  99.3   0.0   0:43.64 ksoftirqd/4
     39 root      20   0       0      0      0 R  99.3   0.0   0:43.65 ksoftirqd/5
     19 root      20   0       0      0      0 R  99.0   0.0   0:43.63 ksoftirqd/1
     29 root      20   0       0      0      0 R  99.0   0.0   0:43.63 ksoftirqd/3
Quoted 5 lines
Quoted 1 line

Any other ideas? :)

bpf_trampoline_put() got significantly changed by e21aa341785c ("bpf:
Fix fexit trampoline. "), it doesn't do synchronize_rcu_tasks() anymore. Please give it a try. It's in bpf tree.

Quoted 5 lines

Ah! I had missed that patch, and only tested this on bpf-next. Yes, that indeed works better; awesome!

And sorry for bothering you with this, Paul; guess I should have looked harder for fixes first... :/

Glad it is now working!

And in any case, my patch needed an s/true/false/. :-/

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Hey, I did say "untested"! ;-)

Haha, right, well at least you run afoul of the 'truth in advertising' committee ;)

If you get a chance, could you please test the (hopefully) corrected patch shown below? This issue might affect other use cases.

Yup, that does seem to help:

Attaching 2 probes...
enter
exit after 136 ms

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Thank you very much! May I please apply your Tested-by?

Sure!

Applied, and thank you!

Awesome! You're welcome, and thank you for the fix (and the quick turnaround)! :)

-Toke