srcu: Queue sdp->work when the delay timer is successfully deleted
Subsystems (3)
Zqiang
reviewerPATCH 1/1 srcu: Queue sdp->work when the delay timer is successfully deleted
In the cleanup_srcu_struct(), when iterating over per-cpu's srcu_data, the timer_delete_sync(&sdp->delay_work) is called to cancel the delay timer before flush_work(&sdp->work).
However, if the timer_delete_sync() returns 1 means that it successfully deleted an pending timer before it had a chance to fire, also means that the sdp->work cannot be queued, the subsequent flush_work(&sdp->work) will returns immediately without waiting for anything, this causes SRCU callbacks to not be processed.
Fix this by checking the return value of timer_delete_sync(), if it returns 1, explicitly queue sdp->work so that the following flush_work() can correctly wait for the work to complete.
--- kernel/rcu/srcutree.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
kernel/rcu/srcutree.c +5 -1
@@ -725,7 +725,11 @@ void cleanup_srcu_struct(struct srcu_struct *ssp)for_each_possible_cpu (cpu) {struct srcu_data *sdp =per_cpu_ptr (ssp->sda, cpu); -timer_delete_sync (&sdp->delay_work); +//In most scenarios, calling srcu_barrier before cleanup +//will not trigger WARN_ON(). +if (WARN_ON (timer_delete_sync (&sdp->delay_work)) && +rcu_cpu_beenfullyonline (sdp->cpu)) +queue_work_on (sdp->cpu, rcu_gp_wq, &sdp->work);flush_work (&sdp->work);if (WARN_ON (rcu_segcblist_n_cbs (&sdp->srcu_cblist)))return ;/* Forgot srcu_barrier(), so just leak it! */
Signature · 1 line
2.17.1
Paul E. McKenney
maintainerQuoted 10 lines · Zqiang
In the
cleanup_srcu_struct(), when iterating over per-cpu'ssrcu_data, thetimer_delete_sync(&sdp->delay_work)is called to cancel the delay timer beforeflush_work(&sdp->work).However, if the
timer_delete_sync()returns 1 means that it successfully deleted an pending timer before it had a chance to fire, also means that thesdp->workcannot be queued, the subsequentflush_work(&sdp->work)will returns immediately without waiting for anything, this causes SRCU callbacks to not be processed.
Quoted 5 lines
Fix this by checking the return value of
timer_delete_sync(), if it returns 1, explicitly queuesdp->workso that the followingflush_work()can correctly wait for the work to complete.Signed-off-by Zqiang <[email protected]>
Good catch! I updated the commit log and comment as shown below. Does that work for you?
Thanx, Paul
Quoted 23 lines
--- kernel/rcu/srcutree.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)kernel/rcu/srcutree.c +5
-1@@ -725,7 +725,11 @@ void cleanup_srcu_struct(struct srcu_struct *ssp)for_each_possible_cpu (cpu) {struct srcu_data *sdp =per_cpu_ptr (ssp->sda, cpu); -timer_delete_sync (&sdp->delay_work); +//In most scenarios, calling srcu_barrier before cleanup +//will not trigger WARN_ON(). +if (WARN_ON (timer_delete_sync (&sdp->delay_work)) && +rcu_cpu_beenfullyonline (sdp->cpu)) +queue_work_on (sdp->cpu, rcu_gp_wq, &sdp->work);flush_work (&sdp->work);if (WARN_ON (rcu_segcblist_n_cbs (&sdp->srcu_cblist)))return ;/* Forgot srcu_barrier(), so just leak it! */ Signature · 1 line
2.17.1
------------------------------------------------------------------------
commit d0b7ba8ff706ad1e3a51ffa46ee79b60ff0203f3
Author: Zqiang <[email protected]> Date: Thu Jul 9 18:06:02 2026 +0800
srcu: Queue sdp->work when the delay timer is successfully deleted
In the cleanup_srcu_struct() function, when iterating over per-cpu's srcu_data, timer_delete_sync(&sdp->delay_work) is called to cancel the delayed work before doing flush_work(&sdp->work).
However, suppose that timer_delete_sync() returns 1, which means that it successfully deleted an pending timer before it had a chance to fire. But this also means that the sdp->work will not be queued, so that the subsequent flush_work(&sdp->work) will returns immediately without waiting for anything. Taken together, all of this means that any recently queued SRCU callbacks to not be invoked, which can result in memory leaks, hangs, or worse.
Fix this by checking the return value of timer_delete_sync(), if it returns 1, explicitly queue sdp->work so that the callbacks will be invoked and the following flush_work() will correctly wait for all of those callbacks to finish executing.
kernel/rcu/srcutree.c +5 -1
@@ -701,7 +701,11 @@ void cleanup_srcu_struct(struct srcu_struct *ssp)for_each_possible_cpu (cpu) {struct srcu_data *sdp =per_cpu_ptr (ssp->sda, cpu); -timer_delete_sync (&sdp->delay_work); +// Call srcu_barrier() before this cleanup_srcu_struct() +// to avoid triggering this WARN_ON(). +if (WARN_ON (timer_delete_sync (&sdp->delay_work)) && +rcu_cpu_beenfullyonline (sdp->cpu)) +queue_work_on (sdp->cpu, rcu_gp_wq, &sdp->work);flush_work (&sdp->work);if (WARN_ON (rcu_segcblist_n_cbs (&sdp->srcu_cblist)))return ;/* Forgot srcu_barrier(), so just leak it! */
Hello Zqiang,
Quoted 24 lines · Zqiang
In the
cleanup_srcu_struct(), when iterating over per-cpu'ssrcu_data, thetimer_delete_sync(&sdp->delay_work)is called to cancel the delay timer beforeflush_work(&sdp->work).However, if the
timer_delete_sync()returns 1 means that it successfully deleted an pending timer before it had a chance to fire, also means that thesdp->workcannot be queued, the subsequentflush_work(&sdp->work)will returns immediately without waiting for anything, this causes SRCU callbacks to not be processed.Fix this by checking the return value of
timer_delete_sync(), if it returns 1, explicitly queuesdp->workso that the followingflush_work()can correctly wait for the work to complete.Signed-off-by Zqiang <[email protected]>--- kernel/rcu/srcutree.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)kernel/rcu/srcutree.c +0
-0
kernel/rcu/srcutree.c void cleanup_srcu_struct(struct srcu_struct *ssp) · line 732
Quoted 9 lines
kernel/rcu/srcutree.c +5
-1@@ -725,7 +725,11 @@ void cleanup_srcu_struct(struct srcu_struct *ssp)for_each_possible_cpu (cpu) {struct srcu_data *sdp =per_cpu_ptr (ssp->sda, cpu); -timer_delete_sync (&sdp->delay_work); +//In most scenarios, calling srcu_barrier before cleanup +//will not trigger WARN_ON(). +if (WARN_ON (timer_delete_sync (&sdp->delay_work)) && +rcu_cpu_beenfullyonline (sdp->cpu)) +queue_work_on (sdp->cpu, rcu_gp_wq, &sdp->work);
I started seeing this on my tests, it is not trivial to decode this one, but, I can try harder if _really_ needed.
07:11:44 Invalid logical block size (65535)
Invalid logical block size (256)
07:11:45 WARNING: kernel/rcu/srcutree.c:706 at cleanup_srcu_struct+0x408/0x7e8, CPU#50: stress-ng-loop/1233713 ^[[36m[warn]^[[0m
Modules linked in: sch_htb(E) cls_u32(E) cls_cgroup(E) cls_basic(E) act_police(E) act_mirred(E) act_bpf(E) sch_fq(E) br_netfilter(E) udp_diag(E) i
CPU: 50 UID: 0 PID: 1233713 Comm: stress-ng-loop Kdump: loaded Tainted: G E 7.2.0-rc5-next-20260730upstream-baseline #1 PREEMPT(f
Tainted: [E]=UNSIGNED_MODULE
pstate: 03401009 (nzcv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--)
pc : cleanup_srcu_struct+0x408/0x7e8
lr : cleanup_srcu_struct+0x3dc/0x7e8
sp : ffff800150357b30
x29: ffff800150357b50 x28: fffffdffbdb51c40 x27: 0000000000000000
x26: dfff800000000000 x25: 1fffe000202daa5f x24: ffff800083bbb010
x23: ffff800083f67210 x22: ffff0001016d52f8 x21: 00007dc98b656c40
x20: 1ffff000107ece42 x19: 0000000000000003 x18: 1fffe006d6db3f20
x17: 0000000000000000 x16: ffff800083f67000 x15: 0000000000000001
x14: 1fffe006d6db6388 x13: 0000000000000000 x12: 0000000000000000
x11: 0000000000000003 x10: fffffdffbdb51e40 x9 : dfff800000000000
x8 : 0000000100000000 x7 : 0000000000000000 x6 : ffff80008148f060
x5 : 0000000000000000 x4 : 0000000000000001 x3 : 0000000000000010
x2 : ffff800150357a60 x1 : ffff80008036827c x0 : 0000000000000001
Call trace:
cleanup_srcu_struct+0x408/0x7e8 (P)
blk_mq_free_tag_set+0x4b4/0x5f0
loop_remove+0x50/0xd8 [loop]
loop_control_ioctl+0x340/0x438 [loop]
__arm64_sys_ioctl+0xc04/0x1310
invoke_syscall+0x74/0x188
do_el0_svc+0x10c/0x198
el0_svc+0x64/0x260
el0t_64_sync_handler+0x84/0x130
el0t_64_sync+0x198/0x1a0
irq event stamp: 0
hardirqs last enabled at (0): [<0000000000000000>] 0x0
hardirqs last disabled at (0): [<ffff800080100734>] copy_process+0x109c/0x31e8
softirqs last enabled at (0): [<ffff800080100760>] copy_process+0x10c8/0x31e8
softirqs last disabled at (0): [<0000000000000000>] 0x0
---[ end trace 0000000000000000 ]---Zqiang
reviewerQuoted 40 lines
Hello Zqiang,
Quoted 28 lines · Zqiang
In the
cleanup_srcu_struct(), when iterating over per-cpu'ssrcu_data,
thetimer_delete_sync(&sdp->delay_work)is called to cancel the delay timer beforeflush_work(&sdp->work).However, if the
timer_delete_sync()returns 1 means that it successfully deleted an pending timer before it had a chance to fire, also means that thesdp->workcannot be queued, the subsequentflush_work(&sdp->work)will returns immediately without waiting for anything, this causes SRCU callbacks to not be processed.Fix this by checking the return value of
timer_delete_sync(), if it returns 1, explicitly queuesdp->workso that the followingflush_work()can correctly wait for the work to complete.Signed-off-by Zqiang <[email protected]>---
kernel/rcu/srcutree.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)diff --git
a/kernel/rcu/srcutree.cb/kernel/rcu/srcutree.cindex7c2f7cc131f7..02c322b7c6f1100644
---a/kernel/rcu/srcutree.c
+++b/kernel/rcu/srcutree.c
@@ -725,7 +725,11 @@ voidcleanup_srcu_struct(struct srcu_struct *ssp)for_each_possible_cpu(cpu){
structsrcu_data*sdp =per_cpu_ptr(ssp->sda, cpu);Quoted 7 lines
-
timer_delete_sync(&sdp->delay_work);
+ //In most scenarios, callingsrcu_barrierbefore cleanup
+ //will not triggerWARN_ON().
+ if (WARN_ON(timer_delete_sync(&sdp->delay_work)) &&
+rcu_cpu_beenfullyonline(sdp->cpu))
+queue_work_on(sdp->cpu, rcu_gp_wq, &sdp->work);
Quoted 3 lines
I started seeing this on my tests, it is not trivial to decode this one, but, I can try harder if _really_ needed.
The scenario I can think of is that we missed the call to srcu_barrier() before cleanup_srcu_struct():
loop_add()
->blk_mq_alloc_tag_set
init_srcu_struct(&set->tags_srcu)
blk_mq_alloc_set_map_and_rqs() {
->__blk_mq_alloc_rq_maps()
->__blk_mq_alloc_map_and_rqs() return error
goto out_unwind: __blk_mq_free_map_and_rqs()
->blk_mq_free_rq_map()
->blk_mq_free_tags()
->call_srcu(&set->tags_srcu, &tags->rcu_head, blk_mq_free_tags_callback);
} return error
goto out_free_mq_map:
....
cleanup_srcu_struct(&set->tags_srcu)
-> trigger WARN_ON(timer_delete_sync(&sdp->delay_work)Can you try the following patch?
block/blk-mq.c +1 -0
@@ -4925,6 +4925,7 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set)kfree (set->tags); set->tags =NULL ; out_cleanup_tags_srcu: +srcu_barrier (&set->tags_srcu);cleanup_srcu_struct (&set->tags_srcu); out_cleanup_srcu:if (set->flags &BLK_MQ_F_BLOCKING )
Thanks
Zqiang
Quoted 39 lines
07:11:44 Invalid logical block size (65535)
Invalid logical block size (256)
07:11:45 WARNING:kernel/rcu/srcutree.c:706atcleanup_srcu_struct+0x408/0x7e8, CPU#50: stress-ng-loop/1233713 ^[[36m[warn]^[[0mModules linked in: sch_htb(E) cls_u32(E) cls_cgroup(E) cls_basic(E) act_police(E) act_mirred(E) act_bpf(E) sch_fq(E) br_netfilter(E) udp_diag(E) i CPU: 50 UID: 0 PID: 1233713 Comm: stress-ng-loop Kdump: loaded Tainted: G E 7.2.0-rc5-next-20260730upstream-baseline #1 PREEMPT(f Tainted: [E]=UNSIGNED_MODULE pstate: 03401009 (nzcv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--) pc : cleanup_srcu_struct+0x408/0x7e8 lr : cleanup_srcu_struct+0x3dc/0x7e8 sp : ffff800150357b30 x29: ffff800150357b50 x28: fffffdffbdb51c40 x27: 0000000000000000 x26: dfff800000000000 x25: 1fffe000202daa5f x24: ffff800083bbb010 x23: ffff800083f67210 x22: ffff0001016d52f8 x21: 00007dc98b656c40 x20: 1ffff000107ece42 x19: 0000000000000003 x18: 1fffe006d6db3f20 x17: 0000000000000000 x16: ffff800083f67000 x15: 0000000000000001 x14: 1fffe006d6db6388 x13: 0000000000000000 x12: 0000000000000000 x11: 0000000000000003 x10: fffffdffbdb51e40 x9 : dfff800000000000 x8 : 0000000100000000 x7 : 0000000000000000 x6 : ffff80008148f060 x5 : 0000000000000000 x4 : 0000000000000001 x3 : 0000000000000010 x2 : ffff800150357a60 x1 : ffff80008036827c x0 : 0000000000000001 Call trace: cleanup_srcu_struct+0x408/0x7e8 (P) blk_mq_free_tag_set+0x4b4/0x5f0 loop_remove+0x50/0xd8 [loop] loop_control_ioctl+0x340/0x438 [loop] __arm64_sys_ioctl+0xc04/0x1310 invoke_syscall+0x74/0x188 do_el0_svc+0x10c/0x198 el0_svc+0x64/0x260 el0t_64_sync_handler+0x84/0x130 el0t_64_sync+0x198/0x1a0 irq event stamp: 0 hardirqs last enabled at (0): [<0000000000000000>] 0x0 hardirqs last disabled at (0): [<ffff800080100734>] copy_process+0x109c/0x31e8 softirqs last enabled at (0): [<ffff800080100760>] copy_process+0x10c8/0x31e8 softirqs last disabled at (0): [<0000000000000000>] 0x0 ---[ end trace 0000000000000000 ]---
Quoted 65 lines · Zqiang
Quoted 40 lines
Hello Zqiang,
Quoted 28 lines · Zqiang
In the
cleanup_srcu_struct(), when iterating over per-cpu'ssrcu_data,
thetimer_delete_sync(&sdp->delay_work)is called to cancel the delay timer beforeflush_work(&sdp->work).However, if the
timer_delete_sync()returns 1 means that it successfully deleted an pending timer before it had a chance to fire, also means that thesdp->workcannot be queued, the subsequentflush_work(&sdp->work)will returns immediately without waiting for anything, this causes SRCU callbacks to not be processed.Fix this by checking the return value of
timer_delete_sync(), if it returns 1, explicitly queuesdp->workso that the followingflush_work()can correctly wait for the work to complete.Signed-off-by Zqiang <[email protected]>---
kernel/rcu/srcutree.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)diff --git
a/kernel/rcu/srcutree.cb/kernel/rcu/srcutree.cindex7c2f7cc131f7..02c322b7c6f1100644
---a/kernel/rcu/srcutree.c
+++b/kernel/rcu/srcutree.c
@@ -725,7 +725,11 @@ voidcleanup_srcu_struct(struct srcu_struct *ssp)for_each_possible_cpu(cpu){
structsrcu_data*sdp =per_cpu_ptr(ssp->sda, cpu);Quoted 7 lines
-
timer_delete_sync(&sdp->delay_work);
+ //In most scenarios, callingsrcu_barrierbefore cleanup
+ //will not triggerWARN_ON().
+ if (WARN_ON(timer_delete_sync(&sdp->delay_work)) &&
+rcu_cpu_beenfullyonline(sdp->cpu))
+queue_work_on(sdp->cpu, rcu_gp_wq, &sdp->work);Quoted 3 lines
I started seeing this on my tests, it is not trivial to decode this one, but, I can try harder if _really_ needed.
The scenario I can think of is that we missed the call to
srcu_barrier()beforecleanup_srcu_struct():
loop_add()
->blk_mq_alloc_tag_setinit_srcu_struct(&set->tags_srcu) blk_mq_alloc_set_map_and_rqs() { ->__blk_mq_alloc_rq_maps() ->__blk_mq_alloc_map_and_rqs() return error goto out_unwind: __blk_mq_free_map_and_rqs() ->blk_mq_free_rq_map() ->blk_mq_free_tags() ->call_srcu(&set->tags_srcu, &tags->rcu_head, blk_mq_free_tags_callback); } return error goto out_free_mq_map: .... cleanup_srcu_struct(&set->tags_srcu) -> trigger WARN_ON(timer_delete_sync(&sdp->delay_work)
Quoted 1 line
Can you try the following patch?
I tried it, and it does not help -- the WARN still fires at the same rate. I think the analysis points at the wrong call site.
Setup: linux-next-20260731 (arm64), 32 vCPU VM, HZ=1000, PROVE_LOCKING and DEBUG_OBJECTS_TIMERS enabled, reproducer stress-ng --loop 32 --timeout 60s.
baseline 3 x WARN srcutree.c:706 in 60s
+ your blk-mq patch 4 x WARN srcutree.c:706 in 60s
(3 vs 4 is just jitter on a one-jiffy race, not a regression.)
The reason it cannot help is that the splat comes from blk_mq_free_tag_set(), not from the blk_mq_alloc_tag_set() error path your patch touches. All four splats in the patched run have the same trace:
cleanup_srcu_struct+0x274/0x450 (P)
blk_mq_free_tag_set+0x1a4/0x1e0
loop_remove+0x2c/0x78
loop_control_ioctl+0x248/0x2a0
__arm64_sys_ioctl+0x9c0/0xb00I also put a pr_warn() at out_cleanup_tags_srcu: to be sure -- it fired zero times over the whole run, so loop_add() never takes that error path in this workload.
Why do you wangt to have this
WARN_ON(timer_delete_sync(&sdp->delay_work)) ?
timer_delete_sync() != 0 means "a timer was armed", not "callbacks are pending".
Zqiang
reviewerQuoted 68 lines · Zqiang
Quoted 66 lines
Hello Zqiang,
Quoted 25 lines · Zqiang
In the
cleanup_srcu_struct(), when iterating over per-cpu'ssrcu_data, thetimer_delete_sync(&sdp->delay_work)is called to cancel the delay timer beforeflush_work(&sdp->work).However, if the
timer_delete_sync()returns 1 means that it successfully deleted an pending timer before it had a chance to fire, also means that thesdp->workcannot be queued, the subsequentflush_work(&sdp->work)will returns immediately without waiting for anything, this causes SRCU callbacks to not be processed.Fix this by checking the return value of
timer_delete_sync(), if it returns 1, explicitly queuesdp->workso that the followingflush_work()can correctly wait for the work to complete.Signed-off-by Zqiang <[email protected]>---
kernel/rcu/srcutree.c| 6 +++++-1 file changed, 5 insertions(+), 1 deletion(-)kernel/rcu/srcutree.c +0
-0kernel/rcu/srcutree.c void cleanup_srcu_struct(struct srcu_struct *ssp) · line 733
Quoted 10 lines
kernel/rcu/srcutree.c +5
-1@@ -725,7 +725,11 @@ void cleanup_srcu_struct(struct srcu_struct *ssp)for_each_possible_cpu (cpu) {struct srcu_data *sdp =per_cpu_ptr (ssp->sda, cpu); -timer_delete_sync (&sdp->delay_work); +//In most scenarios, calling srcu_barrier before cleanup +//will not trigger WARN_ON(). +if (WARN_ON (timer_delete_sync (&sdp->delay_work)) && +rcu_cpu_beenfullyonline (sdp->cpu)) +queue_work_on (sdp->cpu, rcu_gp_wq, &sdp->work);I started seeing this on my tests, it is not trivial to decode this one, but, I can try harder if _really_ needed.
The scenario I can think of is that we missed the call to
srcu_barrier()beforecleanup_srcu_struct():
loop_add()
->blk_mq_alloc_tag_set
init_srcu_struct(&set->tags_srcu)
blk_mq_alloc_set_map_and_rqs(){
->__blk_mq_alloc_rq_maps()
->__blk_mq_alloc_map_and_rqs() return error
gotoout_unwind:__blk_mq_free_map_and_rqs()->blk_mq_free_rq_map()
->blk_mq_free_tags()
->call_srcu(&set->tags_srcu,&tags->rcu_head,blk_mq_free_tags_callback); } return errorgoto
out_free_mq_map:
....
cleanup_srcu_struct(&set->tags_srcu)
-> trigger WARN_ON(timer_delete_sync(&sdp->delay_work)
Quoted 8 lines
Quoted 2 lines
Can you try the following patch?
I tried it, and it does not help -- the WARN still fires at the same rate. I think the analysis points at the wrong call site.
Setup: linux-next-20260731 (arm64), 32 vCPU VM, HZ=1000,
PROVE_LOCKINGandDEBUG_OBJECTS_TIMERSenabled, reproducer stress-ng --loop 32 --timeout 60s.
Thanks for provide testing methods, I will also testing it.
Quoted 10 lines
baseline 3 x WARN srcutree.c:706 in 60s
+ your blk-mq patch 4 x WARN srcutree.c:706 in 60s(3 vs 4 is just jitter on a one-jiffy race, not a regression.)
The reason it cannot help is that the splat comes from
blk_mq_free_tag_set(), not from theblk_mq_alloc_tag_set()error path your patch touches. All four splats in the patched run have the same trace:
Quoted 5 lines
cleanup_srcu_struct+0x274/0x450 (P) blk_mq_free_tag_set+0x1a4/0x1e0 loop_remove+0x2c/0x78 loop_control_ioctl+0x248/0x2a0 __arm64_sys_ioctl+0x9c0/0xb00
This may trigger a new srcu grace period again during the window period between srcu-barrier() and cleanup_srcu_struct().
Quoted 10 lines
I also put a
pr_warn()atout_cleanup_tags_srcu: to be sure -- it fired zero times over the whole run, soloop_add()never takes that error path in this workload.Why do you wangt to have this
WARN_ON(timer_delete_sync(&sdp->delay_work)) ?
timer_delete_sync()!= 0 "a timer was armed", not "callbacks are pending".
There are only two types of return values for timer_delete_sync(), return 0 or 1, the timer_delete_sync() != 0 means that this timer was pending and has been deactivated, right?
Thanks
Zqiang
Quoted 1 line
Zqiang
reviewerQuoted 107 lines
Quoted 69 lines · Zqiang
Hello Zqiang,
Quoted 25 lines · Zqiang
In the
cleanup_srcu_struct(), when iterating over per-cpu'ssrcu_data, thetimer_delete_sync(&sdp->delay_work)is called to cancel the delay timer beforeflush_work(&sdp->work).However, if the
timer_delete_sync()returns 1 means that it successfully deleted an pending timer before it had a chance to fire, also means that thesdp->workcannot be queued, the subsequentflush_work(&sdp->work)will returns immediately without waiting for anything, this causes SRCU callbacks to not be processed.Fix this by checking the return value of
timer_delete_sync(), if it returns 1, explicitly queuesdp->workso that the followingflush_work()can correctly wait for the work to complete.Signed-off-by Zqiang <[email protected]>---
kernel/rcu/srcutree.c| 6 +++++-1 file changed, 5 insertions(+), 1 deletion(-)kernel/rcu/srcutree.c +0
-0kernel/rcu/srcutree.c void cleanup_srcu_struct(struct srcu_struct *ssp) · line 733
Quoted 10 lines
kernel/rcu/srcutree.c +5
-1@@ -725,7 +725,11 @@ void cleanup_srcu_struct(struct srcu_struct *ssp)for_each_possible_cpu (cpu) {struct srcu_data *sdp =per_cpu_ptr (ssp->sda, cpu); -timer_delete_sync (&sdp->delay_work); +//In most scenarios, calling srcu_barrier before cleanup +//will not trigger WARN_ON(). +if (WARN_ON (timer_delete_sync (&sdp->delay_work)) && +rcu_cpu_beenfullyonline (sdp->cpu)) +queue_work_on (sdp->cpu, rcu_gp_wq, &sdp->work);I started seeing this on my tests, it is not trivial to decode this one, but, I can try harder if _really_ needed.
The scenario I can think of is that we missed the call to
srcu_barrier()beforecleanup_srcu_struct():
loop_add()
->blk_mq_alloc_tag_set
init_srcu_struct(&set->tags_srcu)
blk_mq_alloc_set_map_and_rqs(){
->__blk_mq_alloc_rq_maps()
->__blk_mq_alloc_map_and_rqs() return error
gotoout_unwind:__blk_mq_free_map_and_rqs()->blk_mq_free_rq_map()
->blk_mq_free_tags()
->call_srcu(&set->tags_srcu,&tags->rcu_head,blk_mq_free_tags_callback); } return errorgoto
out_free_mq_map:
....
cleanup_srcu_struct(&set->tags_srcu)
-> trigger WARN_ON(timer_delete_sync(&sdp->delay_work)Quoted 8 lines
Can you try the following patch?
I tried it, and it does not help -- the WARN still fires at the same rate. I think the analysis points at the wrong call site.
Setup: linux-next-20260731 (arm64), 32 vCPU VM, HZ=1000,
PROVE_LOCKINGandDEBUG_OBJECTS_TIMERSenabled, reproducer stress-ng --loop 32 --timeout 60s.Thanks for provide testing methods, I will also testing it.
Quoted 11 lines
baseline 3 x WARN srcutree.c:706 in 60s
+ your blk-mq patch 4 x WARN srcutree.c:706 in 60s(3 vs 4 is just jitter on a one-jiffy race, not a regression.)
The reason it cannot help is that the splat comes from
blk_mq_free_tag_set(), not from theblk_mq_alloc_tag_set()error path your patch touches. All four splats in the patched run have the same trace:Quoted 6 lines
cleanup_srcu_struct+0x274/0x450 (P) blk_mq_free_tag_set+0x1a4/0x1e0 loop_remove+0x2c/0x78 loop_control_ioctl+0x248/0x2a0 __arm64_sys_ioctl+0x9c0/0xb00This may trigger a new srcu grace period again during the window period between srcu-barrier() and
cleanup_srcu_struct().Quoted 8 lines
I also put a
pr_warn()atout_cleanup_tags_srcu: to be sure -- it fired
zero times over the whole run, soloop_add()never takes that error path in this workload.Why do you wangt to have this
WARN_ON(timer_delete_sync(&sdp->delay_work)) ?
Quoted 6 lines
Quoted 3 lines
timer_delete_sync()!= 0 "a timer was armed", not "callbacks are pending".There are only two types of return values for
timer_delete_sync(), return 0 or 1, thetimer_delete_sync()!= 0 means that this timer was pending and has been deactivated, right?
How about this?
kernel/rcu/srcutree.c +2 -1
@@ -751,7 +751,8 @@ void cleanup_srcu_struct(struct srcu_struct *ssp)// Call srcu_barrier() before this cleanup_srcu_struct() // to avoid triggering this WARN_ON(). -if (WARN_ON (timer_delete_sync (&sdp->delay_work)) && +if (WARN_ON (rcu_segcblist_n_cbs (&sdp->srcu_cblist) && +timer_delete_sync (&sdp->delay_work)) &&rcu_cpu_beenfullyonline (sdp->cpu))queue_work_on (sdp->cpu, rcu_gp_wq, &sdp->work);flush_work (&sdp->work);
Thanks
Zqiang
Quoted 8 lines
Thanks
ZqiangQuoted 3 lines
Zqiang
reviewerQuoted 121 lines
Quoted 107 lines · Zqiang
Hello Zqiang,
Quoted 25 lines · Zqiang
In the
cleanup_srcu_struct(), when iterating over per-cpu'ssrcu_data, thetimer_delete_sync(&sdp->delay_work)is called to cancel the delay timer beforeflush_work(&sdp->work).However, if the
timer_delete_sync()returns 1 means that it successfully deleted an pending timer before it had a chance to fire, also means that thesdp->workcannot be queued, the subsequentflush_work(&sdp->work)will returns immediately without waiting for anything, this causes SRCU callbacks to not be processed.Fix this by checking the return value of
timer_delete_sync(), if it returns 1, explicitly queuesdp->workso that the followingflush_work()can correctly wait for the work to complete.Signed-off-by Zqiang <[email protected]>---
kernel/rcu/srcutree.c| 6 +++++-1 file changed, 5 insertions(+), 1 deletion(-)kernel/rcu/srcutree.c +0
-0kernel/rcu/srcutree.c void cleanup_srcu_struct(struct srcu_struct *ssp) · line 733
Quoted 10 lines
kernel/rcu/srcutree.c +5
-1@@ -725,7 +725,11 @@ void cleanup_srcu_struct(struct srcu_struct *ssp)for_each_possible_cpu (cpu) {struct srcu_data *sdp =per_cpu_ptr (ssp->sda, cpu); -timer_delete_sync (&sdp->delay_work); +//In most scenarios, calling srcu_barrier before cleanup +//will not trigger WARN_ON(). +if (WARN_ON (timer_delete_sync (&sdp->delay_work)) && +rcu_cpu_beenfullyonline (sdp->cpu)) +queue_work_on (sdp->cpu, rcu_gp_wq, &sdp->work);I started seeing this on my tests, it is not trivial to decode this one, but, I can try harder if _really_ needed.
The scenario I can think of is that we missed the call to
srcu_barrier()beforecleanup_srcu_struct():
loop_add()
->blk_mq_alloc_tag_set
init_srcu_struct(&set->tags_srcu)
blk_mq_alloc_set_map_and_rqs(){
->__blk_mq_alloc_rq_maps()
->__blk_mq_alloc_map_and_rqs() return error
gotoout_unwind:__blk_mq_free_map_and_rqs()->blk_mq_free_rq_map()
->blk_mq_free_tags()
->call_srcu(&set->tags_srcu,&tags->rcu_head,blk_mq_free_tags_callback); } return errorgoto
out_free_mq_map:
....
cleanup_srcu_struct(&set->tags_srcu)
-> trigger WARN_ON(timer_delete_sync(&sdp->delay_work)Can you try the following patch?
I tried it, and it does not help -- the WARN still fires at the same rate. I think the analysis points at the wrong call site.
Setup: linux-next-20260731 (arm64), 32 vCPU VM, HZ=1000,
PROVE_LOCKINGandDEBUG_OBJECTS_TIMERSenabled, reproducer stress-ng --loop 32 --timeout 60s.Thanks for provide testing methods, I will also testing it.
baseline 3 x WARN srcutree.c:706 in 60s
+ your blk-mq patch 4 x WARN srcutree.c:706 in 60s(3 vs 4 is just jitter on a one-jiffy race, not a regression.)
The reason it cannot help is that the splat comes from
blk_mq_free_tag_set(), not from theblk_mq_alloc_tag_set()error path your patch touches. All four splats in the patched run have the same trace:cleanup_srcu_struct+0x274/0x450 (P) blk_mq_free_tag_set+0x1a4/0x1e0 loop_remove+0x2c/0x78 loop_control_ioctl+0x248/0x2a0 __arm64_sys_ioctl+0x9c0/0xb00This may trigger a new srcu grace period again during the window period between srcu-barrier() and
cleanup_srcu_struct().I also put a
pr_warn()atout_cleanup_tags_srcu: to be sure -- it fired zero times over the whole run, soloop_add()never takes that error path in this workload.Why do you wangt to have this
WARN_ON(timer_delete_sync(&sdp->delay_work)) ?Quoted 7 lines
timer_delete_sync()!= 0 "a timer was armed", not "callbacks are pending".There are only two types of return values for
timer_delete_sync(), return 0 or 1, thetimer_delete_sync()!= 0 means that this timer was pending and has been deactivated, right?How about this?
kernel/rcu/srcutree.c +0
-0
kernel/rcu/srcutree.c void cleanup_srcu_struct(struct srcu_struct *ssp) · line 758
Quoted 9 lines
kernel/rcu/srcutree.c +2
-1@@ -751,7 +751,8 @@ void cleanup_srcu_struct(struct srcu_struct *ssp)// Call srcu_barrier() before this cleanup_srcu_struct() // to avoid triggering this WARN_ON(). -if (WARN_ON (timer_delete_sync (&sdp->delay_work)) && +if (WARN_ON (rcu_segcblist_n_cbs (&sdp->srcu_cblist) && +timer_delete_sync (&sdp->delay_work)) &&rcu_cpu_beenfullyonline (sdp->cpu))queue_work_on (sdp->cpu, rcu_gp_wq, &sdp->work);flush_work (&sdp->work);
Please ignore this change, this is my mistake.
There is a scenario where the 'rcu_segcblist_n_cbs(&sdp->srcu_cblist) == 0' srcu_barrier() cannot intercept.
however, for sup->srcu_size_state being between SRCU_SIZE_WAIT_BARRIER and SRCU_SIZE_BIG, we will queue sdp->delay_work for all CPUs belonging to the leaf SRCU node, regardless of whether there is a callback on the current CPU's sdp->srcu_cblist in srcu_gp_end().
so the timer_delete_sync(&sdp->delay_work) should be called unconditionally. to ensure that the timer callback has ended or the timer which in pending status has been successfully deleted.
kernel/rcu/srcutree.c +3 -2
@@ -751,8 +751,9 @@ void cleanup_srcu_struct(struct srcu_struct *ssp)// Call srcu_barrier() before this cleanup_srcu_struct() // to avoid triggering this WARN_ON(). -if (WARN_ON (timer_delete_sync (&sdp->delay_work)) && -rcu_cpu_beenfullyonline (sdp->cpu)) +if (WARN_ON (timer_delete_sync (&sdp->delay_work) && +rcu_segcblist_n_cbs (&sdp->srcu_cblist)) && +rcu_cpu_beenfullyonline (sdp->cpu))queue_work_on (sdp->cpu, rcu_gp_wq, &sdp->work);flush_work (&sdp->work);if (WARN_ON (rcu_segcblist_n_cbs (&sdp->srcu_cblist)))
Thanks
Zqiang
Quoted 9 lines
Thanks
ZqiangQuoted 4 lines
Thanks
Zqiang
kernel/rcu/srcutree.c void cleanup_srcu_struct(struct srcu_struct *ssp) · line 756
Quoted 13 lines · Zqiang
kernel/rcu/srcutree.c +3
-2@@ -751,8 +751,9 @@ void cleanup_srcu_struct(struct srcu_struct *ssp)// Call srcu_barrier() before this cleanup_srcu_struct() // to avoid triggering this WARN_ON(). -if (WARN_ON (timer_delete_sync (&sdp->delay_work)) && -rcu_cpu_beenfullyonline (sdp->cpu)) +if (WARN_ON (timer_delete_sync (&sdp->delay_work) && +rcu_segcblist_n_cbs (&sdp->srcu_cblist)) && +rcu_cpu_beenfullyonline (sdp->cpu))
Isn't the WARN_ON() the same? thus it will hit again?
Zqiang
reviewerQuoted 10 lines · Zqiang
Quoted 8 lines
kernel/rcu/srcutree.c +0
-0index b9fe57ff9100..d13c12f61150 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -751,8 +751,9 @@ void cleanup_srcu_struct(struct srcu_struct *ssp)
Quoted 9 lines
Quoted 8 lines
// Call
srcu_barrier()before thiscleanup_srcu_struct()// to avoid triggering thisWARN_ON().
- if (WARN_ON(timer_delete_sync(&sdp->delay_work)) &&
-rcu_cpu_beenfullyonline(sdp->cpu))
+ if (WARN_ON(timer_delete_sync(&sdp->delay_work) &&
+rcu_segcblist_n_cbs(&sdp->srcu_cblist)) &&
+rcu_cpu_beenfullyonline(sdp->cpu))Isn't the
WARN_ON()the same? thus it will hit again?
WARN_ON(timer_delete_sync(&sdp->delay_work) && rcu_segcblist_n_cbs(&sdp->srcu_cblist))
Thanks
Zqiang