race condition in schedule_on_each_cpu()
Quoted 38 lines · On Fri, May 31, 2013 at 12:07:15PM +0800, [email protected] wrote:
Quoted 4 lines
the only way for them to get stuck is if there aren't enough execution resources (ie. if a new thread can't be created) but OOM killers would have been activated if that were the case.
The following is a detailed description of our scenerio:
1. after turnning off the the disk array, the ps results is shown in *ps*, which indicates the kworker/1:0 kworker/1:2 are stuck
2. the call stack for the kworkers are shown in *stack_xxx.txt*
3. the workqueue operations during that period is shown in *out.txt*, use ftrace
(we added a new trace point /workqueue_queue_work_insert/, immediately beforeinsert_wq_barrier, in the functionstart_flush_work. its implementation is shown in *trace_insert_wq_barrier.txt*)
from the results int *grep_kwork1:0_from_out.txt*, we can see:
kworker/1:0 is stuck after start work
/fc_starget_delete/ at time 360.801271, and catch theinsert_wq_barriertrace_infobehind this4. from out.txt , we can see, there are altogether three /fc_starget_delete/ work enqueued.
atfer the point of deadlock, kworker/1:1 and kworker/1:3 is
executing ...5. if we let the
scsi_transport_fcuses only one worker thread, i.e., changescsi_transport_fc.c:fc_host_setup()alloc_workqueue(fc_host->work_q_name, 0, 0) to alloc_workqueue(fc_host->work_q_name, WQ_UNBOUND, 1) alloc_workqueue(fc_host->devloss_work_q_name, 0, 0) to
alloc_workqueue(fc_host->devloss_work_q_name, WQ_UNBOUND, 1)
Quoted 8 lines
the deadlock won't occur.
Quoted 3 lines
Can you please test a recent kernel? How easily can you reproduce the issue?
it's occured every time when hot remove disk array.
I'll test recent kernel after a while , but this problem in 3.0.30 really confused me
Yeah, it definitely sounds like concurrency depletion. There have been some fixes and substantial changes in the area, so I really wanna find out whether the problem is reproducible in recent vanilla kernel
- say, v3.9 or, even better, v3.10-rc2. Can you please try to reproduce the problem with a newer kernel?
Quoted 2 lines
by the way, I'm wondering about what's the race condition before which doesn't exist now
Before the commit you originally quoted, the calling thread could be preempted and migrated to another CPU before get_online_cpus() thus ending up executing the function twice on the new cpu but skipping the old one.
Thanks.
Signature · 1 line
tejun