Skip to contentSkip to conversation

race condition in schedule_on_each_cpu()

DISCUSSION Tejun Heo 1 participant 1 message May 31, 2013 3 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

Tejun Heo

author
#3 lore raw
Quoted 38 lines · On Fri, May 31, 2013 at 12:07:15PM +0800, [email protected] wrote:
Quoted 4 lines

the only way for them to get stuck is if there aren't enough execution resources (ie. if a new thread can't be created) but OOM killers would have been activated if that were the case.

The following is a detailed description of our scenerio:

1. after turnning off the the disk array, the ps results is shown in *ps*, which indicates the kworker/1:0 kworker/1:2 are stuck

2. the call stack for the kworkers are shown in *stack_xxx.txt*

3. the workqueue operations during that period is shown in *out.txt*, use ftrace
(we added a new trace point /workqueue_queue_work_insert/, immediately before insert_wq_barrier, in the function start_flush_work. its implementation is shown in *trace_insert_wq_barrier.txt*)
from the results int *grep_kwork1:0_from_out.txt*, we can see:
kworker/1:0 is stuck after start work
/fc_starget_delete/ at time 360.801271, and catch the insert_wq_barrier trace_info behind this

4. from out.txt , we can see, there are altogether three /fc_starget_delete/ work enqueued.
atfer the point of deadlock, kworker/1:1 and kworker/1:3 is
executing ...

5. if we let the scsi_transport_fc uses only one worker thread, i.e., change scsi_transport_fc.c : fc_host_setup()

              alloc_workqueue(fc_host->work_q_name, 0, 0) to
                     alloc_workqueue(fc_host->work_q_name, WQ_UNBOUND, 1)

              alloc_workqueue(fc_host->devloss_work_q_name, 0, 0) to

alloc_workqueue(fc_host->devloss_work_q_name, WQ_UNBOUND, 1)

Quoted 8 lines

the deadlock won't occur.

Quoted 3 lines

Can you please test a recent kernel? How easily can you reproduce the issue?

it's occured every time when hot remove disk array.

I'll test recent kernel after a while , but this problem in 3.0.30 really confused me

Yeah, it definitely sounds like concurrency depletion. There have been some fixes and substantial changes in the area, so I really wanna find out whether the problem is reproducible in recent vanilla kernel
- say, v3.9 or, even better, v3.10-rc2. Can you please try to reproduce the problem with a newer kernel?

Quoted 2 lines

by the way, I'm wondering about what's the race condition before which doesn't exist now

Before the commit you originally quoted, the calling thread could be preempted and migrated to another CPU before get_online_cpus() thus ending up executing the function twice on the new cpu but skipping the old one.

Thanks.

Signature · 1 line
tejun