Skip to contentSkip to conversation

cpu stopper threads and setaffinity leads to deadlock

PATCH Sodagudi Prasad 4 participants 7 messages Aug 2, 2018 – Aug 3, 2018

lore
Share & tools
lore
#0 lore raw

Hi Peter and Tglx,

We are observing another deadlock issue due to commit 0b26351b91(stop_machine, sched: Fix migrate_swap() vs. active_balance() deadlock), even after taking the following fix
https://www.mail-archive.com/[email protected]/msg1740526.html on the Linux-4.14.56 kernel.

Here is the scenario that leads to this deadlock.
We have used the stress-ng-64 --affinity test case to reproduce this issue in a controlled environment, while simultaneously running CPU hot plug and task migrations.

Stress-ng-affin (call stack shown below) is changing its own affinity from cpu3 to cpu7. Stress-ng-affin is preempted in the cpu_stop_queue_work() function as soon as the stopper lock for migration/3 is released . At the same time, on CPU 7, cross migration of tasks happens between cpu3 and cpu7.

=======================================================
Process: stress-ng-affin, cpu: 3 pid: 1748 start: 0xffffffd8817e4480
=====================================================

    Task name: stress-ng-affin pid: 1748 cpu: 3 start: ffffffd8817e4480
    state: 0x0 exit_state: 0x0 stack base: 0xffffff801c8e8000 Prio: 120
    Stack:
    [<ffffff87754864f4>] __switch_to+0xb8
    [<ffffff87763ebf8c>] __schedule+0x690
    [<ffffff87763ec388>] preempt_schedule_common+0x100
    [<ffffff87763eb8f4>] preempt_schedule+0x24
    [<ffffff87763f0e58>] _raw_spin_unlock_irqrestore+0x64
    [<ffffff8775574f8c>] cpu_stop_queue_work+0x9c
    [<ffffff8775574dfc>] stop_one_cpu+0x58
    [<ffffff87754e4884>] __set_cpus_allowed_ptr+0x234
    [<ffffff87754e8888>] sched_setaffinity+0x150
    [<ffffff87754e8ad8>] SyS_sched_setaffinity+0xcc
    [<ffffff87754837c0>] el0_svc_naked+0x34
    [<0>] UNKNOWN+0x0

Due to cross migration of tasks between cpu7 and cpu3, migration/7 has started executing and waits for the migration/3 task, so that they can proceed within the multi cpu stop state machine together. Unfortunately stress-ng-affin is affine to cpu7, and since migration 7 has started running, and has monopolized cpu7’s execution, stress-ng will never run on cpu7, and cpu3’s migration task is never woken up.

Essentially:
Due to the nature of the wake_q interface, a thread can only be in at most one wake queue at a time. migration/3 is currently in stress-ng-affin’s wake_q. This means that no other thread can add migration/3 to their wake queue. Thus, even if any attempt is made to stop CPU 3 (e.g. cross-migration, hot plugging, etc), no thread will wake up migration/3.

Below change helped to fix this deadlock.

kernel/stop_machine.c +1 -2
index e190d1e..f932e1e 100644
@@ -87,9 +87,9 @@ static bool cpu_stop_queue_work(unsigned int cpu, struct cpu_stop_work *work)
                __cpu_stop_queue_work(stopper, work, &wakeq);
        else if (work->done)
                cpu_stop_signal_done(work->done);
-       raw_spin_unlock_irqrestore(&stopper->lock, flags);
 
        wake_up_q(&wakeq);
+       raw_spin_unlock_irqrestore(&stopper->lock, flags);
 
 
-Thanks, Prasad
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
Linux Foundation Collaborative Project
#1 lore raw
Quoted 5 lines · Sodagudi Prasad
kernel/stop_machine.c +0 -0
index e190d1e..f932e1e 100644

kernel/stop_machine.c static bool cpu_stop_queue_work(unsigned int cpu, struct · line 94

Quoted 9 lines
kernel/stop_machine.c +1 -1
@@ -87,9 +87,9 @@ static bool cpu_stop_queue_work(unsigned int cpu, struct
 cpu_stop_work *work)
                __cpu_stop_queue_work(stopper, work, &wakeq);
        else if (work->done)
                cpu_stop_signal_done(work->done);
-       raw_spin_unlock_irqrestore(&stopper->lock, flags);
 
        wake_up_q(&wakeq);
+       raw_spin_unlock_irqrestore(&stopper->lock, flags);
 

That puts the wakeup back under stopper lock, which causes another deadlock iirc.

#2 lore raw
Quoted 12 lines · Peter Zijlstra
Quoted 11 lines · Sodagudi Prasad
kernel/stop_machine.c +0 -1
index e190d1e..f932e1e 100644
@@ -87,9 +87,9 @@ static bool cpu_stop_queue_work(unsigned int cpu, struct
 cpu_stop_work *work)
                __cpu_stop_queue_work(stopper, work, &wakeq);
        else if (work->done)
                cpu_stop_signal_done(work->done);
-       raw_spin_unlock_irqrestore(&stopper->lock, flags);
 
Quoted 6 lines

kernel/stop_machine.c static bool cpu_stop_queue_work(unsigned int cpu, struct cpu_stop_work *work) · line 93

Quoted 3 lines
kernel/stop_machine.c +1 -0
        wake_up_q(&wakeq);
+       raw_spin_unlock_irqrestore(&stopper->lock, flags);
 

That puts the wakeup back under stopper lock, which causes another deadlock iirc.

Yup, one you fixed.

0b26351b910fb (Peter Zijlstra 2018-04-20 11:50:05 +0200 92) wake_up_q(&wakeq);

#4 lore raw
In reply to Sodagudi Prasad
Quoted 12 lines · Sodagudi Prasad

Due to cross migration of tasks between cpu7 and cpu3, migration/7 has started executing and waits for the migration/3 task, so that they can proceed within the multi cpu stop state machine together. Unfortunately stress-ng-affin is affine to cpu7, and since migration 7 has started running, and has monopolized cpu7’s execution, stress-ng will never run on cpu7, and cpu3’s migration task is never woken up.

kernel/stop_machine.c +0 -0
index e190d1e..f932e1e 100644

kernel/stop_machine.c static bool cpu_stop_queue_work(unsigned int cpu, struct · line 94

Quoted 9 lines
kernel/stop_machine.c +1 -1
@@ -87,9 +87,9 @@ static bool cpu_stop_queue_work(unsigned int cpu, struct
 cpu_stop_work *work)
                __cpu_stop_queue_work(stopper, work, &wakeq);
        else if (work->done)
                cpu_stop_signal_done(work->done);
-       raw_spin_unlock_irqrestore(&stopper->lock, flags);
 
        wake_up_q(&wakeq);
+       raw_spin_unlock_irqrestore(&stopper->lock, flags);
 

So why didn't you do the 'obvious' parallel to what you did for cpu_stop_queue_two_works(), namely:

kernel/stop_machine.c +2 -0
@@ -81,6 +81,7 @@ static bool cpu_stop_queue_work(unsigned
 	unsigned long flags;
 	bool enabled;
 
+	preempt_disable();
 	raw_spin_lock_irqsave(&stopper->lock, flags);
 	enabled = stopper->enabled;
 	if (enabled)
@@ -90,6 +91,7 @@ static bool cpu_stop_queue_work(unsigned
 	raw_spin_unlock_irqrestore(&stopper->lock, flags);
 
 	wake_up_q(&wakeq);
+	preempt_enable();
 
 	return enabled;
 }
#5 lore raw

Prasad.

Quoted 3 lines · Peter Zijlstra

So why didn't you do the 'obvious' parallel to what you did for cpu_stop_queue_two_works(), namely:

Is that patch fixing the issue for you?

Quoted 19 lines
kernel/stop_machine.c +2 -0
@@ -81,6 +81,7 @@ static bool cpu_stop_queue_work(unsigned
 	unsigned long flags;
 	bool enabled;
 
+	preempt_disable();
 	raw_spin_lock_irqsave(&stopper->lock, flags);
 	enabled = stopper->enabled;
 	if (enabled)
@@ -90,6 +91,7 @@ static bool cpu_stop_queue_work(unsigned
 	raw_spin_unlock_irqrestore(&stopper->lock, flags);
 
 	wake_up_q(&wakeq);
+	preempt_enable();
 
 	return enabled;
 }
#6 lore raw
Quoted 8 lines · Thomas Gleixner

Prasad.

Quoted 3 lines · Peter Zijlstra

So why didn't you do the 'obvious' parallel to what you did for cpu_stop_queue_two_works(), namely:

Is that patch fixing the issue for you?

<Prasad> Hi Thomas and Peter,

Yes. Tested both versions of patches and both variants are working on Qualcomm devices
with stress testing of set affinity and tasks cross-migration, which were previously leading to the deadlock.

-Thanks, Prasad

Quoted 20 lines
Quoted 20 lines
kernel/stop_machine.c +2 -0
@@ -81,6 +81,7 @@ static bool cpu_stop_queue_work(unsigned
 	unsigned long flags;
 	bool enabled;
 
+	preempt_disable();
 	raw_spin_lock_irqsave(&stopper->lock, flags);
 	enabled = stopper->enabled;
 	if (enabled)
@@ -90,6 +91,7 @@ static bool cpu_stop_queue_work(unsigned
 	raw_spin_unlock_irqrestore(&stopper->lock, flags);
 
 	wake_up_q(&wakeq);
+	preempt_enable();
 
 	return enabled;
 }
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
Linux Foundation Collaborative Project