Skip to contentSkip to conversation

3.9.x: Possible race related to stop_machine leads to lockup.

DISCUSSION Ben Greear 2 participants 4 messages Jun 4, 2013 – Jun 5, 2013

lore
Share & tools
lore

Ben Greear

author
#0 lore raw

I've been trying to figure out why I see the migration/* processes hang in a busy loop....

While reading the stop_machine.c file, I think I might have an answer.

The set_state() method sets the thread_ack to the current number of threads. Each thread's state machine then decrements it down to zero where it bumps the state to the next level. This lets each cpu stop in lock-step it seems.

But, from what I can tell, the __stop_machine() method can (re)set the state to STOPMACHINE_PREPARE while the migration processes are in their loop. That would explain why they sometimes loop forever.

Does this make sense?

Any ideas on how to fix this properly?

Thanks,
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
author
#1 lore raw
Quoted 11 lines · Ben Greear

I've been trying to figure out why I see the migration/* processes hang in a busy loop....

While reading the stop_machine.c file, I think I might have an answer.

The set_state() method sets the thread_ack to the current number of threads. Each thread's state machine then decrements it down to zero where it bumps the state to the next level. This lets each cpu stop in lock-step it seems.

Quoted 6 lines

But, from what I can tell, the __stop_machine() method can (re)set the state to STOPMACHINE_PREPARE while the migration processes are in their loop. That would explain why they sometimes loop forever.

Does this make sense?

Err, no..that doesn't make sense. 'smdata' is on the stack.

More printk debugging makes it look like one thread just never notices that smdata->state has been updated by another thread.

There is this comment..maybe cpu_relax only does the chill out part and we need something else to make sure smdata->state is freshly read from the other CPU's cache?

		/* Chill out and ensure we re-read stopmachine_state. */
		cpu_relax();
		if (smdata->state != curstate) {

Gah..way out of my league :P

Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
#2 lore raw
Quoted 29 lines · Ben Greear
Quoted 11 lines · Ben Greear

I've been trying to figure out why I see the migration/* processes hang in a busy loop....

While reading the stop_machine.c file, I think I might have an answer.

The set_state() method sets the thread_ack to the current number of threads. Each thread's state machine then decrements it down to zero where it bumps the state to the next level. This lets each cpu stop in lock-step it seems.

Quoted 6 lines

But, from what I can tell, the __stop_machine() method can (re)set the state to STOPMACHINE_PREPARE while the migration processes are in their loop. That would explain why they sometimes loop forever.

Does this make sense?

Err, no..that doesn't make sense. 'smdata' is on the stack.

More printk debugging makes it look like one thread just never notices that smdata->state has been updated by another thread.

There is this comment..maybe cpu_relax only does the chill out part and we need something else to make sure smdata->state is freshly read from the other CPU's cache?

Quoted 5 lines
		/* Chill out and ensure we re-read stopmachine_state. */
		cpu_relax();
		if (smdata->state != curstate) {

Gah..way out of my league :P

What architecture? Maybe someone didn't get the memo; cpu_relax() should be a read barrier.

Cheers,
Rusty.

author
#3 lore raw
Quoted 30 lines · Rusty Russell
Quoted 29 lines · Ben Greear
Quoted 11 lines · Ben Greear

I've been trying to figure out why I see the migration/* processes hang in a busy loop....

While reading the stop_machine.c file, I think I might have an answer.

The set_state() method sets the thread_ack to the current number of threads. Each thread's state machine then decrements it down to zero where it bumps the state to the next level. This lets each cpu stop in lock-step it seems.

Quoted 6 lines

But, from what I can tell, the __stop_machine() method can (re)set the state to STOPMACHINE_PREPARE while the migration processes are in their loop. That would explain why they sometimes loop forever.

Does this make sense?

Err, no..that doesn't make sense. 'smdata' is on the stack.

More printk debugging makes it look like one thread just never notices that smdata->state has been updated by another thread.

There is this comment..maybe cpu_relax only does the chill out part and we need something else to make sure smdata->state is freshly read from the other CPU's cache?

Quoted 8 lines
Quoted 5 lines
		/* Chill out and ensure we re-read stopmachine_state. */
		cpu_relax();
		if (smdata->state != curstate) {

Gah..way out of my league :P

What architecture? Maybe someone didn't get the memo; cpu_relax() should be a read barrier.

I tried making it and smp read barier, and tried using atomic_t for the state object. No big help.

Latest theory is that one thread gets stuck doing IRQs while rest of CPUs have disabled IRQs and that one CPU/thread never gets back to the cpu shutdown state machine.

I'll post a more complete debugging patch later today, and try to find a better way to reproduce it.

Thanks,
Ben

Quoted 4 lines

Cheers,
Rusty.

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com