Skip to contentSkip to conversation

Documentation/memory-barriers.txt

DISCUSSION Benjamin Poirier 2 participants 3 messages Sep 12, 2011

lore
Share & tools
lore
#0 lore raw

Hello David, Paul,

Thank you for this great piece on memory barriers. I think it made a complex topic approachable. I have two questions:
1)
I had a hard time understanding the second part of the example in the section "Sleep and wake-up functions".

Quoted 5 lines
	set_current_state(TASK_INTERRUPTIBLE);
	if (event_indicated)
		break;
	__set_current_state(TASK_RUNNING);
	do_something(my_data);

I understand the need for memory barriers, but I don't understand what the code is trying to achieve.
Where are the for (;;) loop and the schedule() call gone to?

Quoted 5 lines
	set_current_state(TASK_INTERRUPTIBLE);
	if (event_indicated) {
		smp_rmb();
		do_something(my_data);
	}

Isn't a break; missing here? How come do_something() has moved inside the condition?

I'm thinking these final example code bits should look like this (without and with the smp_rmb), no?:

for (;;) {
	set_current_state(TASK_INTERRUPTIBLE);
	if (event_indicated) {
		smp_rmb();
		do_something(my_data);
		break;
	}
	schedule();
}

__set_current_state(TASK_RUNNING);

2)
On a more general note, why is there a read_barrier_depends() but not a write_barrier_depends()?

l=7
"write_barrier_depends()"
g=&l

---

l=g
read_barrier_depends()
t=*l

Most processors do not reorder dependent loads but do reorder loads after loads. I'm guessing there's no processor that does not reorder dependent stores but that does reorder stores after stores. So there's no point in having write_barrier_depends(), it would always be defined to wmb()?

Thanks,
-Ben

#1 lore raw
Quoted 14 lines · Benjamin Poirier

Hello David, Paul,

Thank you for this great piece on memory barriers. I think it made a complex topic approachable. I have two questions:
1)
I had a hard time understanding the second part of the example in the section "Sleep and wake-up functions".

Quoted 5 lines
	set_current_state(TASK_INTERRUPTIBLE);
	if (event_indicated)
		break;
	__set_current_state(TASK_RUNNING);
	do_something(my_data);
Quoted 3 lines

I understand the need for memory barriers, but I don't understand what the code is trying to achieve.
Where are the for (;;) loop and the schedule() call gone to?

This is a discussion of memory barriers, which handle communication between multiple CPUs. So, yes, in many cases the for-loop is required, but the actual communication will occur on a particular iteration of the for-loop. But there are other use cases, for example, involving prepare_to_wait()/schedule()/finish_wait(), that do not need an enclosing loop. See for example the use in bsg_io_schedule().

Nevertheless, all of the wait/wakeup examples need to enforce proper memory ordering, and we therefore take the least common denominator for the more detailed examples.

Quoted 8 lines
Quoted 5 lines
	set_current_state(TASK_INTERRUPTIBLE);
	if (event_indicated) {
		smp_rmb();
		do_something(my_data);
	}

Isn't a break; missing here? How come do_something() has moved inside the condition?

Again, it depends on the enclosing use case. Keep in mind that even in cases involving a loop, there is only one pass through the loop that actually does anything.

Quoted 12 lines

I'm thinking these final example code bits should look like this (without and with the smp_rmb), no?:

for (;;) {
	set_current_state(TASK_INTERRUPTIBLE);
	if (event_indicated) {
		smp_rmb();
		do_something(my_data);
		break;
	}
	schedule();
}
Quoted 1 line

__set_current_state(TASK_RUNNING);

This example would be correct for a looping case, but is more ornate than required for illustrating the effects of memory barriers. So we took the simpler case without the loop.

Quoted 3 lines

2)
On a more general note, why is there a read_barrier_depends() but not a write_barrier_depends()?

You use rcu_assign_pointer() for write_barrier_depends(). An alternative extremely expensive definition of write_barrier_depends() is to force a memory barrier on all CPUs. This was debated quite some time ago and was rejected.

However, you can get this effect by calling one of the synchronize_rcu() or synchronize_rcu_expedited() family of functions. Please be aware that synchronize_rcu() will impose several milliseconds of latency but minimal CPU overhead, while synchronize_rcu_expedited() will impose only a few tens of microseconds of latency, but will IPI each and every CPU. So both of these are expensive in different ways.

Another way to get this effect is to use smp_call_function(). Like synchronize_rcu_expedited(), this will IPI each and every CPU.

But before going down any of these paths other than rcu_assign_pointer(), you really need to look very carefully at why you need smp_mb() on each and every CPU. Normally, this is a way bigger hammer than you need.

To reiterate, if you think you need write_barrier_depends(), please carefully revisit your design. The odds are that you really do not need it.

Quoted 10 lines

l=7
"write_barrier_depends()"
g=&l

---

l=g
read_barrier_depends()
t=*l

Quoted 5 lines

Most processors do not reorder dependent loads but do reorder loads after loads. I'm guessing there's no processor that does not reorder dependent stores but that does reorder stores after stores. So there's no point in having write_barrier_depends(), it would always be defined to wmb()?

Yes, exactly -- rcu_assign_pointer() does use smp_wmb().

The only CPU that could make good generic use of write_barrier_depends() is DEC Alpha. What we do instead is make Alpha's read_barrier_depends() use smp_rmb().

Thanx, Paul

#2 lore raw
Quoted 3 lines · Paul E. McKenney
Quoted 2 lines · Benjamin Poirier

Hello David, Paul,

[snip]

Quoted 4 lines

This example would be correct for a looping case, but is more ornate than required for illustrating the effects of memory barriers. So we took the simpler case without the loop.

Thanks for the clarification. I was under the false impression that all of the code examples in this section represented the same code segment that was modified from one example to the next. In any case, these examples do demonstrate the use of memory barriers clearly.

-Ben