Skip to contentSkip to conversation

How do I increment a per-CPU variable without warning?

DISCUSSION Paul E. McKenney 4 participants 17 messages Apr 15, 2014 – Apr 17, 2014 Per-CPU memory allocator

lore
Share & tools
lore
#0 lore raw

Hello, Christoph,

I have a patch that currently uses __this_cpu_inc_return() to increment a per-CPU variable, but without preemption disabled. Of course, given that preemption is enabled, it might well end up picking up one CPU's counter, adding one to it, then storing the result into some other CPU's counter. But this is OK, the test can be probabilistic. And when I run this against v3.14 and earlier, it works fine.

But now there is 188a81409ff7 (percpu: add preemption checks to __this_cpu ops), which gives me lots of splats.

My current admittedly crude workaround is as follows:

	static inline bool rcu_should_resched(void)
	{
		int t;

	#ifdef CONFIG_DEBUG_PREEMPT
		preempt_disable();
	#endif /* #ifdef CONFIG_DEBUG_PREEMPT */
		t = __this_cpu_read(rcu_cond_resched_count) + 1;
		if (t < RCU_COND_RESCHED_LIM) {
			__this_cpu_write(rcu_cond_resched_count, t);
	#ifdef CONFIG_DEBUG_PREEMPT
			preempt_enable();
	#endif /* #ifdef CONFIG_DEBUG_PREEMPT */
			return false;
		}
	#ifdef CONFIG_DEBUG_PREEMPT
		preempt_enable();
	#endif /* #ifdef CONFIG_DEBUG_PREEMPT */
		return true;
	}

This is arguably better than the original __this_cpu_read() because it avoids overflow, but I thought I should check to see if there was some better way to do this.

Thanx, Paul

#1 lore raw
Quoted 14 lines · Paul E. McKenney

My current admittedly crude workaround is as follows:

	static inline bool rcu_should_resched(void)
	{
		int t;

	#ifdef CONFIG_DEBUG_PREEMPT
		preempt_disable();
	#endif /* #ifdef CONFIG_DEBUG_PREEMPT */
		t = __this_cpu_read(rcu_cond_resched_count) + 1;
		if (t < RCU_COND_RESCHED_LIM) {
			__this_cpu_write(rcu_cond_resched_count, t);
	#ifdef CONFIG_DEBUG_PREEMPT
			preempt_enable();
Quoted 8 lines
	#endif /* #ifdef CONFIG_DEBUG_PREEMPT */
			return false;
		}
	#ifdef CONFIG_DEBUG_PREEMPT
		preempt_enable();
	#endif /* #ifdef CONFIG_DEBUG_PREEMPT */
		return true;
	}

Won't using DEBUG_PREEMPT instead of just CONFIG_PREEMPT here make this silently do the wrong thing if preemption is enabled, but debugging isn't ?

I'm not seeing why you need the ifdefs at all, unless the implied barrier() is a problem ?

Dave

#2 lore raw
Quoted 25 lines · Dave Jones
Quoted 14 lines · Paul E. McKenney

My current admittedly crude workaround is as follows:

	static inline bool rcu_should_resched(void)
	{
		int t;

	#ifdef CONFIG_DEBUG_PREEMPT
		preempt_disable();
	#endif /* #ifdef CONFIG_DEBUG_PREEMPT */
		t = __this_cpu_read(rcu_cond_resched_count) + 1;
		if (t < RCU_COND_RESCHED_LIM) {
			__this_cpu_write(rcu_cond_resched_count, t);
	#ifdef CONFIG_DEBUG_PREEMPT
			preempt_enable();
Quoted 8 lines
	#endif /* #ifdef CONFIG_DEBUG_PREEMPT */
			return false;
		}
	#ifdef CONFIG_DEBUG_PREEMPT
		preempt_enable();
	#endif /* #ifdef CONFIG_DEBUG_PREEMPT */
		return true;
	}
Quoted 2 lines

Won't using DEBUG_PREEMPT instead of just CONFIG_PREEMPT here make this silently do the wrong thing if preemption is enabled, but debugging isn't ?

If preemption is enabled, but debugging is not, then yes, the above code might force an unnecessary schedule() if the above code was preempted between the __this_cpu_read() and the __this_cpu_write(). Which does not cause a problem, especially given that it won't happen very often.

Quoted 2 lines

I'm not seeing why you need the ifdefs at all, unless the implied barrier() is a problem ?

I don't think that Peter Zijlstra would be too happy about an extra unneeded preempt_disable()/preempt_enable() pair in the cond_resched() fastpath. Not that I necessarily expect him to be particularly happy with the above, but perhaps someone has a better approach.

Thanx, Paul

#3 lore raw
Quoted 34 lines · Paul E. McKenney
Quoted 25 lines · Dave Jones
Quoted 14 lines · Paul E. McKenney

My current admittedly crude workaround is as follows:

	static inline bool rcu_should_resched(void)
	{
		int t;

	#ifdef CONFIG_DEBUG_PREEMPT
		preempt_disable();
	#endif /* #ifdef CONFIG_DEBUG_PREEMPT */
		t = __this_cpu_read(rcu_cond_resched_count) + 1;
		if (t < RCU_COND_RESCHED_LIM) {
			__this_cpu_write(rcu_cond_resched_count, t);
	#ifdef CONFIG_DEBUG_PREEMPT
			preempt_enable();
Quoted 8 lines
	#endif /* #ifdef CONFIG_DEBUG_PREEMPT */
			return false;
		}
	#ifdef CONFIG_DEBUG_PREEMPT
		preempt_enable();
	#endif /* #ifdef CONFIG_DEBUG_PREEMPT */
		return true;
	}
Quoted 2 lines

Won't using DEBUG_PREEMPT instead of just CONFIG_PREEMPT here make this silently do the wrong thing if preemption is enabled, but debugging isn't ?

If preemption is enabled, but debugging is not, then yes, the above code might force an unnecessary schedule() if the above code was preempted between the __this_cpu_read() and the __this_cpu_write(). Which does not cause a problem, especially given that it won't happen very often.

Quoted 7 lines
Quoted 2 lines

I'm not seeing why you need the ifdefs at all, unless the implied barrier() is a problem ?

I don't think that Peter Zijlstra would be too happy about an extra unneeded preempt_disable()/preempt_enable() pair in the cond_resched() fastpath. Not that I necessarily expect him to be particularly happy with the above, but perhaps someone has a better approach.

But falling back on the old ways of doing this at least looks a bit nicer:

	static inline bool rcu_should_resched(void)
	{
		int t;
		int *tp = &per_cpu(rcu_cond_resched_count, raw_smp_processor_id());

		t = ACCESS_ONCE(*tp) + 1;
		if (t < RCU_COND_RESCHED_LIM) {
			ACCESS_ONCE(*tp) = t;
			return false;
		}
		return true;
	}

Other thoughts?

Thanx, Paul

#4 lore raw
Quoted 10 lines · Paul E. McKenney

But falling back on the old ways of doing this at least looks a bit nicer:

	static inline bool rcu_should_resched(void)
	{
		int t;
		int *tp = &per_cpu(rcu_cond_resched_count, raw_smp_processor_id());

		t = ACCESS_ONCE(*tp) + 1;
		if (t < RCU_COND_RESCHED_LIM) {

<here>

Quoted 7 lines
			ACCESS_ONCE(*tp) = t;
			return false;
		}
		return true;
	}

Other thoughts?

Still broken, if A starts out on CPU1, gets migrated to CPU0 at <here>, then B starts the same on CPU1. It is possible for both CPU0 and CPU1 to write a different value into your rcu_cond_resched_count.

You really want to disable preemption around there. The proper old way would've been get_cpu_var()/put_cpu_var().

#5 lore raw
Quoted 20 lines · Peter Zijlstra
Quoted 10 lines · Paul E. McKenney

But falling back on the old ways of doing this at least looks a bit nicer:

	static inline bool rcu_should_resched(void)
	{
		int t;
		int *tp = &per_cpu(rcu_cond_resched_count, raw_smp_processor_id());

		t = ACCESS_ONCE(*tp) + 1;
		if (t < RCU_COND_RESCHED_LIM) {

<here>

Quoted 6 lines
			ACCESS_ONCE(*tp) = t;
			return false;
		}
		return true;
	}
Quoted 5 lines
Quoted 1 line

Other thoughts?

Still broken, if A starts out on CPU1, gets migrated to CPU0 at <here>, then B starts the same on CPU1. It is possible for both CPU0 and CPU1 to write a different value into your rcu_cond_resched_count.

That is actually OK. The values written are guaranteed to be between zero and RCU_COND_RESCHED_LIM-1. In theory, yes, rcu_should_resched() could end up failing due to a horribly unlucky sequence of preemptions, but the probability is -way- lower than that of hardware failure.

However...

Quoted 2 lines

You really want to disable preemption around there. The proper old way would've been get_cpu_var()/put_cpu_var().

If you are OK with unconditional disabling of preemption at this point, that would avoid worrying about probabilities and would be quite a bit simpler.

So unconditional preempt_disable()/preempt_enable() it is.

Thanx, Paul

#6 lore raw
In reply to Peter Zijlstra
Quoted 2 lines · Peter Zijlstra

You really want to disable preemption around there. The proper old way would've been get_cpu_var()/put_cpu_var().

get_cpu_var and put_cpu_var is still the correct way of doing things if you need to access multiple per cpu variables.

The problem that I want to solve is the varied use of __get_cpu_var for address calculations, per cpu variable assignment, structure copying and per cpu variable access. The this_cpu ops can avoid address calculations using segment prefixes. Plus the changes clarify what is actuallly going on.

#7 lore raw
In reply to Paul E. McKenney
Quoted 35 lines · Paul E. McKenney

Hello, Christoph,

I have a patch that currently uses __this_cpu_inc_return() to increment a per-CPU variable, but without preemption disabled. Of course, given that preemption is enabled, it might well end up picking up one CPU's counter, adding one to it, then storing the result into some other CPU's counter. But this is OK, the test can be probabilistic. And when I run this against v3.14 and earlier, it works fine.

But now there is 188a81409ff7 (percpu: add preemption checks to __this_cpu ops), which gives me lots of splats.

My current admittedly crude workaround is as follows:

	static inline bool rcu_should_resched(void)
	{
		int t;

	#ifdef CONFIG_DEBUG_PREEMPT
		preempt_disable();
	#endif /* #ifdef CONFIG_DEBUG_PREEMPT */
		t = __this_cpu_read(rcu_cond_resched_count) + 1;
		if (t < RCU_COND_RESCHED_LIM) {
			__this_cpu_write(rcu_cond_resched_count, t);
	#ifdef CONFIG_DEBUG_PREEMPT
			preempt_enable();
	#endif /* #ifdef CONFIG_DEBUG_PREEMPT */
			return false;
		}
	#ifdef CONFIG_DEBUG_PREEMPT
		preempt_enable();
	#endif /* #ifdef CONFIG_DEBUG_PREEMPT */
		return true;
	}
Quoted 3 lines

This is arguably better than the original __this_cpu_read() because it avoids overflow, but I thought I should check to see if there was some better way to do this.

you could use raw_cpu_{read,write}(). But note that without the unconditional preempt_disable() in there your code can read a different rcu_cond_resched_count than it writes.

So I think you very much want that preempt_disable().

#8 lore raw
In reply to Paul E. McKenney
Quoted 8 lines · Paul E. McKenney

Hello, Christoph,

I have a patch that currently uses __this_cpu_inc_return() to increment a per-CPU variable, but without preemption disabled. Of course, given that preemption is enabled, it might well end up picking up one CPU's counter, adding one to it, then storing the result into some other CPU's counter. But this is OK, the test can be probabilistic. And when I run this against v3.14 and earlier, it works fine.

We introduced raw_cpu_inc_return to squish these warnings.

Quoted 3 lines

This is arguably better than the original __this_cpu_read() because it avoids overflow, but I thought I should check to see if there was some better way to do this.

If this is supposed to be totally race safe then you must disable preemption.

#9 lore raw
Quoted 4 lines · Christoph Lameter
Quoted 2 lines · Paul E. McKenney

Hello, Christoph,

Quoted 8 lines
Quoted 6 lines

I have a patch that currently uses __this_cpu_inc_return() to increment a per-CPU variable, but without preemption disabled. Of course, given that preemption is enabled, it might well end up picking up one CPU's counter, adding one to it, then storing the result into some other CPU's counter. But this is OK, the test can be probabilistic. And when I run this against v3.14 and earlier, it works fine.

We introduced raw_cpu_inc_return to squish these warnings.

Cool, this is a good short-term fix.

Quoted 6 lines
Quoted 3 lines

This is arguably better than the original __this_cpu_read() because it avoids overflow, but I thought I should check to see if there was some better way to do this.

If this is supposed to be totally race safe then you must disable preemption.

Understood!

Thanx, Paul

#10 lore raw
Quoted 12 lines · Paul E. McKenney
Quoted 11 lines · Christoph Lameter
Quoted 8 lines · Paul E. McKenney

Hello, Christoph,

I have a patch that currently uses __this_cpu_inc_return() to increment a per-CPU variable, but without preemption disabled. Of course, given that preemption is enabled, it might well end up picking up one CPU's counter, adding one to it, then storing the result into some other CPU's counter. But this is OK, the test can be probabilistic. And when I run this against v3.14 and earlier, it works fine.

Quoted 3 lines
Quoted 1 line

We introduced raw_cpu_inc_return to squish these warnings.

Cool, this is a good short-term fix.

Or at least it is given the following patch. Was this the intent?

Thanx, Paul

------------------------------------------------------------------------

percpu: Fix raw_cpu_inc_return()

The definition for raw_cpu_add_return() uses the operation prefix "raw_add_return_", but the definitions in the various percpu.h files expect "raw_cpu_add_return_". This commit therefore appropriately adjusts the definition of raw_cpu_add_return().

Cc Christoph Lameter <[email protected]>
include/linux/percpu.h +1 -1
index e7a0b95ed527..495c6543a8f2 100644
@@ -639,7 +639,7 @@ do {									\
 #  define raw_cpu_add_return_8(pcp, val)	raw_cpu_generic_add_return(pcp, val)
 # endif
 # define raw_cpu_add_return(pcp, val)	\
-	__pcpu_size_call_return2(raw_add_return_, pcp, val)
+	__pcpu_size_call_return2(raw_cpu_add_return_, pcp, val)
 #endif
 
 #define raw_cpu_sub_return(pcp, val)	raw_cpu_add_return(pcp, -(typeof(pcp))(val))
#11 lore raw
Quoted 13 lines · Paul E. McKenney
Quoted 12 lines · Paul E. McKenney
Quoted 11 lines · Christoph Lameter
Quoted 8 lines · Paul E. McKenney

Hello, Christoph,

I have a patch that currently uses __this_cpu_inc_return() to increment a per-CPU variable, but without preemption disabled. Of course, given that preemption is enabled, it might well end up picking up one CPU's counter, adding one to it, then storing the result into some other CPU's counter. But this is OK, the test can be probabilistic. And when I run this against v3.14 and earlier, it works fine.

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

We introduced raw_cpu_inc_return to squish these warnings.

Cool, this is a good short-term fix.

Or at least it is given the following patch. Was this the intent?

Correct. Thanks for the fix.

Quoted 28 lines

Thanx, Paul

------------------------------------------------------------------------

percpu: Fix raw_cpu_inc_return()

The definition for raw_cpu_add_return() uses the operation prefix "raw_add_return_", but the definitions in the various percpu.h files expect "raw_cpu_add_return_". This commit therefore appropriately adjusts the definition of raw_cpu_add_return().

Cc Christoph Lameter <[email protected]>
include/linux/percpu.h +1 -1
index e7a0b95ed527..495c6543a8f2 100644
@@ -639,7 +639,7 @@ do {									\
 #  define raw_cpu_add_return_8(pcp, val)	raw_cpu_generic_add_return(pcp, val)
 # endif
 # define raw_cpu_add_return(pcp, val)	\
-	__pcpu_size_call_return2(raw_add_return_, pcp, val)
+	__pcpu_size_call_return2(raw_cpu_add_return_, pcp, val)
 #endif
 
 #define raw_cpu_sub_return(pcp, val)	raw_cpu_add_return(pcp, -(typeof(pcp))(val))
author Applied
#12 lore raw
Quoted 17 lines · Christoph Lameter
Quoted 15 lines · Paul E. McKenney
Quoted 14 lines · Paul E. McKenney
Quoted 12 lines · Christoph Lameter
Quoted 8 lines · Paul E. McKenney

Hello, Christoph,

I have a patch that currently uses __this_cpu_inc_return() to increment a per-CPU variable, but without preemption disabled. Of course, given that preemption is enabled, it might well end up picking up one CPU's counter, adding one to it, then storing the result into some other CPU's counter. But this is OK, the test can be probabilistic. And when I run this against v3.14 and earlier, it works fine.

We introduced raw_cpu_inc_return to squish these warnings.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Cool, this is a good short-term fix.

Or at least it is given the following patch. Was this the intent?

Correct. Thanks for the fix.

Queued, thank you. I don't need this until 3.16, but I am tempted to push it in this cycle in case others need it. Any objections?

Thanx, Paul

Quoted 27 lines
Quoted 26 lines

------------------------------------------------------------------------

percpu: Fix raw_cpu_inc_return()

The definition for raw_cpu_add_return() uses the operation prefix "raw_add_return_", but the definitions in the various percpu.h files expect "raw_cpu_add_return_". This commit therefore appropriately adjusts the definition of raw_cpu_add_return().

Cc Christoph Lameter <[email protected]>
include/linux/percpu.h +1 -1
index e7a0b95ed527..495c6543a8f2 100644
@@ -639,7 +639,7 @@ do {									\
 #  define raw_cpu_add_return_8(pcp, val)	raw_cpu_generic_add_return(pcp, val)
 # endif
 # define raw_cpu_add_return(pcp, val)	\
-	__pcpu_size_call_return2(raw_add_return_, pcp, val)
+	__pcpu_size_call_return2(raw_cpu_add_return_, pcp, val)
 #endif
 
 #define raw_cpu_sub_return(pcp, val)	raw_cpu_add_return(pcp, -(typeof(pcp))(val))
#13 lore raw
Quoted 2 lines · Paul E. McKenney

Queued, thank you. I don't need this until 3.16, but I am tempted to push it in this cycle in case others need it. Any objections?

No its a bug fix.

#14 lore raw
Quoted 6 lines · Christoph Lameter
Quoted 2 lines · Paul E. McKenney

Queued, thank you. I don't need this until 3.16, but I am tempted to push it in this cycle in case others need it. Any objections?

No its a bug fix.

Fair enough! I resent the patch with your Ack to Tejun.

Thanx, Paul

#15 lore raw
Quoted 1 line · Paul E. McKenney

Fair enough! I resent the patch with your Ack to Tejun.

Also note that you may want to use

this_cpu_inc

instead of raw_cpu_inc.

this_cpu_inc will not disable preemption or anything on x86 but just create a single instruction using instruction atomicity to avoid the preempt on/off sequence.

On platforms that cannot emit such an instruction it will fallback to disable interrupts for the sequence of instructions that increments the value.

With such an approach incrementing the counter should be much safer. If the other arch want to avoid irq on/off sequences then they can override the fallback to use atomics or whatever the processor architecture permits to avoid the overhead of interrupt on / off.

#16 lore raw
Quoted 15 lines · Christoph Lameter
Quoted 1 line · Paul E. McKenney

Fair enough! I resent the patch with your Ack to Tejun.

Also note that you may want to use

this_cpu_inc

instead of raw_cpu_inc.

this_cpu_inc will not disable preemption or anything on x86 but just create a single instruction using instruction atomicity to avoid the preempt on/off sequence.

Quoted 8 lines

On platforms that cannot emit such an instruction it will fallback to disable interrupts for the sequence of instructions that increments the value.

With such an approach incrementing the counter should be much safer. If the other arch want to avoid irq on/off sequences then they can override the fallback to use atomics or whatever the processor architecture permits to avoid the overhead of interrupt on / off.

Fair enough, but in this case I don't need it to be safe.

Thanx, Paul