Skip to contentSkip to conversation

lib/percpu_counter.c: disable local irq when updating percpu couter

PATCH In review Ming Lei 2 participants 6 messages Jan 7, 2014 – Jan 8, 2014

Subsystems (2)
lore
Share & tools
lore

Ming Lei

author
#0 lore raw

PATCH 1/1 lib/percpu_counter.c: disable local irq when updating percpu couter

__percpu_counter_add() may be called in softirq/hardirq handler (such as, blk_mq_queue_exit() is typically called in hardirq/softirq handler), so we need to disable local irq when updating the percpu counter, otherwise counts may be lost.

The patch fixes problem that 'rmmod null_blk' may hang in blk_cleanup_queue() because of miscounting of request_queue->mq_usage_counter.

Cc Paul Gortmaker <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Shaohua Li <[email protected]>
Cc Jens Axboe <[email protected]>
Cc Fan Du <[email protected]>
---
 lib/percpu_counter.c |   10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
lib/percpu_counter.c +5 -5
index 7473ee3..2b87bc1 100644
@@ -75,19 +75,19 @@ EXPORT_SYMBOL(percpu_counter_set);
 void __percpu_counter_add(struct percpu_counter *fbc, s64 amount, s32 batch)
 {
 	s64 count;
+	unsigned long flags;
 
-	preempt_disable();
+	raw_local_irq_save(flags);
 	count = __this_cpu_read(*fbc->counters) + amount;
 	if (count >= batch || count <= -batch) {
-		unsigned long flags;
-		raw_spin_lock_irqsave(&fbc->lock, flags);
+		raw_spin_lock(&fbc->lock);
 		fbc->count += count;
-		raw_spin_unlock_irqrestore(&fbc->lock, flags);
+		raw_spin_unlock(&fbc->lock);
 		__this_cpu_write(*fbc->counters, 0);
 	} else {
 		__this_cpu_write(*fbc->counters, count);
 	}
-	preempt_enable();
+	raw_local_irq_restore(flags);
 }
 EXPORT_SYMBOL(__percpu_counter_add);
 
Signature · 1 line
1.7.9.5
maintainer
#1 lore raw
Quoted 4 lines · On Tue, 7 Jan 2014 18:29:27 +0800 Ming Lei <[email protected]> wrote:

__percpu_counter_add() may be called in softirq/hardirq handler (such as, blk_mq_queue_exit() is typically called in hardirq/softirq handler), so we need to disable local irq when updating the percpu counter, otherwise counts may be lost.

OK.

Quoted 21 lines

The patch fixes problem that 'rmmod null_blk' may hang in blk_cleanup_queue() because of miscounting of request_queue->mq_usage_counter.

...

lib/percpu_counter.c +3 -3
@@ -75,19 +75,19 @@ EXPORT_SYMBOL(percpu_counter_set);
 void __percpu_counter_add(struct percpu_counter *fbc, s64 amount, s32 batch)
 {
 	s64 count;
+	unsigned long flags;
 
-	preempt_disable();
+	raw_local_irq_save(flags);
 	count = __this_cpu_read(*fbc->counters) + amount;
 	if (count >= batch || count <= -batch) {
-		unsigned long flags;
-		raw_spin_lock_irqsave(&fbc->lock, flags);
+		raw_spin_lock(&fbc->lock);
 		fbc->count += count;

lib/percpu_counter.c EXPORT_SYMBOL(percpu_counter_set); · line 92

Quoted 10 lines
lib/percpu_counter.c +2 -2
-		raw_spin_unlock_irqrestore(&fbc->lock, flags);
+		raw_spin_unlock(&fbc->lock);
 		__this_cpu_write(*fbc->counters, 0);
 	} else {
 		__this_cpu_write(*fbc->counters, count);
 	}
-	preempt_enable();
+	raw_local_irq_restore(flags);
 }
 EXPORT_SYMBOL(__percpu_counter_add);

Can this be made more efficient?

The this_cpu_foo() documentation is fairly dreadful, but way down at the end of Documentation/this_cpu_ops.txt we find "this_cpu ops are interrupt safe". So I think this is a more efficient fix:

lib/percpu_counter.c renamed from lib/percpu_counter.c~a +2 -2
@@ -82,10 +82,10 @@ void __percpu_counter_add(struct percpu_
 		unsigned long flags;
 		raw_spin_lock_irqsave(&fbc->lock, flags);
 		fbc->count += count;
+		__this_cpu_sub(*fbc->counters, count);
 		raw_spin_unlock_irqrestore(&fbc->lock, flags);
-		__this_cpu_write(*fbc->counters, 0);
 	} else {
-		__this_cpu_write(*fbc->counters, count);
+		this_cpu_add(*fbc->counters, amount);
 	}
 	preempt_enable();
 }

It avoids the local_irq_disable() in the common case, when the CPU supports efficient this_cpu_add(). It will in rare race situations permit the cpu-local counter to exceed `batch', but that should be harmless.

What do you think?

author
#2 lore raw

Hi Andrew,

Quoted 63 lines · Andrew Morton
Quoted 4 lines · On Tue, 7 Jan 2014 18:29:27 +0800 Ming Lei <[email protected]> wrote:

__percpu_counter_add() may be called in softirq/hardirq handler (such as, blk_mq_queue_exit() is typically called in hardirq/softirq handler), so we need to disable local irq when updating the percpu counter, otherwise counts may be lost.

OK.

Quoted 21 lines

The patch fixes problem that 'rmmod null_blk' may hang in blk_cleanup_queue() because of miscounting of request_queue->mq_usage_counter.

...

lib/percpu_counter.c +3 -3
@@ -75,19 +75,19 @@ EXPORT_SYMBOL(percpu_counter_set);
 void __percpu_counter_add(struct percpu_counter *fbc, s64 amount, s32 batch)
 {
      s64 count;
+     unsigned long flags;
 
-     preempt_disable();
+     raw_local_irq_save(flags);
      count = __this_cpu_read(*fbc->counters) + amount;
      if (count >= batch || count <= -batch) {
-             unsigned long flags;
-             raw_spin_lock_irqsave(&fbc->lock, flags);
+             raw_spin_lock(&fbc->lock);
              fbc->count += count;

lib/percpu_counter.c EXPORT_SYMBOL(percpu_counter_set); · line 92

Quoted 10 lines
lib/percpu_counter.c +2 -2
-             raw_spin_unlock_irqrestore(&fbc->lock, flags);
+             raw_spin_unlock(&fbc->lock);
              __this_cpu_write(*fbc->counters, 0);
      } else {
              __this_cpu_write(*fbc->counters, count);
      }
-     preempt_enable();
+     raw_local_irq_restore(flags);
 }
 EXPORT_SYMBOL(__percpu_counter_add);

Can this be made more efficient?

The this_cpu_foo() documentation is fairly dreadful, but way down at the end of Documentation/this_cpu_ops.txt we find "this_cpu ops are interrupt safe". So I think this is a more efficient fix:

lib/percpu_counter.c renamed from lib/percpu_counter.c~a +2 -2
@@ -82,10 +82,10 @@ void __percpu_counter_add(struct percpu_
                unsigned long flags;
                raw_spin_lock_irqsave(&fbc->lock, flags);
                fbc->count += count;
+               __this_cpu_sub(*fbc->counters, count);
                raw_spin_unlock_irqrestore(&fbc->lock, flags);
-               __this_cpu_write(*fbc->counters, 0);
        } else {
-               __this_cpu_write(*fbc->counters, count);
+               this_cpu_add(*fbc->counters, amount);
        }
        preempt_enable();
 }
Quoted 4 lines

It avoids the local_irq_disable() in the common case, when the CPU supports efficient this_cpu_add(). It will in rare race situations permit the cpu-local counter to exceed `batch', but that should be harmless.

I am wondering if the above patch is more efficient, because:

- raw_local_irq_save()/raw_local_irq_restore() should be cheaper than preempt_enable() in theory

- except for x86 and s390, other ARCHs have not their own implementation of this_cpu_foo(), and the generic one just disables local interrupt when operating the percpu variable.

So I suggest to fix it by replacing preempt_* with raw_local_irq_*.

Thanks,

Signature · 1 line
Ming Lei
maintainer
#3 lore raw
Quoted 58 lines · On Wed, 8 Jan 2014 09:12:19 +0800 Ming Lei <[email protected]> wrote:

Hi Andrew,

Quoted 49 lines
Quoted 16 lines
lib/percpu_counter.c +3 -3
@@ -75,19 +75,19 @@ EXPORT_SYMBOL(percpu_counter_set);
 void __percpu_counter_add(struct percpu_counter *fbc, s64 amount, s32 batch)
 {
      s64 count;
+     unsigned long flags;
 
-     preempt_disable();
+     raw_local_irq_save(flags);
      count = __this_cpu_read(*fbc->counters) + amount;
      if (count >= batch || count <= -batch) {
-             unsigned long flags;
-             raw_spin_lock_irqsave(&fbc->lock, flags);
+             raw_spin_lock(&fbc->lock);
              fbc->count += count;

lib/percpu_counter.c EXPORT_SYMBOL(percpu_counter_set); · line 92

Quoted 10 lines
lib/percpu_counter.c +2 -2
-             raw_spin_unlock_irqrestore(&fbc->lock, flags);
+             raw_spin_unlock(&fbc->lock);
              __this_cpu_write(*fbc->counters, 0);
      } else {
              __this_cpu_write(*fbc->counters, count);
      }
-     preempt_enable();
+     raw_local_irq_restore(flags);
 }
 EXPORT_SYMBOL(__percpu_counter_add);

Can this be made more efficient?

The this_cpu_foo() documentation is fairly dreadful, but way down at the end of Documentation/this_cpu_ops.txt we find "this_cpu ops are interrupt safe". So I think this is a more efficient fix:

lib/percpu_counter.c renamed from lib/percpu_counter.c~a +2 -2
@@ -82,10 +82,10 @@ void __percpu_counter_add(struct percpu_
                unsigned long flags;
                raw_spin_lock_irqsave(&fbc->lock, flags);
                fbc->count += count;
+               __this_cpu_sub(*fbc->counters, count);
                raw_spin_unlock_irqrestore(&fbc->lock, flags);
-               __this_cpu_write(*fbc->counters, 0);
        } else {
-               __this_cpu_write(*fbc->counters, count);
+               this_cpu_add(*fbc->counters, amount);
        }
        preempt_enable();
 }
Quoted 4 lines

It avoids the local_irq_disable() in the common case, when the CPU supports efficient this_cpu_add(). It will in rare race situations permit the cpu-local counter to exceed `batch', but that should be harmless.

I am wondering if the above patch is more efficient, because:

Quoted 2 lines

- raw_local_irq_save()/raw_local_irq_restore() should be cheaper than preempt_enable() in theory

Don't think so - local_irq_disable() requires quite some internal synchronization in the CPU and is expensive. preempt_disable() is just an add+barrier, minus the add if the kernel is non-preemptable.

Quoted 3 lines

- except for x86 and s390, other ARCHs have not their own implementation of this_cpu_foo(), and the generic one just disables local interrupt when operating the percpu variable.

Yup. But other CPUs should and will optimise their this_cpu implementations over time.

author
#4 lore raw

Hi Andrew,

Quoted 7 lines · Andrew Morton
Quoted 4 lines

I am wondering if the above patch is more efficient, because:

- raw_local_irq_save()/raw_local_irq_restore() should be cheaper than preempt_enable() in theory

Don't think so - local_irq_disable() requires quite some internal synchronization in the CPU and is expensive. preempt_disable() is just

Yes, it might be a little expensive on some CPUs, but should be arch-dependent(CPU inside things are involved)

Quoted 1 line

an add+barrier, minus the add if the kernel is non-preemptable.

IMO, generally, from software view, local_irq_save() only save the CPU's interrupt mask to the local variable 'flag', and sets irq mask to register, considered local variable can be thought to be in cache, so I think it might be cheaper than preempt_enable() because preempt counter may not be in cache.

Also this_cpu_add() won't work in batch path(slow path), we still need to avoid interrupt coming between reading the percpu counter and resetting it, otherwise counts might be lost too, :-)

Thanks,

Signature · 1 line
Ming Lei
author
#5 lore raw
Quoted 22 lines · Ming Lei

Hi Andrew,

Quoted 7 lines · Andrew Morton
Quoted 4 lines

I am wondering if the above patch is more efficient, because:

- raw_local_irq_save()/raw_local_irq_restore() should be cheaper than preempt_enable() in theory

Don't think so - local_irq_disable() requires quite some internal synchronization in the CPU and is expensive. preempt_disable() is just

Yes, it might be a little expensive on some CPUs, but should be arch-dependent(CPU inside things are involved)

Quoted 1 line

an add+barrier, minus the add if the kernel is non-preemptable.

IMO, generally, from software view, local_irq_save() only save the CPU's interrupt mask to the local variable 'flag', and sets irq mask to register, considered local variable can be thought to be in cache, so I think it might be cheaper than preempt_enable() because preempt counter may not be in cache.

Quoted 3 lines

Also this_cpu_add() won't work in batch path(slow path), we still need to avoid interrupt coming between reading the percpu counter and resetting it, otherwise counts might be lost too, :-)

Sorry, I miss the __this_cpu_sub() in slow path, so it is correct, and even preempt_enable() and preempt_disable() can be removed.

Thanks,

Signature · 1 line
Ming Lei