lib/percpu_counter.c: disable local irq when updating percpu couter
Subsystems (2)
PATCH 1/1 lib/percpu_counter.c: disable local irq when updating percpu couter
__percpu_counter_add() may be called in softirq/hardirq handler (such as, blk_mq_queue_exit() is typically called in hardirq/softirq handler), so we need to disable local irq when updating the percpu counter, otherwise counts may be lost.
The patch fixes problem that 'rmmod null_blk' may hang in blk_cleanup_queue() because of miscounting of request_queue->mq_usage_counter.
--- lib/percpu_counter.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
lib/percpu_counter.c +5 -5
@@ -75,19 +75,19 @@ EXPORT_SYMBOL(percpu_counter_set);void __percpu_counter_add (struct percpu_counter *fbc,s64 amount,s32 batch) {s64 count; +unsigned long flags; -preempt_disable (); +raw_local_irq_save (flags); count =__this_cpu_read (*fbc->counters) + amount;if (count >= batch || count <= -batch) { -unsigned long flags; -raw_spin_lock_irqsave (&fbc->lock, flags); +raw_spin_lock (&fbc->lock); fbc->count += count; -raw_spin_unlock_irqrestore (&fbc->lock, flags); +raw_spin_unlock (&fbc->lock);__this_cpu_write (*fbc->counters,0 ); }else {__this_cpu_write (*fbc->counters, count); } -preempt_enable (); +raw_local_irq_restore (flags); }EXPORT_SYMBOL (__percpu_counter_add);
Signature · 1 line
1.7.9.5
Andrew Morton
maintainerQuoted 4 lines · On Tue, 7 Jan 2014 18:29:27 +0800 Ming Lei <[email protected]> wrote:
__percpu_counter_add()may be called in softirq/hardirq handler (such as,blk_mq_queue_exit()is typically called in hardirq/softirq handler), so we need to disable local irq when updating the percpu counter, otherwise counts may be lost.
OK.
Quoted 21 lines
The patch fixes problem that 'rmmod
null_blk' may hang inblk_cleanup_queue()because of miscounting ofrequest_queue->mq_usage_counter....
lib/percpu_counter.c +3
-3@@ -75,19 +75,19 @@ EXPORT_SYMBOL(percpu_counter_set);void __percpu_counter_add (struct percpu_counter *fbc,s64 amount,s32 batch) {s64 count; +unsigned long flags; -preempt_disable (); +raw_local_irq_save (flags); count =__this_cpu_read (*fbc->counters) + amount;if (count >= batch || count <= -batch) { -unsigned long flags; -raw_spin_lock_irqsave (&fbc->lock, flags); +raw_spin_lock (&fbc->lock); fbc->count += count;
lib/percpu_counter.c EXPORT_SYMBOL(percpu_counter_set); · line 92
Quoted 10 lines
lib/percpu_counter.c +2
-2-raw_spin_unlock_irqrestore (&fbc->lock, flags); +raw_spin_unlock (&fbc->lock);__this_cpu_write (*fbc->counters,0 ); }else {__this_cpu_write (*fbc->counters, count); } -preempt_enable (); +raw_local_irq_restore (flags); }EXPORT_SYMBOL (__percpu_counter_add);
Can this be made more efficient?
The this_cpu_foo() documentation is fairly dreadful, but way down at the end of Documentation/this_cpu_ops.txt we find "this_cpu ops are interrupt safe". So I think this is a more efficient fix:
lib/percpu_counter.c renamed from lib/percpu_counter.c~a +2 -2
@@ -82,10 +82,10 @@ void __percpu_counter_add(struct percpu_unsigned long flags;raw_spin_lock_irqsave (&fbc->lock, flags); fbc->count += count; +__this_cpu_sub (*fbc->counters, count);raw_spin_unlock_irqrestore (&fbc->lock, flags); -__this_cpu_write (*fbc->counters,0 ); }else { -__this_cpu_write (*fbc->counters, count); +this_cpu_add (*fbc->counters, amount); }preempt_enable (); }
It avoids the local_irq_disable() in the common case, when the CPU supports efficient this_cpu_add(). It will in rare race situations permit the cpu-local counter to exceed `batch', but that should be harmless.
What do you think?
Hi Andrew,
Quoted 63 lines · Andrew Morton
Quoted 4 lines · On Tue, 7 Jan 2014 18:29:27 +0800 Ming Lei <[email protected]> wrote:
__percpu_counter_add()may be called in softirq/hardirq handler (such as,blk_mq_queue_exit()is typically called in hardirq/softirq handler), so we need to disable local irq when updating the percpu counter, otherwise counts may be lost.OK.
Quoted 21 lines
The patch fixes problem that 'rmmod
null_blk' may hang inblk_cleanup_queue()because of miscounting ofrequest_queue->mq_usage_counter....
lib/percpu_counter.c +3
-3@@ -75,19 +75,19 @@ EXPORT_SYMBOL(percpu_counter_set);void __percpu_counter_add (struct percpu_counter *fbc,s64 amount,s32 batch) {s64 count; +unsigned long flags; -preempt_disable (); +raw_local_irq_save (flags); count =__this_cpu_read (*fbc->counters) + amount;if (count >= batch || count <= -batch) { -unsigned long flags; -raw_spin_lock_irqsave (&fbc->lock, flags); +raw_spin_lock (&fbc->lock); fbc->count += count;lib/percpu_counter.c EXPORT_SYMBOL(percpu_counter_set); · line 92
Quoted 10 lines
lib/percpu_counter.c +2
-2-raw_spin_unlock_irqrestore (&fbc->lock, flags); +raw_spin_unlock (&fbc->lock);__this_cpu_write (*fbc->counters,0 ); }else {__this_cpu_write (*fbc->counters, count); } -preempt_enable (); +raw_local_irq_restore (flags); }EXPORT_SYMBOL (__percpu_counter_add);Can this be made more efficient?
The
this_cpu_foo()documentation is fairly dreadful, but way down at the end ofDocumentation/this_cpu_ops.txtwe find "this_cpuops are interrupt safe". So I think this is a more efficient fix:lib/percpu_counter.c renamed from lib/percpu_counter.c~a +2
-2@@ -82,10 +82,10 @@ void __percpu_counter_add(struct percpu_unsigned long flags;raw_spin_lock_irqsave (&fbc->lock, flags); fbc->count += count; +__this_cpu_sub (*fbc->counters, count);raw_spin_unlock_irqrestore (&fbc->lock, flags); -__this_cpu_write (*fbc->counters,0 ); }else { -__this_cpu_write (*fbc->counters, count); +this_cpu_add (*fbc->counters, amount); }preempt_enable (); }
Quoted 4 lines
It avoids the
local_irq_disable()in the common case, when the CPU supports efficientthis_cpu_add(). It will in rare race situations permit the cpu-local counter to exceed `batch', but that should be harmless.
I am wondering if the above patch is more efficient, because:
- raw_local_irq_save()/raw_local_irq_restore() should be cheaper than preempt_enable() in theory
- except for x86 and s390, other ARCHs have not their own implementation of this_cpu_foo(), and the generic one just disables local interrupt when operating the percpu variable.
So I suggest to fix it by replacing preempt_* with raw_local_irq_*.
Thanks,
Signature · 1 line
Ming Lei
Andrew Morton
maintainerQuoted 58 lines · On Wed, 8 Jan 2014 09:12:19 +0800 Ming Lei <[email protected]> wrote:
Hi Andrew,
Quoted 49 lines
Quoted 16 lines
lib/percpu_counter.c +3
-3@@ -75,19 +75,19 @@ EXPORT_SYMBOL(percpu_counter_set);void __percpu_counter_add (struct percpu_counter *fbc,s64 amount,s32 batch) {s64 count; +unsigned long flags; -preempt_disable (); +raw_local_irq_save (flags); count =__this_cpu_read (*fbc->counters) + amount;if (count >= batch || count <= -batch) { -unsigned long flags; -raw_spin_lock_irqsave (&fbc->lock, flags); +raw_spin_lock (&fbc->lock); fbc->count += count;lib/percpu_counter.c EXPORT_SYMBOL(percpu_counter_set); · line 92
Quoted 10 lines
lib/percpu_counter.c +2
-2-raw_spin_unlock_irqrestore (&fbc->lock, flags); +raw_spin_unlock (&fbc->lock);__this_cpu_write (*fbc->counters,0 ); }else {__this_cpu_write (*fbc->counters, count); } -preempt_enable (); +raw_local_irq_restore (flags); }EXPORT_SYMBOL (__percpu_counter_add);Can this be made more efficient?
The
this_cpu_foo()documentation is fairly dreadful, but way down at the end ofDocumentation/this_cpu_ops.txtwe find "this_cpuops are interrupt safe". So I think this is a more efficient fix:lib/percpu_counter.c renamed from lib/percpu_counter.c~a +2
-2@@ -82,10 +82,10 @@ void __percpu_counter_add(struct percpu_unsigned long flags;raw_spin_lock_irqsave (&fbc->lock, flags); fbc->count += count; +__this_cpu_sub (*fbc->counters, count);raw_spin_unlock_irqrestore (&fbc->lock, flags); -__this_cpu_write (*fbc->counters,0 ); }else { -__this_cpu_write (*fbc->counters, count); +this_cpu_add (*fbc->counters, amount); }preempt_enable (); }Quoted 4 lines
It avoids the
local_irq_disable()in the common case, when the CPU supports efficientthis_cpu_add(). It will in rare race situations permit the cpu-local counter to exceed `batch', but that should be harmless.I am wondering if the above patch is more efficient, because:
Quoted 2 lines
- raw_local_irq_save()/raw_local_irq_restore() should be cheaper than
preempt_enable()in theory
Don't think so - local_irq_disable() requires quite some internal synchronization in the CPU and is expensive. preempt_disable() is just an add+barrier, minus the add if the kernel is non-preemptable.
Quoted 3 lines
- except for x86 and s390, other ARCHs have not their own implementation of
this_cpu_foo(), and the generic one just disables local interrupt when operating the percpu variable.
Yup. But other CPUs should and will optimise their this_cpu implementations over time.
Hi Andrew,
Quoted 7 lines · Andrew Morton
Quoted 4 lines
I am wondering if the above patch is more efficient, because:
- raw_local_irq_save()/raw_local_irq_restore() should be cheaper than
preempt_enable()in theoryDon't think so -
local_irq_disable()requires quite some internal synchronization in the CPU and is expensive.preempt_disable()is just
Yes, it might be a little expensive on some CPUs, but should be arch-dependent(CPU inside things are involved)
Quoted 1 line
an add+barrier, minus the add if the kernel is non-preemptable.
IMO, generally, from software view, local_irq_save() only save the CPU's interrupt mask to the local variable 'flag', and sets irq mask to register, considered local variable can be thought to be in cache, so I think it might be cheaper than preempt_enable() because preempt counter may not be in cache.
Also this_cpu_add() won't work in batch path(slow path), we still need to avoid interrupt coming between reading the percpu counter and resetting it, otherwise counts might be lost too, :-)
Thanks,
Signature · 1 line
Ming Lei
Quoted 22 lines · Ming Lei
Hi Andrew,
Quoted 7 lines · Andrew Morton
Quoted 4 lines
I am wondering if the above patch is more efficient, because:
- raw_local_irq_save()/raw_local_irq_restore() should be cheaper than
preempt_enable()in theoryDon't think so -
local_irq_disable()requires quite some internal synchronization in the CPU and is expensive.preempt_disable()is justYes, it might be a little expensive on some CPUs, but should be arch-dependent(CPU inside things are involved)
Quoted 1 line
an add+barrier, minus the add if the kernel is non-preemptable.
IMO, generally, from software view,
local_irq_save()only save the CPU's interrupt mask to the local variable 'flag', and sets irq mask to register, considered local variable can be thought to be in cache, so I think it might be cheaper thanpreempt_enable()because preempt counter may not be in cache.
Quoted 3 lines
Also
this_cpu_add()won't work in batch path(slow path), we still need to avoid interrupt coming between reading the percpu counter and resetting it, otherwise counts might be lost too, :-)
Sorry, I miss the __this_cpu_sub() in slow path, so it is correct, and even preempt_enable() and preempt_disable() can be removed.
Thanks,
Signature · 1 line
Ming Lei