Avoid use of spinlock for percpu_counter
Subsystems (2)
PATCH 1/1 Avoid use of spinlock for percpu_counter
The spinlock in struct percpu_counter protects just one counter. It's not obvious why it was done this way (I am guessing it was because earlier, atomic_t was guaranteed 24 bits only on some arches). Since we have atomic_long_t now, I don't see why this cannot be replaced with an atomic_t.
Comments?
include/linux/percpu_counter.h +4 -6
@@ -15,8 +15,7 @@#ifdef CONFIG_SMP struct percpu_counter { -spinlock_t lock; -long count; +atomic_long_t count;long *counters; };
@@ -28,8 +27,7 @@ struct percpu_counter {static inline void percpu_counter_init (struct percpu_counter *fbc) { -spin_lock_init (&fbc->lock); - fbc->count =0 ; +atomic_long_set (&fbc->count,0 ); fbc->counters =alloc_percpu (long); }
@@ -42,7 +40,7 @@ void percpu_counter_mod(struct percpu_costatic inline long percpu_counter_read (struct percpu_counter *fbc) { -return fbc->count; +return atomic_long_read (&fbc->count); }/*
@@ -51,7 +49,7 @@ static inline long percpu_counter_read(s*/ static inline long percpu_counter_read_positive (struct percpu_counter *fbc) { -long ret = fbc->count; +long ret =atomic_long_read (&fbc->count);barrier ();/* Prevent reloads of fbc->count */ if (ret >0 )
mm/swap.c +1 -3
@@ -449,9 +449,7 @@ void percpu_counter_mod(struct percpu_co pcount =per_cpu_ptr (fbc->counters, cpu); count = *pcount + amount;if (count >=FBC_BATCH || count <= -FBC_BATCH ) { -spin_lock (&fbc->lock); - fbc->count += count; -spin_unlock (&fbc->lock); +atomic_long_add (count, &fbc->count); count =0 ; } *pcount = count;
Quoted 6 lines · Ravikiran G Thirumalai
The spinlock in struct
percpu_counterprotects just one counter. It's not obvious why it was done this way (I am guessing it was because earlier,atomic_twas guaranteed 24 bits only on some arches). Since we haveatomic_long_tnow, I don't see why this cannot be replaced with anatomic_t.Comments?
Yes this makes sense.
Furthermore, we could try to fix 'struct percpu_counter' management (if SMP) if alloc_percpu(long) call done in percpu_counter_init() fails. This is currently ignored and can crash.
Something like (hybrid patch, to get the idea) :
mm/swap.c 2006-01-26 16:00:54.000000000 +0100 renamed from mm/swap.c 2006-01-26 15:58:42.000000000 +0100 +5 -2
@@ -472,9 +472,12 @@ { long count; long *pcount; - int cpu = get_cpu(); - pcount = per_cpu_ptr(fbc->counters, cpu); + if (unlikely(fbc->counters == NULL)) { + atomic_long_add(amount, &fbc->count); + return; + } + pcount = per_cpu_ptr(fbc->counters, get_cpu()); count = *pcount + amount; if (count >= FBC_BATCH || count <= -FBC_BATCH) { atomic_long_add(count, &fbc->count);
include/linux/percpu_counter.h 2006-01-26 16:02:53.000000000 +0100 renamed from include/linux/percpu_counter.h 2006-01-26 16:02:31.000000000 +0100 +2 -1
@@ -35,7 +35,8 @@ static inline void percpu_counter_destroy(struct percpu_counter *fbc) { - free_percpu(fbc->counters); + if (fbc->counters) + free_percpu(fbc->counters); } void percpu_counter_mod(struct percpu_counter *fbc, long amount);
Eric
Quoted 19 lines · Eric Dumazet
Quoted 7 lines · Ravikiran G Thirumalai
The spinlock in struct
percpu_counterprotects just one counter. It's not obvious why it was done this way (I am guessing it was because earlier,atomic_twas guaranteed 24 bits only on some arches). Since we haveatomic_long_tnow, I don't see why this cannot be replaced with anatomic_t.Comments?
Yes this makes sense.
Furthermore, we could try to fix 'struct
percpu_counter' management (if SMP) ifalloc_percpu(long)call done inpercpu_counter_init()fails. This is currently ignored and can crash.
Something like (hybrid patch, to get the idea) :
mm/swap.c 2006-01-26 16:00:54.000000000 +0100 line 478
Quoted 9 lines
mm/swap.c 2006-01-26 16:00:54.000000000 +0100 renamed from mm/swap.c 2006-01-26 15:58:42.000000000 +0100 +3
-2@@ -472,9 +472,12 @@ { long count; long *pcount; - int cpu = get_cpu(); - pcount = per_cpu_ptr(fbc->counters, cpu); + if (unlikely(fbc->counters == NULL)) { + atomic_long_add(amount, &fbc->count); + return;
I don't know if adding another branch to the fast path is a good idea, would it not be better if this was handled by returning an error at percpu_counter_init? If we are in agreement, then I can make a patch for the same.
Thanks,
Kiran