Skip to contentSkip to conversation

Avoid use of spinlock for percpu_counter

PATCH In review Ravikiran G Thirumalai 2 participants 3 messages Jan 25, 2006 – Jan 26, 2006

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Avoid use of spinlock for percpu_counter

The spinlock in struct percpu_counter protects just one counter. It's not obvious why it was done this way (I am guessing it was because earlier, atomic_t was guaranteed 24 bits only on some arches). Since we have atomic_long_t now, I don't see why this cannot be replaced with an atomic_t.

Comments?

include/linux/percpu_counter.h +4 -6
Index: linux-2.6.16-rc1/include/linux/percpu_counter.h
===================================================================
@@ -15,8 +15,7 @@
 #ifdef CONFIG_SMP
 
 struct percpu_counter {
-	spinlock_t lock;
-	long count;
+	atomic_long_t count;
 	long *counters;
 };
 
@@ -28,8 +27,7 @@ struct percpu_counter {
 
 static inline void percpu_counter_init(struct percpu_counter *fbc)
 {
-	spin_lock_init(&fbc->lock);
-	fbc->count = 0;
+	atomic_long_set(&fbc->count, 0);
 	fbc->counters = alloc_percpu(long);
 }
 
@@ -42,7 +40,7 @@ void percpu_counter_mod(struct percpu_co
 
 static inline long percpu_counter_read(struct percpu_counter *fbc)
 {
-	return fbc->count;
+	return atomic_long_read(&fbc->count);
 }
 
 /*
@@ -51,7 +49,7 @@ static inline long percpu_counter_read(s
  */
 static inline long percpu_counter_read_positive(struct percpu_counter *fbc)
 {
-	long ret = fbc->count;
+	long ret = atomic_long_read(&fbc->count);
 
 	barrier();		/* Prevent reloads of fbc->count */
 	if (ret > 0)
mm/swap.c +1 -3
Index: linux-2.6.16-rc1/mm/swap.c
===================================================================
@@ -449,9 +449,7 @@ void percpu_counter_mod(struct percpu_co
 	pcount = per_cpu_ptr(fbc->counters, cpu);
 	count = *pcount + amount;
 	if (count >= FBC_BATCH || count <= -FBC_BATCH) {
-		spin_lock(&fbc->lock);
-		fbc->count += count;
-		spin_unlock(&fbc->lock);
+		atomic_long_add(count, &fbc->count);
 		count = 0;
 	}
 	*pcount = count;
#1 lore raw
Quoted 6 lines · Ravikiran G Thirumalai

The spinlock in struct percpu_counter protects just one counter. It's not obvious why it was done this way (I am guessing it was because earlier, atomic_t was guaranteed 24 bits only on some arches). Since we have atomic_long_t now, I don't see why this cannot be replaced with an atomic_t.

Comments?

Yes this makes sense.

Furthermore, we could try to fix 'struct percpu_counter' management (if SMP) if alloc_percpu(long) call done in percpu_counter_init() fails. This is currently ignored and can crash.

Something like (hybrid patch, to get the idea) :

mm/swap.c 2006-01-26 16:00:54.000000000 +0100 renamed from mm/swap.c 2006-01-26 15:58:42.000000000 +0100 +5 -2
@@ -472,9 +472,12 @@
 {
        long count;
        long *pcount;
-       int cpu = get_cpu();
 
-       pcount = per_cpu_ptr(fbc->counters, cpu);
+       if (unlikely(fbc->counters == NULL)) {
+               atomic_long_add(amount, &fbc->count);
+               return;
+       }
+       pcount = per_cpu_ptr(fbc->counters, get_cpu());
        count = *pcount + amount;
        if (count >= FBC_BATCH || count <= -FBC_BATCH) {
                atomic_long_add(count, &fbc->count);
include/linux/percpu_counter.h 2006-01-26 16:02:53.000000000 +0100 renamed from include/linux/percpu_counter.h 2006-01-26 16:02:31.000000000 +0100 +2 -1
@@ -35,7 +35,8 @@
 
 static inline void percpu_counter_destroy(struct percpu_counter *fbc)
 {
-       free_percpu(fbc->counters);
+       if (fbc->counters)
+               free_percpu(fbc->counters);
 }
 
 void percpu_counter_mod(struct percpu_counter *fbc, long amount);

Eric

#2 lore raw
Quoted 19 lines · Eric Dumazet
Quoted 7 lines · Ravikiran G Thirumalai

The spinlock in struct percpu_counter protects just one counter. It's not obvious why it was done this way (I am guessing it was because earlier, atomic_t was guaranteed 24 bits only on some arches). Since we have atomic_long_t now, I don't see why this cannot be replaced with an atomic_t.

Comments?

Yes this makes sense.

Furthermore, we could try to fix 'struct percpu_counter' management (if SMP) if alloc_percpu(long) call done in percpu_counter_init() fails. This is currently ignored and can crash.
Something like (hybrid patch, to get the idea) :

mm/swap.c 2006-01-26 16:00:54.000000000 +0100 line 478

Quoted 9 lines
mm/swap.c 2006-01-26 16:00:54.000000000 +0100 renamed from mm/swap.c 2006-01-26 15:58:42.000000000 +0100 +3 -2
@@ -472,9 +472,12 @@
 {
        long count;
        long *pcount;
-       int cpu = get_cpu();
 
-       pcount = per_cpu_ptr(fbc->counters, cpu);
+       if (unlikely(fbc->counters == NULL)) {
+               atomic_long_add(amount, &fbc->count);
+               return;

I don't know if adding another branch to the fast path is a good idea, would it not be better if this was handled by returning an error at percpu_counter_init? If we are in agreement, then I can make a patch for the same.

Thanks,
Kiran