Skip to contentSkip to conversation

powerpc: Use lwarx hint bit in spinlocks

PATCH In review Anton Blanchard 3 participants 4 messages Feb 10, 2010

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 powerpc: Use lwarx hint bit in spinlocks

Recent versions of the PowerPC architecture added a hint bit to the larx instructions to differentiate between an atomic operation and a lock operation:

Quoted 6 lines

0 Other programs might attempt to modify the word in storage addressed by EA even if the subsequent Store Conditional succeeds.

1 Other programs will not attempt to modify the word in storage addressed by EA until the program that has acquired the lock performs a subsequent store releasing the lock.

To avoid a binutils dependency this patch create macros for the extended lwarx format and uses it in the spinlock code. To test this change I used a simple test case that acquires and releases a global pthread mutex:

	pthread_mutex_lock(&mutex);
	pthread_mutex_unlock(&mutex);

On a 32 core POWER6 running 32 test threads we spend almost all our time in the futex spinlock code (as expected):

    94.37%     perf  [kernel]                     [k] ._raw_spin_lock
               |          
               |--99.95%-- ._raw_spin_lock
               |          |          
               |          |--63.29%-- .futex_wake
               |          |          
               |          |--36.64%-- .futex_wait_setup

which is a good test for this patch. The results (in lock/unlock operations per second) are:

before: 1538203 ops/sec
after:  2189219 ops/sec

An improvement of 42%

---

We've had issues in the past with chips that do the wrong thing and don't ignore instruction reserved bits. One option might be to limit this change to 64bit and verify on that limited set of CPUs.

arch/powerpc/include/asm/ppc-opcode.h +6 -0
Index: powerpc.git/arch/powerpc/include/asm/ppc-opcode.h
===================================================================
@@ -24,6 +24,7 @@
 #define PPC_INST_ISEL_MASK		0xfc00003e
 #define PPC_INST_LSWI			0x7c0004aa
 #define PPC_INST_LSWX			0x7c00042a
+#define PPC_INST_LWARX			0x7c000029
 #define PPC_INST_LWSYNC			0x7c2004ac
 #define PPC_INST_LXVD2X			0x7c000698
 #define PPC_INST_MCRXR			0x7c000400
@@ -55,15 +56,20 @@
 #define __PPC_RA(a)	(((a) & 0x1f) << 16)
 #define __PPC_RB(b)	(((b) & 0x1f) << 11)
 #define __PPC_RS(s)	(((s) & 0x1f) << 21)
+#define __PPC_RT(s)	__PPC_RS(s)
 #define __PPC_XS(s)	((((s) & 0x1f) << 21) | (((s) & 0x20) >> 5))
 #define __PPC_T_TLB(t)	(((t) & 0x3) << 21)
 #define __PPC_WC(w)	(((w) & 0x3) << 21)
+#define __PPC_EH(eh)	(((eh) & 0x1) << 0)
 
 /* Deal with instructions that older assemblers aren't aware of */
 #define	PPC_DCBAL(a, b)		stringify_in_c(.long PPC_INST_DCBAL | \
 					__PPC_RA(a) | __PPC_RB(b))
 #define	PPC_DCBZL(a, b)		stringify_in_c(.long PPC_INST_DCBZL | \
 					__PPC_RA(a) | __PPC_RB(b))
+#define PPC_LWARX(t, a, b, eh)	stringify_in_c(.long PPC_INST_LWARX | \
+					__PPC_RT(t) | __PPC_RA(a) | \
+					__PPC_RB(b) | __PPC_EH(eh))
 #define PPC_MSGSND(b)		stringify_in_c(.long PPC_INST_MSGSND | \
 					__PPC_RB(b))
 #define PPC_RFCI		stringify_in_c(.long PPC_INST_RFCI)
arch/powerpc/include/asm/spinlock.h +4 -3
Index: powerpc.git/arch/powerpc/include/asm/spinlock.h
===================================================================
@@ -27,6 +27,7 @@
 #endif
 #include <asm/asm-compat.h>
 #include <asm/synch.h>
+#include <asm/ppc-opcode.h>
 
 #define arch_spin_is_locked(x)		((x)->slock != 0)
 
@@ -60,7 +61,7 @@ static inline unsigned long __arch_spin_
 
 	token = LOCK_TOKEN;
 	__asm__ __volatile__(
-"1:	lwarx		%0,0,%2\n\
+"1:	" PPC_LWARX(%0,0,%2,1) "\n\
 	cmpwi		0,%0,0\n\
 	bne-		2f\n\
 	stwcx.		%1,0,%2\n\
@@ -186,7 +187,7 @@ static inline long __arch_read_trylock(a
 	long tmp;
 
 	__asm__ __volatile__(
-"1:	lwarx		%0,0,%1\n"
+"1:	" PPC_LWARX(%0,0,%1,1) "\n"
 	__DO_SIGN_EXTEND
 "	addic.		%0,%0,1\n\
 	ble-		2f\n"
@@ -211,7 +212,7 @@ static inline long __arch_write_trylock(
 
 	token = WRLOCK_TOKEN;
 	__asm__ __volatile__(
-"1:	lwarx		%0,0,%2\n\
+"1:	" PPC_LWARX(%0,0,%2,1) "\n\
 	cmpwi		0,%0,0\n\
 	bne-		2f\n"
 	PPC405_ERR77(0,%1)
#1 lore raw
Quoted 35 lines · Anton Blanchard

Recent versions of the PowerPC architecture added a hint bit to the larx instructions to differentiate between an atomic operation and a lock operation:

Quoted 6 lines

0 Other programs might attempt to modify the word in storage addressed by EA even if the subsequent Store Conditional succeeds.

1 Other programs will not attempt to modify the word in storage addressed by EA until the program that has acquired the lock performs a subsequent store releasing the lock.

To avoid a binutils dependency this patch create macros for the extended lwarx format and uses it in the spinlock code. To test this change I used a simple test case that acquires and releases a global pthread mutex:

	pthread_mutex_lock(&mutex);
	pthread_mutex_unlock(&mutex);

On a 32 core POWER6 running 32 test threads we spend almost all our time in the futex spinlock code (as expected):

   94.37%     perf  [kernel]                     [k] ._raw_spin_lock
              |          
              |--99.95%-- ._raw_spin_lock
              |          |          
              |          |--63.29%-- .futex_wake
              |          |          
              |          |--36.64%-- .futex_wait_setup

which is a good test for this patch. The results (in lock/unlock operations per second) are:

before: 1538203 ops/sec
after:  2189219 ops/sec
Quoted 8 lines

An improvement of 42%

---

We've had issues in the past with chips that do the wrong thing and don't ignore instruction reserved bits. One option might be to limit this change to 64bit and verify on that limited set of CPUs.

In the off chance that someone actually does an SMP 44x, I think the hint bit here would just be ignored (I could test possibly if we want to verify). However, I thought the FSL parts didn't like toggling the reserved bits and those already are SMP in some cases.

Kumar?

josh

#2 lore raw
Quoted 3 lines · Josh Boyer
Quoted 2 lines · Anton Blanchard

=20
Recent versions of the PowerPC architecture added a hint bit to the =

larx

Quoted 1 line
Quoted 1 line

instructions to differentiate between an atomic operation and a lock =

operation:

Quoted 2 lines
Quoted 2 lines

=20

Quoted 1 line

0 Other programs might attempt to modify the word in storage =

addressed by EA

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

even if the subsequent Store Conditional succeeds.
=20
1 Other programs will not attempt to modify the word in storage =

addressed by

Quoted 1 line
Quoted 1 line
Quoted 1 line

EA until the program that has acquired the lock performs a =

subsequent store

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

releasing the lock.

=20
To avoid a binutils dependency this patch create macros for the =

extended lwarx

Quoted 1 line
Quoted 1 line

format and uses it in the spinlock code. To test this change I used a =

simple

Quoted 6 lines
Quoted 6 lines

test case that acquires and releases a global pthread mutex:
=20

	pthread_mutex_lock(&mutex);
	pthread_mutex_unlock(&mutex);

=20
On a 32 core POWER6 running 32 test threads we spend almost all our =

time in

Quoted 11 lines
Quoted 11 lines

the futex spinlock code (as expected):
=20

  94.37%     perf  [kernel]                     [k] ._raw_spin_lock
             |         =20
             |--99.95%-- ._raw_spin_lock
             |          |         =20
             |          |--63.29%-- .futex_wake
             |          |         =20
             |          |--36.64%-- .futex_wait_setup

=20
which is a good test for this patch. The results (in lock/unlock =

operations

Quoted 11 lines
Quoted 11 lines

per second) are:
=20

before: 1538203 ops/sec
after:  2189219 ops/sec

=20
An improvement of 42%
=20

---
=20
We've had issues in the past with chips that do the wrong thing and =

don't

Quoted 1 line
Quoted 1 line

ignore instruction reserved bits. One option might be to limit this =

change to

Quoted 3 lines
Quoted 1 line

64bit and verify on that limited set of CPUs.

=20
In the off chance that someone actually does an SMP 44x, I think the =

hint bit

Quoted 1 line

here would just be ignored (I could test possibly if we want to =

verify).

Quoted 1 line

However, I thought the FSL parts didn't like toggling the reserved =

bits and

Quoted 5 lines

those already are SMP in some cases.
=20
Kumar?
=20
josh

Yeah, I need to test this on some older SMP parts that didn't treated = reserved bits being set in the opcode as ignored but illegal.

- k=

#3 lore raw
Quoted 10 lines · Kumar Gala
Quoted 8 lines

In the off chance that someone actually does an SMP 44x, I think the hint bit here would just be ignored (I could test possibly if we want to verify). However, I thought the FSL parts didn't like toggling the reserved bits and those already are SMP in some cases.

Kumar?

josh

Yeah, I need to test this on some older SMP parts that didn't treated reserved bits being set in the opcode as ignored but illegal.

It looks like Anton just did a v2 of the patch targeted at ppc64 instead.

josh