arm64: xchg: Implement cmpxchg_double

The arm64 architecture has the ability to exclusively load and store a pair of registers from an address (ldxp/stxp). Also the SLUB can take advantage of a cmpxchg_double implementation to avoid taking some locks.

This patch provides an implementation of cmpxchg_double for 64-bit pairs, and activates the logic required for the SLUB to use these functions (HAVE_ALIGNED_STRUCT_PAGE and HAVE_CMPXCHG_DOUBLE).

On a Juno platform running on only the A57s I get quite a noticeable performance improvement with hackbench.

Before patch applied:

$ ./hackbench 100 process 1000
Running with 100*40 (== 4000) tasks.
Time: 206.331

After patch applied:

$ ./hackbench 100 process 1000
Running with 100*40 (== 4000) tasks.
Time: 182.396

---
 arch/arm64/Kconfig               |  2 ++
 arch/arm64/include/asm/cmpxchg.h | 55 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 57 insertions(+)

index fd4e81a..4a0f9a1 100644
@@ -31,12 +31,14 @@ config ARM64
 	select GENERIC_STRNLEN_USER
 	select GENERIC_TIME_VSYSCALL
 	select HARDIRQS_SW_RESEND
+	select HAVE_ALIGNED_STRUCT_PAGE if SLUB
 	select HAVE_ARCH_AUDITSYSCALL
 	select HAVE_ARCH_JUMP_LABEL
 	select HAVE_ARCH_KGDB
 	select HAVE_ARCH_TRACEHOOK
 	select HAVE_C_RECORDMCOUNT
 	select HAVE_CC_STACKPROTECTOR
+	select HAVE_CMPXCHG_DOUBLE
 	select HAVE_DEBUG_BUGVERBOSE
 	select HAVE_DEBUG_KMEMLEAK
 	select HAVE_DMA_API_DEBUG

1.9.3

The arm64 architecture has the ability to exclusively load and store a pair of registers from an address (ldxp/stxp). Also the SLUB can take advantage of a cmpxchg_double implementation to avoid taking some locks.

This patch provides an implementation of cmpxchg_double for 64-bit pairs, and activates the logic required for the SLUB to use these functions (HAVE_ALIGNED_STRUCT_PAGE and HAVE_CMPXCHG_DOUBLE).

Also definitions of this_cpu_cmpxchg_8 and this_cpu_cmpxchg_double_8 are wired up to cmpxchg_local and cmpxchg_double_local (rather than the stock implementations that perform non-atomic operations with interrupts disabled) as they are used by the SLUB.

On a Juno platform running on only the A57s I get quite a noticeable performance improvement with 5 runs of hackbench on v3.17:

Baseline | With Patch
-----------------+-----------
Mean 119.2312 | 106.1782
StdDev 0.4919 | 0.4494

(times taken to complete `./hackbench 100 process 1000', in seconds)

---
Changed in V2, added the this_cpu_cmpxchg* definitions, these are used by the fast path of the SLUB (without this our hackbench mean goes up to 111.9 seconds).
Cheers Liviu for pointing out this ommission!

The performance measurements were taken against a newer kernel running on a board with newer firmware, thus the baseline is faster than the one posted in V1.

---
 arch/arm64/Kconfig               |  2 ++
 arch/arm64/include/asm/cmpxchg.h | 71 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 73 insertions(+)

index fd4e81a..4a0f9a1 100644
@@ -31,12 +31,14 @@ config ARM64
 	select GENERIC_STRNLEN_USER
 	select GENERIC_TIME_VSYSCALL
 	select HARDIRQS_SW_RESEND
+	select HAVE_ALIGNED_STRUCT_PAGE if SLUB
 	select HAVE_ARCH_AUDITSYSCALL
 	select HAVE_ARCH_JUMP_LABEL
 	select HAVE_ARCH_KGDB
 	select HAVE_ARCH_TRACEHOOK
 	select HAVE_C_RECORDMCOUNT
 	select HAVE_CC_STACKPROTECTOR
+	select HAVE_CMPXCHG_DOUBLE
 	select HAVE_DEBUG_BUGVERBOSE
 	select HAVE_DEBUG_KMEMLEAK
 	select HAVE_DMA_API_DEBUG

1.9.3

Thanks Steve, I'll queue this for 3.19.

On a related note, I spoke to Christoph at kernel summit and we decided that we should have a go at implementing all of the per-cpu atomics using the atomic instructions, as this is likely to be quicker than disabling interrupts, especially since we don't require any barrier semantics.

Is that something you think you'll have a chance to look at, or shall I keep it on my list?

Cheers,

Will