x86-64, rwsem: Avoid store forwarding hazzard in __downgrade_write
PATCH 1/1 x86-64, rwsem: Avoid store forwarding hazzard in __downgrade_write
The Intel Architecture Optimization Reference Manual states that a short load that follows a long store to the same object will suffer a store forwading penalty, particularly if the two accesses use different addresses. Trivially, a long load that follows a short store will also suffer a penalty.
__downgrade_write() in rwsem incurs both penalties: the increment operation will not be able to reuse a recently-loaded rwsem value, and its result will not be reused by any recently-following rwsem operation.
A comment in the code states that this is because 64-bit immediates are special and expensive; but while they are slightly special (only a single instruction allows them), they aren't expensive: a test shows that two loops, one loading a 32-bit immediate and one loading a 64-bit immediate, both take 1.5 cycles per iteration.
Fix this by changing __downgrade_write to use the same add instruction on i386 and on x86_64, so that it uses the same operand size as all the other rwsem functions.
--- arch/x86/include/asm/rwsem.h | 30 ++++++++++-------------------- 1 files changed, 10 insertions(+), 20 deletions(-)
arch/x86/include/asm/rwsem.h +10 -20
@@ -232,34 +232,24 @@ static inline void __up_write(struct rw_semaphore *sem)*/ static inline void __downgrade_write (struct rw_semaphore *sem) { -#ifdef CONFIG_X86_64 -# if RWSEM_WAITING_BIAS !=-0x100000000 -# error "This code assumes RWSEM_WAITING_BIAS == -2^32" -# endif - -/* 64-bit immediates are special and expensive, and not needed here */ - asmvolatile ("# beginning __downgrade_write\n\t" -LOCK_PREFIX "incl 4(%1)\n\t" -/* transitions 0xZZZZZZZZ00000001 -> 0xYYYYYYYY00000001 */ -" jns 1f\n\t" -" call call_rwsem_downgrade_wake\n" -"1:\n\t" -"# ending __downgrade_write\n" - :"+m" (sem->count) - :"a" (sem) - :"memory" ,"cc" ); -#else asmvolatile ("# beginning __downgrade_write\n\t" LOCK_PREFIX _ASM_ADD"%2,(%1)\n\t" -/* transitions 0xZZZZ0001 -> 0xYYYY0001 */ +/* +* transitions 0xZZZZ0001 -> 0xYYYY0001 (i386) +* 0xZZZZZZZZ00000001 -> 0xYYYYYYYY00000001 (x86_64) +*/ " jns 1f\n\t" " call call_rwsem_downgrade_wake\n" "1:\n\t" "# ending __downgrade_write\n" :"+m" (sem->count) - :"a" (sem),"i" (-RWSEM_WAITING_BIAS ) - :"memory" ,"cc" ); + : "a" (sem), +#ifndef CONFIG_X86_64 + "i" (-RWSEM_WAITING_BIAS) +#else +"r" (-RWSEM_WAITING_BIAS ) #endif + :"memory" ,"cc" ); }/*
Signature · 1 line
1.6.5.3
H. Peter Anvin
reviewerarch/x86/include/asm/rwsem.h static inline void __up_write(struct rw_semaphore *sem) · line 252
Quoted 6 lines · Avi Kivity
arch/x86/include/asm/rwsem.h +5
-0+#ifndef CONFIG_X86_64 + "i" (-RWSEM_WAITING_BIAS) +#else + "r" (-RWSEM_WAITING_BIAS) #endif + : "memory", "cc");
Using an "er" constraint here should work on both architectures.
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.
Quoted 12 lines · H. Peter Anvin
arch/x86/include/asm/rwsem.h static inline void __up_write(struct rw_semaphore *sem) · line 252
Quoted 7 lines · Avi Kivity
arch/x86/include/asm/rwsem.h +5
-0+#ifndef CONFIG_X86_64 + "i" (-RWSEM_WAITING_BIAS) +#else + "r" (-RWSEM_WAITING_BIAS) #endif + : "memory", "cc");Using an "er" constraint here should work on both architectures.
Right. v2 to follow.
Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.