powerpc: Hide some system call labels from profile tools
PATCH 1/4 powerpc: Hide some system call labels from profile tools
syscall_dotrace_cont and syscall_error_cont tend to complicate perf output so make them local.
---
arch/powerpc/kernel/entry_64.S +4 -4
@@ -154,7 +154,7 @@ END_FW_FTR_SECTION_IFSET(FW_FEATURE_SPLP ld r10,TI_FLAGS(r11) andi. r11,r10,_TIF_SYSCALL_T_OR_A bne- syscall_dotrace -syscall_dotrace_cont: +.Lsyscall_dotrace_cont: cmpldi 0,r0,NR_syscalls bge- syscall_enosys
@@ -211,7 +211,7 @@ syscall_exit: cmpld r3,r11 ld r5,_CCR(r1) bge- syscall_error -syscall_error_cont: +.Lsyscall_error_cont: ld r7,_NIP(r1) BEGIN_FTR_SECTION stdcx. r0,0,r1 /* to clear the reservation */
@@ -246,7 +246,7 @@ syscall_error: oris r5,r5,0x1000 /* Set SO bit in CR */ neg r3,r3 std r5,_CCR(r1) - b syscall_error_cont + b .Lsyscall_error_cont /* Traced system call support */ syscall_dotrace:
@@ -268,7 +268,7 @@ syscall_dotrace: addi r9,r1,STACK_FRAME_OVERHEAD clrrdi r10,r1,THREAD_SHIFT ld r10,TI_FLAGS(r10) - b syscall_dotrace_cont + b .Lsyscall_dotrace_cont syscall_enosys: li r3,-ENOSYS
PATCH 2/4 powerpc: No need to save XER in a system call
The XER is a volatile register so there is no need to save and restore it over a system call - zero it out in the exception stack frame instead.
This should fix a 5 cycle stall of the mfxer/std seen on POWER7.
---
arch/powerpc/kernel/entry_64.S +1 -2
@@ -82,6 +82,7 @@ system_call_common: std r11,GPR10(r1) std r11,GPR11(r1) std r11,GPR12(r1) + std r11,_XER(r1) std r9,GPR13(r1) mfcr r9 mflr r10
@@ -89,9 +90,7 @@ system_call_common: std r9,_CCR(r1) std r10,_LINK(r1) std r11,_TRAP(r1) - mfxer r9 mfctr r10 - std r9,_XER(r1) std r10,_CTR(r1) std r3,ORIG_GPR3(r1) ld r2,PACATOC(r13)
PATCH 3/4 powerpc: Better scheduling of CR save code in system call path
In reply to Anton BlanchardAt the moment system call entry looks like:
crclr so
...
mfcr r9
...
std r9,_CCR(r1)
commit bd19c8994a82 ([POWERPC] system call micro optimisation) put some space between the crclr and mfcr in order to avoid a stall.
There is still a stall seen between the mfcr and std. We can avoid the crclr by doing it in a GPR with rlwinm which gives us more room to better schedule the sequence.
---
arch/powerpc/kernel/entry_64.S +7 -9
@@ -63,15 +63,9 @@ system_call_common: std r0,GPR0(r1) std r10,GPR1(r1) ACCOUNT_CPU_USER_ENTRY(r10, r11) - /* - * This "crclr so" clears CR0.SO, which is the error indication on - * return from this system call. There must be no cmp instruction - * between it and the "mfcr r9" below, otherwise if XER.SO is set, - * CR0.SO will get set, causing all system calls to appear to fail. - */ - crclr so std r2,GPR2(r1) std r3,GPR3(r1) + mfcr r2 std r4,GPR4(r1) std r5,GPR5(r1) std r6,GPR6(r1)
@@ -84,15 +78,19 @@ system_call_common: std r11,GPR12(r1) std r11,_XER(r1) std r9,GPR13(r1) - mfcr r9 mflr r10 li r11,0xc01 - std r9,_CCR(r1) + /* + * This clears CR0.SO, which is the error indication on return + * from this system call. + */ + rlwinm r2,r2,0,4,2 std r10,_LINK(r1) std r11,_TRAP(r1) mfctr r10 std r10,_CTR(r1) std r3,ORIG_GPR3(r1) + std r2,_CCR(r1) ld r2,PACATOC(r13) addi r9,r1,STACK_FRAME_OVERHEAD ld r11,exception_marker@toc(r2)
PATCH 4/4 powerpc: No need to preserve count register across system call
In reply to Anton BlanchardThe count register is volatile so we don't need to preserve it. Store zero to the entry in the exception frame.
---
arch/powerpc/kernel/entry_64.S +1 -2
@@ -77,6 +77,7 @@ system_call_common: std r11,GPR11(r1) std r11,GPR12(r1) std r11,_XER(r1) + std r11,_CTR(r1) std r9,GPR13(r1) mflr r10 li r11,0xc01
@@ -87,8 +88,6 @@ system_call_common: rlwinm r2,r2,0,4,2 std r10,_LINK(r1) std r11,_TRAP(r1) - mfctr r10 - std r10,_CTR(r1) std r3,ORIG_GPR3(r1) std r2,_CCR(r1) ld r2,PACATOC(r13)
PATCH v2 3/4 powerpc: Better scheduling of CR save code in system call path
In reply to Anton BlanchardAt the moment system call entry looks like:
crclr so
...
mfcr r9
...
std r9,_CCR(r1)
commit bd19c8994a82 ([POWERPC] system call micro optimisation) put some space between the crclr and mfcr in order to avoid a stall.
There is still a stall seen between the mfcr and std. We can avoid the crclr by doing it in a GPR with rlwinm which gives us more room to better schedule the sequence.
---
v2:
Milton pointed out that rlwinm copies the low 32bits into the top 32bits and this might confuse ptracers. We tossed around a few ideas but his suggestion of using rldimi sounds good to me.
arch/powerpc/kernel/entry_64.S +7 -9
@@ -63,15 +63,9 @@ system_call_common: std r0,GPR0(r1) std r10,GPR1(r1) ACCOUNT_CPU_USER_ENTRY(r10, r11) - /* - * This "crclr so" clears CR0.SO, which is the error indication on - * return from this system call. There must be no cmp instruction - * between it and the "mfcr r9" below, otherwise if XER.SO is set, - * CR0.SO will get set, causing all system calls to appear to fail. - */ - crclr so std r2,GPR2(r1) std r3,GPR3(r1) + mfcr r2 std r4,GPR4(r1) std r5,GPR5(r1) std r6,GPR6(r1)
@@ -84,15 +78,19 @@ system_call_common: std r11,GPR12(r1) std r11,_XER(r1) std r9,GPR13(r1) - mfcr r9 mflr r10 + /* + * This clears CR0.SO (bit 28), which is the error indication on + * return from this system call. + */ + rldimi r2,r11,28,(63-28) li r11,0xc01 - std r9,_CCR(r1) std r10,_LINK(r1) std r11,_TRAP(r1) mfctr r10 std r10,_CTR(r1) std r3,ORIG_GPR3(r1) + std r2,_CCR(r1) ld r2,PACATOC(r13) addi r9,r1,STACK_FRAME_OVERHEAD ld r11,exception_marker@toc(r2)