s390: Improve this_cpu operations
Subsystems (3)
Heiko Carstens
maintainerPATCH v4 7/8 s390/percpu: Provide arch_this_cpu_write() implementation
In reply to Heiko CarstensProvide an s390 specific implementation of arch_this_cpu_write() instead of the generic variant. The generic variant uses a quite expensive raw_local_irq_save() / raw_local_irq_restore() pair.
Get rid of this by providing an own variant which makes use of the new percpu code section infrastructure.
With this the text size of the kernel image is reduced by ~1k (defconfig).
--- arch/s390/include/asm/percpu.h | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+)
arch/s390/include/asm/percpu.h +30 -0
@@ -279,6 +279,36 @@ do { \#define this_cpu_read_4 (pcp) arch_this_cpu_read(pcp, "l")#define this_cpu_read_8 (pcp) arch_this_cpu_read(pcp, "lg") +#define arch_this_cpu_write (pcp, val, op) \ +do { \ + unsigned long lc_pcpr, lc_pcpo; \ + typedef typeof(pcp) pcp_op_T__; \ + pcp_op_T__ *ptr__, val__ = (val); \ + \ + lc_pcpr = offsetof(struct lowcore, percpu_register); \ + lc_pcpo = offsetof(struct lowcore, percpu_offset); \ + ptr__ = PERCPU_PTR(&(pcp)); \ + asm_inline volatile( \ + MVIY_PERCPU("%[disppcpr]", "%[dispaltpcpr]", "%[ptr__]")\ + AG_ALT("%[disppcpo]", "%[dispaltpcpo]", "%[ptr__]") \ + op " %[val__],0(%[ptr__])\n" \ + MVIY_ALT("%[disppcpr]", "%[dispaltpcpr]") \ + : [ptr__] "+&a" (ptr__), "=m" (*ptr__), \ + "=m" (((struct lowcore *)0)->percpu_register) \ + : [val__] "d" (val__), \ + [disppcpr] "i" (lc_pcpr), \ + [disppcpo] "i" (lc_pcpo), \ + [dispaltpcpr] "i" (lc_pcpr + LOWCORE_ALT_ADDRESS), \ + [dispaltpcpo] "i" (lc_pcpo + LOWCORE_ALT_ADDRESS), \ + "m" (((struct lowcore *)0)->percpu_offset) \ + : "cc"); \ +} while (0) + +#define this_cpu_write_1 (pcp, val) arch_this_cpu_write(pcp, val, "stc") +#define this_cpu_write_2 (pcp, val) arch_this_cpu_write(pcp, val, "sth") +#define this_cpu_write_4 (pcp, val) arch_this_cpu_write(pcp, val, "st") +#define this_cpu_write_8 (pcp, val) arch_this_cpu_write(pcp, val, "stg") +#define arch_this_cpu_cmpxchg (pcp, oval, nval) \ ({ \ typedef typeof(pcp) pcp_op_T__; \
Signature · 1 line
2.51.0
Heiko Carstens
maintainerv4 cover s390: Improve this_cpu operations
v4:
- Drop alternatives approach and extract percpu base register number for mviy at compile time [David Laight]
- Fix logic for percpu code section detection, as well as interruption/exception/nmi path [Sashiko [5]]
[5] https://sashiko.dev/#/patchset/20260520092243.264847-1-hca%40linux.ibm.com
v3:
- Fix various typos [Juergen Christ]
- Add missing kprobe detection / handling [Sashiko [3]] [FWIW, this made me also aware of that the current general s390 kprobes code seems to be racy against concurrent removal of a kprobe while a probe hit on a different CPU. But that is a different story.]
- Fix various minor findings [Sashiko [3]]
- All of this might be dropped / exchanged in future in favor of the percpu page table approach proposed by Yang Shi [4].
[3] https://sashiko.dev/#/patchset/[email protected]
[4] https://lore.kernel.org/all/[email protected]/
v2:
- Add proper PERCPU_PTR cast to most patches to avoid tons of sparse warnings
- Add missing __packed attribute to insn structure [Sashiko [2]]
- Fix inverted if condition [Sashiko [2]]
- Add missing user_mode() check [Sashiko [2]]
- Move percpu_entry() call in front of irqentry_enter() call in all entry paths to avoid that potential this_cpu() operations overwrite the not-yet saved percpu code section indicator [Sashiko [2]]
[2] https://sashiko.dev/#/patchset/20260317195436.2276810-1-hca%40linux.ibm.com
v1:
This is a follow-up to Peter Zijlstra's in-kernel rseq RFC [1].
With the intended removal of PREEMPT_NONE this_cpu operations based on atomic instructions, guarded with preempt_disable()/preempt_enable() pairs, become more expensive: the preempt_disable() / preempt_enable() pairs are not optimized away anymore during compile time.
In particular the conditional call to preempt_schedule_notrace() after preempt_enable() adds additional code and register pressure.
To avoid this Peter suggested an in-kernel rseq approach. While this would certainly work, this series tries to come up with a solution which uses less instructions and doesn't require to repeat instruction sequences.
The idea is that this_cpu operations based on atomic instructions are guarded with mvyi instructions:
- The first mvyi instruction writes the register number, which contains the percpu address variable to lowcore. This also indicates that a percpu code section is executed.
- The first instruction following the mvyi instruction must be the ag instruction which adds the percpu offset to the percpu address register.
- Afterwards the atomic percpu operation follows.
- Then a second mvyi instruction writes a zero to lowcore, which indicates the end of the percpu code section.
- In case of an interrupt/exception/nmi the register number which was written to lowcore is copied to the exception frame (pt_regs), and a zero is written to lowcore.
- On return to the previous context it is checked if a percpu code section was executed (saved register number not zero), and if the process was migrated to a different cpu. If the percpu offset was already added to the percpu address register (instruction address does _not_ point to the ag instruction) the content of the percpu address register is adjusted so it points to percpu variable of the new cpu.
All of this seems to work, but of course it could still be broken since I missed some detail.
In total this series results in a kernel text size reduction of ~106kb. The number of preempt_schedule_notrace() call sites is reduced from 7089 to 1577.
Note: this comes without any huge performance analysis, however all microbenchmarks confirmed that the new code is at least as fast as the old code, like expected.
Heiko Carstens (8):
s390/percpu: Infrastructure for more efficient this_cpu operations s390/percpu: Add missing do { } while (0) constructs s390/percpu: Use new percpu code section for arch_this_cpu_add() s390/percpu: Use new percpu code section for arch_this_cpu_add_return() s390/percpu: Use new percpu code section for arch_this_cpu_[and|or]() s390/percpu: Provide arch_this_cpu_read() implementation s390/percpu: Provide arch_this_cpu_write() implementation s390/percpu: Remove one and two byte this_cpu operation implementation
arch/s390/include/asm/entry-percpu.h | 78 ++++++++ arch/s390/include/asm/lowcore.h | 3 +- arch/s390/include/asm/percpu.h | 257 +++++++++++++++++++++------ arch/s390/include/asm/ptrace.h | 2 + arch/s390/kernel/irq.c | 24 ++- arch/s390/kernel/nmi.c | 5 + arch/s390/kernel/traps.c | 5 + 7 files changed, 315 insertions(+), 59 deletions(-) create mode 100644 arch/s390/include/asm/entry-percpu.h
Signature · 1 line
2.51.0
Heiko Carstens
maintainerPATCH v4 4/8 s390/percpu: Use new percpu code section for arch_this_cpu_add_return()
Convert arch_this_cpu_add_return() to make use of the new percpu code section infrastructure.
With this the text size of the kernel image is reduced by ~4k (defconfig). Also 66 generated preempt_schedule_notrace() function calls within the kernel image (modules not counted) are removed.
--- arch/s390/include/asm/percpu.h | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-)
arch/s390/include/asm/percpu.h +19 -7
@@ -185,17 +185,29 @@ do { \#define arch_this_cpu_add_return (pcp, val, op) \ ({ \ + unsigned long lc_pcpr, lc_pcpo; \ typedef typeof(pcp) pcp_op_T__; \ pcp_op_T__ val__ = (val); \ pcp_op_T__ old__, *ptr__; \ - preempt_disable_notrace(); \ - ptr__ = raw_cpu_ptr(&(pcp)); \ - asm volatile( \ - op " %[old__],%[val__],%[ptr__]" \ - : [old__] "=d" (old__), [ptr__] "+Q" (*ptr__) \ - : [val__] "d" (val__) \ + \ + lc_pcpr = offsetof(struct lowcore, percpu_register); \ + lc_pcpo = offsetof(struct lowcore, percpu_offset); \ + ptr__ = PERCPU_PTR(&(pcp)); \ + asm_inline volatile( \ + MVIY_PERCPU("%[disppcpr]", "%[dispaltpcpr]", "%[ptr__]")\ + AG_ALT("%[disppcpo]", "%[dispaltpcpo]", "%[ptr__]") \ + op " %[old__],%[val__],0(%[ptr__])\n" \ + MVIY_ALT("%[disppcpr]", "%[dispaltpcpr]") \ + : [old__] "=&d" (old__), \ + [ptr__] "+&a" (ptr__), "+m" (*ptr__), \ + "=m" (((struct lowcore *)0)->percpu_register) \ + : [val__] "d" (val__), \ + [disppcpr] "i" (lc_pcpr), \ + [disppcpo] "i" (lc_pcpo), \ + [dispaltpcpr] "i" (lc_pcpr + LOWCORE_ALT_ADDRESS), \ + [dispaltpcpo] "i" (lc_pcpo + LOWCORE_ALT_ADDRESS), \ + "m" (((struct lowcore *)0)->percpu_offset) \ : "cc"); \ - preempt_enable_notrace(); \ old__ + val__; \ })
Signature · 1 line
2.51.0
Heiko Carstens
maintainerPATCH v4 5/8 s390/percpu: Use new percpu code section for arch_this_cpu_[and|or]()
In reply to Heiko CarstensConvert arch_this_cpu_[and|or]() to make use of the new percpu code section infrastructure.
There is no user of this_cpu_and() and only one user of this_cpu_or() within the kernel. Therefore this conversion has hardly any effect, and also removes only preempt_schedule_notrace() function call.
--- arch/s390/include/asm/percpu.h | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-)
arch/s390/include/asm/percpu.h +19 -7
@@ -216,17 +216,29 @@ do { \#define arch_this_cpu_to_op (pcp, val, op) \ do { \ + unsigned long lc_pcpr, lc_pcpo; \ typedef typeof(pcp) pcp_op_T__; \ pcp_op_T__ val__ = (val); \ pcp_op_T__ old__, *ptr__; \ - preempt_disable_notrace(); \ - ptr__ = raw_cpu_ptr(&(pcp)); \ - asm volatile( \ - op " %[old__],%[val__],%[ptr__]" \ - : [old__] "=d" (old__), [ptr__] "+Q" (*ptr__) \ - : [val__] "d" (val__) \ + \ + lc_pcpr = offsetof(struct lowcore, percpu_register); \ + lc_pcpo = offsetof(struct lowcore, percpu_offset); \ + ptr__ = PERCPU_PTR(&(pcp)); \ + asm_inline volatile( \ + MVIY_PERCPU("%[disppcpr]", "%[dispaltpcpr]", "%[ptr__]")\ + AG_ALT("%[disppcpo]", "%[dispaltpcpo]", "%[ptr__]") \ + op " %[old__],%[val__],0(%[ptr__])\n" \ + MVIY_ALT("%[disppcpr]", "%[dispaltpcpr]") \ + : [old__] "=&d" (old__), \ + [ptr__] "+&a" (ptr__), "+m" (*ptr__), \ + "=m" (((struct lowcore *)0)->percpu_register) \ + : [val__] "d" (val__), \ + [disppcpr] "i" (lc_pcpr), \ + [disppcpo] "i" (lc_pcpo), \ + [dispaltpcpr] "i" (lc_pcpr + LOWCORE_ALT_ADDRESS), \ + [dispaltpcpo] "i" (lc_pcpo + LOWCORE_ALT_ADDRESS), \ + "m" (((struct lowcore *)0)->percpu_offset) \ : "cc"); \ - preempt_enable_notrace(); \ } while (0)#define this_cpu_and_4 (pcp, val) arch_this_cpu_to_op(pcp, val, "lan")
Signature · 1 line
2.51.0
Heiko Carstens
maintainerPATCH v4 3/8 s390/percpu: Use new percpu code section for arch_this_cpu_add()
In reply to Heiko CarstensConvert arch_this_cpu_add() to make use of the new percpu code section infrastructure.
With this the text size of the kernel image is reduced by ~76kb (defconfig). Also more than 5300 generated preempt_schedule_notrace() function calls within the kernel image (modules not counted) are removed.
With:
DEFINE_PER_CPU(long, foo);
void bar(long a) { this_cpu_add(foo, a); }
Old arch_this_cpu_add() looks like this:
00000000000000c0 <bar>:
c0: c0 04 00 00 00 00 jgnop c0 <bar>
c6: eb 01 03 a8 00 6a asi 936,1
cc: c4 18 00 00 00 00 lgrl %r1,cc <bar+0xc>
ce: R_390_GOTENT foo+0x2
d2: e3 10 03 b8 00 08 ag %r1,952
d8: eb 22 10 00 00 e8 laag %r2,%r2,0(%r1)
de: eb ff 03 a8 00 6e alsi 936,-1
e4: a7 a4 00 05 jhe ee <bar+0x2e>
e8: c0 f4 00 00 00 00 jg e8 <bar+0x28>
ea: R_390_PC32DBL __s390_indirect_jump_r14+0x2
ee: c0 f4 00 00 00 00 jg ee <bar+0x2e>
f0: R_390_PLT32DBL preempt_schedule_notrace+0x2New arch_this_cpu_add() looks like this:
00000000000000c0 <bar>:
c0: c0 04 00 00 00 00 jgnop c0 <bar>
c6: c4 38 00 00 00 00 lgrl %r3,c6 <bar+0x6>
c8: R_390_GOTENT foo+0x2
cc: b9 04 00 43 lgr %r4,%r3
d0: eb 00 43 c0 00 52 mviy 960(%r0),4
d6: e3 40 03 b8 00 08 ag %r4,952
dc: eb 52 40 00 00 e8 laag %r5,%r2,0(%r4)
e2: eb 00 03 c0 00 52 mviy 960,0
e8: c0 f4 00 00 00 00 jg e8 <bar+0x28>
ea: R_390_PC32DBL __s390_indirect_jump_r14+0x2Note that the conditional function call is removed.
--- arch/s390/include/asm/percpu.h | 65 ++++++++++++++++++++++------------ 1 file changed, 43 insertions(+), 22 deletions(-)
arch/s390/include/asm/percpu.h +43 -22
@@ -135,28 +135,49 @@#else /* MARCH_HAS_Z196_FEATURES */ -#define arch_this_cpu_add (pcp, val, op1, op2, szcast) \ -do { \ - typedef typeof(pcp) pcp_op_T__; \ - pcp_op_T__ val__ = (val); \ - pcp_op_T__ old__, *ptr__; \ - preempt_disable_notrace(); \ - ptr__ = raw_cpu_ptr(&(pcp)); \ - if (__builtin_constant_p(val__) && \ - ((szcast)val__ > -129) && ((szcast)val__ < 128)) { \ - asm volatile( \ - op2 " %[ptr__],%[val__]" \ - : [ptr__] "+Q" (*ptr__) \ - : [val__] "i" ((szcast)val__) \ - : "cc"); \ - } else { \ - asm volatile( \ - op1 " %[old__],%[val__],%[ptr__]" \ - : [old__] "=d" (old__), [ptr__] "+Q" (*ptr__) \ - : [val__] "d" (val__) \ - : "cc"); \ - } \ - preempt_enable_notrace(); \ +#define arch_this_cpu_add (pcp, val, op1, op2, szcast) \ +do { \ + unsigned long lc_pcpr, lc_pcpo; \ + typedef typeof(pcp) pcp_op_T__; \ + pcp_op_T__ val__ = (val); \ + pcp_op_T__ old__, *ptr__; \ + \ + lc_pcpr = offsetof(struct lowcore, percpu_register); \ + lc_pcpo = offsetof(struct lowcore, percpu_offset); \ + ptr__ = PERCPU_PTR(&(pcp)); \ + if (__builtin_constant_p(val__) && \ + ((szcast)val__ > -129) && ((szcast)val__ < 128)) { \ + asm volatile( \ + MVIY_PERCPU("%[disppcpr]", "%[dispaltpcpr]", "%[ptr__]")\ + AG_ALT("%[disppcpo]", "%[dispaltpcpo]", "%[ptr__]") \ + op2 " 0(%[ptr__]),%[val__]\n" \ + MVIY_ALT("%[disppcpr]", "%[dispaltpcpr]") \ + : [ptr__] "+&a" (ptr__), "+m" (*ptr__), \ + "=m" (((struct lowcore *)0)->percpu_register) \ + : [val__] "i" ((szcast)val__), \ + [disppcpr] "i" (lc_pcpr), \ + [disppcpo] "i" (lc_pcpo), \ + [dispaltpcpr] "i" (lc_pcpr + LOWCORE_ALT_ADDRESS), \ + [dispaltpcpo] "i" (lc_pcpo + LOWCORE_ALT_ADDRESS), \ + "m" (((struct lowcore *)0)->percpu_offset) \ + : "cc"); \ + } else { \ + asm volatile( \ + MVIY_PERCPU("%[disppcpr]", "%[dispaltpcpr]", "%[ptr__]")\ + AG_ALT("%[disppcpo]", "%[dispaltpcpo]", "%[ptr__]") \ + op1 " %[old__],%[val__],0(%[ptr__])\n" \ + MVIY_ALT("%[disppcpr]", "%[dispaltpcpr]") \ + : [old__] "=&d" (old__), \ + [ptr__] "+&a" (ptr__), "+m" (*ptr__), \ + "=m" (((struct lowcore *)0)->percpu_register) \ + : [val__] "d" (val__), \ + [disppcpr] "i" (lc_pcpr), \ + [disppcpo] "i" (lc_pcpo), \ + [dispaltpcpr] "i" (lc_pcpr + LOWCORE_ALT_ADDRESS), \ + [dispaltpcpo] "i" (lc_pcpo + LOWCORE_ALT_ADDRESS), \ + "m" (((struct lowcore *)0)->percpu_offset) \ + : "cc"); \ + } \ } while (0)#define this_cpu_add_4 (pcp, val) arch_this_cpu_add(pcp, val, "laa", "asi", int)
Signature · 1 line
2.51.0
Heiko Carstens
maintainerPATCH v4 1/8 s390/percpu: Infrastructure for more efficient this_cpu operations
In reply to Heiko CarstensWith the intended removal of PREEMPT_NONE this_cpu operations based on atomic instructions, guarded with preempt_disable()/preempt_enable() pairs become more expensive: the preempt_disable() / preempt_enable() pairs are not optimized away anymore during compile time.
In particular the conditional call to preempt_schedule_notrace() after preempt_enable() adds additional code and register pressure.
E.g. this simple C code sequence
DEFINE_PER_CPU(long, foo);
long bar(long a) { return this_cpu_add_return(foo, a); }
generates this code:
11a976: eb af f0 68 00 24 stmg %r10,%r15,104(%r15) 11a97c: b9 04 00 ef lgr %r14,%r15 11a980: b9 04 00 b2 lgr %r11,%r2 11a984: e3 f0 ff c8 ff 71 lay %r15,-56(%r15) 11a98a: e3 e0 f0 98 00 24 stg %r14,152(%r15) 11a990: eb 01 03 a8 00 6a asi 936,1 <- __preempt_count_add(1) 11a996: c0 10 00 d2 ac b5 larl %r1,1b70300 <- address of percpu var 11a9a0: e3 10 23 b8 00 08 ag %r1,952 <- add percpu offset 11a9a6: eb ab 10 00 00 e8 laag %r10,%r11,0(%r1) <- atomic op 11a9ac: eb ff 03 a8 00 6e alsi 936,-1 <- __preempt_count_dec_and_test() 11a9b2: a7 54 00 05 jnhe 11a9bc <bar+0x4c> 11a9b6: c0 e5 00 76 d1 bd brasl %r14,ff4d30 <preempt_schedule_notrace> 11a9bc: b9 e8 b0 2a agrk %r2,%r10,%r11 11a9c0: eb af f0 a0 00 04 lmg %r10,%r15,160(%r15) 11a9c6 07 fe br %r14
Even though the above example is more or less the worst case, since the branch to preempt_schedule_notrace() requires a stackframe, which otherwise wouldn't be necessary, there is also the conditional jnhe branch instruction.
Get rid of the conditional branch with the following code sequence:
11a8e6: c0 30 00 d0 c5 0d larl %r3,1b33300 11a8ec: b9 04 00 43 lgr %r4,%r3 11a8f0: eb 00 43 c0 00 52 mviy 960,4 11a8f6: e3 40 03 b8 00 08 ag %r4,952 11a8fc: eb 52 40 00 00 e8 laag %r5,%r2,0(%r4) 11a902: eb 00 03 c0 00 52 mviy 960,0 11a908: b9 08 00 25 agr %r2,%r5 11a90c 07 fe br %r14
The general idea is that this_cpu operations based on atomic instructions are guarded with mviy instructions:
- The first mviy instruction writes the register number, which contains the percpu address variable to lowcore. This also indicates that a percpu code section is executed.
- The first instruction following the mviy instruction must be the ag instruction which adds the percpu offset to the percpu address register.
- Afterwards the atomic percpu operation follows.
- Then a second mviy instruction writes a zero to lowcore, which indicates the end of the percpu code section.
- In case of an interrupt/exception/nmi the register number which was written to lowcore is copied to the exception frame (pt_regs), and a zero is written to lowcore.
- On return to the previous context it is checked if a percpu code section was executed (saved register number not zero), and if the process was migrated to a different cpu. If the percpu offset was already added to the percpu address register (instruction address does _not_ point to the ag instruction) the content of the percpu address register is adjusted so it points to percpu variable of the new cpu.
--- arch/s390/include/asm/entry-percpu.h | 78 ++++++++++++++++++++++++++++ arch/s390/include/asm/lowcore.h | 3 +- arch/s390/include/asm/percpu.h | 62 ++++++++++++++++++++++ arch/s390/include/asm/ptrace.h | 2 + arch/s390/kernel/irq.c | 24 ++++++--- arch/s390/kernel/nmi.c | 5 ++ arch/s390/kernel/traps.c | 5 ++ 7 files changed, 171 insertions(+), 8 deletions(-) create mode 100644 arch/s390/include/asm/entry-percpu.h
arch/s390/include/asm/entry-percpu.h new file +78 -0
@@ -0,0 +1,78 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef ARCH_S390_ENTRY_PERCPU_H +#define ARCH_S390_ENTRY_PERCPU_H + +#include <linux/kprobes.h> +#include <linux/percpu.h> +#include <asm/lowcore.h> +#include <asm/ptrace.h> +#include <asm/asm-offsets.h> + +static __always_inline void percpu_entry(struct pt_regs *regs) +{ +struct lowcore *lc =get_lowcore (); + +if (user_mode (regs)) +return ; + regs->cpu = lc->cpu_nr; + regs->percpu_register = lc->percpu_register; + lc->percpu_register =0 ; +} + +static __always_inline bool percpu_code_check(struct pt_regs *regs) +{ +unsigned int insn, disp; +struct kprobe *p; + +if (likely (user_mode (regs) || !regs->percpu_register)) +return false; +/* +* Within a percpu code section - check if the percpu base register +* needs to be updated. This is the case if the PSW does not point to +* the ADD instruction within the section. +* - AG %rx,percpu_offset_in_lowcore(%r0,%r0) +* which adds the percpu offset to the percpu base register. +*/ +lockdep_assert_preemption_disabled (); +again: + insn =READ_ONCE (*(u16 *)psw_bits (regs->psw).ia); +if (unlikely (insn ==BREAKPOINT_INSTRUCTION )) { + p =get_kprobe ((void *)psw_bits (regs->psw).ia); +/* +* If the kprobe is concurrently removed on a different CPU +* it might not be found anymore. However text must have +* been restored - try again. +*/ +if (!p) + goto again; + insn = p->opcode; + } +if ((insn &0xff0f ) !=0xe300 ) +return true; + disp = offsetof(struct lowcore , percpu_offset); +if (machine_has_relocated_lowcore ()) + disp +=LOWCORE_ALT_ADDRESS ; + insn = (disp &0xff000 ) >>4 | (disp &0x00fff ) <<16 |0x8 ; +if (*(u32 *)(psw_bits (regs->psw).ia +2 ) != insn) +return true; +return false; +} + +static __always_inline void percpu_exit(struct pt_regs *regs, bool needs_fixup) +{ +struct lowcore *lc =get_lowcore (); +unsigned char reg; + + reg = regs->percpu_register; + lc->percpu_register = reg; +if (likely (!needs_fixup)) +return ; +/* Check if process has been migrated to a different CPU. */ +if (regs->cpu == lc->cpu_nr) +return ; +/* Fixup percpu base register */ + regs->gprs[reg] -= __per_cpu_offset[regs->cpu]; + regs->gprs[reg] += lc->percpu_offset; +} + +#endif
arch/s390/include/asm/lowcore.h +2 -1
@@ -165,7 +165,8 @@ struct lowcore {__u32 spinlock_index;/* 0x03b0 */ __u8 pad_0x03b4[0x03b8 -0x03b4 ];/* 0x03b4 */ __u64 percpu_offset;/* 0x03b8 */ -__u8 pad_0x03c0[0x0400 -0x03c0 ];/* 0x03c0 */ +__u8 percpu_register;/* 0x03c0 */ +__u8 pad_0x03c1[0x0400 -0x03c1 ];/* 0x03c1 */ __u32 return_lpswe;/* 0x0400 */ __u32 return_mcck_lpswe;/* 0x0404 */
arch/s390/include/asm/percpu.h +62 -0
68 lines not on this page · raw message
arch/s390/include/asm/ptrace.h +2 -0
@@ -134,6 +134,8 @@ struct pt_regs { };unsigned long flags;unsigned long last_break; +unsigned int cpu; +unsigned char percpu_register; };/*
arch/s390/kernel/irq.c +17 -7
@@ -33,6 +33,7 @@#include <asm/softirq_stack.h> #include <asm/vtime.h> #include <asm/asm.h> +#include <asm/entry-percpu.h> #include "entry.h" DEFINE_PER_CPU_SHARED_ALIGNED(struct irq_stat, irq_stat);
@@ -142,10 +143,13 @@ static int irq_pending(struct pt_regs *regs) void noinstr do_io_irq(struct pt_regs *regs) { -irqentry_state_t state =irqentry_enter (regs); -struct pt_regs *old_regs =set_irq_regs (regs); -bool from_idle; +bool from_idle, percpu_needs_fixup; +struct pt_regs *old_regs; +irqentry_state_t state; +percpu_entry (regs); + state =irqentry_enter (regs); + old_regs =set_irq_regs (regs); from_idle =test_and_clear_cpu_flag (CIF_ENABLED_WAIT );if (from_idle)update_timer_idle ();
@@ -170,21 +174,25 @@ void noinstr do_io_irq(struct pt_regs *regs)do_irq_async (regs,IO_INTERRUPT ); }while (machine_is_lpar () &&irq_pending (regs)); + percpu_needs_fixup =percpu_code_check (regs);irq_exit_rcu (); -set_irq_regs (old_regs);irqentry_exit (regs, state);if (from_idle) regs->psw.mask &= ~(PSW_MASK_EXT |PSW_MASK_IO |PSW_MASK_WAIT ); +percpu_exit (regs, percpu_needs_fixup); } void noinstr do_ext_irq(struct pt_regs *regs) { -irqentry_state_t state =irqentry_enter (regs); -struct pt_regs *old_regs =set_irq_regs (regs); -bool from_idle; +bool from_idle, percpu_needs_fixup; +struct pt_regs *old_regs; +irqentry_state_t state; +percpu_entry (regs); + state =irqentry_enter (regs); + old_regs =set_irq_regs (regs); from_idle =test_and_clear_cpu_flag (CIF_ENABLED_WAIT );if (from_idle)update_timer_idle ();
@@ -206,12 +214,14 @@ void noinstr do_ext_irq(struct pt_regs *regs)do_irq_async (regs,EXT_INTERRUPT ); + percpu_needs_fixup =percpu_code_check (regs);irq_exit_rcu ();set_irq_regs (old_regs);irqentry_exit (regs, state);if (from_idle) regs->psw.mask &= ~(PSW_MASK_EXT |PSW_MASK_IO |PSW_MASK_WAIT ); +percpu_exit (regs, percpu_needs_fixup); }static void show_msi_interrupt (struct seq_file *p,int irq)
arch/s390/kernel/nmi.c +5 -0
@@ -22,6 +22,7 @@#include <linux/module.h> #include <linux/sched/signal.h> #include <linux/kvm_host.h> +#include <asm/entry-percpu.h> #include <asm/lowcore.h> #include <asm/ctlreg.h> #include <asm/fpu.h>
@@ -363,6 +364,7 @@ NOKPROBE_SYMBOL(s390_backup_mcck_info);*/ void notrace s390_do_machine_check(struct pt_regs *regs) { +bool percpu_needs_fixup;static int ipd_count;static DEFINE_SPINLOCK (ipd_lock);static unsigned long long last_ipd;
@@ -374,6 +376,7 @@ void notrace s390_do_machine_check(struct pt_regs *regs)unsigned long mcck_dam_code;int mcck_pending =0 ; +percpu_entry (regs); irq_state =irqentry_nmi_enter (regs);if (user_mode (regs))
@@ -495,7 +498,9 @@ void notrace s390_do_machine_check(struct pt_regs *regs)if (mcck_pending)schedule_mcck_handler (); + percpu_needs_fixup =percpu_code_check (regs);irqentry_nmi_exit (regs, irq_state); +percpu_exit (regs, percpu_needs_fixup); }NOKPROBE_SYMBOL (s390_do_machine_check);
arch/s390/kernel/traps.c +5 -0
@@ -24,6 +24,7 @@#include <linux/entry-common.h> #include <linux/kmsan.h> #include <linux/bug.h> +#include <asm/entry-percpu.h> #include <asm/asm-extable.h> #include <asm/irqflags.h> #include <asm/ptrace.h>
@@ -329,6 +330,7 @@ static void (*pgm_check_table[128])(struct pt_regs *regs); void noinstr __do_pgm_check(struct pt_regs *regs) {struct lowcore *lc =get_lowcore (); +bool percpu_needs_fixup;irqentry_state_t state;unsigned int trapnr;union teid teid;
@@ -349,6 +351,7 @@ void noinstr __do_pgm_check(struct pt_regs *regs) current->thread.gmap_int_code = regs->int_code &0xffff ;return ; } +percpu_entry (regs); state =irqentry_enter (regs);if (user_mode (regs)) {update_timer_sys ();
@@ -385,7 +388,9 @@ void noinstr __do_pgm_check(struct pt_regs *regs) pgm_check_table[trapnr](regs); out:local_irq_disable (); + percpu_needs_fixup =percpu_code_check (regs);irqentry_exit (regs, state); +percpu_exit (regs, percpu_needs_fixup); }/*
Signature · 1 line
2.51.0
Heiko Carstens
maintainerPATCH v4 2/8 s390/percpu: Add missing do { } while (0) constructs
In reply to Heiko CarstensAdd missing do { } while (0) constructs in order to avoid potential build failures.
--- arch/s390/include/asm/percpu.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
arch/s390/include/asm/percpu.h +4 -4
@@ -136,7 +136,7 @@#else /* MARCH_HAS_Z196_FEATURES */ #define arch_this_cpu_add (pcp, val, op1, op2, szcast) \ -{ \ +do { \ typedef typeof(pcp) pcp_op_T__; \ pcp_op_T__ val__ = (val); \ pcp_op_T__ old__, *ptr__; \
@@ -157,7 +157,7 @@ : "cc"); \ } \preempt_enable_notrace (); \ -} +} while (0)#define this_cpu_add_4 (pcp, val) arch_this_cpu_add(pcp, val, "laa", "asi", int)#define this_cpu_add_8 (pcp, val) arch_this_cpu_add(pcp, val, "laag", "agsi", long)
@@ -182,7 +182,7 @@#define this_cpu_add_return_8 (pcp, val) arch_this_cpu_add_return(pcp, val, "laag")#define arch_this_cpu_to_op (pcp, val, op) \ -{ \ +do { \ typedef typeof(pcp) pcp_op_T__; \ pcp_op_T__ val__ = (val); \ pcp_op_T__ old__, *ptr__; \
@@ -194,7 +194,7 @@ : [val__] "d" (val__) \ : "cc"); \preempt_enable_notrace (); \ -} +} while (0)#define this_cpu_and_4 (pcp, val) arch_this_cpu_to_op(pcp, val, "lan")#define this_cpu_and_8 (pcp, val) arch_this_cpu_to_op(pcp, val, "lang")
Signature · 1 line
2.51.0
Heiko Carstens
maintainerPATCH v4 6/8 s390/percpu: Provide arch_this_cpu_read() implementation
In reply to Heiko CarstensProvide an s390 specific implementation of arch_this_cpu_read() instead of the generic variant. The generic variant uses preempt_disable() / preempt_enable() pair and READ_ONCE().
Get rid of the preempt_disable() / preempt_enable() pairs by providing an own variant which makes use of the new percpu code section infrastructure.
With this the text size of the kernel image is reduced by ~1k (defconfig). Also 87 generated preempt_schedule_notrace() function calls within the kernel image (modules not counted) are removed.
--- arch/s390/include/asm/percpu.h | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
arch/s390/include/asm/percpu.h +31 -0
@@ -248,6 +248,37 @@ do { \#endif /* MARCH_HAS_Z196_FEATURES */ +#define arch_this_cpu_read (pcp, op) \ +({ \ + unsigned long lc_pcpr, lc_pcpo, res__; \ + typedef typeof(pcp) pcp_op_T__; \ + pcp_op_T__ *ptr__; \ + \ + lc_pcpr = offsetof(struct lowcore, percpu_register); \ + lc_pcpo = offsetof(struct lowcore, percpu_offset); \ + ptr__ = PERCPU_PTR(&(pcp)); \ + asm_inline volatile( \ + MVIY_PERCPU("%[disppcpr]", "%[dispaltpcpr]", "%[ptr__]")\ + AG_ALT("%[disppcpo]", "%[dispaltpcpo]", "%[ptr__]") \ + op " %[res__],0(%[ptr__])\n" \ + MVIY_ALT("%[disppcpr]", "%[dispaltpcpr]") \ + : [res__] "=&d" (res__), [ptr__] "+&a" (ptr__), \ + "=m" (((struct lowcore *)0)->percpu_register) \ + : [disppcpr] "i" (lc_pcpr), \ + [disppcpo] "i" (lc_pcpo), \ + [dispaltpcpr] "i" (lc_pcpr + LOWCORE_ALT_ADDRESS), \ + [dispaltpcpo] "i" (lc_pcpo + LOWCORE_ALT_ADDRESS), \ + "m" (*ptr__), \ + "m" (((struct lowcore *)0)->percpu_offset) \ + : "cc"); \ + (pcp_op_T__)res__; \ +}) + +#define this_cpu_read_1 (pcp) arch_this_cpu_read(pcp, "ic") +#define this_cpu_read_2 (pcp) arch_this_cpu_read(pcp, "lh") +#define this_cpu_read_4 (pcp) arch_this_cpu_read(pcp, "l") +#define this_cpu_read_8 (pcp) arch_this_cpu_read(pcp, "lg") +#define arch_this_cpu_cmpxchg (pcp, oval, nval) \ ({ \ typedef typeof(pcp) pcp_op_T__; \
Signature · 1 line
2.51.0
Heiko Carstens
maintainerPATCH v4 8/8 s390/percpu: Remove one and two byte this_cpu operation implementation
In reply to Heiko CarstensThere are no one and two byte this_cpu operations within the kernel (defconfig). However even if there would be, the s390 implementation, which uses a cmpxchg loop, generates a very large code sequence due to the lack of native one and two byte cmpxchg instructions.
Remove the s390 implementation and use the generic implementation.
--- arch/s390/include/asm/percpu.h | 13 ------------- 1 file changed, 13 deletions(-)
arch/s390/include/asm/percpu.h +0 -13
@@ -51,15 +51,6 @@ new__; \ }) -#define this_cpu_add_1 (pcp, val) arch_this_cpu_to_op_simple(pcp, val, +) -#define this_cpu_add_2 (pcp, val) arch_this_cpu_to_op_simple(pcp, val, +) -#define this_cpu_add_return_1 (pcp, val) arch_this_cpu_to_op_simple(pcp, val, +) -#define this_cpu_add_return_2 (pcp, val) arch_this_cpu_to_op_simple(pcp, val, +) -#define this_cpu_and_1 (pcp, val) arch_this_cpu_to_op_simple(pcp, val, &) -#define this_cpu_and_2 (pcp, val) arch_this_cpu_to_op_simple(pcp, val, &) -#define this_cpu_or_1 (pcp, val) arch_this_cpu_to_op_simple(pcp, val, |) -#define this_cpu_or_2 (pcp, val) arch_this_cpu_to_op_simple(pcp, val, |) -/* * Macros to be used for percpu code section based on atomic instructions. *
@@ -321,8 +312,6 @@ do { \ ret__; \ }) -#define this_cpu_cmpxchg_1 (pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) -#define this_cpu_cmpxchg_2 (pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval)#define this_cpu_cmpxchg_4 (pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval)#define this_cpu_cmpxchg_8 (pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval)
@@ -353,8 +342,6 @@ do { \ ret__; \ }) -#define this_cpu_xchg_1 (pcp, nval) arch_this_cpu_xchg(pcp, nval) -#define this_cpu_xchg_2 (pcp, nval) arch_this_cpu_xchg(pcp, nval)#define this_cpu_xchg_4 (pcp, nval) arch_this_cpu_xchg(pcp, nval)#define this_cpu_xchg_8 (pcp, nval) arch_this_cpu_xchg(pcp, nval)
Signature · 1 line
2.51.0
Quoted 4 lines · On Fri, 22 May 2026 16:12:49 +0200 Heiko Carstens <[email protected]> wrote:
v4:
- Drop alternatives approach and extract percpu base register number for mviy at compile time [David Laight]
Definitely looks better.
I'm sure I managed to understand it once :-)
-- David
Quoted 115 lines
- Fix logic for percpu code section detection, as well as interruption/exception/nmi path [Sashiko [5]]
[5] https://sashiko.dev/#/patchset/20260520092243.264847-1-hca%40linux.ibm.com
v3:
- Fix various typos [Juergen Christ]
- Add missing kprobe detection / handling [Sashiko [3]] [FWIW, this made me also aware of that the current general s390 kprobes code seems to be racy against concurrent removal of a kprobe while a probe hit on a different CPU. But that is a different story.]
- Fix various minor findings [Sashiko [3]]
- All of this might be dropped / exchanged in future in favor of the percpu page table approach proposed by Yang Shi [4].
[3] https://sashiko.dev/#/patchset/[email protected]
[4] https://lore.kernel.org/all/[email protected]/v2:
- Add proper
PERCPU_PTRcast to most patches to avoid tons of sparse warnings- Add missing
__packedattribute to insn structure [Sashiko [2]]- Fix inverted if condition [Sashiko [2]]
- Add missing
user_mode()check [Sashiko [2]]- Move
percpu_entry()call in front ofirqentry_enter()call in all entry paths to avoid that potentialthis_cpu()operations overwrite the not-yet saved percpu code section indicator [Sashiko [2]][2] https://sashiko.dev/#/patchset/20260317195436.2276810-1-hca%40linux.ibm.com
v1:
This is a follow-up to Peter Zijlstra's in-kernel rseq RFC [1].
With the intended removal of
PREEMPT_NONEthis_cpuoperations based on atomic instructions, guarded with preempt_disable()/preempt_enable() pairs, become more expensive: thepreempt_disable()/preempt_enable()pairs are not optimized away anymore during compile time.In particular the conditional call to
preempt_schedule_notrace()afterpreempt_enable()adds additional code and register pressure.To avoid this Peter suggested an in-kernel rseq approach. While this would certainly work, this series tries to come up with a solution which uses less instructions and doesn't require to repeat instruction sequences.
The idea is that
this_cpuoperations based on atomic instructions are guarded with mvyi instructions:- The first mvyi instruction writes the register number, which contains the percpu address variable to lowcore. This also indicates that a percpu code section is executed.
- The first instruction following the mvyi instruction must be the ag instruction which adds the percpu offset to the percpu address register.
- Afterwards the atomic percpu operation follows.
- Then a second mvyi instruction writes a zero to lowcore, which indicates the end of the percpu code section.
- In case of an interrupt/exception/nmi the register number which was written to lowcore is copied to the exception frame (
pt_regs), and a zero is written to lowcore.- On return to the previous context it is checked if a percpu code section was executed (saved register number not zero), and if the process was migrated to a different cpu. If the percpu offset was already added to the percpu address register (instruction address does _not_ point to the ag instruction) the content of the percpu address register is adjusted so it points to percpu variable of the new cpu.
All of this seems to work, but of course it could still be broken since I missed some detail.
In total this series results in a kernel text size reduction of ~106kb. The number of
preempt_schedule_notrace()call sites is reduced from 7089 to 1577.Note: this comes without any huge performance analysis, however all microbenchmarks confirmed that the new code is at least as fast as the old code, like expected.
Heiko Carstens (8):
s390/percpu: Infrastructure for more efficientthis_cpuoperations s390/percpu: Add missing do { } while (0) constructs s390/percpu: Use new percpu code section forarch_this_cpu_add()s390/percpu: Use new percpu code section forarch_this_cpu_add_return()s390/percpu: Use new percpu code section for arch_this_cpu_[and|or]() s390/percpu: Providearch_this_cpu_read()implementation s390/percpu: Providearch_this_cpu_write()implementation s390/percpu: Remove one and two bytethis_cpuoperation implementationarch/s390/include/asm/entry-percpu.h | 78 ++++++++ arch/s390/include/asm/lowcore.h | 3 +- arch/s390/include/asm/percpu.h | 257 +++++++++++++++++++++------ arch/s390/include/asm/ptrace.h | 2 + arch/s390/kernel/irq.c | 24 ++- arch/s390/kernel/nmi.c | 5 + arch/s390/kernel/traps.c | 5 + 7 files changed, 315 insertions(+), 59 deletions(-) create mode 100644 arch/s390/include/asm/entry-percpu.h