ARM: enable IRQ stacks and vmap'ed stacks for UP
Subsystems (6)
v2 cover ARM: enable IRQ stacks and vmap'ed stacks for UP
Enable the use of the TLS register to hold the 'current' pointer for all configurations that can support it, including non-SMP ones that target v6k or later CPUs, and multi-platform SMP ones that also support v6 based UP systems.
The remaining configurations are all strictly UP, which means we can switch to a global variable to hold the current pointer. By doing this, we can enable THREAD_INFO_IN_TASK, which moves thread info off the stack, protecting it from overflows. It also permits us to enable IRQ stacks and vmap'ed stacks for UP configurations as well.
Supporting v6 cores without SMP extensions in SMP configurations (e.g., omap2plus_defconfig or imx_v6_v7_defconfig) makes this a bit tricky, and this is a feature we may consider dropping entirely in the future. But for the time being, we can support this mode as well.
The accesses to the global variable holding 'current' are constructed in a way that ensures that no literal pool accesses (and associated D-cache misses) are needed unless the access is from a module and module PLTs are enabled. This means that accessing 'current' is just as costly as before, as it used to require some arithmetic involving the stack pointer and a load from the thread_info::task field.
However, accessing thread_info itself now also involves a load, although it should be noted that all thread_info and current accesses now go via the same variable, which is therefore expected to be hot in the caches at all times.
Changes since RFC/v1:
- add five preparatory patches that move RiscPC, IOP32x and Footbridge to GENERIC_IRQ_MULTI_HANDLER so that even these ancient platforms can benefit from the IRQ stacks changes for UP that this series proposes (contributed by Arnd)
- fix various issues related to SMP+v6 corner cases that were caught by kernelci testing;
- add acks from Nico and Linus (thanks!)
Ard Biesheuvel (7):
ARM: entry: preserve thread_info pointer in switch_to
ARM: module: implement support for PC-relative group relocations
ARM: assembler: add optimized ldr/str macros to load variables from
memory
ARM: percpu: add SMP_ON_UP support
ARM: use TLS register for 'current' on !SMP as well
ARM: smp: defer TPIDRURO update for SMP v6 configurations too
ARM: implement THREAD_INFO_IN_TASK for uniprocessor systems
Arnd Bergmann (5):
ARM: riscpc: use GENERIC_IRQ_MULTI_HANDLER
ARM: footbridge: use GENERIC_IRQ_MULTI_HANDLER
ARM: iop32x: offset IRQ numbers by 1
ARM: iop32x: use GENERIC_IRQ_MULTI_HANDLER
ARM: remove old-style irq entry
arch/arm/Kconfig | 22 +-- arch/arm/include/asm/assembler.h | 185 ++++++++++++++++---- arch/arm/include/asm/current.h | 37 ++-- arch/arm/include/asm/elf.h | 3 + arch/arm/include/asm/entry-macro-multi.S | 16 -- arch/arm/include/asm/hardware/entry-macro-iomd.S | 131 -------------- arch/arm/include/asm/insn.h | 24 +++ arch/arm/include/asm/irq.h | 1 - arch/arm/include/asm/mach/arch.h | 2 - arch/arm/include/asm/percpu.h | 25 ++- arch/arm/include/asm/switch_to.h | 3 +- arch/arm/include/asm/thread_info.h | 27 --- arch/arm/include/asm/tls.h | 13 +- arch/arm/kernel/asm-offsets.c | 3 - arch/arm/kernel/entry-armv.S | 48 ++--- arch/arm/kernel/entry-common.S | 10 +- arch/arm/kernel/entry-header.S | 13 +- arch/arm/kernel/head-common.S | 4 +- arch/arm/kernel/irq.c | 17 -- arch/arm/kernel/module.c | 63 +++++++ arch/arm/kernel/process.c | 7 +- arch/arm/kernel/sleep.S | 4 +- arch/arm/kernel/smp.c | 11 ++ arch/arm/kernel/traps.c | 4 + arch/arm/mach-footbridge/common.c | 87 +++++++++ arch/arm/mach-footbridge/include/mach/entry-macro.S | 107 ----------- arch/arm/mach-iop32x/cp6.c | 10 +- arch/arm/mach-iop32x/include/mach/entry-macro.S | 31 ---- arch/arm/mach-iop32x/include/mach/irqs.h | 2 +- arch/arm/mach-iop32x/iop3xx.h | 1 + arch/arm/mach-iop32x/irq.c | 23 +++ arch/arm/mach-iop32x/irqs.h | 60 ++++--- arch/arm/mach-rpc/fiq.S | 5 +- arch/arm/mach-rpc/include/mach/entry-macro.S | 13 -- arch/arm/mach-rpc/irq.c | 145 +++++++++++++++ arch/arm/mm/Kconfig | 1 + 36 files changed, 656 insertions(+), 502 deletions(-) delete mode 100644 arch/arm/include/asm/entry-macro-multi.S delete mode 100644 arch/arm/include/asm/hardware/entry-macro-iomd.S delete mode 100644 arch/arm/mach-footbridge/include/mach/entry-macro.S delete mode 100644 arch/arm/mach-iop32x/include/mach/entry-macro.S delete mode 100644 arch/arm/mach-rpc/include/mach/entry-macro.S
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 1/12 ARM: riscpc: use GENERIC_IRQ_MULTI_HANDLER
From: Arnd Bergmann <[email protected]>
This is one of the last platforms using the old entry path. While this code path is spread over a few files, it is fairly straightforward to convert it into an equivalent C version, leaving the existing algorithm and all the priority handling the same.
Unlike most irqchip drivers, this means reading the status register(s) in a loop and always handling the highest-priority irq first.
The IOMD_IRQREQC and IOMD_IRQREQD registers are not actaully used here, but I left the code in place for the time being, to keep the conversion as direct as possible. It could be removed in a cleanup on top.
--- arch/arm/Kconfig | 1 + arch/arm/include/asm/hardware/entry-macro-iomd.S | 131 ------------------ arch/arm/mach-rpc/fiq.S | 5 +- arch/arm/mach-rpc/include/mach/entry-macro.S | 13 -- arch/arm/mach-rpc/irq.c | 145 ++++++++++++++++++++ 5 files changed, 149 insertions(+), 146 deletions(-)
arch/arm/Kconfig +1 -0
@@ -444,6 +444,7 @@ config ARCH_RPC select ARM_HAS_SG_CHAIN select CPU_SA110 select FIQ + select GENERIC_IRQ_MULTI_HANDLER select HAVE_PATA_PLATFORM select ISA_DMA_API select LEGACY_TIMER_TICK
arch/arm/include/asm/hardware/entry-macro-iomd.S deleted +0 -131
131 lines not on this page · raw message
arch/arm/mach-rpc/fiq.S +3 -2
@@ -2,10 +2,11 @@ #include <linux/linkage.h> #include <asm/assembler.h> #include <mach/hardware.h> -#include <mach/entry-macro.S> - .text + .equ ioc_base_high, IOC_BASE & 0xff000000 + .equ ioc_base_low, IOC_BASE & 0x00ff0000 + .text .global rpc_default_fiq_end ENTRY(rpc_default_fiq_start) mov r12, #ioc_base_high
arch/arm/mach-rpc/include/mach/entry-macro.S deleted +0 -13
@@ -1,13 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#include <mach/hardware.h> -#include <asm/hardware/entry-macro-iomd.S> - - .equ ioc_base_high, IOC_BASE & 0xff000000 - .equ ioc_base_low, IOC_BASE & 0x00ff0000 - - .macro get_irqnr_preamble, base, tmp - mov \base, #ioc_base_high @ point at IOC - .if ioc_base_low - orr \base, \base, #ioc_base_low - .endif - .endm
arch/arm/mach-rpc/irq.c +145 -0
157 lines not on this page · raw message
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 2/12 ARM: footbridge: use GENERIC_IRQ_MULTI_HANDLER
In reply to Ard BiesheuvelFrom: Arnd Bergmann <[email protected]>
Footbridge still uses the classic IRQ entry path in assembler, but this is easily converted into an equivalent C version.
In this case, the correlation between IRQ numbers and bits in the status register is non-obvious, and the priorities are handled by manually checking each bit in a static order, re-reading the status register after each handled event.
I moved the code into the new file and edited the syntax without changing this sequence to keep the behavior as close as possible to what it traditionally did.
--- arch/arm/Kconfig | 1 + arch/arm/mach-footbridge/common.c | 87 ++++++++++++++++ arch/arm/mach-footbridge/include/mach/entry-macro.S | 107 -------------------- 3 files changed, 88 insertions(+), 107 deletions(-)
arch/arm/Kconfig +1 -0
@@ -362,6 +362,7 @@ config ARCH_FOOTBRIDGE select FOOTBRIDGE select NEED_MACH_IO_H if !MMU select NEED_MACH_MEMORY_H + select GENERIC_IRQ_MULTI_HANDLER help Support for systems based on the DC21285 companion chip ("FootBridge"), such as the Simtec CATS and the Rebel NetWinder.
arch/arm/mach-footbridge/common.c +87 -0
99 lines not on this page · raw message
arch/arm/mach-footbridge/include/mach/entry-macro.S deleted +0 -107
107 lines not on this page · raw message
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v2 02/12] ARM: footbridge: use GENERIC_IRQ_MULTI_HANDLER
Quoted 10 lines · Ard Biesheuvel
From: Arnd Bergmann <[email protected]>
Footbridge still uses the classic IRQ entry path in assembler, but this is easily converted into an equivalent C version.
In this case, the correlation between IRQ numbers and bits in the status register is non-obvious, and the priorities are handled by manually checking each bit in a static order, re-reading the status register after each handled event.
Quoted 6 lines
I moved the code into the new file and edited the syntax without changing this sequence to keep the behavior as close as possible to what it traditionally did.
Signed-off-by Arnd Bergmann <[email protected]>Signed-off-by Ard Biesheuvel <[email protected]>
This generally looks good to me, if I had time to go into my office I could also test it but not right now because of external factors.
Yours,
Linus Walleij
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 3/12 ARM: iop32x: offset IRQ numbers by 1
In reply to Ard BiesheuvelFrom: Arnd Bergmann <[email protected]>
iop32x is one of the last platforms to use IRQ 0, and this has apparently stopped working in a 2014 cleanup without anyone noticing. This interrupt is used for the DMA engine, so most likely this has not actually worked in the past 7 years, but it's also not essential for using this board.
I'm splitting out this change from my GENERIC_IRQ_MULTI_HANDLER conversion so it can be backported if anyone cares.
--- arch/arm/mach-iop32x/include/mach/entry-macro.S | 2 +- arch/arm/mach-iop32x/include/mach/irqs.h | 2 +- arch/arm/mach-iop32x/irqs.h | 60 +++++++++++--------- 3 files changed, 34 insertions(+), 30 deletions(-)
arch/arm/mach-iop32x/include/mach/entry-macro.S +1 -1
@@ -20,7 +20,7 @@ mrc p6, 0, \irqstat, c8, c0, 0 @ Read IINTSRC cmp \irqstat, #0 clzne \irqnr, \irqstat - rsbne \irqnr, \irqnr, #31 + rsbne \irqnr, \irqnr, #32 .endm .macro arch_ret_to_user, tmp1, tmp2
arch/arm/mach-iop32x/include/mach/irqs.h +1 -1
@@ -9,6 +9,6 @@#ifndef __IRQS_H#define __IRQS_H -#define NR_IRQS 32 +#define NR_IRQS 33#endif
arch/arm/mach-iop32x/irqs.h +32 -28
68 lines not on this page · raw message
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v2 03/12] ARM: iop32x: offset IRQ numbers by 1
Quoted 7 lines · Ard Biesheuvel
From: Arnd Bergmann <[email protected]>
iop32x is one of the last platforms to use IRQ 0, and this has apparently stopped working in a 2014 cleanup without anyone noticing. This interrupt is used for the DMA engine, so most likely this has not actually worked in the past 7 years, but it's also not essential for using this board.
Quoted 6 lines
I'm splitting out this change from my
GENERIC_IRQ_MULTI_HANDLERconversion so it can be backported if anyone cares.Fixes a71b092a9c68 ("ARM: Convert handle_IRQ to use __handle_domain_irq")Signed-off-by Arnd Bergmann <[email protected]>Signed-off-by Ard Biesheuvel <[email protected]>
Good riddance.
Yours,
Linus Walleij
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 4/12 ARM: iop32x: use GENERIC_IRQ_MULTI_HANDLER
In reply to Ard BiesheuvelFrom: Arnd Bergmann <[email protected]>
iop32x uses the entry-macro.S file for both the IRQ entry and for hooking into the arch_ret_to_user code path. This is done because the cp6 registers have to be enabled before accessing any of the interrupt controller registers but have to be disabled when running in user space.
There is also a lazy-enable logic in cp6.c, but during a hardirq, we know it has to be enabled.
Both the cp6-enable code and the code to read the IRQ status can be lifted into the normal generic_handle_arch_irq() path, but the cp6-disable code has to remain in the user return code. As nothing other than iop32x uses this hook, just open-code it there with an ifdef for the platform that can eventually be removed when iop32x has reached the end of its life.
The cp6-enable path in the IRQ entry has an extra cp_wait barrier that the trap version does not have, but it is harmless to do it in both cases to simplify the logic here at the cost of a few extra cycles for the trap.
I'm leaving the logic completely unchanged, including the way this platform starts its IRQs at zero, but add a note that this can cause problems.
--- arch/arm/Kconfig | 5 +--- arch/arm/kernel/entry-common.S | 10 ++++--- arch/arm/mach-iop32x/cp6.c | 10 ++++++- arch/arm/mach-iop32x/include/mach/entry-macro.S | 31 -------------------- arch/arm/mach-iop32x/iop3xx.h | 1 + arch/arm/mach-iop32x/irq.c | 23 +++++++++++++++ 6 files changed, 40 insertions(+), 40 deletions(-)
arch/arm/Kconfig +1 -4
@@ -227,9 +227,6 @@ config GENERIC_ISA_DMA config FIQ bool -config NEED_RET_TO_USER - bool - config ARCH_MTD_XIP bool
@@ -371,9 +368,9 @@ config ARCH_IOP32X bool "IOP32x-based" depends on MMU select CPU_XSCALE + select GENERIC_IRQ_MULTI_HANDLER select GPIO_IOP select GPIOLIB - select NEED_RET_TO_USER select FORCE_PCI select PLAT_IOP help
arch/arm/kernel/entry-common.S +6 -4
@@ -16,12 +16,14 @@ .equ NR_syscalls, __NR_syscalls -#ifdef CONFIG_NEED_RET_TO_USER -#include <mach/entry-macro.S> -#else .macro arch_ret_to_user, tmp1, tmp2 - .endm +#ifdef CONFIG_ARCH_IOP32X + mrc p15, 0, \tmp1, c15, c1, 0 + ands \tmp2, \tmp1, #(1 << 6) + bicne \tmp1, \tmp1, #(1 << 6) + mcrne p15, 0, \tmp1, c15, c1, 0 @ Disable cp6 access #endif + .endm #include "entry-header.S"
arch/arm/mach-iop32x/cp6.c +9 -1
@@ -7,7 +7,7 @@#include <asm/traps.h> #include <asm/ptrace.h> -static int cp6_trap (struct pt_regs *regs,unsigned int instr) +void iop_enable_cp6 (void ) {u32 temp;
@@ -16,7 +16,15 @@ static int cp6_trap(struct pt_regs *regs, unsigned int instr) "mrc p15, 0, %0, c15, c1, 0\n\t" "orr %0, %0, #(1 << 6)\n\t" "mcr p15, 0, %0, c15, c1, 0\n\t" + "mrc p15, 0, %0, c15, c1, 0\n\t" + "mov %0, %0\n\t" + "sub pc, pc, #4 @ cp_wait\n\t" : "=r"(temp)); +} + +static int cp6_trap (struct pt_regs *regs,unsigned int instr) +{ +iop_enable_cp6 ();return 0 ; }
arch/arm/mach-iop32x/include/mach/entry-macro.S deleted +0 -31
@@ -1,31 +0,0 @@ -/* - * arch/arm/mach-iop32x/include/mach/entry-macro.S - * - * Low-level IRQ helper macros for IOP32x-based platforms - * - * This file is licensed under the terms of the GNU General Public - * License version 2. This program is licensed "as is" without any - * warranty of any kind, whether express or implied. - */ - .macro get_irqnr_preamble, base, tmp - mrc p15, 0, \tmp, c15, c1, 0 - orr \tmp, \tmp, #(1 << 6) - mcr p15, 0, \tmp, c15, c1, 0 @ Enable cp6 access - mrc p15, 0, \tmp, c15, c1, 0 - mov \tmp, \tmp - sub pc, pc, #4 @ cp_wait - .endm - - .macro get_irqnr_and_base, irqnr, irqstat, base, tmp - mrc p6, 0, \irqstat, c8, c0, 0 @ Read IINTSRC - cmp \irqstat, #0 - clzne \irqnr, \irqstat - rsbne \irqnr, \irqnr, #32 - .endm - - .macro arch_ret_to_user, tmp1, tmp2 - mrc p15, 0, \tmp1, c15, c1, 0 - ands \tmp2, \tmp1, #(1 << 6) - bicne \tmp1, \tmp1, #(1 << 6) - mcrne p15, 0, \tmp1, c15, c1, 0 @ Disable cp6 access - .endm
arch/arm/mach-iop32x/iop3xx.h +1 -0
@@ -225,6 +225,7 @@ extern int iop3xx_get_init_atu(void);#include <linux/reboot.h> void iop3xx_map_io (void ); +void iop_enable_cp6 (void );void iop_init_cp6_handler (void );void iop_init_time (unsigned long tickrate);void iop3xx_restart (enum reboot_mode ,const char *);
arch/arm/mach-iop32x/irq.c +23 -0
@@ -29,6 +29,15 @@ static void intstr_write(u32 val) asmvolatile ("mcr p6, 0, %0, c4, c0, 0" : :"r" (val)); } +static u32 iintsrc_read (void ) +{ +int irq; + + asmvolatile ("mrc p6, 0, %0, c8, c0, 0" :"=r" (irq)); + +return irq; +} +static void iop32x_irq_mask (struct irq_data *d) {
@@ -50,11 +59,25 @@ struct irq_chip ext_chip = { .irq_unmask = iop32x_irq_unmask, }; +void iop_handle_irq (struct pt_regs *regs) +{ +u32 mask; + +iop_enable_cp6 (); + +do { + mask =iintsrc_read (); +if (mask) +generic_handle_irq (fls (mask)); + }while (mask); +} +void __init iop32x_init_irq (void ) {int i;iop_init_cp6_handler (); +set_handle_irq (iop_handle_irq);intctl_write (0 );intstr_write (0 );
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v2 04/12] ARM: iop32x: use GENERIC_IRQ_MULTI_HANDLER
Quoted 11 lines · Ard Biesheuvel
From: Arnd Bergmann <[email protected]>
The cp6-enable path in the IRQ entry has an extra
cp_waitbarrier that the trap version does not have, but it is harmless to do it in both cases to simplify the logic here at the cost of a few extra cycles for the trap.I'm leaving the logic completely unchanged, including the way this platform starts its IRQs at zero, but add a note that this can cause problems.
(Replying to my own patch here)
The last paragraph is now obsolete, please remove it.
Arnd
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 5/12 ARM: remove old-style irq entry
In reply to Ard BiesheuvelFrom: Arnd Bergmann <[email protected]>
The last user of arch_irq_handler_default is gone now, so the entry-macro-multi.S file and all references to mach/entry-macro.S can be removed, as well as the asm_do_IRQ() entrypoint into the interrupt handling routines implemented in C.
Note: The ARMv7-M entry still uses its own top-level IRQ entry, calling nvic_handle_irq() from assembly. This could be changed to go through generic_handle_arch_irq() as well, but it's unclear to me if there are any benefits.
--- arch/arm/Kconfig | 14 ++------------ arch/arm/include/asm/entry-macro-multi.S | 16 ---------------- arch/arm/include/asm/irq.h | 1 - arch/arm/include/asm/mach/arch.h | 2 -- arch/arm/kernel/entry-armv.S | 8 -------- arch/arm/kernel/irq.c | 17 ----------------- 6 files changed, 2 insertions(+), 56 deletions(-)
arch/arm/Kconfig +2 -12
@@ -58,6 +58,7 @@ config ARM select GENERIC_CPU_AUTOPROBE select GENERIC_EARLY_IOREMAP select GENERIC_IDLE_POLL_SETUP + select GENERIC_IRQ_MULTI_HANDLER if MMU select GENERIC_IRQ_PROBE select GENERIC_IRQ_SHOW select GENERIC_IRQ_SHOW_LEVEL
@@ -320,7 +321,6 @@ config ARCH_MULTIPLATFORM select AUTO_ZRELADDR select TIMER_OF select COMMON_CLK - select GENERIC_IRQ_MULTI_HANDLER select HAVE_PCI select PCI_DOMAINS_GENERIC if PCI select SPARSE_IRQ
@@ -344,7 +344,6 @@ config ARCH_EP93XX select ARM_AMBA imply ARM_PATCH_PHYS_VIRT select ARM_VIC - select GENERIC_IRQ_MULTI_HANDLER select AUTO_ZRELADDR select CLKSRC_MMIO select CPU_ARM920T
@@ -359,7 +358,6 @@ config ARCH_FOOTBRIDGE select FOOTBRIDGE select NEED_MACH_IO_H if !MMU select NEED_MACH_MEMORY_H - select GENERIC_IRQ_MULTI_HANDLER help Support for systems based on the DC21285 companion chip ("FootBridge"), such as the Simtec CATS and the Rebel NetWinder.
@@ -368,7 +366,6 @@ config ARCH_IOP32X bool "IOP32x-based" depends on MMU select CPU_XSCALE - select GENERIC_IRQ_MULTI_HANDLER select GPIO_IOP select GPIOLIB select FORCE_PCI
@@ -384,7 +381,6 @@ config ARCH_IXP4XX select ARCH_SUPPORTS_BIG_ENDIAN select CPU_XSCALE select DMABOUNCE if PCI - select GENERIC_IRQ_MULTI_HANDLER select GPIO_IXP4XX select GPIOLIB select HAVE_PCI
@@ -400,7 +396,6 @@ config ARCH_IXP4XX config ARCH_DOVE bool "Marvell Dove" select CPU_PJ4 - select GENERIC_IRQ_MULTI_HANDLER select GPIOLIB select HAVE_PCI select MVEBU_MBUS
@@ -423,7 +418,6 @@ config ARCH_PXA select CLKSRC_MMIO select TIMER_OF select CPU_XSCALE if !CPU_XSC3 - select GENERIC_IRQ_MULTI_HANDLER select GPIO_PXA select GPIOLIB select IRQ_DOMAIN
@@ -442,7 +436,6 @@ config ARCH_RPC select ARM_HAS_SG_CHAIN select CPU_SA110 select FIQ - select GENERIC_IRQ_MULTI_HANDLER select HAVE_PATA_PLATFORM select ISA_DMA_API select LEGACY_TIMER_TICK
@@ -463,7 +456,6 @@ config ARCH_SA1100 select COMMON_CLK select CPU_FREQ select CPU_SA1100 - select GENERIC_IRQ_MULTI_HANDLER select GPIOLIB select IRQ_DOMAIN select ISA
@@ -478,7 +470,6 @@ config ARCH_S3C24XX select CLKSRC_SAMSUNG_PWM select GPIO_SAMSUNG select GPIOLIB - select GENERIC_IRQ_MULTI_HANDLER select HAVE_S3C2410_I2C if I2C select NEED_MACH_IO_H select S3C2410_WATCHDOG
@@ -497,7 +488,6 @@ config ARCH_OMAP1 select ARCH_OMAP select CLKSRC_MMIO select GENERIC_IRQ_CHIP - select GENERIC_IRQ_MULTI_HANDLER select GPIOLIB select HAVE_LEGACY_CLK select IRQ_DOMAIN
@@ -1168,7 +1158,7 @@ config CURRENT_POINTER_IN_TPIDRURO config IRQSTACKS def_bool y - depends on GENERIC_IRQ_MULTI_HANDLER && THREAD_INFO_IN_TASK + depends on THREAD_INFO_IN_TASK select HAVE_IRQ_EXIT_ON_IRQ_STACK select HAVE_SOFTIRQ_ON_OWN_STACK
arch/arm/include/asm/entry-macro-multi.S deleted +0 -16
@@ -1,16 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#include <asm/assembler.h> - -/* - * Interrupt handling. Preserves r7, r8, r9 - */ - .macro arch_irq_handler_default - get_irqnr_preamble r6, lr -1: get_irqnr_and_base r0, r2, r6, lr - movne r1, sp - @ - @ routine called with r0 = irq number, r1 = struct pt_regs * - @ - badrne lr, 1b - bne asm_do_IRQ - .endm
arch/arm/include/asm/irq.h +0 -1
@@ -26,7 +26,6 @@struct irqaction ;struct pt_regs ; -extern void asm_do_IRQ (unsigned int ,struct pt_regs *);void handle_IRQ (unsigned int ,struct pt_regs *);void init_IRQ (void );
arch/arm/include/asm/mach/arch.h +0 -2
@@ -56,9 +56,7 @@ struct machine_desc {void (*init_time)(void );void (*init_machine)(void );void (*init_late)(void ); -#ifdef CONFIG_GENERIC_IRQ_MULTI_HANDLER void (*handle_irq)(struct pt_regs *); -#endif void (*restart)(enum reboot_mode ,const char *); };
arch/arm/kernel/entry-armv.S +0 -8
@@ -19,9 +19,6 @@ #include <asm/glue-df.h> #include <asm/glue-pf.h> #include <asm/vfpmacros.h> -#ifndef CONFIG_GENERIC_IRQ_MULTI_HANDLER -#include <mach/entry-macro.S> -#endif #include <asm/thread_notify.h> #include <asm/unwind.h> #include <asm/unistd.h>
@@ -30,14 +27,12 @@ #include <asm/uaccess-asm.h> #include "entry-header.S" -#include <asm/entry-macro-multi.S> #include <asm/probes.h> /* * Interrupt handling. */ .macro irq_handler, from_user:req -#ifdef CONFIG_GENERIC_IRQ_MULTI_HANDLER mov r0, sp #ifdef CONFIG_IRQSTACKS mov_l r2, irq_stack_ptr @ Take base address
@@ -92,9 +87,6 @@ UNWIND( .setfp fpreg, sp ) mov sp, r9 @ Restore original SP #endif // CONFIG_UNWINDER_ARM #endif // CONFIG_IRQSTACKS -#else - arch_irq_handler_default -#endif .endm .macro pabt_helper
arch/arm/kernel/irq.c +0 -17
@@ -120,23 +120,6 @@ void handle_IRQ(unsigned int irq, struct pt_regs *regs)ack_bad_irq (irq); } -/* -* asm_do_IRQ is the interface to be used from assembly code. -*/ -asmlinkage void __exception_irq_entry -asm_do_IRQ (unsigned int irq,struct pt_regs *regs) -{ -struct pt_regs *old_regs; - -irq_enter (); - old_regs =set_irq_regs (regs); - -handle_IRQ (irq, regs); - -set_irq_regs (old_regs); -irq_exit (); -} -void __init init_IRQ (void ) {int ret;
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v2 05/12] ARM: remove old-style irq entry
Quoted 7 lines · Ard Biesheuvel
From: Arnd Bergmann <[email protected]>
The last user of
arch_irq_handler_defaultis gone now, so theentry-macro-multi.Sfile and all references tomach/entry-macro.Scan be removed, as well as theasm_do_IRQ()entrypoint into the interrupt handling routines implemented in C.
Quoted 8 lines
Note: The ARMv7-M entry still uses its own top-level IRQ entry, calling
nvic_handle_irq()from assembly. This could be changed to go throughgeneric_handle_arch_irq()as well, but it's unclear to me if there are any benefits.Signed-off-by Arnd Bergmann <[email protected]>[ardb: keep irq_handler macro as it carries all the IRQ stack handling]Signed-off-by Ard Biesheuvel <[email protected]>
Yours,
Linus Walleij
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 6/12 ARM: entry: preserve thread_info pointer in switch_to
In reply to Ard BiesheuvelTweak the UP stack protector handling code so that the thread info pointer is preserved in R7 until set_current is called. This is needed for a subsequent patch that implements THREAD_INFO_IN_TASK and set_current for UP as well.
This also means we will prefer the per-task protector on UP systems that implement the thread ID registers, so tweak the preprocessor conditionals to reflect this.
--- arch/arm/kernel/entry-armv.S | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-)
arch/arm/kernel/entry-armv.S +9 -8
@@ -806,16 +806,16 @@ ENTRY(__switch_to) ldr r6, [r2, #TI_CPU_DOMAIN] #endif switch_tls r1, r4, r5, r3, r7 -#if defined(CONFIG_STACKPROTECTOR) && !defined(CONFIG_SMP) - ldr r7, [r2, #TI_TASK] +#if defined(CONFIG_STACKPROTECTOR) && !defined(CONFIG_SMP) && \ + !defined(CONFIG_STACKPROTECTOR_PER_TASK) + ldr r9, [r2, #TI_TASK] ldr r8, =__stack_chk_guard .if (TSK_STACK_CANARY > IMM12_MASK) - add r7, r7, #TSK_STACK_CANARY & ~IMM12_MASK + add r9, r9, #TSK_STACK_CANARY & ~IMM12_MASK .endif - ldr r7, [r7, #TSK_STACK_CANARY & IMM12_MASK] -#elif defined(CONFIG_CURRENT_POINTER_IN_TPIDRURO) - mov r7, r2 @ Preserve 'next' + ldr r9, [r9, #TSK_STACK_CANARY & IMM12_MASK] #endif + mov r7, r2 @ Preserve 'next' #ifdef CONFIG_CPU_USE_DOMAINS mcr p15, 0, r6, c3, c0, 0 @ Set domain register #endif
@@ -824,8 +824,9 @@ ENTRY(__switch_to) ldr r0, =thread_notify_head mov r1, #THREAD_NOTIFY_SWITCH bl atomic_notifier_call_chain -#if defined(CONFIG_STACKPROTECTOR) && !defined(CONFIG_SMP) - str r7, [r8] +#if defined(CONFIG_STACKPROTECTOR) && !defined(CONFIG_SMP) && \ + !defined(CONFIG_STACKPROTECTOR_PER_TASK) + str r9, [r8] #endif mov r0, r5 #if !defined(CONFIG_THUMB2_KERNEL) && !defined(CONFIG_VMAP_STACK)
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 7/12 ARM: module: implement support for PC-relative group relocations
In reply to Ard BiesheuvelAdd support for the R_ARM_ALU_PC_Gn_NC and R_ARM_LDR_PC_G2 group relocations so we can use them in modules. These will be used to load the current task pointer from a global variable without having to rely on a literal pool entry to carry the address of this variable, which would be slightly less efficient.
--- arch/arm/include/asm/elf.h | 3 + arch/arm/kernel/module.c | 63 ++++++++++++++++++++ 2 files changed, 66 insertions(+)
arch/arm/include/asm/elf.h +3 -0
@@ -61,6 +61,9 @@ typedef struct user_fp elf_fpregset_t;#define R_ARM_MOVT_ABS 44#define R_ARM_MOVW_PREL_NC 45#define R_ARM_MOVT_PREL 46 +#define R_ARM_ALU_PC_G0_NC 57 +#define R_ARM_ALU_PC_G1_NC 59 +#define R_ARM_LDR_PC_G2 63#define R_ARM_THM_CALL 10#define R_ARM_THM_JUMP24 30
arch/arm/kernel/module.c +63 -0
81 lines not on this page · raw message
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 8/12 ARM: assembler: add optimized ldr/str macros to load variables from memory
In reply to Ard BiesheuvelWe will be adding variable loads to various hot paths, so it makes sense to add a helper macro that can load variables from asm code without the use of literal pool entries. On v7 or later, we can simply use MOVW/MOVT pairs, but on earlier cores, this requires a bit of hackery to emit a instruction sequence that implements this using a sequence of ADD/LDR instructions.
--- arch/arm/include/asm/assembler.h | 45 ++++++++++++++++++-- arch/arm/kernel/entry-armv.S | 2 +- arch/arm/kernel/entry-header.S | 2 +- 3 files changed, 43 insertions(+), 6 deletions(-)
arch/arm/include/asm/assembler.h +41 -4
@@ -568,12 +568,12 @@ THUMB( orr \reg , \reg , #PSR_T_BIT )/* * mov_l - move a constant value or [relocated] address into a register */ - .macro mov_l, dst:req, imm:req + .macro mov_l, dst:req, imm:req, cond .if __LINUX_ARM_ARCH__ < 7 - ldr \dst, =\imm + ldr\cond \dst, =\imm .else - movw \dst, #:lower16:\imm - movt \dst, #:upper16:\imm + movw\cond \dst, #:lower16:\imm + movt\cond \dst, #:upper16:\imm .endif .endm
@@ -611,6 +611,43 @@ THUMB( orr \reg , \reg , #PSR_T_BIT ) __adldst_l str, \src, \sym, \tmp, \cond .endm + .macro __ldst_va, op, reg, tmp, sym, cond +#if __LINUX_ARM_ARCH__ >= 7 || \ + (defined(MODULE) && defined(CONFIG_ARM_MODULE_PLTS)) || \ + (defined(CONFIG_LD_IS_LLD) && CONFIG_LLD_VERSION < 140000) + mov_l \tmp, \sym, \cond + \op\cond \reg, [\tmp] +#else + /* + * Avoid a literal load, by emitting a sequence of ADD/LDR instructions + * with the appropriate relocations. The combined sequence has a range + * of -/+ 256 MiB, which should be sufficient for the core kernel and + * for modules loaded into the module region. + */ + .globl \sym + .reloc .L0_\@, R_ARM_ALU_PC_G0_NC, \sym + .reloc .L1_\@, R_ARM_ALU_PC_G1_NC, \sym + .reloc .L2_\@, R_ARM_LDR_PC_G2, \sym +.L0_\@: sub\cond \tmp, pc, #8 +.L1_\@: sub\cond \tmp, \tmp, #4 +.L2_\@: \op\cond \reg, [\tmp, #0] +#endif + .endm + + /* + * ldr_va - load a 32-bit word from the virtual address of \sym + */ + .macro ldr_va, rd:req, sym:req, cond + __ldst_va ldr, \rd, \rd, \sym, \cond + .endm + + /* + * str_va - store a 32-bit word to the virtual address of \sym + */ + .macro str_va, rn:req, sym:req, tmp:req, cond + __ldst_va str, \rn, \tmp, \sym, \cond + .endm +/* * rev_l - byte-swap a 32-bit value *
arch/arm/kernel/entry-armv.S +1 -1
@@ -53,7 +53,7 @@ UNWIND( .setfp fpreg, sp ) subs r2, sp, r0 @ SP above bottom of IRQ stack? rsbscs r2, r2, #THREAD_SIZE @ ... and below the top? #ifdef CONFIG_VMAP_STACK - ldr_l r2, high_memory, cc @ End of the linear region + ldr_va r2, high_memory, cc @ End of the linear region cmpcc r2, r0 @ Stack pointer was below it? #endif movcs sp, r0 @ If so, revert to incoming SP
arch/arm/kernel/entry-header.S +1 -1
@@ -445,7 +445,7 @@ THUMB( it ne ) @ in such cases so just carry on. @ str ip, [r0, #12] @ Stash IP on the mode stack - ldr_l ip, high_memory @ Start of VMALLOC space + ldr_va ip, high_memory @ Start of VMALLOC space ARM( cmp sp, ip ) @ SP in vmalloc space? THUMB( cmp r1, ip ) THUMB( itt lo )
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 9/12 ARM: percpu: add SMP_ON_UP support
In reply to Ard BiesheuvelPermit the use of the TPIDRPRW system register for carrying the per-CPU offset in generic SMP configurations that also target non-SMP capable ARMv6 cores. This uses the SMP_ON_UP code patching framework to turn all TPIDRPRW accesses into reads/writes of entry #0 in the __per_cpu_offset array.
While at it, switch over some existing direct TPIDRPRW accesses in asm code to invocations of a new helper that is patched in the same way when necessary.
Note that CPU_V6+SMP without SMP_ON_UP results in a kernel that does not boot on v6 CPUs without SMP extensions, so add this dependency to Kconfig as well.
--- arch/arm/include/asm/assembler.h | 59 +++++++++++++++++++- arch/arm/include/asm/insn.h | 24 ++++++++ arch/arm/include/asm/percpu.h | 25 ++++++++- arch/arm/kernel/entry-armv.S | 16 +----- arch/arm/kernel/sleep.S | 4 +- arch/arm/mm/Kconfig | 1 + 6 files changed, 107 insertions(+), 22 deletions(-)
arch/arm/include/asm/assembler.h +56 -3
@@ -220,9 +220,7 @@ THUMB( fpreg .req r7 ) .macro reload_current, t1:req, t2:req#ifdef CONFIG_CURRENT_POINTER_IN_TPIDRURO - adr_l \t1, __entry_task @ get __entry_task base address - mrc p15, 0, \t2, c13, c0, 4 @ get per-CPU offset - ldr \t1, [\t1, \t2] @ load variable + ldr_this_cpu \t1, __entry_task, \t1, \t2 mcr p15, 0, \t1, c13, c0, 3 @ store in TPIDRURO #endif .endm
@@ -312,6 +310,26 @@ THUMB( fpreg .req r7 )#define ALT_UP_B (label) b label#endif +/* +* this_cpu_offset - load the per-CPU offset of this CPU into +* register 'rd' +*/ + .macro this_cpu_offset, rd:req +#ifdef CONFIG_SMP +ALT_SMP(mrc p15, 0, \rd, c13, c0, 4) +#ifdef CONFIG_CPU_V6 +ALT_UP_B(.L1_\@) +.L0_\@: + .subsection 1 +.L1_\@: ldr_va \rd, __per_cpu_offset + b .L0_\@ + .previous +#endif +#else + mov \rd, #0 +#endif + .endm +/* * Instruction barrier */
@@ -648,6 +666,41 @@ THUMB( orr \reg , \reg , #PSR_T_BIT ) __ldst_va str, \rn, \tmp, \sym, \cond .endm + /* + * ldr_this_cpu_armv6 - Load a 32-bit word from the per-CPU variable 'sym', + * without using a temp register. Supported in ARM mode + * only. + */ + .macro ldr_this_cpu_armv6, rd:req, sym:req + this_cpu_offset \rd + .globl \sym + .reloc .L0_\@, R_ARM_ALU_PC_G0_NC, \sym + .reloc .L1_\@, R_ARM_ALU_PC_G1_NC, \sym + .reloc .L2_\@, R_ARM_LDR_PC_G2, \sym + add \rd, \rd, pc +.L0_\@: sub \rd, \rd, #4 +.L1_\@: sub \rd, \rd, #0 +.L2_\@: ldr \rd, [\rd, #4] + .endm + + /* + * ldr_this_cpu - Load a 32-bit word from the per-CPU variable 'sym' + * into register 'rd', which may be the stack pointer, + * using 't1' and 't2' as general temp registers. These + * are permitted to overlap with 'rd' if != sp + */ + .macro ldr_this_cpu, rd:req, sym:req, t1:req, t2:req +#if __LINUX_ARM_ARCH__ >= 7 || \ + (defined(MODULE) && defined(CONFIG_ARM_MODULE_PLTS)) || \ + (defined(CONFIG_LD_IS_LLD) && CONFIG_LLD_VERSION < 140000) + this_cpu_offset \t1 + mov_l \t2, \sym + ldr \rd, [\t1, \t2] +#else + ldr_this_cpu_armv6 \rd, \sym +#endif + .endm +/* * rev_l - byte-swap a 32-bit value *
arch/arm/include/asm/insn.h +24 -0
@@ -2,6 +2,30 @@#ifndef __ASM_ARM_INSN_H#define __ASM_ARM_INSN_H +#include <linux/types.h> + +/* +* Avoid a literal load by emitting a sequence of ADD/LDR instructions with the +* appropriate relocations. The combined sequence has a range of -/+ 256 MiB, +* which should be sufficient for the core kernel as well as modules loaded +* into the module region. (Not supported by LLD before release 14) +*/ +#if !(defined(MODULE ) && defined(CONFIG_ARM_MODULE_PLTS )) && \ + !(defined(CONFIG_LD_IS_LLD ) &&CONFIG_LLD_VERSION <140000 ) +#define LOAD_SYM_ARMV6 (reg, sym) \ + " .globl " #sym " \n\t" \ + " .reloc 10f, R_ARM_ALU_PC_G0_NC, " #sym " \n\t" \ + " .reloc 11f, R_ARM_ALU_PC_G1_NC, " #sym " \n\t" \ + " .reloc 12f, R_ARM_LDR_PC_G2, " #sym " \n\t" \ + "10: sub " #reg ", pc, #8 \n\t" \ + "11: sub " #reg ", " #reg ", #4 \n\t" \ + "12: ldr " #reg ", [" #reg ", #0] \n\t" +#else +#define LOAD_SYM_ARMV6 (reg, sym) \ + " ldr " #reg ", =" #sym " \n\t" \ + " ldr " #reg ", [" #reg "] \n\t" +#endif +static inline unsigned long arm_gen_nop (void ) {
arch/arm/include/asm/percpu.h +22 -3
@@ -5,15 +5,22 @@#ifndef _ASM_ARM_PERCPU_H_#define _ASM_ARM_PERCPU_H_ +#include <asm/insn.h> + registerunsigned long current_stack_pointer asm ("sp" );/* * Same as asm-generic/percpu.h, except that we store the per cpu offset * in the TPIDRPRW. TPIDRPRW only exists on V6K and V7 */ -#if defined(CONFIG_SMP ) && !defined(CONFIG_CPU_V6 ) +#ifdef CONFIG_SMP +extern unsigned int smp_on_up; +static inline void set_my_cpu_offset (unsigned long off) { +if (IS_ENABLED (CONFIG_CPU_V6 ) && !smp_on_up) +return ; +/* Set TPIDRPRW */ asmvolatile ("mcr p15, 0, %0, c13, c0, 4" : :"r" (off) :"memory" ); }
@@ -27,8 +34,20 @@ static inline unsigned long __my_cpu_offset(void)* We want to allow caching the value, so avoid using volatile and * instead use a fake stack read to hazard against barrier(). */ - asm("mrc p15, 0, %0, c13, c0, 4" :"=r" (off) - :"Q" (*(const unsigned long *)current_stack_pointer)); + asm("0: mrc p15, 0, %0, c13, c0, 4 \n\t" +#ifdef CONFIG_CPU_V6 +"1: \n\t" +" .subsection 1 \n\t" + "2: " LOAD_SYM_ARMV6(%0, __per_cpu_offset) " \n\t" + " b 1b \n\t" + " .previous \n\t" + " .pushsection \".alt.smp.init\", \"a\" \n\t" + " .long 0b - . \n\t" + " b . + (2b - 0b) \n\t" + " .popsection \n\t" +#endif + : "=r" (off) + : "Q" (*(const unsigned long *)current_stack_pointer));return off; }
arch/arm/kernel/entry-armv.S +3 -13
@@ -35,15 +35,14 @@ .macro irq_handler, from_user:req mov r0, sp #ifdef CONFIG_IRQSTACKS - mov_l r2, irq_stack_ptr @ Take base address - mrc p15, 0, r3, c13, c0, 4 @ Get CPU offset #ifdef CONFIG_UNWINDER_ARM mov fpreg, sp @ Preserve original SP #else mov r8, fp @ Preserve original FP mov r9, sp @ Preserve original SP #endif - ldr sp, [r2, r3] @ Load SP from per-CPU var + ldr_this_cpu sp, irq_stack_ptr, r2, r3 + .if \from_user == 0 UNWIND( .setfp fpreg, sp ) @
@@ -876,16 +875,7 @@ __bad_stack: THUMB( bx pc ) THUMB( nop ) THUMB( .arm ) - mrc p15, 0, ip, c13, c0, 4 @ Get per-CPU offset - - .globl overflow_stack_ptr - .reloc 0f, R_ARM_ALU_PC_G0_NC, overflow_stack_ptr - .reloc 1f, R_ARM_ALU_PC_G1_NC, overflow_stack_ptr - .reloc 2f, R_ARM_LDR_PC_G2, overflow_stack_ptr - add ip, ip, pc -0: add ip, ip, #-4 -1: add ip, ip, #0 -2: ldr ip, [ip, #4] + ldr_this_cpu_armv6 ip, overflow_stack_ptr str sp, [ip, #-4]! @ Preserve original SP value mov sp, ip @ Switch to overflow stack
arch/arm/kernel/sleep.S +1 -3
@@ -71,9 +71,7 @@ ENTRY(__cpu_suspend) @ Run the suspend code from the overflow stack so we don't have to rely @ on vmalloc-to-phys conversions anywhere in the arch suspend code. @ The original SP value captured in R5 will be restored on the way out. - mov_l r6, overflow_stack_ptr @ Base pointer - mrc p15, 0, r7, c13, c0, 4 @ Get per-CPU offset - ldr sp, [r6, r7] @ Address of this CPU's overflow stack + ldr_this_cpu sp, overflow_stack_ptr, r6, r7 #endif add r4, r4, #12 @ Space for pgd, virt sp, phys resume fn sub sp, sp, r4 @ allocate CPU state on stack
arch/arm/mm/Kconfig +1 -0
@@ -386,6 +386,7 @@ config CPU_V6 select CPU_PABRT_V6 select CPU_THUMB_CAPABLE select CPU_TLB_V6 if MMU + select SMP_ON_UP if SMP # ARMv6k config CPU_V6K
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Russell King (Oracle)
maintainerRe: [PATCH v2 09/12] ARM: percpu: add SMP_ON_UP support
Quoted 9 lines · Ard Biesheuvel
arch/arm/include/asm/percpu.h +1
-0@@ -5,15 +5,22 @@#ifndef _ASM_ARM_PERCPU_H_#define _ASM_ARM_PERCPU_H_ +#include <asm/insn.h>
arch/arm/include/asm/percpu.h line 17
Quoted 10 lines
arch/arm/include/asm/percpu.h +3
-1+ registerunsigned long current_stack_pointer asm ("sp" );/* * Same as asm-generic/percpu.h, except that we store the per cpu offset * in the TPIDRPRW. TPIDRPRW only exists on V6K and V7 */ -#if defined(CONFIG_SMP ) && !defined(CONFIG_CPU_V6 ) +#ifdef CONFIG_SMP +extern unsigned int smp_on_up;
I would much rather we kept this hidden from global view by placing it inside the set_my_cpu_offset() body.
Signature · 7 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last! _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v2 09/12] ARM: percpu: add SMP_ON_UP support
Quoted 14 lines · Russell King (Oracle)
Quoted 12 lines · Ard Biesheuvel
arch/arm/include/asm/percpu.h +2
-0@@ -5,15 +5,22 @@#ifndef _ASM_ARM_PERCPU_H_#define _ASM_ARM_PERCPU_H_ +#include <asm/insn.h> + registerunsigned long current_stack_pointer asm ("sp" );
Quoted 11 lines
arch/arm/include/asm/percpu.h line 17
Quoted 7 lines
arch/arm/include/asm/percpu.h +2
-1/* * Same as asm-generic/percpu.h, except that we store the per cpu offset * in the TPIDRPRW. TPIDRPRW only exists on V6K and V7 */ -#if defined(CONFIG_SMP ) && !defined(CONFIG_CPU_V6 ) +#ifdef CONFIG_SMP +extern unsigned int smp_on_up;I would much rather we kept this hidden from global view by placing it inside the
set_my_cpu_offset()body.
Good point, I'll change that.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 10/12 ARM: use TLS register for 'current' on !SMP as well
In reply to Ard BiesheuvelEnable the use of the TLS register to hold the 'current' pointer also on non-SMP configurations that target v6k or later CPUs. This will permit the use of THREAD_INFO_IN_TASK as well as IRQ stacks and vmap'ed stacks for such configurations.
--- arch/arm/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm/Kconfig +1 -1
@@ -1154,7 +1154,7 @@ config SMP_ON_UP config CURRENT_POINTER_IN_TPIDRURO def_bool y - depends on SMP && CPU_32v6K && !CPU_V6 + depends on CPU_32v6K && !CPU_V6 config IRQSTACKS def_bool y
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v2 10/12] ARM: use TLS register for 'current' on !SMP as well
Quoted 18 lines · Ard Biesheuvel
Enable the use of the TLS register to hold the 'current' pointer also on non-SMP configurations that target v6k or later CPUs. This will permit the use of
THREAD_INFO_IN_TASKas well as IRQ stacks and vmap'ed stacks for such configurations.Acked-by Linus Walleij <[email protected]>Acked-by Nicolas Pitre <[email protected]>Signed-off-by Ard Biesheuvel <[email protected]>--- arch/arm/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/arm/Kconfig +0
-0
arch/arm/Kconfig config SMP_ON_UP · line 1157
Quoted 5 lines
arch/arm/Kconfig +1
-1@@ -1154,7 +1154,7 @@ config SMP_ON_UP config CURRENT_POINTER_IN_TPIDRURO def_bool y - depends on SMP && CPU_32v6K && !CPU_V6 + depends on CPU_32v6K && !CPU_V6
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 11/12 ARM: smp: defer TPIDRURO update for SMP v6 configurations too
In reply to Ard BiesheuvelDefer TPIDURO updates for user space until exit also for CPU_V6+SMP configurations so that we can decide at runtime whether to use it to carry the current pointer, provided that we are running on a CPU that actually implements this register. This is needed for THREAD_INFO_IN_TASK support for UP systems, which requires that all SMP capable systems use the TPIDRURO based access to 'current' as the only remaining alternative will be a global variable which only works on UP.
--- arch/arm/include/asm/tls.h | 13 +++++++------ arch/arm/kernel/entry-header.S | 11 ++++++++++- 2 files changed, 17 insertions(+), 7 deletions(-)
arch/arm/include/asm/tls.h +7 -6
@@ -18,13 +18,14 @@ .endm .macro switch_tls_v6, base, tp, tpuser, tmp1, tmp2 - ldr \tmp1, =elf_hwcap - ldr \tmp1, [\tmp1, #0] + ldr_va \tmp1, elf_hwcap mov \tmp2, #0xffff0fff tst \tmp1, #HWCAP_TLS @ hardware TLS available? streq \tp, [\tmp2, #-15] @ set TLS value at 0xffff0ff0 mrcne p15, 0, \tmp2, c13, c0, 2 @ get the user r/w register +#ifndef CONFIG_SMP mcrne p15, 0, \tp, c13, c0, 3 @ yes, set TLS register +#endif mcrne p15, 0, \tpuser, c13, c0, 2 @ set user r/w register strne \tmp2, [\base, #TI_TP_VALUE + 4] @ save it .endm
@@ -43,7 +44,7 @@#elif defined(CONFIG_CPU_V6 )#define tls_emu 0#define has_tls_reg (elf_hwcap & HWCAP_TLS) -#define defer_tls_reg_update 0 +#define defer_tls_reg_update IS_ENABLED(CONFIG_SMP)#define switch_tls switch_tls_v6#elif defined(CONFIG_CPU_32v6K)#define tls_emu 0
@@ -81,11 +82,11 @@ static inline void set_tls(unsigned long val)*/ barrier (); -if (!tls_emu && !defer_tls_reg_update) { -if (has_tls_reg) { +if (!tls_emu) { +if (has_tls_reg && !defer_tls_reg_update) { asm("mcr p15, 0, %0, c13, c0, 3" : :"r" (val)); - }else { + }else if (!has_tls_reg) {#ifdef CONFIG_KUSER_HELPERS /* * User space must never try to access this
arch/arm/kernel/entry-header.S +10 -1
@@ -292,12 +292,21 @@ .macro restore_user_regs, fast = 0, offset = 0 -#if defined(CONFIG_CPU_32v6K) && !defined(CONFIG_CPU_V6) +#if defined(CONFIG_CPU_32v6K) || defined(CONFIG_SMP) +#if defined(CONFIG_CPU_V6) && defined(CONFIG_SMP) +ALT_SMP(b .L1_\@ ) +ALT_UP( nop ) + ldr_va r1, elf_hwcap + tst r1, #HWCAP_TLS @ hardware TLS available? + beq .L2_\@ +.L1_\@: +#endif @ The TLS register update is deferred until return to user space so we @ can use it for other things while running in the kernel get_thread_info r1 ldr r1, [r1, #TI_TP_VALUE] mcr p15, 0, r1, c13, c0, 3 @ set TLS register +.L2_\@: #endif uaccess_enable r1, isb=0
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 12/12 ARM: implement THREAD_INFO_IN_TASK for uniprocessor systems
In reply to Ard BiesheuvelOn UP systems, only a single task can be 'current' at the same time, which means we can use a global variable to track it. This means we can also enable THREAD_INFO_IN_TASK for those systems, as in that case, thread_info is accessed via current rather than the other way around, removing the need to store thread_info at the base of the task stack. This, in turn, permits us to enable IRQ stacks and vmap'ed stacks on UP systems as well.
To partially mitigate the performance overhead of this arrangement, use a ADD/ADD/LDR sequence with the appropriate PC-relative group relocations to load the value of current when needed. This means that accessing current will still only require a single load as before, avoiding the need for a literal to carry the address of the global variable in each function. However, accessing thread_info will now require this load as well.
--- arch/arm/Kconfig | 7 +- arch/arm/include/asm/assembler.h | 83 +++++++++++++------- arch/arm/include/asm/current.h | 37 +++++---- arch/arm/include/asm/switch_to.h | 3 +- arch/arm/include/asm/thread_info.h | 27 ------- arch/arm/kernel/asm-offsets.c | 3 - arch/arm/kernel/entry-armv.S | 11 +-- arch/arm/kernel/head-common.S | 4 +- arch/arm/kernel/process.c | 7 +- arch/arm/kernel/smp.c | 11 +++ arch/arm/kernel/traps.c | 4 + 11 files changed, 109 insertions(+), 88 deletions(-)
arch/arm/Kconfig +3 -4
@@ -127,8 +127,8 @@ config ARM select PERF_USE_VMALLOC select RTC_LIB select SYS_SUPPORTS_APM_EMULATION - select THREAD_INFO_IN_TASK if CURRENT_POINTER_IN_TPIDRURO - select HAVE_ARCH_VMAP_STACK if MMU && THREAD_INFO_IN_TASK && (!LD_IS_LLD || LLD_VERSION >= 140000) + select THREAD_INFO_IN_TASK + select HAVE_ARCH_VMAP_STACK if MMU && (!LD_IS_LLD || LLD_VERSION >= 140000) select TRACE_IRQFLAGS_SUPPORT if !CPU_V7M # Above selects are sorted alphabetically; please add new ones # according to that. Thanks.
@@ -1158,7 +1158,6 @@ config CURRENT_POINTER_IN_TPIDRURO config IRQSTACKS def_bool y - depends on THREAD_INFO_IN_TASK select HAVE_IRQ_EXIT_ON_IRQ_STACK select HAVE_SOFTIRQ_ON_OWN_STACK
@@ -1608,7 +1607,7 @@ config CC_HAVE_STACKPROTECTOR_TLS config STACKPROTECTOR_PER_TASK bool "Use a unique stack canary value for each task" - depends on STACKPROTECTOR && THREAD_INFO_IN_TASK && !XIP_DEFLATED_DATA + depends on STACKPROTECTOR && CURRENT_POINTER_IN_TPIDRURO && !XIP_DEFLATED_DATA depends on GCC_PLUGINS || CC_HAVE_STACKPROTECTOR_TLS select GCC_PLUGIN_ARM_SSP_PER_TASK if !CC_HAVE_STACKPROTECTOR_TLS default y
arch/arm/include/asm/assembler.h +54 -29
101 lines not on this page · raw message
arch/arm/include/asm/current.h +22 -15
@@ -8,25 +8,18 @@#define _ASM_ARM_CURRENT_H#ifndef __ASSEMBLY__ +#include <asm/insn.h> struct task_struct ; -static inline void set_current(struct task_struct *cur) -{ - if (!IS_ENABLED(CONFIG_CURRENT_POINTER_IN_TPIDRURO)) - return; - - /* Set TPIDRURO */ - asm("mcr p15, 0, %0, c13, c0, 3" :: "r"(cur) : "memory"); -} +extern struct task_struct *__current; -#ifdef CONFIG_CURRENT_POINTER_IN_TPIDRURO - -static inline struct task_struct *get_current(void) +static inline __attribute_const__ struct task_struct *get_current(void) {struct task_struct *cur;#if __has_builtin (__builtin_thread_pointer) && \ + defined(CONFIG_CURRENT_POINTER_IN_TPIDRURO ) && \ !(defined(CONFIG_THUMB2_KERNEL ) && \ defined(CONFIG_CC_IS_CLANG ) &&CONFIG_CLANG_VERSION <130001 )/*
@@ -39,16 +32,30 @@ static inline struct task_struct *get_current(void)* https://github.com/ClangBuiltLinux/linux/issues/1485 */ cur =__builtin_thread_pointer (); +#elif defined(CONFIG_CURRENT_POINTER_IN_TPIDRURO ) || defined(CONFIG_SMP ) + asm("0: mrc p15, 0, %0, c13, c0, 3 \n\t" +#ifdef CONFIG_CPU_V6 +"1: \n\t" +" .subsection 1 \n\t" + "2: " LOAD_SYM_ARMV6(%0, __current) " \n\t" + " b 1b \n\t" + " .previous \n\t" + " .pushsection \".alt.smp.init\", \"a\" \n\t" + " .long 0b - . \n\t" + " b . + (2b - 0b) \n\t" + " .popsection \n\t" +#endif + : "=r"(cur)); +#elif __LINUX_ARM_ARCH__>=7 || \ + (defined(MODULE ) && defined(CONFIG_ARM_MODULE_PLTS )) + cur = __current;#else - asm("mrc p15, 0, %0, c13, c0, 3" :"=r" (cur)); + asm(LOAD_SYM_ARMV6(%0, __current) : "=r"(cur));#endif return cur; }#define current get_current() -#else -#include <asm-generic/current.h> -#endif /* CONFIG_CURRENT_POINTER_IN_TPIDRURO */ #endif /* __ASSEMBLY__ */
arch/arm/include/asm/switch_to.h +2 -1
@@ -40,7 +40,8 @@ static inline void set_ti_cpu(struct task_struct *p)do { \__complete_pending_tlbi (); \set_ti_cpu (next); \ -if (IS_ENABLED (CONFIG_CURRENT_POINTER_IN_TPIDRURO )) \ +if (IS_ENABLED (CONFIG_CURRENT_POINTER_IN_TPIDRURO ) || \ +IS_ENABLED (CONFIG_SMP )) \__this_cpu_write (__entry_task, next); \ last =__switch_to (prev,task_thread_info (prev),task_thread_info (next)); \ }while (0 )
arch/arm/include/asm/thread_info.h +0 -27
@@ -62,9 +62,6 @@ struct cpu_context_save {struct thread_info {unsigned long flags;/* low level flags */ int preempt_count;/* 0 => preemptable, <0 => bug */ -#ifndef CONFIG_THREAD_INFO_IN_TASK -struct task_struct *task;/* main task structure */ -#endif __u32 cpu;/* cpu */ __u32 cpu_domain;/* cpu domain */ struct cpu_context_save cpu_context;/* cpu context */
@@ -80,39 +77,15 @@ struct thread_info {#define INIT_THREAD_INFO (tsk) \ { \ - INIT_THREAD_INFO_TASK(tsk) \ .flags = 0, \ .preempt_count = INIT_PREEMPT_COUNT, \ } -#ifdef CONFIG_THREAD_INFO_IN_TASK -#define INIT_THREAD_INFO_TASK (tsk) -static inline struct task_struct *thread_task (struct thread_info * ti) {return (struct task_struct *)ti; } -#else -#define INIT_THREAD_INFO_TASK (tsk) .task = &(tsk), - -static inline struct task_struct *thread_task (struct thread_info * ti) -{ -return ti->task; -} - -/* -* how to get the thread information struct from C -*/ -static inline struct thread_info *current_thread_info (void ) __attribute_const__; - -static inline struct thread_info *current_thread_info (void ) -{ -return (struct thread_info *) - (current_stack_pointer & ~(THREAD_SIZE -1 )); -} -#endif -#define thread_saved_pc (tsk) \ ((unsigned long)(task_thread_info(tsk)->cpu_context.pc))#define thread_saved_sp (tsk) \
arch/arm/kernel/asm-offsets.c +0 -3
@@ -43,9 +43,6 @@ int main(void)BLANK ();DEFINE (TI_FLAGS , offsetof(struct thread_info , flags));DEFINE (TI_PREEMPT , offsetof(struct thread_info , preempt_count)); -#ifndef CONFIG_THREAD_INFO_IN_TASK -DEFINE (TI_TASK , offsetof(struct thread_info , task)); -#endif DEFINE (TI_CPU , offsetof(struct thread_info , cpu));DEFINE (TI_CPU_DOMAIN , offsetof(struct thread_info , cpu_domain));DEFINE (TI_CPU_SAVE , offsetof(struct thread_info , cpu_context));
arch/arm/kernel/entry-armv.S +6 -5
@@ -807,12 +807,13 @@ ENTRY(__switch_to) switch_tls r1, r4, r5, r3, r7 #if defined(CONFIG_STACKPROTECTOR) && !defined(CONFIG_SMP) && \ !defined(CONFIG_STACKPROTECTOR_PER_TASK) - ldr r9, [r2, #TI_TASK] ldr r8, =__stack_chk_guard .if (TSK_STACK_CANARY > IMM12_MASK) - add r9, r9, #TSK_STACK_CANARY & ~IMM12_MASK - .endif + add r9, r2, #TSK_STACK_CANARY & ~IMM12_MASK ldr r9, [r9, #TSK_STACK_CANARY & IMM12_MASK] + .else + ldr r9, [r2, #TSK_STACK_CANARY & IMM12_MASK] + .endif #endif mov r7, r2 @ Preserve 'next' #ifdef CONFIG_CPU_USE_DOMAINS
@@ -829,7 +830,7 @@ ENTRY(__switch_to) #endif mov r0, r5 #if !defined(CONFIG_THUMB2_KERNEL) && !defined(CONFIG_VMAP_STACK) - set_current r7 + set_current r7, r8 ldmia r4, {r4 - sl, fp, sp, pc} @ Load all regs saved previously #else mov r1, r7
@@ -851,7 +852,7 @@ ENTRY(__switch_to) @ switches us to another stack, with few other side effects. In order @ to prevent this distinction from causing any inconsistencies, let's @ keep the 'set_current' call as close as we can to the update of SP. - set_current r1 + set_current r1, r2 mov sp, ip ret lr #endif
arch/arm/kernel/head-common.S +1 -3
@@ -105,10 +105,8 @@ __mmap_switched: mov r1, #0 bl __memset @ clear .bss -#ifdef CONFIG_CURRENT_POINTER_IN_TPIDRURO adr_l r0, init_task @ get swapper task_struct - set_current r0 -#endif + set_current r0, r1 ldmia r4, {r0, r1, r2, r3} str r9, [r0] @ Save processor ID
arch/arm/kernel/process.c +6 -1
@@ -36,7 +36,7 @@#include "signal.h" -#ifdef CONFIG_CURRENT_POINTER_IN_TPIDRURO +#if defined(CONFIG_CURRENT_POINTER_IN_TPIDRURO ) || defined(CONFIG_SMP ) DEFINE_PER_CPU(struct task_struct *, __entry_task);#endif
@@ -46,6 +46,11 @@ unsigned long __stack_chk_guard __read_mostly;EXPORT_SYMBOL (__stack_chk_guard);#endif +#ifndef CONFIG_CURRENT_POINTER_IN_TPIDRURO +asmlinkage struct task_struct *__current; +EXPORT_SYMBOL (__current); +#endif +static const char *processor_modes[]__maybe_unused = {"USER_26" ,"FIQ_26" ,"IRQ_26" ,"SVC_26" ,"UK4_26" ,"UK5_26" ,"UK6_26" ,"UK7_26" ,"UK8_26" ,"UK9_26" ,"UK10_26" ,"UK11_26" ,"UK12_26" ,"UK13_26" ,"UK14_26" ,"UK15_26" ,
arch/arm/kernel/smp.c +11 -0
@@ -403,6 +403,17 @@ static void smp_store_cpu_info(unsigned int cpuid)check_cpu_icache_size (cpuid); } +static void set_current (struct task_struct *cur) +{ +if (!IS_ENABLED (CONFIG_CURRENT_POINTER_IN_TPIDRURO ) && !is_smp ()) { + __current = cur; +return ; + } + +/* Set TPIDRURO */ + asm("mcr p15, 0, %0, c13, c0, 3" ::"r" (cur) :"memory" ); +} +/* * This is the secondary CPU boot entry. We're using this CPUs * idle thread stack, but a set of temporary page tables.
arch/arm/kernel/traps.c +4 -0
@@ -865,7 +865,9 @@ early_initcall(allocate_overflow_stacks); asmlinkage void handle_bad_stack(struct pt_regs *regs) {unsigned long tsk_stk = (unsigned long )current->stack; +#ifdef CONFIG_IRQSTACKS unsigned long irq_stk = (unsigned long )this_cpu_read (irq_stack_ptr); +#endif unsigned long ovf_stk = (unsigned long )this_cpu_read (overflow_stack_ptr);console_verbose ();
@@ -873,8 +875,10 @@ asmlinkage void handle_bad_stack(struct pt_regs *regs)pr_emerg ("Task stack: [0x%08lx..0x%08lx]\n" , tsk_stk, tsk_stk +THREAD_SIZE ); +#ifdef CONFIG_IRQSTACKS pr_emerg ("IRQ stack: [0x%08lx..0x%08lx]\n" , irq_stk -THREAD_SIZE , irq_stk); +#endif pr_emerg ("Overflow stack: [0x%08lx..0x%08lx]\n" , ovf_stk -OVERFLOW_STACK_SIZE , ovf_stk);
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v2 12/12] ARM: implement THREAD_INFO_IN_TASK for uniprocessor systems
Quoted 5 lines · Ard Biesheuvel
config IRQSTACKS
def_booly
- depends onTHREAD_INFO_IN_TASK
selectHAVE_IRQ_EXIT_ON_IRQ_STACK
selectHAVE_SOFTIRQ_ON_OWN_STACK
Doesn't this still need a 'depends on MMU' or the corresponding patch to entry-v7m.S? I could only find your patch to entry-armv.S and it looks like you need the same thing for both.
Arnd
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v2 12/12] ARM: implement THREAD_INFO_IN_TASK for uniprocessor systems
Quoted 9 lines · Arnd Bergmann
Quoted 5 lines · Ard Biesheuvel
config IRQSTACKS
def_booly
- depends onTHREAD_INFO_IN_TASK
selectHAVE_IRQ_EXIT_ON_IRQ_STACK
selectHAVE_SOFTIRQ_ON_OWN_STACK
Quoted 4 lines
Doesn't this still need a 'depends on MMU' or the corresponding patch to
entry-v7m.S? I could only find your patch toentry-armv.Sand it looks like you need the same thing for both.
Yes, I suppose you're right. I'll add the 'depends on MMU' here.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 24 lines · Ard Biesheuvel
Enable the use of the TLS register to hold the 'current' pointer for all configurations that can support it, including non-SMP ones that target v6k or later CPUs, and multi-platform SMP ones that also support v6 based UP systems.
The remaining configurations are all strictly UP, which means we can switch to a global variable to hold the current pointer. By doing this, we can enable
THREAD_INFO_IN_TASK, which moves thread info off the stack, protecting it from overflows. It also permits us to enable IRQ stacks and vmap'ed stacks for UP configurations as well.Supporting v6 cores without SMP extensions in SMP configurations (e.g.,
omap2plus_defconfigorimx_v6_v7_defconfig) makes this a bit tricky, and this is a feature we may consider dropping entirely in the future. But for the time being, we can support this mode as well.The accesses to the global variable holding 'current' are constructed in a way that ensures that no literal pool accesses (and associated D-cache misses) are needed unless the access is from a module and module PLTs are enabled. This means that accessing 'current' is just as costly as before, as it used to require some arithmetic involving the stack pointer and a load from the
thread_info::taskfield.
Quoted 4 lines
However, accessing
thread_infoitself now also involves a load, although it should be noted that allthread_infoand current accesses now go via the same variable, which is therefore expected to be hot in the caches at all times.
Most of your assembler changes are beyond my understanding of that code, but everything that I do understand moves the code in the right direction, this looks really good overall.
I'll add this to my randconfig compile test setup to see if that finds any corner cases you may have missed.
Arnd
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel