arm64: Allow the compat vdso to be disabled at runtime
Marc Zyngier
maintainerv2 cover arm64: Allow the compat vdso to be disabled at runtime
The relatively recent introduction of the compat vdso on arm64 has overlooked its interactions with some of the interesting errata workarounds, such as ARM64_ERRATUM_1418040 (and its older 1188873 incarnation).
This erratum requires the 64bit kernel to trap 32bit accesses to the virtual counter and emulate it. When the workaround was introduced, the compat vdso simply wasn't a thing. Now that the patches have landed in mainline, we trap the CVTVCT accesses from the vdso.
This can end-up in a nasty loop in the vdso, where the sequence number changes on each trap, never stabilising, and leaving userspace in a bit of a funny state (which is why we disable the vdso in most similar cases). This erratum mentionned above is a bit special in the sense that in only requires to trap AArch32 accesses, and 64bit tasks can be left alone. Consequently, the vdso is never disabled and AArch32 tasks are affected.
Obviously, we really want to retain the 64bit vdso in this case. To that effect, this series offers a way to disable the 32bit view of the vdso without impacting its 64bit counterpart, by providing a "no-compat" vdso clock_mode, and plugging this feature into the 1418040 detection code.
Lastly, I've tagged a rework of the 1414080 workaround (which had been posted separately) at the end of the series so that it limits its effect to 32bit tasks exclusively (so far, it forces the userspace access bit on 64bit tasks, and we may need to leave it disabled in the future...).
* From v1:
- Reworked following Mark's feedback (patches #2 and #3)
- Reworked patch #4 after Will's comments
- patches #1 to #3 are now cc stable
- Applied Mark's AB to patch #1
Marc Zyngier (4):
arm64: Introduce a way to disable the 32bit vdso
arm64: arch_timer: Allow an workaround descriptor to disable compat
vdso
arm64: arch_timer: Disable the compat vdso for cores affected by
ARM64_WORKAROUND_1418040
arm64: Rework ARM_ERRATUM_1414080 handling
arch/arm64/include/asm/arch_timer.h | 1 + arch/arm64/include/asm/vdso/clocksource.h | 7 +++- .../include/asm/vdso/compat_gettimeofday.h | 8 +++- arch/arm64/kernel/entry.S | 40 +++++++++++-------- drivers/clocksource/arm_arch_timer.c | 11 +++++ 5 files changed, 48 insertions(+), 19 deletions(-)
Signature · 1 line
2.27.0
Marc Zyngier
maintainerPATCH v2 1/4 arm64: Introduce a way to disable the 32bit vdso
We have a class of errata (grouped under the ARM64_WORKAROUND_1418040 banner) that force the trapping of counter access from 32bit EL0.
We would normally disable the whole vdso for such defect, except that it would disable it for 64bit userspace as well, which is a shame.
Instead, add a new vdso_clock_mode, which signals that the vdso isn't usable for compat tasks. This gets checked in the new vdso_clocksource_ok() helper, now provided for the 32bit vdso.
--- arch/arm64/include/asm/vdso/clocksource.h | 7 +++++-- arch/arm64/include/asm/vdso/compat_gettimeofday.h | 8 +++++++- 2 files changed, 12 insertions(+), 3 deletions(-)
arch/arm64/include/asm/vdso/clocksource.h +5 -2
@@ -2,7 +2,10 @@#ifndef __ASM_VDSOCLOCKSOURCE_H#define __ASM_VDSOCLOCKSOURCE_H -#define VDSO_ARCH_CLOCKMODES \ - VDSO_CLOCKMODE_ARCHTIMER +#define VDSO_ARCH_CLOCKMODES \ + /* vdso clocksource for both 32 and 64bit tasks */ \ + VDSO_CLOCKMODE_ARCHTIMER, \ + /* vdso clocksource for 64bit tasks only */ \ +VDSO_CLOCKMODE_ARCHTIMER_NOCOMPAT #endif
arch/arm64/include/asm/vdso/compat_gettimeofday.h +7 -1
@@ -111,7 +111,7 @@ static __always_inline u64 __arch_get_hw_counter(s32 clock_mode)* update. Return something. Core will do another round and then * see the mode change and fallback to the syscall. */ -if (clock_mode ==VDSO_CLOCKMODE_NONE ) +if (clock_mode !=VDSO_CLOCKMODE_ARCHTIMER )return 0 ;/*
@@ -152,6 +152,12 @@ static __always_inline const struct vdso_data *__arch_get_vdso_data(void)return ret; } +static inline bool vdso_clocksource_ok (const struct vdso_data *vd) +{ +return vd->clock_mode ==VDSO_CLOCKMODE_ARCHTIMER ; +} +#define vdso_clocksource_ok vdso_clocksource_ok +#endif /* !__ASSEMBLY__ */ #endif /* __ASM_VDSO_GETTIMEOFDAY_H */
Signature · 1 line
2.27.0
Sasha Levin
maintainerRe: [PATCH v2 1/4] arm64: Introduce a way to disable the 32bit vdso
Hi
[This is an automated email]
This commit has been processed because it contains a -stable tag. The stable tag indicates that it's relevant for the following trees: all
The bot has tested the following trees: v5.7.7, v5.4.50, v4.19.131, v4.14.187, v4.9.229, v4.4.229.
v5.7.7: Build OK!
v5.4.50: Failed to apply! Possible dependencies:
04bb96427d4ee ("ARM: 8947/1: Fix __arch_get_hw_counter() access to CNTVCT")
20e2fc42312f9 ("ARM: 8930/1: Add support for generic vDSO")
31fdcac07f679 ("arm64: Introduce asm/vdso/clocksource.h")
5e3c6a312a094 ("ARM/arm64: vdso: Use common vdso clock mode storage")v4.19.131: Failed to apply! Possible dependencies:
31fdcac07f679 ("arm64: Introduce asm/vdso/clocksource.h")
5e3c6a312a094 ("ARM/arm64: vdso: Use common vdso clock mode storage") 6bf752daca07c ("powerpc: implement CONFIG_DEBUG_VIRTUAL")
942fa985e9f16 ("32-bit userspace ABI: introduce ARCH_32BIT_OFF_T config option") aef0f78e7460c ("binfmt_flat: add a ARCH_HAS_BINFMT_FLAT option") bdd15a288492f ("binfmt_flat: replace flat_argvp_envp_on_stack with a Kconfig variable") c32e64e852f3f ("csky: Build infrastructure")
e0a9317d90042 ("hexagon: use generic dma_noncoherent_ops")
ea6a37373f9ac ("MIPS: Avoid FP ELF checks when CONFIG_MIPS_FP_SUPPORT=n") f406f222d4b21 ("hexagon: implement the sync_sg_for_device DMA operation")
v4.14.187: Failed to apply! Possible dependencies:
104daea149c45 ("kconfig: reference environment variables directly and remove 'option env='")
18492685e479f ("kconfig: use yylineno option instead of manual lineno increments")
21c54b7747447 ("kconfig: show compiler version text in the top comment")
31fdcac07f679 ("arm64: Introduce asm/vdso/clocksource.h")
32a94b8b0c3e5 ("kconfig: remove duplicated file name and lineno of recursive inclusion")
379a8eb8eb1a5 ("kconfig: detect recursive inclusion earlier")
5ae6fcc4bb82b ("kconfig: fix line number in recursive inclusion error message")
5e3c6a312a094 ("ARM/arm64: vdso: Use common vdso clock mode storage")
79b05c1f31e2e ("um: stop abusing KBUILD_KCONFIG")
7f5c1ea3b707f ("c6x: use generic dma_noncoherent_ops")
87a4c375995ed ("kconfig: include kernel/Kconfig.preempt from init/Kconfig")
942fa985e9f16 ("32-bit userspace ABI: introduce ARCH_32BIT_OFF_T config option")
9e3e10c725360 ("kconfig: send error messages to stderr")
c32e64e852f3f ("csky: Build infrastructure")
cd81fc82b93fa ("kconfig: add xstrdup() helper")
d717f24d8c680 ("kconfig: add xrealloc() helper")
e0a9317d90042 ("hexagon: use generic dma_noncoherent_ops")
e2c75e7667c73 ("kconfig: tests: test if recursive inclusion is detected")
e71ea3badae55 ("nds32: Build infrastructure")
f163977d21a2b ("um: cleanup Kconfig files")
fbe934d69eb7e ("RISC-V: Build Infrastructure")v4.9.229: Failed to apply! Possible dependencies:
2fbadc3002c5f ("arm/arm64: xen: Move shared architecture headers to include/xen/arm")
31fdcac07f679 ("arm64: Introduce asm/vdso/clocksource.h")
5299709d0a873 ("treewide: Constify most dma_map_ops structures")
5e3c6a312a094 ("ARM/arm64: vdso: Use common vdso clock mode storage")
7f5c1ea3b707f ("c6x: use generic dma_noncoherent_ops")
942fa985e9f16 ("32-bit userspace ABI: introduce ARCH_32BIT_OFF_T config option")
eb17726b00b32 ("m32r: add simple dma")v4.4.229: Failed to apply! Possible dependencies:
0cb0786bac159 ("ARM64: PCI: Support ACPI-based PCI host controller")
1a4f93f7112fd ("PCI: Factor DT-specific pci_bus_find_domain_nr() code out")
1bd37a6835bef ("iommu/arm-smmu: Workaround for ThunderX erratum #27704")
1d8f51d41fc71 ("arm/arm64: arch_timer: Use archdata to indicate vdso suitability")
21266be9ed542 ("arch: consolidate CONFIG_STRICT_DEVM in lib/Kconfig.debug")
2ab51ddeca2fc ("ARM64: PCI: Add acpi_pci_bus_find_domain_nr()")
31fdcac07f679 ("arm64: Introduce asm/vdso/clocksource.h")
46fd5c6b30594 ("clocksource/drivers/arm_arch_timer: Control the evtstrm via the cmdline")
4e3e9b6997b24 ("iommu/arm-smmu: Add support for 16 bit VMID")
5e3c6a312a094 ("ARM/arm64: vdso: Use common vdso clock mode storage")
75df1386557c2 ("iommu/arm-smmu: Invalidate TLBs properly")
942fa985e9f16 ("32-bit userspace ABI: introduce ARCH_32BIT_OFF_T config option")
9c7cb891ecfea ("PCI: Refactor pci_bus_assign_domain_nr() for CONFIG_PCI_DOMAINS_GENERIC")
cd5f22d7967f6 ("arm64: arch_timer: simplify accessors")
f6dc1576cd517 ("arm64: arch_timer: Work around QorIQ Erratum A-008585")NOTE: The patch will not be queued to stable trees until it is upstream.
How should we proceed with this patch?
Signature · 2 lines
Thanks Sasha
Marc Zyngier
maintainerPATCH v2 2/4 arm64: arch_timer: Allow an workaround descriptor to disable compat vdso
In reply to Marc ZyngierAs we are about to disable the vdso for compat tasks in some circumstances, let's allow a workaround descriptor to express exactly that.
--- arch/arm64/include/asm/arch_timer.h | 1 + drivers/clocksource/arm_arch_timer.c | 3 +++ 2 files changed, 4 insertions(+)
arch/arm64/include/asm/arch_timer.h +1 -0
@@ -58,6 +58,7 @@ struct arch_timer_erratum_workaround {u64 (*read_cntvct_el0)(void);int (*set_next_event_phys)(unsigned long ,struct clock_event_device *);int (*set_next_event_virt)(unsigned long ,struct clock_event_device *); +bool disable_compat_vdso; }; DECLARE_PER_CPU(const struct arch_timer_erratum_workaround *,
drivers/clocksource/arm_arch_timer.c +3 -0
@@ -566,6 +566,9 @@ void arch_timer_enable_workaround(const struct arch_timer_erratum_workaround *waif (wa->read_cntvct_el0) { clocksource_counter.vdso_clock_mode =VDSO_CLOCKMODE_NONE ; vdso_default =VDSO_CLOCKMODE_NONE ; + }else if (wa->disable_compat_vdso && vdso_default !=VDSO_CLOCKMODE_NONE ) { + vdso_default =VDSO_CLOCKMODE_ARCHTIMER_NOCOMPAT ; + clocksource_counter.vdso_clock_mode = vdso_default; } }
Signature · 1 line
2.27.0
Re: [PATCH v2 2/4] arm64: arch_timer: Allow an workaround descriptor to disable compat vdso
Quoted 5 lines · Marc Zyngier
As we are about to disable the vdso for compat tasks in some circumstances, let's allow a workaround descriptor to express exactly that.
Signed-off-by Marc Zyngier <[email protected]>
Mark.
Quoted 34 lines
--- arch/arm64/include/asm/arch_timer.h | 1 + drivers/clocksource/arm_arch_timer.c | 3 +++ 2 files changed, 4 insertions(+)arch/arm64/include/asm/arch_timer.h +1
-0@@ -58,6 +58,7 @@ struct arch_timer_erratum_workaround {u64 (*read_cntvct_el0)(void);int (*set_next_event_phys)(unsigned long ,struct clock_event_device *);int (*set_next_event_virt)(unsigned long ,struct clock_event_device *); +bool disable_compat_vdso; }; DECLARE_PER_CPU(const struct arch_timer_erratum_workaround *,drivers/clocksource/arm_arch_timer.c +3
-0@@ -566,6 +566,9 @@ void arch_timer_enable_workaround(const struct arch_timer_erratum_workaround *waif (wa->read_cntvct_el0) { clocksource_counter.vdso_clock_mode =VDSO_CLOCKMODE_NONE ; vdso_default =VDSO_CLOCKMODE_NONE ; + }else if (wa->disable_compat_vdso && vdso_default !=VDSO_CLOCKMODE_NONE ) { + vdso_default =VDSO_CLOCKMODE_ARCHTIMER_NOCOMPAT ; + clocksource_counter.vdso_clock_mode = vdso_default; } }Signature · 1 line
2.27.0
Marc Zyngier
maintainerPATCH v2 3/4 arm64: arch_timer: Disable the compat vdso for cores affected by ARM64_WORKAROUND_1418040
In reply to Marc ZyngierARM64_WORKAROUND_1418040 requires that AArch32 EL0 accesses to the virtual counter register are trapped and emulated by the kernel. This makes the vdso pretty pointless, and in some cases livelock prone.
Provide a workaround entry that limits the vdso to 64bit tasks.
--- drivers/clocksource/arm_arch_timer.c | 8 ++++++++ 1 file changed, 8 insertions(+)
drivers/clocksource/arm_arch_timer.c +8 -0
@@ -480,6 +480,14 @@ static const struct arch_timer_erratum_workaround ool_workarounds[] = { .set_next_event_virt = erratum_set_next_event_tval_virt, },#endif +#ifdef CONFIG_ARM64_ERRATUM_1418040 + { + .match_type = ate_match_local_cap_id, + .id = (void *)ARM64_WORKAROUND_1418040, + .desc = "ARM erratum 1418040", + .disable_compat_vdso = true, + }, +#endif }; typedef bool (*ate_match_fn_t)(const struct arch_timer_erratum_workaround *,
Signature · 1 line
2.27.0
Re: [PATCH v2 3/4] arm64: arch_timer: Disable the compat vdso for cores affected by ARM64_WORKAROUND_1418040
Quoted 9 lines · Marc Zyngier
ARM64_WORKAROUND_1418040requires that AArch32 EL0 accesses to the virtual counter register are trapped and emulated by the kernel. This makes the vdso pretty pointless, and in some cases livelock prone.Provide a workaround entry that limits the vdso to 64bit tasks.
Signed-off-by Marc Zyngier <[email protected]>
Mark.
Quoted 26 lines
--- drivers/clocksource/arm_arch_timer.c | 8 ++++++++ 1 file changed, 8 insertions(+)drivers/clocksource/arm_arch_timer.c +8
-0@@ -480,6 +480,14 @@ static const struct arch_timer_erratum_workaround ool_workarounds[] = { .set_next_event_virt = erratum_set_next_event_tval_virt, },#endif +#ifdef CONFIG_ARM64_ERRATUM_1418040 + { + .match_type = ate_match_local_cap_id, + .id = (void *)ARM64_WORKAROUND_1418040, + .desc = "ARM erratum 1418040", + .disable_compat_vdso = true, + }, +#endif }; typedef bool (*ate_match_fn_t)(const struct arch_timer_erratum_workaround *,Signature · 1 line
2.27.0
Marc Zyngier
maintainerPATCH v2 4/4 arm64: Rework ARM_ERRATUM_1414080 handling
In reply to Marc ZyngierThe current handling of erratum 1414080 has the side effect that cntkctl_el1 can get changed for both 32 and 64bit tasks.
This isn't a problem so far, but if we ever need to mitigate another of these errata on the 64bit side, we'd better keep the messing with cntkctl_el1 local to 32bit tasks.
For that, make sure that on entering the kernel from a 32bit tasks, userspace access to cntvct gets enabled, and disabled returning to userspace, while it never gets changed for 64bit tasks.
--- arch/arm64/kernel/entry.S | 40 +++++++++++++++++++++++---------------- 1 file changed, 24 insertions(+), 16 deletions(-)
arch/arm64/kernel/entry.S +24 -16
@@ -167,6 +167,19 @@ alternative_cb_end stp x28, x29, [sp, #16 * 14] .if \el == 0 + .if \regsize == 32 + // If we come back from a 32bit task on a system affected by + // 1418040, let's reenable userspace access to the virtual counter. +#ifdef CONFIG_ARM64_ERRATUM_1418040 +alternative_if_not ARM64_WORKAROUND_1418040 + b .L__entry_skip_wa_1418040\@ +alternative_else_nop_endif + mrs x0, cntkctl_el1 + orr x0, x0, #2 // ARCH_TIMER_USR_VCT_ACCESS_EN + msr cntkctl_el1, x0 +.L__entry_skip_wa_1418040\@: +#endif + .endif clear_gp_regs mrs x21, sp_el0 ldr_this_cpu tsk, __entry_task, x20
@@ -318,7 +331,17 @@ alternative_else_nop_endif ldr x23, [sp, #S_SP] // load return stack pointer msr sp_el0, x23 tst x22, #PSR_MODE32_BIT // native task? - b.eq 3f + b.eq 4f + +#ifdef CONFIG_ARM64_ERRATUM_1418040 +alternative_if_not ARM64_WORKAROUND_1418040 + b 3f +alternative_else_nop_endif + mrs x1, cntkctl_el1 + bic x1, x1, #2 // ARCH_TIMER_USR_VCT_ACCESS_EN + msr cntkctl_el1, x1 +3: +#endif #ifdef CONFIG_ARM64_ERRATUM_845719 alternative_if ARM64_WORKAROUND_845719
@@ -330,22 +353,7 @@ alternative_if ARM64_WORKAROUND_845719 #endif alternative_else_nop_endif #endif -3: -#ifdef CONFIG_ARM64_ERRATUM_1418040 -alternative_if_not ARM64_WORKAROUND_1418040 - b 4f -alternative_else_nop_endif - /* - * if (x22.mode32 == cntkctl_el1.el0vcten) - * cntkctl_el1.el0vcten = ~cntkctl_el1.el0vcten - */ - mrs x1, cntkctl_el1 - eon x0, x1, x22, lsr #3 - tbz x0, #1, 4f - eor x1, x1, #2 // ARCH_TIMER_USR_VCT_ACCESS_EN - msr cntkctl_el1, x1 4: -#endif scs_save tsk, x0 /* No kernel C function calls after this as user keys are set. */
Signature · 1 line
2.27.0
Re: [PATCH v2 4/4] arm64: Rework ARM_ERRATUM_1414080 handling
Quoted 7 lines · Marc Zyngier
The current handling of erratum 1414080 has the side effect that
cntkctl_el1can get changed for both 32 and 64bit tasks.This isn't a problem so far, but if we ever need to mitigate another of these errata on the 64bit side, we'd better keep the messing with
cntkctl_el1local to 32bit tasks.
Quoted 5 lines
For that, make sure that on entering the kernel from a 32bit tasks, userspace access to cntvct gets enabled, and disabled returning to userspace, while it never gets changed for 64bit tasks.
Signed-off-by Marc Zyngier <[email protected]>
The asm looks sound to me. Suggestion below, but either way:
Quoted 14 lines
--- arch/arm64/kernel/entry.S | 40 +++++++++++++++++++++++---------------- 1 file changed, 24 insertions(+), 16 deletions(-)arch/arm64/kernel/entry.S +2
-0@@ -167,6 +167,19 @@ alternative_cb_end stp x28, x29, [sp, #16 * 14] .if \el == 0 + .if \regsize == 32 + // If we come back from a 32bit task on a system affected by
arch/arm64/kernel/entry.S alternative_cb_end · line 181
Quoted 10 lines
arch/arm64/kernel/entry.S +10
-0+ // 1418040, let's reenable userspace access to the virtual counter. +#ifdef CONFIG_ARM64_ERRATUM_1418040 +alternative_if_not ARM64_WORKAROUND_1418040 + b .L__entry_skip_wa_1418040\@ +alternative_else_nop_endif + mrs x0, cntkctl_el1 + orr x0, x0, #2 // ARCH_TIMER_USR_VCT_ACCESS_EN + msr cntkctl_el1, x0 +.L__entry_skip_wa_1418040\@: +#endif
Given this is only 3 instructions, it might be clearer to remove the branch and label and NOP the whole thing for legibility:
| #ifdef CONFIG_ARM64_ERRATUM_1418040 | alternative_if ARM64_WORKAROUND_1418040 | mrs x0. cntkctl_el1 | orr x0, x0, #2 // ARCH_TIMER_USR_VCT_ACCESS_EN | msr cntkctl_el1, x0 | alternative_else_nop_endif | #endif
... and likewise for the clearing case.
Regardless, this looks sound to me, so take your pick. :)
Mark.
Quoted 49 lines
arch/arm64/kernel/entry.S +12
-16+ .endif clear_gp_regs mrs x21, sp_el0 ldr_this_cpu tsk, __entry_task, x20@@ -318,7 +331,17 @@ alternative_else_nop_endif ldr x23, [sp, #S_SP] // load return stack pointer msr sp_el0, x23 tst x22, #PSR_MODE32_BIT // native task? - b.eq 3f + b.eq 4f + +#ifdef CONFIG_ARM64_ERRATUM_1418040 +alternative_if_not ARM64_WORKAROUND_1418040 + b 3f +alternative_else_nop_endif + mrs x1, cntkctl_el1 + bic x1, x1, #2 // ARCH_TIMER_USR_VCT_ACCESS_EN + msr cntkctl_el1, x1 +3: +#endif #ifdef CONFIG_ARM64_ERRATUM_845719 alternative_if ARM64_WORKAROUND_845719@@ -330,22 +353,7 @@ alternative_if ARM64_WORKAROUND_845719 #endif alternative_else_nop_endif #endif -3: -#ifdef CONFIG_ARM64_ERRATUM_1418040 -alternative_if_not ARM64_WORKAROUND_1418040 - b 4f -alternative_else_nop_endif - /* - * if (x22.mode32 == cntkctl_el1.el0vcten) - * cntkctl_el1.el0vcten = ~cntkctl_el1.el0vcten - */ - mrs x1, cntkctl_el1 - eon x0, x1, x22, lsr #3 - tbz x0, #1, 4f - eor x1, x1, #2 // ARCH_TIMER_USR_VCT_ACCESS_EN - msr cntkctl_el1, x1 4: -#endif scs_save tsk, x0 /* No kernel C function calls after this as user keys are set. */Signature · 1 line
2.27.0
Quoted 11 lines · Marc Zyngier
The relatively recent introduction of the compat vdso on arm64 has overlooked its interactions with some of the interesting errata workarounds, such as
ARM64_ERRATUM_1418040(and its older 1188873 incarnation).This erratum requires the 64bit kernel to trap 32bit accesses to the virtual counter and emulate it. When the workaround was introduced, the compat vdso simply wasn't a thing. Now that the patches have landed in mainline, we trap the CVTVCT accesses from the vdso.
[...]
Applied to arm64 (for-next/fixes), thanks!
[1/4] arm64: Introduce a way to disable the 32bit vdso
https://git.kernel.org/arm64/c/97884ca8c292 [2/4] arm64: arch_timer: Allow an workaround descriptor to disable compat vdso https://git.kernel.org/arm64/c/c1fbec4ac0d7 [3/4] arm64: arch_timer: Disable the compat vdso for cores affected by ARM64_WORKAROUND_1418040 https://git.kernel.org/arm64/c/4b661d6133c5 [4/4] arm64: Rework ARM_ERRATUM_1414080 handling https://git.kernel.org/arm64/c/dc802f2bc020
Cheers,
Signature · 5 lines
Will https://fixes.arm64.dev https://next.arm64.dev https://will.arm64.dev