arm64: Allow erratum 1418040 for late CPUs
v2 cover arm64: Allow erratum 1418040 for late CPUs
Erratum 1418040 currently prevents a late CPU from booting if none of the early CPUs are affected by it. This is because the handling is implemented as alternatives, and we have already got rid of them by the time userspace onlines a new CPU.
A solution to this is to move everything into C code, and rely on static keys instead. Once this is done, the feature can be allowed for late CPUs.
Note that CPUs affected by 1418040 also tend to miss AArch32-EL1, meaning they cannot be used as late CPUs when KVM is enabled and that their sibblings have AArch32-EL1.
* From v1:
- Dropped check for kernel threads
- Added comment describing the switching logic
- Made the errata handling function __always_inline
Marc Zyngier (2):
arm64: Move handling of erratum 1418040 into C code
arm64: Allow booting of late CPUs affected by erratum 1418040
arch/arm64/kernel/cpu_errata.c | 2 ++ arch/arm64/kernel/entry.S | 21 -------------------- arch/arm64/kernel/process.c | 35 ++++++++++++++++++++++++++++++++++ 3 files changed, 37 insertions(+), 21 deletions(-)
Signature · 7 lines
2.27.0 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 1/2 arm64: Move handling of erratum 1418040 into C code
Instead of dealing with erratum 1418040 on each entry and exit, let's move the handling to __switch_to() instead, which has several advantages:
- It can be applied when it matters (switching between 32 and 64 bit tasks).
- It is written in C (yay!)
- It can rely on static keys rather than alternatives
--- arch/arm64/kernel/entry.S | 21 --------------------- arch/arm64/kernel/process.c | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+), 21 deletions(-)
arch/arm64/kernel/entry.S +0 -21
@@ -169,19 +169,6 @@ alternative_cb_end stp x28, x29, [sp, #16 * 14] .if \el == 0 - .if \regsize == 32 - /* - * If we're returning from a 32-bit task on a system affected by - * 1418040 then re-enable userspace access to the virtual counter. - */ -#ifdef CONFIG_ARM64_ERRATUM_1418040 -alternative_if ARM64_WORKAROUND_1418040 - mrs x0, cntkctl_el1 - orr x0, x0, #2 // ARCH_TIMER_USR_VCT_ACCESS_EN - msr cntkctl_el1, x0 -alternative_else_nop_endif -#endif - .endif clear_gp_regs mrs x21, sp_el0 ldr_this_cpu tsk, __entry_task, x20
@@ -337,14 +324,6 @@ alternative_else_nop_endif tst x22, #PSR_MODE32_BIT // native task? b.eq 3f -#ifdef CONFIG_ARM64_ERRATUM_1418040 -alternative_if ARM64_WORKAROUND_1418040 - mrs x0, cntkctl_el1 - bic x0, x0, #2 // ARCH_TIMER_USR_VCT_ACCESS_EN - msr cntkctl_el1, x0 -alternative_else_nop_endif -#endif - #ifdef CONFIG_ARM64_ERRATUM_845719 alternative_if ARM64_WORKAROUND_845719 #ifdef CONFIG_PID_IN_CONTEXTIDR
arch/arm64/kernel/process.c +35 -0
@@ -515,6 +515,40 @@ static void entry_task_switch(struct task_struct *next)__this_cpu_write (__entry_task, next); } +/* +* ARM erratum 1418040 handling, affecting the 32bit view of CNTVCT. +* Assuming the virtual counter is enabled at the beginning of times: +* +* - disable access when switching from a 64bit task to a 32bit task +* - enable access when switching from a 32bit task to a 64bit task +*/ +static __always_inline +void erratum_1418040_thread_switch(struct task_struct *prev, +struct task_struct *next) +{ +bool prev32, next32; +u64 val; + +if (!(IS_ENABLED (CONFIG_ARM64_ERRATUM_1418040 ) && +cpus_have_const_cap (ARM64_WORKAROUND_1418040 ))) +return ; + + prev32 =is_compat_thread (task_thread_info (prev)); + next32 =is_compat_thread (task_thread_info (next)); + +if (prev32 == next32) +return ; + + val =read_sysreg (cntkctl_el1); + +if (prev32 & !next32) + val |=ARCH_TIMER_USR_VCT_ACCESS_EN ; +else + val &= ~ARCH_TIMER_USR_VCT_ACCESS_EN ; + +write_sysreg (val, cntkctl_el1); +} +/* * Thread switching. */
@@ -530,6 +564,7 @@ __notrace_funcgraph struct task_struct *__switch_to(struct task_struct *prev,entry_task_switch (next);uao_thread_switch (next);ssbs_thread_switch (next); +erratum_1418040_thread_switch (prev, next);/* * Complete any pending TLB or cache maintenance on this CPU in case
Signature · 7 lines
2.27.0 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Will Deacon
maintainerRe: [PATCH v2 1/2] arm64: Move handling of erratum 1418040 into C code
Quoted 15 lines · Marc Zyngier
Instead of dealing with erratum 1418040 on each entry and exit, let's move the handling to
__switch_to()instead, which has several advantages:- It can be applied when it matters (switching between 32 and 64 bit tasks).
- It is written in C (yay!)
- It can rely on static keys rather than alternativesTested-by Sai Prakash Ranjan <[email protected]>Signed-off-by Marc Zyngier <[email protected]>--- arch/arm64/kernel/entry.S | 21 --------------------- arch/arm64/kernel/process.c | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+), 21 deletions(-)
[...]
Quoted 6 lines
arch/arm64/kernel/process.c +0
-0@@ -515,6 +515,40 @@ static void entry_task_switch(struct task_struct *next)__this_cpu_write (__entry_task, next);
arch/arm64/kernel/process.c static void entry_task_switch(struct task_struct *next) · line 525
Quoted 10 lines
arch/arm64/kernel/process.c +8
-0} +/* +* ARM erratum 1418040 handling, affecting the 32bit view of CNTVCT. +* Assuming the virtual counter is enabled at the beginning of times: +* +* - disable access when switching from a 64bit task to a 32bit task +* - enable access when switching from a 32bit task to a 64bit task +*/ +static __always_inline
Do we need the __always_inline? None of the other calls from __switch_to() have it.
Quoted 9 lines
arch/arm64/kernel/process.c +9
-0+void erratum_1418040_thread_switch (struct task_struct *prev, +struct task_struct *next) +{ +bool prev32, next32; +u64 val; + +if (!(IS_ENABLED (CONFIG_ARM64_ERRATUM_1418040 ) && +cpus_have_const_cap (ARM64_WORKAROUND_1418040 ))) +return ;
arch/arm64/kernel/process.c static void entry_task_switch(struct task_struct *next) · line 544
Quoted 10 lines
arch/arm64/kernel/process.c +10
-0+ + prev32 =is_compat_thread (task_thread_info (prev)); + next32 =is_compat_thread (task_thread_info (next)); + +if (prev32 == next32) +return ; + + val =read_sysreg (cntkctl_el1); + +if (prev32 & !next32)
I know they're bools but this is perverse!
Why can't it just be:
if (next32) val &= ~ARCH_TIMER_USR_VCT_ACCESS_EN; else val |= ARCH_TIMER_USR_VCT_ACCESS_EN;
?
Will
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v2 1/2] arm64: Move handling of erratum 1418040 into C code
Quoted 28 lines · Will Deacon
Quoted 15 lines · Marc Zyngier
Instead of dealing with erratum 1418040 on each entry and exit, let's move the handling to
__switch_to()instead, which has several advantages:- It can be applied when it matters (switching between 32 and 64 bit tasks).
- It is written in C (yay!)
- It can rely on static keys rather than alternativesTested-by Sai Prakash Ranjan <[email protected]>Signed-off-by Marc Zyngier <[email protected]>--- arch/arm64/kernel/entry.S | 21 --------------------- arch/arm64/kernel/process.c | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+), 21 deletions(-)[...]
Quoted 9 lines
arch/arm64/kernel/process.c +0
-0@@ -515,6 +515,40 @@ static void entry_task_switch(struct task_struct *next)__this_cpu_write (__entry_task, next); }
Quoted 11 lines
arch/arm64/kernel/process.c static void entry_task_switch(struct task_struct *next) · line 525
Quoted 8 lines
arch/arm64/kernel/process.c +8
-0+/* +* ARM erratum 1418040 handling, affecting the 32bit view of CNTVCT. +* Assuming the virtual counter is enabled at the beginning of times: +* +* - disable access when switching from a 64bit task to a 32bit task +* - enable access when switching from a 32bit task to a 64bit task +*/ +static __always_inlineDo we need the
__always_inline? None of the other calls from__switch_to()have it.
Suggestion from Stephen. In my experience, it doesn't change much as most things get inlined anyway. Happy to drop it.
Quoted 14 lines
Quoted 14 lines
arch/arm64/kernel/process.c +13
-0+void erratum_1418040_thread_switch (struct task_struct *prev, +struct task_struct *next) +{ +bool prev32, next32; +u64 val; + +if (!(IS_ENABLED (CONFIG_ARM64_ERRATUM_1418040 ) && +cpus_have_const_cap (ARM64_WORKAROUND_1418040 ))) +return ; + + prev32 =is_compat_thread (task_thread_info (prev)); + next32 =is_compat_thread (task_thread_info (next)); +
Quoted 8 lines
arch/arm64/kernel/process.c static void entry_task_switch(struct task_struct *next) · line 544
Quoted 6 lines
arch/arm64/kernel/process.c +6
-0+if (prev32 == next32) +return ; + + val =read_sysreg (cntkctl_el1); + +if (prev32 & !next32)I know they're bools but this is perverse!
Well, this is me writing this code, so don't attribute to perversity what can adequately be explained by a silly typo... ;-)
Quoted 8 lines
Why can't it just be:
if (next32) val &= ~ARCH_TIMER_USR_VCT_ACCESS_EN; else val |= ARCH_TIMER_USR_VCT_ACCESS_EN;?
Yup, that's better. Do you want to apply these changes directly, or should I repin it?
M.
Signature · 6 lines
Jazz is not dead. It just smells funny... _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Will Deacon
maintainerRe: [PATCH v2 1/2] arm64: Move handling of erratum 1418040 into C code
Quoted 24 lines · Marc Zyngier
Quoted 10 lines · Will Deacon
Quoted 9 lines · Marc Zyngier
arch/arm64/kernel/process.c +0
-0@@ -515,6 +515,40 @@ static void entry_task_switch(struct task_struct *next)__this_cpu_write (__entry_task ,next ); }Quoted 12 lines
arch/arm64/kernel/process.c static void entry_task_switch(struct task_struct *next) · line 525
Quoted 8 lines
arch/arm64/kernel/process.c +8
-0+/* +* ARM erratum 1418040 handling, affecting the 32bit view of CNTVCT. +* Assuming the virtual counter is enabled at the beginning of times: +* +* - disable access when switching from a 64bit task to a 32bit task +* - enable access when switching from a 32bit task to a 64bit task +*/ +static __always_inlineDo we need the
__always_inline? None of the other calls from__switch_to()
have it.
Quoted 2 lines
Suggestion from Stephen. In my experience, it doesn't change much as most things get inlined anyway. Happy to drop it.
Yes, please. We can add it back if it's shown to be a problem.
Quoted 32 lines
Quoted 13 lines
Quoted 13 lines
arch/arm64/kernel/process.c +13
-0+void erratum_1418040_thread_switch (struct task_struct *prev, +struct task_struct *next) +{ +bool prev32, next32; +u64 val; + +if (!(IS_ENABLED (CONFIG_ARM64_ERRATUM_1418040 ) && +cpus_have_const_cap (ARM64_WORKAROUND_1418040 ))) +return ; + + prev32 =is_compat_thread (task_thread_info (prev)); + next32 =is_compat_thread (task_thread_info (next)); +Quoted 8 lines
arch/arm64/kernel/process.c static void entry_task_switch(struct task_struct *next) · line 544
Quoted 6 lines
arch/arm64/kernel/process.c +6
-0+if (prev32 == next32) +return ; + + val =read_sysreg (cntkctl_el1); + +if (prev32 & !next32)I know they're bools but this is perverse!
Well, this is me writing this code, so don't attribute to perversity what can adequately be explained by a silly typo... ;-)
Quoted 7 lines
Why can't it just be:
if (next32) val &= ~ARCH_TIMER_USR_VCT_ACCESS_EN; else val |= ARCH_TIMER_USR_VCT_ACCESS_EN;
Quoted 4 lines
Quoted 1 line
?
Yup, that's better. Do you want to apply these changes directly, or should I repin it?
If you don't mind respinning, that would be best please. You can add my Ack as well, as I think this is one for Catalin now (I don't know if we're getting an -rc8 or not).
Will
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v2 1/2] arm64: Move handling of erratum 1418040 into C code
Quoting Will Deacon (2020-07-31 04:32:28)
Quoted 25 lines · Marc Zyngier
Quoted 24 lines · Will Deacon
Quoted 22 lines · Marc Zyngier
Quoted 7 lines
arch/arm64/kernel/process.c +0
-0@@ -515,6 +515,40 @@ static void entry_task_switch(struct task_struct *next)__this_cpu_write (__entry_task ,next );arch/arm64/kernel/process.c static void entry_task_switch(struct · line 526
Quoted 10 lines
arch/arm64/kernel/process.c +8
-0} +/* +* ARM erratum 1418040 handling, affecting the 32bit view of CNTVCT. +* Assuming the virtual counter is enabled at the beginning of times: +* +* - disable access when switching from a 64bit task to a 32bit task +* - enable access when switching from a 32bit task to a 64bit task +*/ +static __always_inlineDo we need the
__always_inline? None of the other calls from__switch_to()
have it.
Quoted 4 lines
Quoted 2 lines
Suggestion from Stephen. In my experience, it doesn't change much as most things get inlined anyway. Happy to drop it.
Yes, please. We can add it back if it's shown to be a problem.
Just for my own edification, why is __always_inline undesirable? Should there be an always inline version of the function that has the static key so that the erratum path is kept out of the switch_to() default path?
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Catalin Marinas
maintainerRe: [PATCH v2 1/2] arm64: Move handling of erratum 1418040 into C code
Quoted 29 lines · Stephen Boyd
Quoting Will Deacon (2020-07-31 04:32:28)
Quoted 28 lines · Marc Zyngier
Quoted 26 lines · Will Deacon
Quoted 10 lines · Marc Zyngier
Quoted 9 lines
arch/arm64/kernel/process.c +0
-0@@ -515,6 +515,40 @@ static void entry_task_switch(struct task_struct *next)__this_cpu_write (__entry_task ,next ); }Quoted 12 lines
arch/arm64/kernel/process.c static void entry_task_switch(struct task_struct *next) · line 525
Quoted 8 lines
arch/arm64/kernel/process.c +8
-0+/* +* ARM erratum 1418040 handling, affecting the 32bit view of CNTVCT. +* Assuming the virtual counter is enabled at the beginning of times: +* +* - disable access when switching from a 64bit task to a 32bit task +* - enable access when switching from a 32bit task to a 64bit task +*/ +static __always_inlineDo we need the
__always_inline? None of the other calls from__switch_to()
have it.Suggestion from Stephen. In my experience, it doesn't change much as most things get inlined anyway. Happy to drop it.
Quoted 6 lines
Quoted 1 line
Yes, please. We can add it back if it's shown to be a problem.
Just for my own edification, why is
__always_inlineundesirable? Should there be an always inline version of the function that has the static key so that the erratum path is kept out of theswitch_to()default path?
It's rather unnecessary, so it just makes the code a bit more unreadable. I usually go by two questions:
1. Is the function required to be inlined for correct execution?
2. Is there a visible performance benefit by using __always_inline? (e.g. the compiler does a bad job)
For a static function called only in one place, the compiler usually inlines it.
I can't speak on behalf of Will but from my perspective I'm against adding __always_inline just because it's harmless (and occasionally I edit it out of patches I apply manually).
Signature · 6 lines
Catalin _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 2/2 arm64: Allow booting of late CPUs affected by erratum 1418040
In reply to Marc ZyngierAs we can now switch from a system that isn't affected by 1418040 to a system that globally is affected, let's allow affected CPUs to come in at a later time.
--- arch/arm64/kernel/cpu_errata.c | 2 ++ 1 file changed, 2 insertions(+)
arch/arm64/kernel/cpu_errata.c +2 -0
@@ -910,6 +910,8 @@ const struct arm64_cpu_capabilities arm64_errata[] = { .desc = "ARM erratum 1418040", .capability =ARM64_WORKAROUND_1418040 ,ERRATA_MIDR_RANGE_LIST (erratum_1418040_list), + .type = (ARM64_CPUCAP_SCOPE_LOCAL_CPU | +ARM64_CPUCAP_PERMITTED_FOR_LATE_CPU ), }, #endif #ifdef CONFIG_ARM64_WORKAROUND_SPECULATIVE_AT
Signature · 7 lines
2.27.0 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel