Skip to contentSkip to conversation

powerpc: Fix kuap warnings

PATCH In review Shrikanth Hegde 3 participants 7 messages Jan 9 – Feb 17 PowerPC

lore
Share & tools
lore

Shrikanth Hegde

author reviewer
#0 lore raw

cover powerpc: Fix kuap warnings

Recently stumbled upon these kuap warnings. This happens with preempt=full/lazy kernel with function tracing enabled. What irked me was kernel compilation was getting failed when i had tracing enabled. It doesn't fail everytime. While running stress-ng memory class it threw same warnings. So that helped to narrow it down.

So one possible way is to disable tracing for these enter/exit vmx_usercopy. That seems to fix the bug/warnings. But that will make them as non trace-able. If there is a better way to fix these warning while keeping them as trace-able, please let me know.

Anyone with insights on amr, vmx and tracing, please advise.

Shrikanth Hegde (1):
powerpc: Fix kuap warnings on lazy/full preemption with tracing

 arch/powerpc/kernel/process.c | 10 +++++-----
 arch/powerpc/lib/vmx-helper.c |  4 ++--
 2 files changed, 7 insertions(+), 7 deletions(-)
Signature · 1 line
2.47.3
author reviewer
#1 lore raw

PATCH 1/1 powerpc: Fix kuap warnings on lazy/full preemption with tracing

These KUAP bugs/Warnings were seen often when tracing was enabled. It happens with preempt=full/lazy. It is easily hit.

How to trigger:
echo lazy > /sys/kernel/debug/sched/preempt echo function > /sys/kernel/debug/tracing/current_tracer stress-ng --class memory --all 1 -t 3

Bug: Write fault blocked by KUAP!
WARNING: [] arch/powerpc/mm/fault.c:231 at bad_kernel_fault.constprop.0+0x1a8/0x2c8, CPU#9: stress-ng-vm-rw/5477
NIP [c00000000008cdec] bad_kernel_fault.constprop.0+0x1a8/0x2c8

Call Trace:
 bad_kernel_fault.constprop.0+0x1a4/0x2c8 (unreliable)
 ___do_page_fault+0x688/0xa54
 do_page_fault+0x30/0x70
 data_access_common_virt+0x210/0x220
---- interrupt: 300 at __copy_tofrom_user_power7+0x410/0x7ac
NIP [c0000000000b3b44] __copy_tofrom_user_power7+0x410/0x7ac
LR [c0000000009a7d78] _copy_to_iter+0x134/0x9c4

Enabled CONFIG_PPC_KUAP_DEBUG=y, which prints out below warnings.

WARNING: ./arch/powerpc/include/asm/book3s/64/kup.h:93 at _switch+0x80/0x12c,
CPU#9: stress-ng-vm-rw/5477
NIP [c000000000013ce4] _switch+0x80/0x12c
LR [c00000000001f968] __switch_to+0x148/0x230
Call Trace:
__switch_to+0x148/0x230
__schedule+0x270/0x700
preempt_schedule_notrace+0x64/0xd8
function_trace_call+0x180/0x204
ftrace_call+0x4/0x4c
enter_vmx_usercopy+0x10/0x74
__copy_tofrom_user_power7+0x278/0x7ac
_copy_to_iter+0x134/0x9c4
copy_page_to_iter+0xe4/0x1c4
process_vm_rw_single_vec.constprop.0+0x1cc/0x3b4
process_vm_rw_core.constprop.0+0x168/0x30c
process_vm_rw+0x128/0x184
system_call_exception+0x128/0x390
system_call_vectored_common+0x15c/0x2ec

enter/exit vmx_usercopy clearly says it shouldn't call schedule. Doing so will end up corrupting AMR registers. When function tracer is enabled, the entry point, i.e enter_vmx_usercopy could be in preemptible context. First thing enter_vmx_usercopy does is, preempt_disable and again function exit of exit_vmx_usercopy maybe preemptible too. So make these as notrace to avoid these bug reports.

WARNING: [amr != AMR_KUAP_BLOCKED] ./arch/powerpc/include/asm/book3s/64/kup.h:293 at arch_local_irq_restore.part.0+0x1e8/0x224, CPU#15: stress-ng-pipe/11623 NIP [c000000000038830] arch_local_irq_restore.part.0+0x1e8/0x224
LR [c00000000003871c] arch_local_irq_restore.part.0+0xd4/0x224

Call Trace:
return_to_handler+0x0/0x4c (unreliable)
__rb_reserve_next+0x198/0x4f8
ring_buffer_lock_reserve+0x1a8/0x51c
trace_buffer_lock_reserve+0x30/0x80
__graph_entry.isra.0+0x118/0x140
function_graph_enter_regs+0x1ec/0x408
ftrace_graph_func+0x50/0xcc
ftrace_call+0x4/0x4c
enable_kernel_altivec+0x10/0xd0
enter_vmx_usercopy+0x58/0x74
return_to_handler+0x0/0x4c (__copy_tofrom_user_power7+0x278/0x7ac)
_copy_from_iter+0x134/0x9bc
copy_page_from_iter+0xd4/0x1a0

Since AMR registers aren't set to BLOCKED state, warnings could be seen if there is any unlock involved, which gets triggered via arch_local_irq_restore. So had to for that enable_kernel_altivec too. Similarly for check_if_tm_restore_required, giveup_altivec.

---
 arch/powerpc/kernel/process.c | 10 +++++-----
 arch/powerpc/lib/vmx-helper.c |  4 ++--
 2 files changed, 7 insertions(+), 7 deletions(-)
arch/powerpc/kernel/process.c +5 -5
index a45fe147868b..7bf2fe3e5878 100644
@@ -80,7 +80,7 @@
  */
 bool tm_suspend_disabled __ro_after_init = false;
 
-static void check_if_tm_restore_required(struct task_struct *tsk)
+notrace static void check_if_tm_restore_required(struct task_struct *tsk)
 {
 	/*
 	 * If we are saving the current thread's registers, and the
@@ -98,7 +98,7 @@ static void check_if_tm_restore_required(struct task_struct *tsk)
 }
 
 #else
-static inline void check_if_tm_restore_required(struct task_struct *tsk) { }
+static __always_inline void check_if_tm_restore_required(struct task_struct *tsk) { }
 #endif /* CONFIG_PPC_TRANSACTIONAL_MEM */
 
 bool strict_msr_control;
@@ -231,7 +231,7 @@ static inline void __giveup_fpu(struct task_struct *tsk) { }
 #endif /* CONFIG_PPC_FPU */
 
 #ifdef CONFIG_ALTIVEC
-static void __giveup_altivec(struct task_struct *tsk)
+notrace static void __giveup_altivec(struct task_struct *tsk)
 {
 	unsigned long msr;
 
@@ -243,7 +243,7 @@ static void __giveup_altivec(struct task_struct *tsk)
 	regs_set_return_msr(tsk->thread.regs, msr);
 }
 
-void giveup_altivec(struct task_struct *tsk)
+notrace void giveup_altivec(struct task_struct *tsk)
 {
 	check_if_tm_restore_required(tsk);
 
@@ -253,7 +253,7 @@ void giveup_altivec(struct task_struct *tsk)
 }
 EXPORT_SYMBOL(giveup_altivec);
 
-void enable_kernel_altivec(void)
+notrace void enable_kernel_altivec(void)
 {
 	unsigned long cpumsr;
 
arch/powerpc/lib/vmx-helper.c +2 -2
index 54340912398f..a0c041c148e4 100644
@@ -10,7 +10,7 @@
 #include <linux/hardirq.h>
 #include <asm/switch_to.h>
 
-int enter_vmx_usercopy(void)
+notrace int enter_vmx_usercopy(void)
 {
 	if (in_interrupt())
 		return 0;
@@ -32,7 +32,7 @@ int enter_vmx_usercopy(void)
  * This function must return 0 because we tail call optimise when calling
  * from __copy_tofrom_user_power7 which returns 0 on success.
  */
-int exit_vmx_usercopy(void)
+notrace int exit_vmx_usercopy(void)
 {
 	disable_kernel_altivec();
 	pagefault_enable();
Signature · 1 line
2.47.3
#2 lore raw
In reply to Shrikanth Hegde

Le 09/01/2026 à 07:49, Shrikanth Hegde a écrit :

Quoted 6 lines

Recently stumbled upon these kuap warnings. This happens with preempt=full/lazy kernel with function tracing enabled. What irked me was kernel compilation was getting failed when i had tracing enabled. It doesn't fail everytime. While running stress-ng memory class it threw same warnings. So that helped to narrow it down.

Quoted 6 lines

So one possible way is to disable tracing for these enter/exit vmx_usercopy. That seems to fix the bug/warnings. But that will make them as non trace-able. If there is a better way to fix these warning while keeping them as trace-able, please let me know.

Anyone with insights on amr, vmx and tracing, please advise.

The main principle with KUAP is to not call subfunctions once userspace access enabled. There are a few exceptions like __copy_tofrom_user() that are allowed in order to optimise large copies. However this needs to be handled very carefully, and in principle we don't expect __copy_tofrom_user() to call other functions.

So it might require wider rework but we should narrow as much as possible the period during which access to userspace is opened, with something like:

raw_coy_to_user_power7()

{
	enter_vmx_usercopy();
	allow_write_to_user(to, n);
	ret = __copy_tofrom_user_power7();
	prevent_write_to_user(to, n);
	exit_vmx_usercopy();
	return ret;
}

raw_copy_to_user()

{
	if (cpu_has_feature(CPU_FTR_VMX_COPY))
		raw_copy_to_user_power7();

	allow_write_to_user(to, n);
	ret = __copy_tofrom_user(to, (__force const void __user *)from, n);
	prevent_write_to_user(to, n);
	return ret;
}
author reviewer
#3 lore raw

Hi Christophe.

Quoted 13 lines · Christophe Leroy (CS GROUP)

Le 09/01/2026 à 07:49, Shrikanth Hegde a écrit :

Quoted 10 lines

Recently stumbled upon these kuap warnings. This happens with preempt=full/lazy kernel with function tracing enabled. What irked me was kernel compilation was getting failed when i had tracing enabled. It doesn't fail everytime. While running stress-ng memory class it threw same warnings. So that helped to narrow it down.
So one possible way is to disable tracing for these enter/exit vmx_usercopy. That seems to fix the bug/warnings. But that will make them as non trace-able. If there is a better way to fix these warning while keeping them as trace-able, please let me know.

Quoted 4 lines
Quoted 1 line

Anyone with insights on amr, vmx and tracing, please advise.

The main principle with KUAP is to not call subfunctions once userspace access enabled. There are a few exceptions like __copy_tofrom_user() that are allowed in order to optimise large copies. However this needs to be handled very carefully, and in principle we don't expect __copy_tofrom_user() to call other functions.

I didn't understand. My knowledge is quite limited in this space. Could you please explain how this will help us avoid the warnings? or are you saying we have more callsites which needs to worked upon.

Quoted 5 lines

So it might require wider rework but we should narrow as much as possible the period during which access to userspace is opened, with something like:

raw_coy_to_user_power7()
{
    enter_vmx_usercopy();

I think the problem is when it comes here, it has some AMR state, but it is preemptible. So shouldn't call schedule IIUC.

Quoted 17 lines

    allow_write_to_user(to, n);
    ret = __copy_tofrom_user_power7();
    prevent_write_to_user(to, n);
    exit_vmx_usercopy();
    return ret;
}

raw_copy_to_user()
{
    if (cpu_has_feature(CPU_FTR_VMX_COPY))
        raw_copy_to_user_power7();

    allow_write_to_user(to, n);
    ret = __copy_tofrom_user(to, (__force const void __user *)from, n); prevent_write_to_user(to, n);
    return ret;
}

#4 lore raw

Hi Shrikanth,

Le 09/01/2026 à 13:19, Shrikanth Hegde a écrit :

Quoted 16 lines

Hi Christophe.

Quoted 13 lines · Christophe Leroy (CS GROUP)

Le 09/01/2026 à 07:49, Shrikanth Hegde a écrit :

Quoted 10 lines

Recently stumbled upon these kuap warnings. This happens with preempt=full/lazy kernel with function tracing enabled. What irked me was kernel compilation was getting failed when i had tracing enabled. It doesn't fail everytime. While running stress-ng memory class it threw same warnings. So that helped to narrow it down.
So one possible way is to disable tracing for these enter/exit vmx_usercopy. That seems to fix the bug/warnings. But that will make them as non trace-able. If there is a better way to fix these warning while keeping them as trace-able, please let me know.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Anyone with insights on amr, vmx and tracing, please advise.

The main principle with KUAP is to not call subfunctions once userspace access enabled. There are a few exceptions like __copy_tofrom_user() that are allowed in order to optimise large copies. However this needs to be handled very carefully, and in principle we don't expect __copy_tofrom_user() to call other functions.

I didn't understand. My knowledge is quite limited in this space. Could you please explain how this will help us avoid the warnings? or are you saying we have more callsites which needs to worked upon.

Read tools/objtool/Documentation/objtool.txt section "Objtool warning" item 9.

Unfortunately powerpc doesn't yet implement objtool to detect it, but the principle applies anyway.

Quoted 9 lines
Quoted 6 lines

So it might require wider rework but we should narrow as much as possible the period during which access to userspace is opened, with something like:

raw_coy_to_user_power7()
{
     enter_vmx_usercopy();

I think the problem is when it comes here, it has some AMR state, but it is preemptible. So shouldn't call schedule IIUC.

See commit 00ff1eaac129 ("powerpc: Fix reschedule bug in KUAP-unlocked user copy")

The problem is because enter_vmx_usercopy() is called _after_ allow_write_to_user() which changes AMR.

If you call enter_vmx_usercopy() _before_ allow_write_to_user() and call exit_vmx_usercopy() _after_ prevent_write_to_user() the problem is solved.

Quoted 19 lines
Quoted 18 lines

     allow_write_to_user(to, n);
     ret = __copy_tofrom_user_power7();
     prevent_write_to_user(to, n);
     exit_vmx_usercopy();
     return ret;
}

raw_copy_to_user()
{
     if (cpu_has_feature(CPU_FTR_VMX_COPY))
         raw_copy_to_user_power7();

     allow_write_to_user(to, n);
     ret = __copy_tofrom_user(to, (__force const void __user *)from, n); prevent_write_to_user(to, n);
     return ret;
}

author reviewer
#5 lore raw

Hi Christophe.

Quoted 39 lines · Christophe Leroy (CS GROUP)

Hi Shrikanth,

Le 09/01/2026 à 13:19, Shrikanth Hegde a écrit :

Quoted 16 lines

Hi Christophe.

Quoted 13 lines · Christophe Leroy (CS GROUP)

Le 09/01/2026 à 07:49, Shrikanth Hegde a écrit :

Quoted 10 lines

Recently stumbled upon these kuap warnings. This happens with preempt=full/lazy kernel with function tracing enabled. What irked me was kernel compilation was getting failed when i had tracing enabled. It doesn't fail everytime. While running stress-ng memory class it threw same warnings. So that helped to narrow it down.
So one possible way is to disable tracing for these enter/exit vmx_usercopy. That seems to fix the bug/warnings. But that will make them as non trace-able. If there is a better way to fix these warning while keeping them as trace-able, please let me know.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Anyone with insights on amr, vmx and tracing, please advise.

The main principle with KUAP is to not call subfunctions once userspace access enabled. There are a few exceptions like __copy_tofrom_user() that are allowed in order to optimise large copies. However this needs to be handled very carefully, and in principle we don't expect __copy_tofrom_user() to call other functions.

I didn't understand. My knowledge is quite limited in this space. Could you please explain how this will help us avoid the warnings? or are you saying we have more callsites which needs to worked upon.

Read tools/objtool/Documentation/objtool.txt section "Objtool warning" item 9.

Unfortunately powerpc doesn't yet implement objtool to detect it, but the principle applies anyway.

Quoted 7 lines
Quoted 6 lines

So it might require wider rework but we should narrow as much as possible the period during which access to userspace is opened, with something like:

raw_coy_to_user_power7()
{
     enter_vmx_usercopy();

Quoted 29 lines
Quoted 2 lines

I think the problem is when it comes here, it has some AMR state, but it is preemptible. So shouldn't call schedule IIUC.

See commit 00ff1eaac129 ("powerpc: Fix reschedule bug in KUAP-unlocked user copy")

The problem is because enter_vmx_usercopy() is called _after_ allow_write_to_user() which changes AMR.

If you call enter_vmx_usercopy() _before_ allow_write_to_user() and call exit_vmx_usercopy() _after_ prevent_write_to_user() the problem is solved.

Quoted 19 lines
Quoted 18 lines

     allow_write_to_user(to, n);
     ret = __copy_tofrom_user_power7();
     prevent_write_to_user(to, n);
     exit_vmx_usercopy();
     return ret;
}

raw_copy_to_user()
{
     if (cpu_has_feature(CPU_FTR_VMX_COPY))
         raw_copy_to_user_power7();

     allow_write_to_user(to, n);
     ret = __copy_tofrom_user(to, (__force const void __user *)from, n); prevent_write_to_user(to, n);
     return ret;
}

Sorry, I forgot to update.

I spoke to ritesh a while ago and someone with better knowledge in this area will work on a proper fix for this.

#6 lore raw
Quoted 5 lines · Shrikanth Hegde

Sorry, I forgot to update.

I spoke to ritesh a while ago and someone with better knowledge in this area will work on a proper fix for this.

Sorry about the delay in getting back to this. Sayali from our team has been looking into this. She is soon going to post the fix which is basd upon Christophe suggestion.

-ritesh