Skip to contentSkip to conversation

Regression on linux-next (next-20250721)

DISCUSSION Borah, Chaitanya Kumar 3 participants 5 messages Jul 28, 2025 – Jul 31, 2025

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hello John,

Hope you are doing well. I am Chaitanya from the linux graphics team in Intel.

This mail is regarding a regression we are seeing in our CI runs[1] on linux-next repository.

Since the version next-20250721 [2], we are seeing the following regression

`````````````````````````````````````````````````````````````````````````````````
<3>[   10.781401] INFO: trying to register non-static key.
<3>[   10.781411] The code is fine but needs lockdep annotation, or maybe
<3>[   10.781414] you didn't initialize this object before use?
<3>[   10.781416] turning off the locking correctness validator.
<4>[   10.781420] CPU: 1 UID: 0 PID: 745 Comm: rsyslogd Not tainted 6.16.0-rc7-next-20250721-next-20250721-g979875200256+ #1 PREEMPT(voluntary)
<4>[   10.781422] Hardware name: Intel Corporation Arrow Lake Client Platform/ARL-H Lp5x T4 RVP, BIOS MTLPFWI1.R00.5204.D80.2505151204 05/15/2025
<4>[   10.781423] Call Trace:
<4>[   10.781424]  <TASK>
<4>[   10.781425]  dump_stack_lvl+0x91/0xf0
<4>[   10.781430]  dump_stack+0x10/0x20
<4>[   10.781431]  register_lock_class+0x491/0x4a0
<4>[   10.781434]  ? prepend_path+0x330/0x510
<4>[   10.781438]  ? prepend_path+0x8f/0x510
<4>[   10.781440]  __lock_acquire+0xa8/0x2650
<4>[   10.781444]  lock_acquire+0xc4/0x2e0
<4>[   10.781445]  ? update_file_ctx+0x1e/0x110
<4>[   10.781450]  _raw_spin_lock+0x2f/0x60
<4>[   10.781452]  ? update_file_ctx+0x1e/0x110
<4>[   10.781453]  update_file_ctx+0x1e/0x110
<4>[   10.781455]  aa_file_perm+0x68f/0x7f0
<4>[   10.781457]  common_file_perm+0x88/0x150
<4>[   10.781460]  apparmor_mmap_file+0x5f/0x80
<4>[   10.781462]  security_mmap_file+0x1b0/0x490
<4>[   10.781465]  vm_mmap_pgoff+0x5d/0x220
<4>[   10.781468]  ksys_mmap_pgoff+0x17a/0x250
<4>[   10.781471]  __x64_sys_mmap+0x33/0x70
<4>[   10.781474]  x64_sys_call+0x1eda/0x2680
<4>[   10.781477]  do_syscall_64+0x93/0xa20
<4>[   10.781480]  ? irqentry_exit+0x77/0xb0
<4>[   10.781482]  ? exc_page_fault+0xbd/0x2c0
<4>[   10.781484]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
`````````````````````````````````````````````````````````````````````````````````

Details log can be found in [3].

After bisecting the tree, the following patch [4] seems to be the first "bad" commit

`````````````````````````````````````````````````````````````````````````````````````````````````````````
commit 88fec3526e84123997ecebd6bb6778eb4ce779b7

Author: John Johansen [email protected]

Date:   Thu Jun 19 22:11:52 2025 -0700


    apparmor: make sure unix socket labeling is correctly updated.
`````````````````````````````````````````````````````````````````````````````````````````````````````````

We also verified that if we revert the patch the issue is not seen.

Could you please check why the patch causes this regression and provide a fix if necessary?

Thank you.

Regards

Chaitanya

[1]
https://intel-gfx-ci.01.org/tree/linux-next/combined-alt.html?
[2] https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20250721
[3]
https://intel-gfx-ci.01.org/tree/linux-next/next-20250721/bat-arlh-2/boot0.txt
[4] https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20250721&id=88fec3526e84123997ecebd6bb6778eb4ce779b7

maintainer
#1 lore raw
Quoted 57 lines · Chaitanya Kumar

Hello John,

Hope you are doing well. I am Chaitanya from the linux graphics team in Intel.

This mail is regarding a regression we are seeing in our CI runs[1] on linux-next repository.

Since the version next-20250721 [2], we are seeing the following regression

`````````````````````````````````````````````````````````````````````````````````
<3>[   10.781401] INFO: trying to register non-static key.
<3>[   10.781411] The code is fine but needs lockdep annotation, or maybe
<3>[   10.781414] you didn't initialize this object before use?
<3>[   10.781416] turning off the locking correctness validator.
<4>[   10.781420] CPU: 1 UID: 0 PID: 745 Comm: rsyslogd Not tainted 6.16.0-rc7-next-20250721-next-20250721-g979875200256+ #1 PREEMPT(voluntary)
<4>[   10.781422] Hardware name: Intel Corporation Arrow Lake Client Platform/ARL-H Lp5x T4 RVP, BIOS MTLPFWI1.R00.5204.D80.2505151204 05/15/2025
<4>[   10.781423] Call Trace:
<4>[   10.781424]  <TASK>
<4>[   10.781425]  dump_stack_lvl+0x91/0xf0
<4>[   10.781430]  dump_stack+0x10/0x20
<4>[   10.781431]  register_lock_class+0x491/0x4a0
<4>[   10.781434]  ? prepend_path+0x330/0x510
<4>[   10.781438]  ? prepend_path+0x8f/0x510
<4>[   10.781440]  __lock_acquire+0xa8/0x2650
<4>[   10.781444]  lock_acquire+0xc4/0x2e0
<4>[   10.781445]  ? update_file_ctx+0x1e/0x110
<4>[   10.781450]  _raw_spin_lock+0x2f/0x60
<4>[   10.781452]  ? update_file_ctx+0x1e/0x110
<4>[   10.781453]  update_file_ctx+0x1e/0x110
<4>[   10.781455]  aa_file_perm+0x68f/0x7f0
<4>[   10.781457]  common_file_perm+0x88/0x150
<4>[   10.781460]  apparmor_mmap_file+0x5f/0x80
<4>[   10.781462]  security_mmap_file+0x1b0/0x490
<4>[   10.781465]  vm_mmap_pgoff+0x5d/0x220
<4>[   10.781468]  ksys_mmap_pgoff+0x17a/0x250
<4>[   10.781471]  __x64_sys_mmap+0x33/0x70
<4>[   10.781474]  x64_sys_call+0x1eda/0x2680
<4>[   10.781477]  do_syscall_64+0x93/0xa20
<4>[   10.781480]  ? irqentry_exit+0x77/0xb0
<4>[   10.781482]  ? exc_page_fault+0xbd/0x2c0
<4>[   10.781484]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
`````````````````````````````````````````````````````````````````````````````````

Details log can be found in [3].

After bisecting the tree, the following patch [4] seems to be the first "bad" commit

`````````````````````````````````````````````````````````````````````````````````````````````````````````
commit 88fec3526e84123997ecebd6bb6778eb4ce779b7

Author: John Johansen [email protected]

Date:   Thu Jun 19 22:11:52 2025 -0700


     apparmor: make sure unix socket labeling is correctly updated.
`````````````````````````````````````````````````````````````````````````````````````````````````````````
Quoted 3 lines

We also verified that if we revert the patch the issue is not seen.

Could you please check why the patch causes this regression and provide a fix if necessary?

yep thanks, fix made. That patch accidentally removed the spinlock initialization from apparmor_file_alloc_security() while testing spin lock changes to the very similar apparmor_sk_alloc_security(), and it wasn't caught :(

anyways fix is being pushed

Quoted 13 lines

Thank you.

Regards

Chaitanya

[1]
https://intel-gfx-ci.01.org/tree/linux-next/combined-alt.html?
[2] https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20250721
[3]
https://intel-gfx-ci.01.org/tree/linux-next/next-20250721/bat-arlh-2/boot0.txt
[4] https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20250721&id=88fec3526e84123997ecebd6bb6778eb4ce779b7

#2 lore raw
Quoted 15 lines · John Johansen
Quoted 15 lines

After bisecting the tree, the following patch [4] seems to be the first "bad" commit

`````````````````````````````````````````````````````````````````````````````````````````````````````````
commit 88fec3526e84123997ecebd6bb6778eb4ce779b7

Author: John Johansen [email protected]

Date:   Thu Jun 19 22:11:52 2025 -0700


     apparmor: make sure unix socket labeling is correctly updated.
`````````````````````````````````````````````````````````````````````````````````````````````````````````

We also verified that if we revert the patch the issue is not seen.

Quoted 7 lines
Quoted 1 line

Could you please check why the patch causes this regression and provide a fix if necessary?

yep thanks, fix made. That patch accidentally removed the spinlock initialization from apparmor_file_alloc_security() while testing spin lock changes to the very similar apparmor_sk_alloc_security(), and it wasn't caught 🙁

anyways fix is being pushed

Thank you John for your response. Kindly share the patch whenever it is available. So that we can verify the fix.

Regards

Chaitanya

maintainer
#3 lore raw
Quoted 20 lines · Chaitanya Kumar
Quoted 17 lines · John Johansen
Quoted 16 lines

After bisecting the tree, the following patch [4] seems to be the first "bad" commit

`````````````````````````````````````````````````````````````````````````````````````````````````````````
commit 88fec3526e84123997ecebd6bb6778eb4ce779b7

Author: John Johansen [email protected]

Date:   Thu Jun 19 22:11:52 2025 -0700


     apparmor: make sure unix socket labeling is correctly updated.
`````````````````````````````````````````````````````````````````````````````````````````````````````````

We also verified that if we revert the patch the issue is not seen.

Could you please check why the patch causes this regression and provide a fix if necessary?

Quoted 8 lines
Quoted 5 lines

yep thanks, fix made. That patch accidentally removed the spinlock initialization from apparmor_file_alloc_security() while testing spin lock changes to the very similar apparmor_sk_alloc_security(), and it wasn't caught 🙁

anyways fix is being pushed

Thank you John for your response. Kindly share the patch whenever it is available. So that we can verify the fix.

in apparmor-next it is
43584e993293 apparmor: fix Regression on linux-next (next-20250721)

but also included below

From 43584e993293326cfc508e664fe81f56a65f6240 Mon Sep 17 00:00:00 2001
From: John Johansen <[email protected]>
Date: Wed, 30 Jul 2025 03:47:07 -0700
Subject: [PATCH] apparmor: fix Regression on linux-next (next-20250721)

sk lock initialization was incorrectly removed, from apparmor_file_alloc_security() while testing changes to changes to apparmor_sk_alloc_security()

resulting in the following regression.

[   48.056654] INFO: trying to register non-static key.
[   48.057480] The code is fine but needs lockdep annotation, or maybe
[   48.058416] you didn't initialize this object before use?
[   48.059209] turning off the locking correctness validator.
[   48.060040] CPU: 0 UID: 0 PID: 648 Comm: chronyd Not tainted 6.16.0-rc7-test-next-20250721-11410-g1ee809985e11-dirty #577 NONE
[   48.060049] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[   48.060055] Call Trace:
[   48.060059]  <TASK>
[   48.060063] dump_stack_lvl (lib/dump_stack.c:122)
[   48.060075] register_lock_class (kernel/locking/lockdep.c:988 kernel/locking/lockdep.c:1302)
[   48.060084] ? path_name (security/apparmor/file.c:159)
[   48.060093] __lock_acquire (kernel/locking/lockdep.c:5116)
[   48.060103] lock_acquire (kernel/locking/lockdep.c:473 (discriminator 4) kernel/locking/lockdep.c:5873 (discriminator 4) kernel/locking/lockdep.c:5828 (discriminator 4))
[   48.060109] ? update_file_ctx (security/apparmor/file.c:464)
[   48.060115] ? __pfx_profile_path_perm (security/apparmor/file.c:247)
[   48.060121] _raw_spin_lock (include/linux/spinlock_api_smp.h:134 kernel/locking/spinlock.c:154)
[   48.060130] ? update_file_ctx (security/apparmor/file.c:464)
[   48.060134] update_file_ctx (security/apparmor/file.c:464)
[   48.060140] aa_file_perm (security/apparmor/file.c:532 (discriminator 1) security/apparmor/file.c:642 (discriminator 1))
[   48.060147] ? __pfx_aa_file_perm (security/apparmor/file.c:607)
[   48.060152] ? do_mmap (mm/mmap.c:558)
[   48.060160] ? __pfx_userfaultfd_unmap_complete (fs/userfaultfd.c:841)
[   48.060170] ? __lock_acquire (kernel/locking/lockdep.c:4677 (discriminator 1) kernel/locking/lockdep.c:5194 (discriminator 1))
[   48.060176] ? common_file_perm (security/apparmor/lsm.c:535 (discriminator 1))
[   48.060185] security_mmap_file (security/security.c:3012 (discriminator 2))
[   48.060192] vm_mmap_pgoff (mm/util.c:574 (discriminator 1))
[   48.060200] ? find_held_lock (kernel/locking/lockdep.c:5353 (discriminator 1))
[   48.060206] ? __pfx_vm_mmap_pgoff (mm/util.c:568)
[   48.060212] ? lock_release (kernel/locking/lockdep.c:5539 kernel/locking/lockdep.c:5892 kernel/locking/lockdep.c:5878)
[   48.060219] ? __fget_files (arch/x86/include/asm/preempt.h:85 (discriminator 13) include/linux/rcupdate.h:100 (discriminator 13) include/linux/rcupdate.h:873 (discriminator 13) fs/file.c:1072 (discriminator 13))
[   48.060229] ksys_mmap_pgoff (mm/mmap.c:604)
[   48.060239] do_syscall_64 (arch/x86/entry/syscall_64.c:63 (discriminator 1) arch/x86/entry/syscall_64.c:94 (discriminator 1))
[   48.060248] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130)
[   48.060254] RIP: 0033:0x7fb6920e30a2
[ 48.060265] Code: 08 00 04 00 00 eb e2 90 41 f7 c1 ff 0f 00 00 75 27 55 89 cd 53 48 89 fb 48 85 ff 74 33 41 89 ea 48 89 df b8 09 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5e 5b 5d c3 0f 1f 00 c7 05 e6 41 01 00 16 00
All code
========
   0:	08 00                	or     %al,(%rax)
   2:	04 00                	add    $0x0,%al
   4:	00 eb                	add    %ch,%bl
   6:	e2 90                	loop   0xffffffffffffff98
   8:	41 f7 c1 ff 0f 00 00 	test   $0xfff,%r9d
   f:	75 27                	jne    0x38
  11:	55                   	push   %rbp
  12:	89 cd                	mov    %ecx,%ebp
  14:	53                   	push   %rbx
  15:	48 89 fb             	mov    %rdi,%rbx
  18:	48 85 ff             	test   %rdi,%rdi
  1b:	74 33                	je     0x50
  1d:	41 89 ea             	mov    %ebp,%r10d
  20:	48 89 df             	mov    %rbx,%rdi
  23:	b8 09 00 00 00       	mov    $0x9,%eax
  28:	0f 05                	syscall
  2a:*	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax		<-- trapping instruction
  30:	77 5e                	ja     0x90
  32:	5b                   	pop    %rbx
  33:	5d                   	pop    %rbp
  34:	c3                   	ret
  35:	0f 1f 00             	nopl   (%rax)
  38:	c7                   	.byte 0xc7
  39:	05 e6 41 01 00       	add    $0x141e6,%eax
  3e:	16                   	(bad)
	...

Code starting with the faulting instruction
===========================================

   0:	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax
   6:	77 5e                	ja     0x66
   8:	5b                   	pop    %rbx
   9:	5d                   	pop    %rbp
   a:	c3                   	ret
   b:	0f 1f 00             	nopl   (%rax)
   e:	c7                   	.byte 0xc7
   f:	05 e6 41 01 00       	add    $0x141e6,%eax
  14:	16                   	(bad)
	...
[   48.060270] RSP: 002b:00007ffd2c0d3528 EFLAGS: 00000206 ORIG_RAX: 0000000000000009
[   48.060279] RAX: ffffffffffffffda RBX: 00007fb691fc8000 RCX: 00007fb6920e30a2
[   48.060283] RDX: 0000000000000005 RSI: 000000000007d000 RDI: 00007fb691fc8000
[   48.060287] RBP: 0000000000000812 R08: 0000000000000003 R09: 0000000000011000
[   48.060290] R10: 0000000000000812 R11: 0000000000000206 R12: 00007ffd2c0d3578
[   48.060293] R13: 00007fb6920b6160 R14: 00007ffd2c0d39f0 R15: 00000fffa581a6a8
Fixes 88fec3526e84 ("apparmor: make sure unix socket labeling is correctly updated.")
---
 security/apparmor/lsm.c | 1 +
 1 file changed, 1 insertion(+)
security/apparmor/lsm.c +1 -0
index e4b2944431e4..f385913e7d0e 100644
@@ -505,6 +505,7 @@ static int apparmor_file_alloc_security(struct file *file)
 	struct aa_file_ctx *ctx = file_ctx(file);
 	struct aa_label *label = begin_current_label_crit_section();
 
+	spin_lock_init(&ctx->lock);
 	rcu_assign_pointer(ctx->label, aa_get_label(label));
 	end_current_label_crit_section(label);
 	return 0;
Signature · 1 line
2.43.0
#4 lore raw

✗ Fi.CI.BUILD: failure for Regression on linux-next (next-20250721)

In reply to Borah, Chaitanya Kumar
== Series Details ==

== Series Details ==

Series: Regression on linux-next (next-20250721)
URL : https://patchwork.freedesktop.org/series/152326/
State : failure

== Summary ==

Error: patch https://patchwork.freedesktop.org/api/1.0/series/152326/revisions/1/mbox/ not applied
Applying: Regression on linux-next (next-20250721)
error: patch failed: security/apparmor/lsm.c:505
error: security/apparmor/lsm.c: patch does not apply
error: Did you hand edit your patch?
It does not apply to blobs recorded in its index.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Using index info to reconstruct a base tree...
M security/apparmor/lsm.c
Patch failed at 0001 Regression on linux-next (next-20250721)
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
Build failed, no error log produced