Skip to contentSkip to conversation

net: cxgb4: Call Trace reported with PREEMPT_RT: BUG: using smp_processor_id() in preemptible [00000000] code: ethtool/78718

PATCH Applied John B. Wyatt IV 4 participants 4 messages Apr 23, 2024 – May 1, 2024

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Hello Raju, Hello Sebastian,

Red Hat QE found this issue with cxgb4 only when the kernel has PREEMPT_RT set with the preempt-rt patchset:

git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git

We are also seeing this in the Real-time builds of RHEL9 and 8.

The specific build is an internal build that was pulled from the mirror Clark Williams setup for Fedora and RHEL testing.

https://gitlab.com/cki-project/kernel-ark/-/tree/os-build-rt?ref_type=heads

We use the branch: os-build-rt

I was unable to find the cause of this and I thought I should report it.

Please let me if you have any questions or you need any testing done.

Call trace is below:

kernel-rt-6.9.0-0.rc4.f8dba31b0a82.38.test.eln136.x86_64

 BUG: using smp_processor_id() in preemptible [00000000] code: ethtool/78718
 caller is cxgb4_selftest_lb_pkt+0x3d/0x3a0 cxgb4
 Hardware name: Dell Inc. PowerEdge R750/0WT8Y6, BIOS 1.5.4 12/17/2021
 Call Trace:
  <TASK>
 dump_stack_lvl (lib/dump_stack.c:116) 
 check_preemption_disabled (lib/smp_processor_id.c:49) 
 cxgb4_selftest_lb_pkt+0x3d/0x3a0 cxgb4
 cxgb4_self_test+0x8f/0xe0 cxgb4
 ethtool_self_test (net/ethtool/ioctl.c:2002) 
 __dev_ethtool (net/ethtool/ioctl.c:2997) 
 ? migrate_enable (./include/linux/preempt.h:480 (discriminator 3) ./include/linux/preempt.h:480 (discriminator 3) kernel/sched/core.c:2472 (discriminator 3)) 
 ? kmalloc_trace (./arch/x86/include/asm/jump_label.h:55 ./include/linux/memcontrol.h:1839 mm/slub.c:1980 mm/slub.c:3807 mm/slub.c:3845 mm/slub.c:3992) 
 dev_ethtool (net/ethtool/ioctl.c:3177) 
 dev_ioctl (net/core/dev_ioctl.c:724) 
 sock_do_ioctl (net/socket.c:1236) 
 __x64_sys_ioctl (fs/ioctl.c:51 fs/ioctl.c:904 fs/ioctl.c:890 fs/ioctl.c:890) 
 do_syscall_64 (arch/x86/entry/common.c:52 (discriminator 1) arch/x86/entry/common.c:83 (discriminator 1)) 
 ? mod_objcg_state (mm/memcontrol.c:3421) 
 ? migrate_enable (./include/linux/preempt.h:480 (discriminator 3) ./include/linux/preempt.h:480 (discriminator 3) kernel/sched/core.c:2472 (discriminator 3)) 
 ? try_charge_memcg (mm/memcontrol.c:2745) 
 ? __mod_node_page_state (./include/linux/preempt.h:477 (discriminator 3) mm/vmstat.c:405 (discriminator 3)) 
 ? migrate_enable (./include/linux/preempt.h:480 (discriminator 3) ./include/linux/preempt.h:480 (discriminator 3) kernel/sched/core.c:2472 (discriminator 3)) 
 ? rt_spin_unlock (kernel/locking/rtmutex.c:230 (discriminator 5) kernel/locking/spinlock_rt.c:84 (discriminator 5)) 
 ? do_anonymous_page (./include/linux/pgtable.h:114 mm/memory.c:4490) 
 ? __handle_mm_fault (mm/memory.c:3878 mm/memory.c:5300 mm/memory.c:5441) 
 ? syscall_exit_to_user_mode (kernel/entry/common.c:221) 
 ? __count_memcg_events (./include/linux/preempt.h:477 (discriminator 3) mm/memcontrol.c:704 (discriminator 3) mm/memcontrol.c:963 (discriminator 3)) 
 ? handle_mm_fault (mm/memory.c:5483 mm/memory.c:5622) 
 ? do_user_addr_fault (arch/x86/mm/fault.c:1443 (discriminator 1)) 
 ? clear_bhb_loop (arch/x86/entry/entry_64.S:1539) 
 ? clear_bhb_loop (arch/x86/entry/entry_64.S:1539) 
 entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130) 
 RIP: 0033:0x7f55216c557b
 Code: ff ff ff 85 c0 79 9b 49 c7 c4 ff ff ff ff 5b 5d 4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 75 68 0f 00 f7 d8 64 89 01 48
All code
========
   0:	ff                   	(bad)
   1:	ff                   	(bad)
   2:	ff 85 c0 79 9b 49    	incl   0x499b79c0(%rbp)
   8:	c7 c4 ff ff ff ff    	mov    $0xffffffff,%esp
   e:	5b                   	pop    %rbx
   f:	5d                   	pop    %rbp
  10:	4c 89 e0             	mov    %r12,%rax
  13:	41 5c                	pop    %r12
  15:	c3                   	ret
  16:	66 0f 1f 84 00 00 00 	nopw   0x0(%rax,%rax,1)
  1d:	00 00 
  1f:	f3 0f 1e fa          	endbr64
  23:	b8 10 00 00 00       	mov    $0x10,%eax
  28:	0f 05                	syscall
  2a:*	48 3d 01 f0 ff ff    	cmp    $0xfffffffffffff001,%rax		<-- trapping instruction
  30:	73 01                	jae    0x33
  32:	c3                   	ret
  33:	48 8b 0d 75 68 0f 00 	mov    0xf6875(%rip),%rcx        # 0xf68af
  3a:	f7 d8                	neg    %eax
  3c:	64 89 01             	mov    %eax,%fs:(%rcx)

3f: 48 rex.W

Code starting with the faulting instruction
===========================================

   0:	48 3d 01 f0 ff ff    	cmp    $0xfffffffffffff001,%rax
   6:	73 01                	jae    0x9
   8:	c3                   	ret
   9:	48 8b 0d 75 68 0f 00 	mov    0xf6875(%rip),%rcx        # 0xf6885
  10:	f7 d8                	neg    %eax
  12:	64 89 01             	mov    %eax,%fs:(%rcx)
  15:	48                   	rex.W
 RSP: 002b:00007ffd867a78f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
 RAX: ffffffffffffffda RBX: 00007ffd867a7980 RCX: 00007f55216c557b
 RDX: 00007ffd867a7990 RSI: 0000000000008946 RDI: 0000000000000003
 RBP: 0000556fe43632e0 R08: 0000000000000003 R09: 0000000000000001
 R10: 0000000000000fff R11: 0000000000000246 R12: 0000556fe43632a0
 R13: 0000000000000018 R14: 0000000000000001 R15: 0000000000000000
  </TASK>
Signature · 4 lines
Sincerely,
John Wyatt
Software Engineer, Core Kernel
Red Hat
#1 lore raw
Quoted 23 lines · John B. Wyatt IV

Hello Raju, Hello Sebastian,

Red Hat QE found this issue with cxgb4 only when the kernel has PREEMPT_RT set with the preempt-rt patchset:

git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git

We are also seeing this in the Real-time builds of RHEL9 and 8.

The specific build is an internal build that was pulled from the mirror Clark Williams setup for Fedora and RHEL testing.

https://gitlab.com/cki-project/kernel-ark/-/tree/os-build-rt?ref_type=heads

We use the branch: os-build-rt

I was unable to find the cause of this and I thought I should report it.

Please let me if you have any questions or you need any testing done.

Call trace is below:

kernel-rt-6.9.0-0.rc4.f8dba31b0a82.38.test.eln136.x86_64

Quoted 11 lines
 BUG: using smp_processor_id() in preemptible [00000000] code: ethtool/78718
 caller is cxgb4_selftest_lb_pkt+0x3d/0x3a0 cxgb4
 Hardware name: Dell Inc. PowerEdge R750/0WT8Y6, BIOS 1.5.4 12/17/2021
 Call Trace:
  <TASK>
 dump_stack_lvl (lib/dump_stack.c:116) 
 check_preemption_disabled (lib/smp_processor_id.c:49) 
 cxgb4_selftest_lb_pkt+0x3d/0x3a0 cxgb4
 cxgb4_self_test+0x8f/0xe0 cxgb4
 ethtool_self_test (net/ethtool/ioctl.c:2002) 
 __dev_ethtool (net/ethtool/ioctl.c:2997) 

Hi John,

The patch below is untested but should fix the problem you reported:

======

cxgb4: fix smp_processor_id() usage in selftests

When PREEMPT_RT is enabled the following call can result in a "BUG: using smp_processor_id() in preemptible [00000000] code: ethtool/xxxx" error message:

    ethtool_self_test()
        cxgb4_self_test()
            cxgb4_selftest_lb_pkt()
                __netif_tx_lock(q->txq, smp_processor_id());  <--- BOOM

Replacing smp_processor_id() by raw_smp_processor_id() is safe in this context given that __netif_tx_lock() is an inline function that takes a spinlock and then uses the cpu value.

drivers/net/ethernet/chelsio/cxgb4/sge.c +1 -1
index 1948b7bf99661..803dc62a4db04 100644
@@ -2674,7 +2674,7 @@ int cxgb4_selftest_lb_pkt(struct net_device *netdev)
 	lb->loopback = 1;
 
 	q = &adap->sge.ethtxq[pi->first_qset];
-	__netif_tx_lock(q->txq, smp_processor_id());
+	__netif_tx_lock(q->txq, raw_smp_processor_id());
 
 	reclaim_completed_tx(adap, &q->q, -1, true);
 	credits = txq_avail(&q->q) - ndesc;
#2 lore raw

PATCH 1/1 cxgb4: Properly lock TX queue for the selftest.

In reply to John B. Wyatt IV

The selftest for the driver sends a dummy packet and checks if the packet will be received properly as it should be. The regular TX path and the selftest can use the same network queue so locking is required and was missing in the selftest path. This was addressed in the commit cited below.
Unfortunately locking the TX queue requires BH to be disabled which is not the case in selftest path which is invoked in process context. Lockdep should be complaining about this.

Use __netif_tx_lock_bh() for TX queue locking.

Fixes c650e04898072 ("cxgb4: Fix race between loopback and normal Tx path")
---
 drivers/net/ethernet/chelsio/cxgb4/sge.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/net/ethernet/chelsio/cxgb4/sge.c +3 -3
index 49d5808b7d11d..de52bcb884c41 100644
@@ -2670,12 +2670,12 @@ int cxgb4_selftest_lb_pkt(struct net_device *netdev)
 	lb->loopback = 1;
 
 	q = &adap->sge.ethtxq[pi->first_qset];
-	__netif_tx_lock(q->txq, smp_processor_id());
+	__netif_tx_lock_bh(q->txq);
 
 	reclaim_completed_tx(adap, &q->q, -1, true);
 	credits = txq_avail(&q->q) - ndesc;
 	if (unlikely(credits < 0)) {
-		__netif_tx_unlock(q->txq);
+		__netif_tx_unlock_bh(q->txq);
 		return -ENOMEM;
 	}
 
@@ -2710,7 +2710,7 @@ int cxgb4_selftest_lb_pkt(struct net_device *netdev)
 	init_completion(&lb->completion);
 	txq_advance(&q->q, ndesc);
 	cxgb4_ring_tx_db(adap, &q->q, ndesc);
-	__netif_tx_unlock(q->txq);
+	__netif_tx_unlock_bh(q->txq);
 
 	/* wait for the pkt to return */
 	ret = wait_for_completion_timeout(&lb->completion, 10 * HZ);
Signature · 1 line
2.43.0
#3 lore raw

Re: [PATCH net] cxgb4: Properly lock TX queue for the selftest.

This patch was applied to netdev/net.git (main) by Jakub Kicinski <[email protected]>:

Hello:

This patch was applied to netdev/net.git (main)
by Jakub Kicinski <[email protected]>:

Quoted 10 lines · On Mon, 29 Apr 2024 11:11:47 +0200 you wrote:

The selftest for the driver sends a dummy packet and checks if the packet will be received properly as it should be. The regular TX path and the selftest can use the same network queue so locking is required and was missing in the selftest path. This was addressed in the commit cited below.
Unfortunately locking the TX queue requires BH to be disabled which is not the case in selftest path which is invoked in process context. Lockdep should be complaining about this.

[...]

Here is the summary with links:
- [net] cxgb4: Properly lock TX queue for the selftest.

    https://git.kernel.org/netdev/net/c/9067eccdd784

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html