Skip to contentSkip to conversation

ufs: core: Avoid IRQ thread wakeup during active UIC command

PATCH Applied [email protected] 4 participants 15 messages Mar 6 – Mar 20

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 ufs: core: Avoid IRQ thread wakeup during active UIC command

From: Peter Wang <[email protected]>

Only return IRQ_WAKE_THREAD when MCQ and ESI are not enabled and no UIC command is active. The default UIC command timeout is 500ms, Using threaded IRQs during an active UIC command increases the risk of timeout due to possible preemption by other system IRQs.

---
 drivers/ufs/core/ufshcd.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
drivers/ufs/core/ufshcd.c +6 -2
index 9908375b2f98..6554e1db3343 100644
@@ -7200,8 +7200,12 @@ static irqreturn_t ufshcd_intr(int irq, void *__hba)
 	struct ufs_hba *hba = __hba;
 	u32 intr_status, enabled_intr_status;
 
-	/* Move interrupt handling to thread when MCQ & ESI are not enabled */
-	if (!hba->mcq_enabled || !hba->mcq_esi_enabled)
+	/*
+	 * Handle interrupt in thread if MCQ or ESI is disabled,
+	 * and no active UIC command.
+	 */
+	if ((!hba->mcq_enabled || !hba->mcq_esi_enabled) &&
+	    !hba->active_uic_cmd)
 		return IRQ_WAKE_THREAD;
 
 	intr_status = ufshcd_readl(hba, REG_INTERRUPT_STATUS);
Signature · 1 line
2.45.2
reviewer Reviewed-by on v2 1/1
#1 lore raw
Quoted 9 lines · On 3/5/26 11:43 PM, [email protected] wrote:
drivers/ufs/core/ufshcd.c +0 -1
index 9908375b2f98..6554e1db3343 100644
@@ -7200,8 +7200,12 @@ static irqreturn_t ufshcd_intr(int irq, void *__hba)
  	struct ufs_hba *hba = __hba;
  	u32 intr_status, enabled_intr_status;
 
-	/* Move interrupt handling to thread when MCQ & ESI are not enabled */

drivers/ufs/core/ufshcd.c static irqreturn_t ufshcd_intr(int irq, void *__hba) · line 7211

Quoted 10 lines
drivers/ufs/core/ufshcd.c +6 -1
-	if (!hba->mcq_enabled || !hba->mcq_esi_enabled)
+	/*
+	 * Handle interrupt in thread if MCQ or ESI is disabled,
+	 * and no active UIC command.
+	 */
+	if ((!hba->mcq_enabled || !hba->mcq_esi_enabled) &&
+	    !hba->active_uic_cmd)
  		return IRQ_WAKE_THREAD;
 
  	intr_status = ufshcd_readl(hba, REG_INTERRUPT_STATUS);
#2 lore raw
In reply to [email protected]

Peter,

Quoted 4 lines

Only return IRQ_WAKE_THREAD when MCQ and ESI are not enabled and no UIC command is active. The default UIC command timeout is 500ms, Using threaded IRQs during an active UIC command increases the risk of timeout due to possible preemption by other system IRQs.

Applied to 7.1/scsi-staging, thanks!

Signature · 1 line
Martin K. Petersen
#3 lore raw
In reply to [email protected]
Quoted 8 lines · On Fri, 06 Mar 2026 13:43:02 +0800, [email protected] wrote:

Only return IRQ_WAKE_THREAD when MCQ and ESI are not enabled and no UIC command is active. The default UIC command timeout is 500ms, Using threaded IRQs during an active UIC command increases the risk of timeout due to possible preemption by other system IRQs.

[...]

Applied to 7.1/scsi-queue, thanks!

[1/1] ufs: core: Avoid IRQ thread wakeup during active UIC command

      https://git.kernel.org/mkp/scsi/c/6475cfb81fc4
Signature · 1 line
Martin K. Petersen
#4 lore raw
In reply to [email protected]

Hi

Quoted 9 lines · On 06.03.2026 06:43, [email protected] wrote:

From: Peter Wang <[email protected]>

Only return IRQ_WAKE_THREAD when MCQ and ESI are not enabled and no UIC command is active. The default UIC command timeout is 500ms, Using threaded IRQs during an active UIC command increases the risk of timeout due to possible preemption by other system IRQs.

This patch landed in linux-next as commit 6475cfb81fc4 ("scsi: ufs:
core: Avoid IRQ thread wakeup during active UIC command"). In my tests I found that it causes the following regression on QCom RB5 board (arch/arm64/boot/dts/qcom/qrb5165-rb5.dts):

=============================
[ BUG: Invalid wait context ] 7.0.0-rc4-next-20260316 #16535 Not tainted
-----------------------------
swapper/0/0 is trying to lock:
ffff000089f58048 (shost->host_lock){....}-{3:3}, at:
ufshcd_sl_intr+0x3c0/0x6b4
other info that might help us debug this:
context-{2:2}
no locks held by swapper/0/0.

stack backtrace:
CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 7.0.0-rc4-next-20260316 
#16535 PREEMPT
Hardware name: Qualcomm Technologies, Inc. Robotics RB5 (DT)
Call trace:
  show_stack+0x18/0x24 (C)
  dump_stack_lvl+0x90/0xd0
  dump_stack+0x18/0x24
  __lock_acquire+0xa40/0x2254
  lock_acquire+0x1c4/0x3fc
  _raw_spin_lock_irqsave+0x60/0x88
  ufshcd_sl_intr+0x3c0/0x6b4
  ufshcd_intr+0x7c/0x90
  __handle_irq_event_percpu+0xa0/0x4c4
  handle_irq_event+0x4c/0xf8
  handle_fasteoi_irq+0x108/0x198
  handle_irq_desc+0x40/0x58
  generic_handle_domain_irq+0x18/0x24
  gic_handle_irq+0x4c/0x110
  call_on_irq_stack+0x30/0x48
  do_interrupt_handler+0x80/0x84
  el1_interrupt+0x3c/0x60
  el1h_64_irq_handler+0x18/0x24
  el1h_64_irq+0x6c/0x70
  cpuidle_enter_state+0xf8/0x41c (P)
  cpuidle_enter+0x38/0x50
  do_idle+0x170/0x2ac
  cpu_startup_entry+0x34/0x3c
  rest_init+0xf8/0x188
  start_kernel+0x818/0x8ec
  __primary_switched+0x88/0x90

scsi host0: ufshcd

Reverting $subject on top of linux-next fixes this issue.

Quoted 13 lines
---
  drivers/ufs/core/ufshcd.c | 8 ++++++--
  1 file changed, 6 insertions(+), 2 deletions(-)
drivers/ufs/core/ufshcd.c +0 -1
index 9908375b2f98..6554e1db3343 100644
@@ -7200,8 +7200,12 @@ static irqreturn_t ufshcd_intr(int irq, void *__hba)
  	struct ufs_hba *hba = __hba;
  	u32 intr_status, enabled_intr_status;
 
-	/* Move interrupt handling to thread when MCQ & ESI are not enabled */

drivers/ufs/core/ufshcd.c static irqreturn_t ufshcd_intr(int irq, void *__hba) · line 7211

Quoted 10 lines
drivers/ufs/core/ufshcd.c +6 -1
-	if (!hba->mcq_enabled || !hba->mcq_esi_enabled)
+	/*
+	 * Handle interrupt in thread if MCQ or ESI is disabled,
+	 * and no active UIC command.
+	 */
+	if ((!hba->mcq_enabled || !hba->mcq_esi_enabled) &&
+	    !hba->active_uic_cmd)
  		return IRQ_WAKE_THREAD;
 
  	intr_status = ufshcd_readl(hba, REG_INTERRUPT_STATUS);

Best regards

Signature · 2 lines
Marek Szyprowski, PhD
Samsung R&D Institute Poland
#5 lore raw
Quoted 5 lines · Marek Szyprowski

This patch landed in linux-next as commit 6475cfb81fc4 ("scsi: ufs:
core: Avoid IRQ thread wakeup during active UIC command"). In my tests I
found that it causes the following regression on QCom RB5 board (arch/arm64/boot/dts/qcom/qrb5165-rb5.dts):

Hi Marek,

If this issue can always be reproduced with this patch? I have doubts because, in your log, it seems to be ISR trying to acquire the spinlock (shost->host_lock), but cannot obtain it in time, causing a timeout.

ufshcd-qcom 1d84000.ufshc: uic cmd 0x1 with arg3 0x0 completion timeout ufshcd-qcom 1d84000.ufshc: dme-get: sttr-id 0x41 failed 0 retries no locks held by swapper/0/0.
ufshcd-qcom 1d84000.ufshc: ufs_wcom_check_hibern8: unable to get TX_FSM_STATE, err -110 stack backtrace:
ufshcd-qcom 1d84000.ufshc: No active UIC command. Maybe a timeout occurred?
ufshcd-qcom 1d84000.ufshc: ufshcd_threaded_intr: Unhadled interrupt 0x00000000 (0x00000400, 0x00000400)

Regardless of whether it runs in IRQ or thread IRQ context, the outcome is the same (host_lock cannot be acquired in time). So, could you check why this spinlock cannot be acquired first? Additionally, ufs_wcom_check_hibern8 does not appear in the current code base, so there might be some other unknown code that has already acquired host_lock in your code base?

Thanks
Peter

#6 lore raw

Hi Peter,

Quoted 9 lines · Peter Wang (王信友)
Quoted 5 lines · Marek Szyprowski

This patch landed in linux-next as commit 6475cfb81fc4 ("scsi: ufs:
core: Avoid IRQ thread wakeup during active UIC command"). In my tests I
found that it causes the following regression on QCom RB5 board (arch/arm64/boot/dts/qcom/qrb5165-rb5.dts):

Hi Marek,

If this issue can always be reproduced with this patch?

The issue I've observed is 100% reproducible. I've checked again by compiling the kernel directly from commit 6475cfb81fc4:

=============================
[ BUG: Invalid wait context ]
7.0.0-rc1+ #16536 Not tainted
-----------------------------
swapper/0/0 is trying to lock:
ffff000080bdc048 (shost->host_lock){....}-{3:3}, at:
ufshcd_sl_intr+0x3c0/0x6b4
other info that might help us debug this:
context-{2:2}
no locks held by swapper/0/0.

stack backtrace:
CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 7.0.0-rc1+ #16536 PREEMPT
Hardware name: Qualcomm Technologies, Inc. Robotics RB5 (DT)
Call trace:
  show_stack+0x18/0x24 (C)
  dump_stack_lvl+0x90/0xd0
  dump_stack+0x18/0x24
  __lock_acquire+0xa40/0x2254
  lock_acquire+0x1c4/0x3fc
  _raw_spin_lock_irqsave+0x60/0x88
  ufshcd_sl_intr+0x3c0/0x6b4
  ufshcd_intr+0x7c/0x90
  __handle_irq_event_percpu+0xa0/0x4c4
  handle_irq_event+0x4c/0xf8
  handle_fasteoi_irq+0x108/0x198
  handle_irq_desc+0x40/0x58
  generic_handle_domain_irq+0x18/0x24
  gic_handle_irq+0x4c/0x110
  call_on_irq_stack+0x30/0x48
  do_interrupt_handler+0x80/0x84
  el1_interrupt+0x3c/0x60
  el1h_64_irq_handler+0x18/0x24
  el1h_64_irq+0x6c/0x70
  cpuidle_enter_state+0xf8/0x41c (P)
  cpuidle_enter+0x38/0x50
  do_idle+0x208/0x290
  cpu_startup_entry+0x34/0x3c
  rest_init+0xf8/0x188
  start_kernel+0x810/0x8e4
  __primary_switched+0x88/0x90

scsi host0: ufshcd

Quoted 14 lines

I have doubts because, in your log, it seems to be ISR trying to acquire the spinlock (shost->host_lock), but cannot obtain it in time, causing a timeout.

ufshcd-qcom 1d84000.ufshc: uic cmd 0x1 with arg3 0x0 completion timeout ufshcd-qcom 1d84000.ufshc: dme-get: sttr-id 0x41 failed 0 retries no locks held by swapper/0/0.
ufshcd-qcom 1d84000.ufshc: ufs_wcom_check_hibern8: unable to get TX_FSM_STATE, err -110 stack backtrace:
ufshcd-qcom 1d84000.ufshc: No active UIC command. Maybe a timeout occurred?
ufshcd-qcom 1d84000.ufshc: ufshcd_threaded_intr: Unhadled interrupt 0x00000000 (0x00000400, 0x00000400)

Quoted 6 lines

Regardless of whether it runs in IRQ or thread IRQ context, the outcome is the same (host_lock cannot be acquired in time). So, could you check why this spinlock cannot be acquired first? Additionally, ufs_wcom_check_hibern8 does not appear in the current code base, so there might be some other unknown code that has already acquired host_lock in your code base?

You have probably mixed my report with this one
https://lore.kernel.org/all/[email protected]/

Best regards

Signature · 2 lines
Marek Szyprowski, PhD
Samsung R&D Institute Poland
#7 lore raw
Quoted 3 lines · Marek Szyprowski
CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 7.0.0-rc1+ #16536

PREEMPT

Hi Marek,

I stressed the UIC command for 24 hours but could not reproduce the issue. I also found that the line containing "PREEMPT" might be a key point.

So I suspect that lockdep might be misjudging with "PREEMPT" setting. After all, using spin_lock_irqsave inside the ISR should be fine:

    guard(spinlock_irqsave)(hba->host->host_lock);

Would it be possible to try disabling lockdep and see what happens? It seems lockdep is even getting the lock wrong (hba->host->host_lock, not shost->host_lock).

Quoted 5 lines

You have probably mixed my report with this one
https://lore.kernel.org/all/[email protected]/

Best regards

Yes, I mixed it up with this one because the call trace looks the same.

Thanks
Peter

#8 lore raw
Quoted 2 lines · Peter Wang (王信友)

So I suspect that lockdep might be misjudging with "PREEMPT" setting.

If this issue would be caused by a bug in lockdep, which I doubt, there should be other reports of this lockdep bug, isn't it? Even if this would be a lockdep bug, this should be fixed because lockdep is an essential debugging tool.

I propose to revert patch "ufs: core: Avoid IRQ thread wakeup during active UIC command" if the root cause of this issue is not found in the next few days.

Thanks,

Bart.

#9 lore raw
Quoted 5 lines · Bart Van Assche

If this issue would be caused by a bug in lockdep, which I doubt, there should be other reports of this lockdep bug, isn't it? Even if this would be a lockdep bug, this should be fixed because lockdep is an essential debugging tool.

Hi Bart,

I absolutely agree with you, and I just want to conduct some DOE experiments to find out the root cause of this issue. Because it is really weird and cannot be reproduced at my site.

Quoted 8 lines

I propose to revert patch "ufs: core: Avoid IRQ thread wakeup during active UIC command" if the root cause of this issue is not found in the
next few days.

Thanks,

Bart.

Sure, reverting is a quick solution. But I am concerned that it would hide the real problem.

Thanks.
Peter

#10 lore raw
In reply to Marek Szyprowski
Quoted 11 lines · Marek Szyprowski

This patch landed in linux-next as commit 6475cfb81fc4 ("scsi: ufs:
core: Avoid IRQ thread wakeup during active UIC command"). In my tests I found that it causes the following regression on QCom RB5 board (arch/arm64/boot/dts/qcom/qrb5165-rb5.dts):

=============================
[ BUG: Invalid wait context ] 7.0.0-rc4-next-20260316 #16535 Not tainted
-----------------------------
swapper/0/0 is trying to lock:
ffff000089f58048 (shost->host_lock){....}-{3:3}, at:

This line is a mystery to me. Are there perhaps any local changes in your kernel tree on top of linux-next? I haven't been able to find the text "shost->host_lock" in the UIC completion path. Instead, this is what I found:

	guard(spinlock_irqsave)(hba->host->host_lock);
Quoted 1 line

ufshcd_sl_intr+0x3c0/0x6b4

Can you please help with translating this information into a line number? Tools like addr2line, llvm-addr2line or llvm-objdump -d -l -S can be used to perform such a conversion.

Thanks,

Bart.

#11 lore raw
Quoted 13 lines · Bart Van Assche
Quoted 11 lines · Marek Szyprowski

This patch landed in linux-next as commit 6475cfb81fc4 ("scsi: ufs:
core: Avoid IRQ thread wakeup during active UIC command"). In my tests I found that it causes the following regression on QCom RB5 board (arch/arm64/boot/dts/qcom/qrb5165-rb5.dts):

=============================
[ BUG: Invalid wait context ] 7.0.0-rc4-next-20260316 #16535 Not tainted
-----------------------------
swapper/0/0 is trying to lock:
ffff000089f58048 (shost->host_lock){....}-{3:3}, at:

Quoted 3 lines

This line is a mystery to me. Are there perhaps any local changes in your kernel tree on top of linux-next? I haven't been able to find the text "shost->host_lock" in the UIC completion path.

I don't have any local changes, code is at commit 6475cfb81fc4. After looking at the code this 'shost' indeed looks a bit mysterious, but maybe it got that name after some inlining or code optimization.

Quoted 10 lines

Instead, this is
what I found:

    guard(spinlock_irqsave)(hba->host->host_lock);

Quoted 1 line

ufshcd_sl_intr+0x3c0/0x6b4

Can you please help with translating this information into a line number? Tools like addr2line, llvm-addr2line or llvm-objdump -d -l -S can be used to perform such a conversion.

ufshcd_clk_scaling_allow() in drivers/ufs/core/ufshcd.c:6492 (code checkout at git commit 6475cfb81fc4)

Best regards

Signature · 2 lines
Marek Szyprowski, PhD
Samsung R&D Institute Poland
#12 lore raw
Quoted 27 lines · Marek Szyprowski
Quoted 13 lines · Bart Van Assche
Quoted 11 lines · Marek Szyprowski

This patch landed in linux-next as commit 6475cfb81fc4 ("scsi: ufs:
core: Avoid IRQ thread wakeup during active UIC command"). In my tests I found that it causes the following regression on QCom RB5 board (arch/arm64/boot/dts/qcom/qrb5165-rb5.dts):

=============================
[ BUG: Invalid wait context ] 7.0.0-rc4-next-20260316 #16535 Not tainted
-----------------------------
swapper/0/0 is trying to lock:
ffff000089f58048 (shost->host_lock){....}-{3:3}, at:

Quoted 3 lines

This line is a mystery to me. Are there perhaps any local changes in your kernel tree on top of linux-next? I haven't been able to find the text "shost->host_lock" in the UIC completion path.

I don't have any local changes, code is at commit 6475cfb81fc4. After looking at the code this 'shost' indeed looks a bit mysterious, but maybe it got that name after some inlining or code optimization.

Quoted 5 lines

Instead, this is
what I found:

    guard(spinlock_irqsave)(hba->host->host_lock);

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

ufshcd_sl_intr+0x3c0/0x6b4

Can you please help with translating this information into a line number? Tools like addr2line, llvm-addr2line or llvm-objdump -d -l -S can be used to perform such a conversion.

ufshcd_clk_scaling_allow() in drivers/ufs/core/ufshcd.c:6492 (code checkout at git commit 6475cfb81fc4)

Hi Marek,

I haven't been able to find any call chain from ufshcd_sl_intr() into ufshcd_clk_scaling_allow(). Am I perhaps overlooking something?

Thanks,

Bart.

#13 lore raw
Quoted 35 lines · Bart Van Assche
Quoted 34 lines · Marek Szyprowski
Quoted 14 lines · Bart Van Assche
Quoted 6 lines · Marek Szyprowski

This patch landed in linux-next as commit 6475cfb81fc4 ("scsi: ufs:
core: Avoid IRQ thread wakeup during active UIC command"). In my tests I
found that it causes the following regression on QCom RB5 board (arch/arm64/boot/dts/qcom/qrb5165-rb5.dts):

Quoted 6 lines

=============================
[ BUG: Invalid wait context ] 7.0.0-rc4-next-20260316 #16535 Not tainted
-----------------------------
swapper/0/0 is trying to lock:
ffff000089f58048 (shost->host_lock){....}-{3:3}, at:

Quoted 3 lines

This line is a mystery to me. Are there perhaps any local changes in your kernel tree on top of linux-next? I haven't been able to find the text "shost->host_lock" in the UIC completion path.

I don't have any local changes, code is at commit 6475cfb81fc4. After looking at the code this 'shost' indeed looks a bit mysterious, but maybe it got that name after some inlining or code optimization.

Quoted 10 lines

Instead, this is
what I found:

     guard(spinlock_irqsave)(hba->host->host_lock);

Quoted 1 line

ufshcd_sl_intr+0x3c0/0x6b4

Can you please help with translating this information into a line number? Tools like addr2line, llvm-addr2line or llvm-objdump -d -l -S can be used to perform such a conversion.

Quoted 7 lines
Quoted 2 lines

ufshcd_clk_scaling_allow() in drivers/ufs/core/ufshcd.c:6492 (code checkout at git commit 6475cfb81fc4)

Hi Marek,

I haven't been able to find any call chain from ufshcd_sl_intr() into ufshcd_clk_scaling_allow(). Am I perhaps overlooking something?

I must have mixed something while calculating offsets for addr2line. I checked again and I found that there is already a script doing that. I've recompiled kernel with -Os (assuming that this way less code will be inlined) and this is the result:

=============================
[ BUG: Invalid wait context ]
7.0.0-rc1+ #16537 Not tainted
-----------------------------
swapper/0/0 is trying to lock:
ffff000080a04048 (shost->host_lock){....}-{3:3}, at:
ufshcd_sl_intr+0x50/0x598
other info that might help us debug this:
context-{2:2}
no locks held by swapper/0/0.

stack backtrace:
CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 7.0.0-rc1+ #16537 PREEMPT
Hardware name: Qualcomm Technologies, Inc. Robotics RB5 (DT)
Call trace:
  show_stack+0x18/0x24 (C)
  dump_stack_lvl+0x6c/0x94
  dump_stack+0x18/0x24
  __lock_acquire+0x3f8/0x10ac
  lock_acquire+0x29c/0x2ec
  _raw_spin_lock_irqsave+0x58/0x78
  ufshcd_sl_intr+0x50/0x598
  ufshcd_intr+0x64/0x78
  __handle_irq_event_percpu+0x1d4/0x354
  handle_irq_event_percpu+0x18/0x4c
  handle_irq_event+0x48/0x90
  handle_fasteoi_irq+0xc4/0xe8
  handle_irq_desc+0x48/0x58
  generic_handle_domain_irq+0x18/0x24
  gic_handle_irq+0xa4/0x108
  call_on_irq_stack+0x30/0x48
  do_interrupt_handler+0x88/0x94
  el1_interrupt+0x3c/0x60
  el1h_64_irq_handler+0x18/0x24
  el1h_64_irq+0x6c/0x70
  cpuidle_enter_state+0x1c0/0x2f0 (P)
  cpuidle_enter+0x38/0x50
  do_idle+0x23c/0x260
  cpu_startup_entry+0x34/0x38
  kernel_init+0x0/0x12c
  start_kernel+0x7e4/0x7f0
  __primary_switched+0x88/0x90
# ./scripts/faddr2line vmlinux "ufshcd_sl_intr+0x50/0x598"
ufshcd_sl_intr+0x50/0x598:
ufshcd_uic_cmd_compl at drivers/ufs/core/ufshcd.c:5570
(inlined by) ufshcd_sl_intr at drivers/ufs/core/ufshcd.c:7144

Best regards

Signature · 2 lines
Marek Szyprowski, PhD
Samsung R&D Institute Poland
#14 lore raw
Quoted 11 lines · Marek Szyprowski

I must have mixed something while calculating offsets for addr2line. I checked again and I found that there is already a script doing that. I've recompiled kernel with -Os (assuming that this way less code will be inlined) and this is the result:

[ ... ]

# ./scripts/faddr2line vmlinux "ufshcd_sl_intr+0x50/0x598"
ufshcd_sl_intr+0x50/0x598:
ufshcd_uic_cmd_compl at drivers/ufs/core/ufshcd.c:5570
(inlined by) ufshcd_sl_intr at drivers/ufs/core/ufshcd.c:7144

Thanks! So it's the guard(spinlock_irqsave)(hba->host->host_lock) statement. Not sure why it triggers an "invalid wait context" complaint. Is CONFIG_PREEMPT_RT perhaps enabled in your kernel configuration?

Thanks,

Bart.