Skip to contentSkip to conversation

ufs-qcom: Reduce interrupt latency

PATCH In review Bart Van Assche 3 participants 10 messages Apr 2 – May 14

Subsystems (4)
lore
Share & tools
lore

Bart Van Assche

author reviewer
#0 lore raw

v2 cover ufs-qcom: Reduce interrupt latency

Hi Martin,

On Android systems it is important to keep the time spent in interrupts short. This keeps the user interface responsive and prevents audio stuttering. Hence this patch series to reduce the time spent in the UFS interrupt handler. Please consider this patch series for the next merge window after test results have been shared by Qualcomm.

Thanks,

Bart.

Changes compared to v1:
- Dropped the ufshcd_mcq_compl_all_cqes_lock() changes.
- Renamed ufshcd_mcq_poll_cqe_lock_n() into ufshcd_mcq_poll_n_cqe_lock().

Bart Van Assche (2):
ufs: core: Introduce ufshcd_mcq_poll_n_cqe_lock()
ufs: qcom: Reduce interrupt latency

 drivers/ufs/core/ufs-mcq.c  | 14 +++++++++++---
 drivers/ufs/host/ufs-qcom.c | 33 +++++++++++++++++++++++++++++----
 include/ufs/ufshcd.h        |  3 +++
 3 files changed, 43 insertions(+), 7 deletions(-)
author reviewer
#1 lore raw

PATCH v2 1/2 ufs: core: Introduce ufshcd_mcq_poll_n_cqe_lock()

Introduce a new function for processing completions that accepts an upper limit for the number of completions to poll. Tell ufshcd_mcq_poll_cqe_lock() to poll at most hwq->max_entries. This is sufficient to poll all pending completions since there are never more than hwq->max_entries - 1 completions on a completion queue. This patch prepares for reducing the interrupt latency.

---
 drivers/ufs/core/ufs-mcq.c | 14 +++++++++++---
 include/ufs/ufshcd.h       |  3 +++
 2 files changed, 14 insertions(+), 3 deletions(-)
drivers/ufs/core/ufs-mcq.c +11 -3
index 1b3062577945..e6081b97ed74 100644
@@ -340,15 +340,16 @@ void ufshcd_mcq_compl_all_cqes_lock(struct ufs_hba *hba,
 	spin_unlock_irqrestore(&hwq->cq_lock, flags);
 }
 
-unsigned long ufshcd_mcq_poll_cqe_lock(struct ufs_hba *hba,
-				       struct ufs_hw_queue *hwq)
+unsigned long ufshcd_mcq_poll_n_cqe_lock(struct ufs_hba *hba,
+					 struct ufs_hw_queue *hwq,
+					 unsigned int max_compl)
 {
 	unsigned long completed_reqs = 0;
 	unsigned long flags;
 
 	spin_lock_irqsave(&hwq->cq_lock, flags);
 	ufshcd_mcq_update_cq_tail_slot(hwq);
-	while (!ufshcd_mcq_is_cq_empty(hwq)) {
+	while (!ufshcd_mcq_is_cq_empty(hwq) && completed_reqs < max_compl) {
 		ufshcd_mcq_process_cqe(hba, hwq);
 		ufshcd_mcq_inc_cq_head_slot(hwq);
 		completed_reqs++;
@@ -360,6 +361,13 @@ unsigned long ufshcd_mcq_poll_cqe_lock(struct ufs_hba *hba,
 
 	return completed_reqs;
 }
+EXPORT_SYMBOL_GPL(ufshcd_mcq_poll_n_cqe_lock);
+
+unsigned long ufshcd_mcq_poll_cqe_lock(struct ufs_hba *hba,
+				       struct ufs_hw_queue *hwq)
+{
+	return ufshcd_mcq_poll_n_cqe_lock(hba, hwq, hwq->max_entries);
+}
 EXPORT_SYMBOL_GPL(ufshcd_mcq_poll_cqe_lock);
 
 void ufshcd_mcq_make_queues_operational(struct ufs_hba *hba)
include/ufs/ufshcd.h +3 -0
index cfbc75d8df83..aa5f9b31ea86 100644
@@ -1475,6 +1475,9 @@ void ufshcd_mcq_config_mac(struct ufs_hba *hba, u32 max_active_cmds);
 unsigned int ufshcd_mcq_queue_cfg_addr(struct ufs_hba *hba);
 u32 ufshcd_mcq_read_cqis(struct ufs_hba *hba, int i);
 void ufshcd_mcq_write_cqis(struct ufs_hba *hba, u32 val, int i);
+unsigned long ufshcd_mcq_poll_n_cqe_lock(struct ufs_hba *hba,
+					 struct ufs_hw_queue *hwq,
+					 unsigned int max_compl);
 unsigned long ufshcd_mcq_poll_cqe_lock(struct ufs_hba *hba,
 					 struct ufs_hw_queue *hwq);
 void ufshcd_mcq_make_queues_operational(struct ufs_hba *hba);
#2 lore raw

Re: [PATCH v2 1/2] ufs: core: Introduce ufshcd_mcq_poll_n_cqe_lock()

Quoted 11 lines · Bart Van Assche

Introduce a new function for processing completions that accepts an upper limit for the number of completions to poll. Tell ufshcd_mcq_poll_cqe_lock() to poll at most hwq->max_entries. This is sufficient to poll all pending completions since there are never more than hwq->max_entries - 1 completions on a completion queue. This patch
prepares for reducing the interrupt latency.

---

Hi Bart,

May I ask if you have any plans to continue upstreaming this patch in the future?

Thanks.
Peter

author reviewer
#3 lore raw

Re: [PATCH v2 1/2] ufs: core: Introduce ufshcd_mcq_poll_n_cqe_lock()

Quoted 2 lines · Peter Wang (王信友)

May I ask if you have any plans to continue upstreaming this patch in the future?

Hi Peter,

After I posted this patch series I learned that even processing a single completion can cause interrupts to be disabled for too long. I think the root cause is in F2FS (f2fs_write_end_io()). I have reported this to the F2FS team and I'm waiting for their feedback. After F2FS has been improved I will repeat my measurements and reduce the number of completions processed in interrupt context if that is still necessary.

Thanks,

Bart.

#4 lore raw

Re: [PATCH v2 1/2] ufs: core: Introduce ufshcd_mcq_poll_n_cqe_lock()

Quoted 17 lines · Bart Van Assche
Quoted 2 lines · Peter Wang (王信友)

May I ask if you have any plans to continue upstreaming this patch in the future?

Hi Peter,

After I posted this patch series I learned that even processing a single
completion can cause interrupts to be disabled for too long. I think the
root cause is in F2FS (f2fs_write_end_io()). I have reported this to the
F2FS team and I'm waiting for their feedback. After F2FS has been improved I will repeat my measurements and reduce the number of completions processed in interrupt context if that is still necessary.

Quoted 3 lines

Thanks,

Bart.

Hi Bart,

I'm sure we're facing the same issue. F2FS hooks a heavy function in the UFS ISR callback. However, the UFS ISR still cannot acquire the spinlock and executes an unknown task that disables IRQs, the duration of which is controlled by F2FS.
I plan to fine-tune this by posting another patch.

Thanks.
Peter

author reviewer
#5 lore raw

Re: [PATCH v2 1/2] ufs: core: Introduce ufshcd_mcq_poll_n_cqe_lock()

Quoted 3 lines · Peter Wang (王信友)

However, the UFS ISR still cannot acquire
the spinlock and executes an unknown task that disables IRQs, the duration of which is controlled by F2FS.

I see this as an F2FS bug. It is widely known that bi_end_io callbacks may be called from interrupt context and hence should finish quickly.

Thanks,

Bart.

author reviewer
#6 lore raw

PATCH v2 2/2 ufs: qcom: Reduce interrupt latency

In reply to Bart Van Assche

Defer completion processing to thread context on slower CPU cores to prevent interrupt latency spikes. On the fastest CPU cores, keep processing all completions in interrupt context.

---
 drivers/ufs/host/ufs-qcom.c | 33 +++++++++++++++++++++++++++++----
 1 file changed, 29 insertions(+), 4 deletions(-)
drivers/ufs/host/ufs-qcom.c +29 -4
index 5a58ffef3d27..4aabbd14631e 100644
@@ -2370,6 +2370,16 @@ struct ufs_qcom_irq {
 	struct ufs_hba		*hba;
 };
 
+static irqreturn_t ufs_qcom_mcq_threaded_esi_handler(int irq, void *data)
+{
+	struct ufs_qcom_irq *qi = data;
+	struct ufs_hba *hba = qi->hba;
+
+	ufshcd_mcq_poll_cqe_lock(hba, &hba->uhq[qi->idx]);
+
+	return IRQ_HANDLED;
+}
+
 static irqreturn_t ufs_qcom_mcq_esi_handler(int irq, void *data)
 {
 	struct ufs_qcom_irq *qi = data;
@@ -2377,9 +2387,22 @@ static irqreturn_t ufs_qcom_mcq_esi_handler(int irq, void *data)
 	struct ufs_hw_queue *hwq = &hba->uhq[qi->idx];
 
 	ufshcd_mcq_write_cqis(hba, 0x1, qi->idx);
-	ufshcd_mcq_poll_cqe_lock(hba, hwq);
 
-	return IRQ_HANDLED;
+	if (arch_scale_cpu_capacity(raw_smp_processor_id()) ==
+	    SCHED_CAPACITY_SCALE) {
+		ufshcd_mcq_poll_cqe_lock(hba, hwq);
+		return IRQ_HANDLED;
+	}
+
+	if (ufshcd_mcq_poll_n_cqe_lock(hba, hwq, 4) < 4)
+		return IRQ_HANDLED;
+
+	/*
+	 * Defer further completion processing to thread context because
+	 * processing a large number of completions in interrupt context on
+	 * slower CPU cores can result in unacceptably high interrupt latencies.
+	 */
+	return IRQ_WAKE_THREAD;
 }
 
 static int ufs_qcom_config_esi(struct ufs_hba *hba)
@@ -2415,8 +2438,10 @@ static int ufs_qcom_config_esi(struct ufs_hba *hba)
 		qi[idx].idx = idx;
 		qi[idx].hba = hba;
 
-		ret = devm_request_irq(hba->dev, qi[idx].irq, ufs_qcom_mcq_esi_handler,
-				       IRQF_SHARED, "qcom-mcq-esi", qi + idx);
+		ret = devm_request_threaded_irq(hba->dev, qi[idx].irq,
+			ufs_qcom_mcq_esi_handler,
+			ufs_qcom_mcq_threaded_esi_handler,
+			IRQF_SHARED | IRQF_ONESHOT, "qcom-mcq-esi", qi + idx);
 		if (ret) {
 			dev_err(hba->dev, "%s: Failed to request IRQ for %d, err = %d\n",
 				__func__, qi[idx].irq, ret);
#7 lore raw

Re: [PATCH v2 2/2] ufs: qcom: Reduce interrupt latency

Hi Bart!

drivers/ufs/host/ufs-qcom.c static irqreturn_t ufs_qcom_mcq_esi_handler(int irq, void *data) · line 2398

Quoted 2 lines
drivers/ufs/host/ufs-qcom.c +2 -0
+	if (ufshcd_mcq_poll_n_cqe_lock(hba, hwq, 4) < 4)
+		return IRQ_HANDLED;

Magic numbers...

Signature · 1 line
Martin K. Petersen
#8 lore raw
In reply to Bart Van Assche

Bart,

Quoted 6 lines

On Android systems it is important to keep the time spent in interrupts short. This keeps the user interface responsive and prevents audio stuttering. Hence this patch series to reduce the time spent in the UFS interrupt handler. Please consider this patch series for the next merge window after test results have been shared by Qualcomm.

https://sashiko.dev/#/patchset/20260402171404.3008494-1-bvanassche%40acm.org

Signature · 1 line
Martin K. Petersen
author reviewer
#9 lore raw
Quoted 8 lines · Martin K. Petersen
Quoted 6 lines

On Android systems it is important to keep the time spent in interrupts short. This keeps the user interface responsive and prevents audio stuttering. Hence this patch series to reduce the time spent in the UFS interrupt handler. Please consider this patch series for the next merge window after test results have been shared by Qualcomm.

https://sashiko.dev/#/patchset/20260402171404.3008494-1-bvanassche%40acm.org

Right, the threaded interrupt handler needs more work.

PS: I asked the Sashiko owner to enable Sashiko for the SCSI and block mailing lists.

Thanks,

Bart.