ufs-qcom: Reduce interrupt latency
Subsystems (4)
Bart Van Assche
reviewerv2 cover ufs-qcom: Reduce interrupt latency
Hi Martin,
On Android systems it is important to keep the time spent in interrupts short. This keeps the user interface responsive and prevents audio stuttering. Hence this patch series to reduce the time spent in the UFS interrupt handler. Please consider this patch series for the next merge window after test results have been shared by Qualcomm.
Thanks,
Bart.
Changes compared to v1:
- Dropped the ufshcd_mcq_compl_all_cqes_lock() changes.
- Renamed ufshcd_mcq_poll_cqe_lock_n() into ufshcd_mcq_poll_n_cqe_lock().
Bart Van Assche (2):
ufs: core: Introduce ufshcd_mcq_poll_n_cqe_lock()
ufs: qcom: Reduce interrupt latency
drivers/ufs/core/ufs-mcq.c | 14 +++++++++++--- drivers/ufs/host/ufs-qcom.c | 33 +++++++++++++++++++++++++++++---- include/ufs/ufshcd.h | 3 +++ 3 files changed, 43 insertions(+), 7 deletions(-)
Bart Van Assche
reviewerPATCH v2 1/2 ufs: core: Introduce ufshcd_mcq_poll_n_cqe_lock()
Introduce a new function for processing completions that accepts an upper limit for the number of completions to poll. Tell ufshcd_mcq_poll_cqe_lock() to poll at most hwq->max_entries. This is sufficient to poll all pending completions since there are never more than hwq->max_entries - 1 completions on a completion queue. This patch prepares for reducing the interrupt latency.
--- drivers/ufs/core/ufs-mcq.c | 14 +++++++++++--- include/ufs/ufshcd.h | 3 +++ 2 files changed, 14 insertions(+), 3 deletions(-)
drivers/ufs/core/ufs-mcq.c +11 -3
@@ -340,15 +340,16 @@ void ufshcd_mcq_compl_all_cqes_lock(struct ufs_hba *hba,spin_unlock_irqrestore (&hwq->cq_lock, flags); } -unsigned long ufshcd_mcq_poll_cqe_lock (struct ufs_hba *hba, -struct ufs_hw_queue *hwq) +unsigned long ufshcd_mcq_poll_n_cqe_lock (struct ufs_hba *hba, +struct ufs_hw_queue *hwq, +unsigned int max_compl) {unsigned long completed_reqs =0 ;unsigned long flags;spin_lock_irqsave (&hwq->cq_lock, flags);ufshcd_mcq_update_cq_tail_slot (hwq); -while (!ufshcd_mcq_is_cq_empty (hwq)) { +while (!ufshcd_mcq_is_cq_empty (hwq) && completed_reqs < max_compl) {ufshcd_mcq_process_cqe (hba, hwq);ufshcd_mcq_inc_cq_head_slot (hwq); completed_reqs++;
@@ -360,6 +361,13 @@ unsigned long ufshcd_mcq_poll_cqe_lock(struct ufs_hba *hba,return completed_reqs; } +EXPORT_SYMBOL_GPL (ufshcd_mcq_poll_n_cqe_lock); + +unsigned long ufshcd_mcq_poll_cqe_lock (struct ufs_hba *hba, +struct ufs_hw_queue *hwq) +{ +return ufshcd_mcq_poll_n_cqe_lock (hba, hwq, hwq->max_entries); +}EXPORT_SYMBOL_GPL (ufshcd_mcq_poll_cqe_lock);void ufshcd_mcq_make_queues_operational (struct ufs_hba *hba)
include/ufs/ufshcd.h +3 -0
@@ -1475,6 +1475,9 @@ void ufshcd_mcq_config_mac(struct ufs_hba *hba, u32 max_active_cmds);unsigned int ufshcd_mcq_queue_cfg_addr (struct ufs_hba *hba);u32 ufshcd_mcq_read_cqis (struct ufs_hba *hba,int i);void ufshcd_mcq_write_cqis (struct ufs_hba *hba,u32 val,int i); +unsigned long ufshcd_mcq_poll_n_cqe_lock (struct ufs_hba *hba, +struct ufs_hw_queue *hwq, +unsigned int max_compl);unsigned long ufshcd_mcq_poll_cqe_lock (struct ufs_hba *hba,struct ufs_hw_queue *hwq);void ufshcd_mcq_make_queues_operational (struct ufs_hba *hba);
Re: [PATCH v2 1/2] ufs: core: Introduce ufshcd_mcq_poll_n_cqe_lock()
Quoted 11 lines · Bart Van Assche
Introduce a new function for processing completions that accepts an upper limit for the number of completions to poll. Tell
ufshcd_mcq_poll_cqe_lock()to poll at mosthwq->max_entries. This is sufficient to poll all pending completions since there are never more thanhwq->max_entries- 1 completions on a completion queue. This patch
prepares for reducing the interrupt latency.Reviewed-by Peter Wang <[email protected]>Signed-off-by Bart Van Assche <[email protected]>---
Hi Bart,
May I ask if you have any plans to continue upstreaming this patch in the future?
Thanks.
Peter
Bart Van Assche
reviewerRe: [PATCH v2 1/2] ufs: core: Introduce ufshcd_mcq_poll_n_cqe_lock()
Quoted 2 lines · Peter Wang (王信友)
May I ask if you have any plans to continue upstreaming this patch in the future?
Hi Peter,
After I posted this patch series I learned that even processing a single completion can cause interrupts to be disabled for too long. I think the root cause is in F2FS (f2fs_write_end_io()). I have reported this to the F2FS team and I'm waiting for their feedback. After F2FS has been improved I will repeat my measurements and reduce the number of completions processed in interrupt context if that is still necessary.
Thanks,
Bart.
Re: [PATCH v2 1/2] ufs: core: Introduce ufshcd_mcq_poll_n_cqe_lock()
Quoted 17 lines · Bart Van Assche
Quoted 2 lines · Peter Wang (王信友)
May I ask if you have any plans to continue upstreaming this patch in the future?
Hi Peter,
After I posted this patch series I learned that even processing a single
completion can cause interrupts to be disabled for too long. I think the
root cause is in F2FS (f2fs_write_end_io()). I have reported this to the
F2FS team and I'm waiting for their feedback. After F2FS has been improved I will repeat my measurements and reduce the number of completions processed in interrupt context if that is still necessary.
Quoted 3 lines
Thanks,
Bart.
Hi Bart,
I'm sure we're facing the same issue. F2FS hooks a heavy function in the UFS ISR callback. However, the UFS ISR still cannot acquire the spinlock and executes an unknown task that disables IRQs, the duration of which is controlled by F2FS.
I plan to fine-tune this by posting another patch.
Thanks.
Peter
Bart Van Assche
reviewerRe: [PATCH v2 1/2] ufs: core: Introduce ufshcd_mcq_poll_n_cqe_lock()
Quoted 3 lines · Peter Wang (王信友)
However, the UFS ISR still cannot acquire
the spinlock and executes an unknown task that disables IRQs, the duration of which is controlled by F2FS.
I see this as an F2FS bug. It is widely known that bi_end_io callbacks may be called from interrupt context and hence should finish quickly.
Thanks,
Bart.
Bart Van Assche
reviewerPATCH v2 2/2 ufs: qcom: Reduce interrupt latency
In reply to Bart Van AsscheDefer completion processing to thread context on slower CPU cores to prevent interrupt latency spikes. On the fastest CPU cores, keep processing all completions in interrupt context.
--- drivers/ufs/host/ufs-qcom.c | 33 +++++++++++++++++++++++++++++---- 1 file changed, 29 insertions(+), 4 deletions(-)
drivers/ufs/host/ufs-qcom.c +29 -4
@@ -2370,6 +2370,16 @@ struct ufs_qcom_irq {struct ufs_hba *hba; }; +static irqreturn_t ufs_qcom_mcq_threaded_esi_handler (int irq,void *data) +{ +struct ufs_qcom_irq *qi = data; +struct ufs_hba *hba = qi->hba; + +ufshcd_mcq_poll_cqe_lock (hba, &hba->uhq[qi->idx]); + +return IRQ_HANDLED ; +} +static irqreturn_t ufs_qcom_mcq_esi_handler (int irq,void *data) {struct ufs_qcom_irq *qi = data;
@@ -2377,9 +2387,22 @@ static irqreturn_t ufs_qcom_mcq_esi_handler(int irq, void *data)struct ufs_hw_queue *hwq = &hba->uhq[qi->idx];ufshcd_mcq_write_cqis (hba,0x1 , qi->idx); -ufshcd_mcq_poll_cqe_lock (hba, hwq); -return IRQ_HANDLED ; +if (arch_scale_cpu_capacity (raw_smp_processor_id ()) == +SCHED_CAPACITY_SCALE ) { +ufshcd_mcq_poll_cqe_lock (hba, hwq); +return IRQ_HANDLED ; + } + +if (ufshcd_mcq_poll_n_cqe_lock (hba, hwq,4 ) <4 ) +return IRQ_HANDLED ; + +/* +* Defer further completion processing to thread context because +* processing a large number of completions in interrupt context on +* slower CPU cores can result in unacceptably high interrupt latencies. +*/ +return IRQ_WAKE_THREAD ; }static int ufs_qcom_config_esi (struct ufs_hba *hba)
@@ -2415,8 +2438,10 @@ static int ufs_qcom_config_esi(struct ufs_hba *hba) qi[idx].idx = idx; qi[idx].hba = hba; - ret =devm_request_irq (hba->dev, qi[idx].irq, ufs_qcom_mcq_esi_handler, -IRQF_SHARED ,"qcom-mcq-esi" , qi + idx); + ret =devm_request_threaded_irq (hba->dev, qi[idx].irq, + ufs_qcom_mcq_esi_handler, + ufs_qcom_mcq_threaded_esi_handler, +IRQF_SHARED |IRQF_ONESHOT ,"qcom-mcq-esi" , qi + idx);if (ret) {dev_err (hba->dev,"%s: Failed to request IRQ for %d, err = %d\n" , __func__, qi[idx].irq, ret);
Re: [PATCH v2 2/2] ufs: qcom: Reduce interrupt latency
Hi Bart!
drivers/ufs/host/ufs-qcom.c static irqreturn_t ufs_qcom_mcq_esi_handler(int irq, void *data) · line 2398
Quoted 2 lines
drivers/ufs/host/ufs-qcom.c +2
-0+if (ufshcd_mcq_poll_n_cqe_lock (hba, hwq,4 ) <4 ) +return IRQ_HANDLED ;
Magic numbers...
Signature · 1 line
Martin K. Petersen
Bart,
Quoted 6 lines
On Android systems it is important to keep the time spent in interrupts short. This keeps the user interface responsive and prevents audio stuttering. Hence this patch series to reduce the time spent in the UFS interrupt handler. Please consider this patch series for the next merge window after test results have been shared by Qualcomm.
https://sashiko.dev/#/patchset/20260402171404.3008494-1-bvanassche%40acm.org
Signature · 1 line
Martin K. Petersen
Bart Van Assche
reviewerQuoted 8 lines · Martin K. Petersen
Quoted 6 lines
On Android systems it is important to keep the time spent in interrupts short. This keeps the user interface responsive and prevents audio stuttering. Hence this patch series to reduce the time spent in the UFS interrupt handler. Please consider this patch series for the next merge window after test results have been shared by Qualcomm.
https://sashiko.dev/#/patchset/20260402171404.3008494-1-bvanassche%40acm.org
Right, the threaded interrupt handler needs more work.
PS: I asked the Sashiko owner to enable Sashiko for the SCSI and block mailing lists.
Thanks,
Bart.