Prepare for upstreaming Pixel 6 UFS support
Subsystems (2)
Bart Van Assche
reviewerv2 cover Prepare for upstreaming Pixel 6 UFS support
Hi Martin,
The UFS controller in the Google Pixel 6 phone requires that SCSI command processing is suspended while reprogramming encryption keys. The patches in this series are a first step towards integrating support in the upstream kernel for the UFS controller in the Pixel 6. Please consider these patches for inclusion in kernel v5.19.
Note: instructions for downloading the Pixel 6 source code are available at
https://source.android.com/setup/build/building-kernels.
Thanks,
Bart.
Changes compared to v1:
- Addressed Avi's review comments.
- Added patch "Allow UFS host drivers to override the sg entry size".
Bart Van Assche (4):
scsi: ufs: Reduce the clock scaling latency
scsi: ufs: Move a clock scaling check
scsi: ufs: Pass the clock scaling timeout as an argument
scsi: ufs: Add suspend/resume SCSI command processing support
Eric Biggers (1):
scsi: ufs: Allow UFS host drivers to override the sg entry size
drivers/ufs/core/ufshcd.c | 89 +++++++++++++++++++++++++++------------ drivers/ufs/host/Kconfig | 10 +++++ include/ufs/ufshcd.h | 35 +++++++++++++++ include/ufs/ufshci.h | 9 +++- 4 files changed, 114 insertions(+), 29 deletions(-)
Bart Van Assche
reviewerPATCH v2 1/5 scsi: ufs: Reduce the clock scaling latency
Wait at most 20 ms before rechecking the doorbells instead of waiting for a potentially long time between doorbell checks.
--- drivers/ufs/core/ufshcd.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/ufs/core/ufshcd.c +1 -1
@@ -1149,7 +1149,7 @@ static int ufshcd_wait_for_doorbell_clr(struct ufs_hba *hba, }spin_unlock_irqrestore (hba->host->host_lock, flags); -schedule (); +io_schedule_timeout (msecs_to_jiffies (20 ));if (ktime_to_us (ktime_sub (ktime_get (), start)) > wait_timeout_us) { timeout = true;
Re: [PATCH v2 1/5] scsi: ufs: Reduce the clock scaling latency
Quoted 3 lines · Bart Van Assche
Wait at most 20 ms before rechecking the doorbells instead of waiting for a potentially long time between doorbell checks.
Bart Van Assche
reviewerPATCH v2 2/5 scsi: ufs: Move a clock scaling check
In reply to Bart Van AsscheMove a check related to clock scaling into ufshcd_devfreq_scale(). This patch prepares for adding a second ufshcd_clock_scaling_prepare() caller in a function not related to clock scaling.
--- drivers/ufs/core/ufshcd.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
drivers/ufs/core/ufshcd.c +9 -2
@@ -1231,8 +1231,7 @@ static int ufshcd_clock_scaling_prepare(struct ufs_hba *hba)ufshcd_scsi_block_requests (hba);down_write (&hba->clk_scaling_lock); -if (!hba->clk_scaling.is_allowed || -ufshcd_wait_for_doorbell_clr (hba,DOORBELL_CLR_TOUT_US )) { +if (ufshcd_wait_for_doorbell_clr (hba,DOORBELL_CLR_TOUT_US )) { ret = -EBUSY ;up_write (&hba->clk_scaling_lock);ufshcd_scsi_unblock_requests (hba);
@@ -1270,10 +1269,18 @@ static int ufshcd_devfreq_scale(struct ufs_hba *hba, bool scale_up)int ret =0 ;bool is_writelock = true; +if (!hba->clk_scaling.is_allowed) +return -EBUSY ; + ret =ufshcd_clock_scaling_prepare (hba);if (ret)return ret; +if (!hba->clk_scaling.is_allowed) { + ret = -EBUSY ; + goto out_unprepare; + } +/* scale down the gear before scaling down clocks */ if (!scale_up) { ret =ufshcd_scale_gear (hba, false);
Bart Van Assche
reviewerPATCH v2 3/5 scsi: ufs: Pass the clock scaling timeout as an argument
In reply to Bart Van AsschePrepare for adding an additional ufshcd_clock_scaling_prepare() call with a different timeout.
--- drivers/ufs/core/ufshcd.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)
drivers/ufs/core/ufshcd.c +15 -4
@@ -1116,6 +1116,12 @@ static u32 ufshcd_pending_cmds(struct ufs_hba *hba)return pending; } +/* +* Wait until all pending SCSI commands and TMFs have finished or the timeout +* has expired. +* +* Return: 0 upon success; -EBUSY upon timeout. +*/ static int ufshcd_wait_for_doorbell_clr (struct ufs_hba *hba,u64 wait_timeout_us) {
@@ -1220,9 +1226,14 @@ static int ufshcd_scale_gear(struct ufs_hba *hba, bool scale_up)return ret; } -static int ufshcd_clock_scaling_prepare (struct ufs_hba *hba) +/* +* Wait until all pending SCSI commands and TMFs have finished or the timeout +* has expired. +* +* Return: 0 upon success; -EBUSY upon timeout. +*/ +static int ufshcd_clock_scaling_prepare (struct ufs_hba *hba,u64 timeout_us) { -#define DOORBELL_CLR_TOUT_US (1000 * 1000)/* 1 sec */ int ret =0 ;/* * make sure that there are no outstanding requests when
@@ -1231,7 +1242,7 @@ static int ufshcd_clock_scaling_prepare(struct ufs_hba *hba)ufshcd_scsi_block_requests (hba);down_write (&hba->clk_scaling_lock); -if (ufshcd_wait_for_doorbell_clr (hba,DOORBELL_CLR_TOUT_US )) { +if (ufshcd_wait_for_doorbell_clr (hba, timeout_us)) { ret = -EBUSY ;up_write (&hba->clk_scaling_lock);ufshcd_scsi_unblock_requests (hba);
@@ -1272,7 +1283,7 @@ static int ufshcd_devfreq_scale(struct ufs_hba *hba, bool scale_up)if (!hba->clk_scaling.is_allowed)return -EBUSY ; - ret =ufshcd_clock_scaling_prepare (hba); + ret =ufshcd_clock_scaling_prepare (hba,1 *USEC_PER_SEC );if (ret)return ret;
Bart Van Assche
reviewerPATCH v2 4/5 scsi: ufs: Add suspend/resume SCSI command processing support
In reply to Bart Van AsscheThis functionality is needed by UFS drivers to e.g. suspend SCSI command processing while reprogramming encryption keys if the hardware does not support concurrent I/O and key reprogramming.
--- drivers/ufs/core/ufshcd.c | 20 ++++++++++++++++++++ include/ufs/ufshcd.h | 3 +++ 2 files changed, 23 insertions(+)
drivers/ufs/core/ufshcd.c +20 -0
@@ -1700,6 +1700,26 @@ static void ufshcd_ungate_work(struct work_struct *work)ufshcd_scsi_unblock_requests (hba); } +/* +* Block processing of new SCSI commands and wait until pending SCSI +* commands and TMFs have finished. ufshcd_exec_dev_cmd() and +* ufshcd_issue_devman_upiu_cmd() are not affected by this function. +* +* Return: 0 upon success; -EBUSY upon timeout. +*/ +int ufshcd_freeze_scsi_devs (struct ufs_hba *hba,u64 timeout_us) +{ +return ufshcd_clock_scaling_prepare (hba, timeout_us); +} +EXPORT_SYMBOL_GPL (ufshcd_freeze_scsi_devs); + +/* Resume processing of SCSI commands. */ +void ufshcd_unfreeze_scsi_devs (struct ufs_hba *hba) +{ +ufshcd_clock_scaling_unprepare (hba, true); +} +EXPORT_SYMBOL_GPL (ufshcd_unfreeze_scsi_devs); +/** * ufshcd_hold - Enable clocks that were gated earlier due to ufshcd_release. * Also, exit from hibern8 mode and set the link as active.
include/ufs/ufshcd.h +3 -0
@@ -1194,6 +1194,9 @@ void ufshcd_release(struct ufs_hba *hba);void ufshcd_clkgate_delay_set (struct device *dev,unsigned long value); +int ufshcd_freeze_scsi_devs (struct ufs_hba *hba,u64 timeout_us); +void ufshcd_unfreeze_scsi_devs (struct ufs_hba *hba); +void ufshcd_map_desc_id_to_length (struct ufs_hba *hba,enum desc_idn desc_id,int *desc_length);
RE: [PATCH v2 4/5] scsi: ufs: Add suspend/resume SCSI command processing support
Quoted 5 lines
This functionality is needed by UFS drivers to e.g. suspend SCSI command processing while reprogramming encryption keys if the hardware does not support concurrent I/O and key reprogramming.
Signed-off-by Bart Van Assche <[email protected]>
Quoted 51 lines
--- drivers/ufs/core/ufshcd.c | 20 ++++++++++++++++++++ include/ufs/ufshcd.h | 3 +++ 2 files changed, 23 insertions(+)drivers/ufs/core/ufshcd.c +20
-0@@ -1700,6 +1700,26 @@ static void ufshcd_ungate_work(struct work_struct *work)ufshcd_scsi_unblock_requests (hba ); } +/* +* Block processing of new SCSI commands and wait until pending SCSI +* commands and TMFs have finished. ufshcd_exec_dev_cmd() and +* ufshcd_issue_devman_upiu_cmd() are not affected by this function. +* +* Return: 0 upon success; -EBUSY upon timeout. +*/ +int ufshcd_freeze_scsi_devs (struct ufs_hba *hba,u64 timeout_us) +{ +return ufshcd_clock_scaling_prepare (hba, timeout_us); +} +EXPORT_SYMBOL_GPL (ufshcd_freeze_scsi_devs); + +/* Resume processing of SCSI commands. */ +void ufshcd_unfreeze_scsi_devs (struct ufs_hba *hba) +{ +ufshcd_clock_scaling_unprepare (hba, true); +} +EXPORT_SYMBOL_GPL (ufshcd_unfreeze_scsi_devs); +/** * ufshcd_hold - Enable clocks that were gated earlier due to ufshcd_release. * Also, exit from hibern8 mode and set the link as active.
include/ufs/ufshcd.h +3
-0@@ -1194,6 +1194,9 @@ void ufshcd_release(struct ufs_hba *hba);void ufshcd_clkgate_delay_set (struct device *dev,unsigned long value); +int ufshcd_freeze_scsi_devs (struct ufs_hba *hba,u64 timeout_us); +void ufshcd_unfreeze_scsi_devs (struct ufs_hba *hba); +void ufshcd_map_desc_id_to_length (struct ufs_hba *hba,enum desc_idn desc_id,int *desc_length);
Bart Van Assche
reviewerPATCH v2 5/5 scsi: ufs: Allow UFS host drivers to override the sg entry size
In reply to Bart Van AsscheFrom: Eric Biggers <[email protected]>
Modify the UFSHCD core to allow 'struct ufshcd_sg_entry' to be variable-length. The default is the standard length, but variants can override ufs_hba::sg_entry_size with a larger value if there are vendor-specific fields following the standard ones.
This is needed to support inline encryption with ufs-exynos (FMP).
--- drivers/ufs/core/ufshcd.c | 39 ++++++++++++++++++--------------------- drivers/ufs/host/Kconfig | 10 ++++++++++ include/ufs/ufshcd.h | 32 ++++++++++++++++++++++++++++++++ include/ufs/ufshci.h | 9 +++++++-- 4 files changed, 67 insertions(+), 23 deletions(-)
drivers/ufs/core/ufshcd.c +18 -21
@@ -523,7 +523,7 @@ void ufshcd_print_trs(struct ufs_hba *hba, unsigned long bitmap, bool pr_prdt) prdt_length =le16_to_cpu ( lrbp->utr_descriptor_ptr->prd_table_length);if (hba->quirks &UFSHCD_QUIRK_PRDT_BYTE_GRAN ) - prdt_length /=sizeof (struct ufshcd_sg_entry ); + prdt_length /=ufshcd_sg_entry_size (hba); dev_err(hba->dev, "UPIU[%d] - PRDT - %d entries phys@0x%llx\n",
@@ -532,7 +532,7 @@ void ufshcd_print_trs(struct ufs_hba *hba, unsigned long bitmap, bool pr_prdt)if (pr_prdt)ufshcd_hex_dump ("UPIU PRDT: " , lrbp->ucd_prdt_ptr, -sizeof (struct ufshcd_sg_entry ) * prdt_length); +ufshcd_sg_entry_size (hba) * prdt_length); } }
@@ -2437,7 +2437,7 @@ int ufshcd_send_uic_cmd(struct ufs_hba *hba, struct uic_command *uic_cmd)*/ static int ufshcd_map_sg (struct ufs_hba *hba,struct ufshcd_lrb *lrbp) { -struct ufshcd_sg_entry *prd_table; +struct ufshcd_sg_entry *prd;struct scatterlist *sg;struct scsi_cmnd *cmd;int sg_segments;
@@ -2452,13 +2452,12 @@ static int ufshcd_map_sg(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)if (hba->quirks &UFSHCD_QUIRK_PRDT_BYTE_GRAN ) lrbp->utr_descriptor_ptr->prd_table_length = -cpu_to_le16 ((sg_segments * -sizeof (struct ufshcd_sg_entry ))); +cpu_to_le16 (sg_segments *ufshcd_sg_entry_size (hba));else lrbp->utr_descriptor_ptr->prd_table_length =cpu_to_le16 (sg_segments); - prd_table = lrbp->ucd_prdt_ptr; + prd = lrbp->ucd_prdt_ptr;scsi_for_each_sg (cmd, sg, sg_segments, i) {const unsigned int len =sg_dma_len (sg);
@@ -2472,9 +2471,10 @@ static int ufshcd_map_sg(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)* indicates 4 bytes, '7' indicates 8 bytes, etc." */ WARN_ONCE (len >256 *1024 ,"len = %#x\n" , len); - prd_table[i].size =cpu_to_le32 (len -1 ); - prd_table[i].addr =cpu_to_le64 (sg->dma_address); - prd_table[i].reserved =0 ; + prd->size =cpu_to_le32 (len -1 ); + prd->addr =cpu_to_le64 (sg->dma_address); + prd->reserved =0 ; + prd = (void *)prd +ufshcd_sg_entry_size (hba); } }else { lrbp->utr_descriptor_ptr->prd_table_length =0 ;
@@ -2767,10 +2767,11 @@ static int ufshcd_map_queues(struct Scsi_Host *shost)static void ufshcd_init_lrb (struct ufs_hba *hba,struct ufshcd_lrb *lrb,int i) { -struct utp_transfer_cmd_desc *cmd_descp = hba->ucdl_base_addr; +struct utp_transfer_cmd_desc *cmd_descp = (void *)hba->ucdl_base_addr + + i *sizeof_utp_transfer_cmd_desc (hba);struct utp_transfer_req_desc *utrdlp = hba->utrdl_base_addr;dma_addr_t cmd_desc_element_addr = hba->ucdl_dma_addr + - i *sizeof (struct utp_transfer_cmd_desc ); + i *sizeof_utp_transfer_cmd_desc (hba);u16 response_offset = offsetof(struct utp_transfer_cmd_desc , response_upiu);u16 prdt_offset = offsetof(struct utp_transfer_cmd_desc , prd_table);
@@ -2778,11 +2779,11 @@ static void ufshcd_init_lrb(struct ufs_hba *hba, struct ufshcd_lrb *lrb, int i) lrb->utr_descriptor_ptr = utrdlp + i; lrb->utrd_dma_addr = hba->utrdl_dma_addr + i *sizeof (struct utp_transfer_req_desc ); - lrb->ucd_req_ptr = (struct utp_upiu_req *)(cmd_descp + i); + lrb->ucd_req_ptr = (struct utp_upiu_req *)cmd_descp; lrb->ucd_req_dma_addr = cmd_desc_element_addr; - lrb->ucd_rsp_ptr = (struct utp_upiu_rsp *)cmd_descp[i].response_upiu; + lrb->ucd_rsp_ptr = (struct utp_upiu_rsp *)cmd_descp->response_upiu; lrb->ucd_rsp_dma_addr = cmd_desc_element_addr + response_offset; - lrb->ucd_prdt_ptr = cmd_descp[i].prd_table; + lrb->ucd_prdt_ptr = (struct ufshcd_sg_entry *)cmd_descp->prd_table; lrb->ucd_prdt_dma_addr = cmd_desc_element_addr + prdt_offset; }
@@ -3669,7 +3670,7 @@ static int ufshcd_memory_alloc(struct ufs_hba *hba)size_t utmrdl_size, utrdl_size, ucdl_size;/* Allocate memory for UTP command descriptors */ - ucdl_size = (sizeof (struct utp_transfer_cmd_desc ) * hba->nutrs); + ucdl_size =sizeof_utp_transfer_cmd_desc (hba) * hba->nutrs; hba->ucdl_base_addr = dmam_alloc_coherent(hba->dev, ucdl_size, &hba->ucdl_dma_addr,
@@ -3763,7 +3764,7 @@ static void ufshcd_host_memory_configure(struct ufs_hba *hba) prdt_offset = offsetof(struct utp_transfer_cmd_desc , prd_table); - cmd_desc_size =sizeof (struct utp_transfer_cmd_desc ); + cmd_desc_size =sizeof_utp_transfer_cmd_desc (hba); cmd_desc_dma_addr = hba->ucdl_dma_addr;for (i =0 ; i < hba->nutrs; i++) {
@@ -9601,6 +9602,7 @@ int ufshcd_alloc_host(struct device *dev, struct ufs_hba **hba_handle) hba->dev = dev; hba->dev_ref_clk_freq =REF_CLK_FREQ_INVAL ; hba->nop_out_timeout =NOP_OUT_TIMEOUT ; +ufshcd_set_sg_entry_size (hba,sizeof (struct ufshcd_sg_entry ));INIT_LIST_HEAD (&hba->clk_list_head);spin_lock_init (&hba->outstanding_lock);
@@ -9979,11 +9981,6 @@ static int __init ufshcd_core_init(void) {int ret; -/* Verify that there are no gaps in struct utp_transfer_cmd_desc. */ -static_assert (sizeof (struct utp_transfer_cmd_desc ) == -2 *ALIGNED_UPIU_SIZE + -SG_ALL *sizeof (struct ufshcd_sg_entry )); -ufs_debugfs_init (); ret =scsi_register_driver (&ufs_dev_wlun_template.gendrv);
drivers/ufs/host/Kconfig +10 -0
@@ -124,3 +124,13 @@ config SCSI_UFS_EXYNOS Select this if you have UFS host controller on Samsung Exynos SoC. If unsure, say N. + +config SCSI_UFS_VARIABLE_SG_ENTRY_SIZE + bool "Variable size UTP physical region descriptor" + help + In the UFSHCI 3.0 standard the Physical Region Descriptor (PRD) is a + data structure used for transferring data between host and UFS + device. This data structure describes a single region in physical + memory. Although the standard requires that this data structure has a + size of 16 bytes, for some controllers this data structure has a + different size. Enable this option for UFS controllers that need it.
include/ufs/ufshcd.h +32 -0
@@ -744,6 +744,9 @@ struct ufs_hba_monitor {* @vops: pointer to variant specific operations * @vps: pointer to variant specific parameters * @priv: pointer to variant specific private data +#ifdef CONFIG_SCSI_UFS_VARIABLE_SG_ENTRY_SIZE +* @sg_entry_size: size of struct ufshcd_sg_entry (may include variant fields) +#endif * @irq: Irq number of the controller * @is_irq_enabled: whether or not the UFS controller interrupt is enabled. * @dev_ref_clk_freq: reference clock frequency
@@ -865,6 +868,9 @@ struct ufs_hba {const struct ufs_hba_variant_ops *vops;struct ufs_hba_variant_params *vps;void *priv; +#ifdef CONFIG_SCSI_UFS_VARIABLE_SG_ENTRY_SIZE +size_t sg_entry_size; +#endif unsigned int irq;bool is_irq_enabled;enum ufs_ref_clk_freq dev_ref_clk_freq;
@@ -967,6 +973,32 @@ struct ufs_hba {bool complete_put; }; +#ifdef CONFIG_SCSI_UFS_VARIABLE_SG_ENTRY_SIZE +static inline size_t ufshcd_sg_entry_size (const struct ufs_hba *hba) +{ +return hba->sg_entry_size; +} + +static inline void ufshcd_set_sg_entry_size (struct ufs_hba *hba,size_t sg_entry_size) +{ +WARN_ON_ONCE (sg_entry_size <sizeof (struct ufshcd_sg_entry )); + hba->sg_entry_size = sg_entry_size; +} +#else +static inline size_t ufshcd_sg_entry_size (const struct ufs_hba *hba) +{ +return sizeof (struct ufshcd_sg_entry ); +} + +#define ufshcd_set_sg_entry_size (hba, sg_entry_size) \ + ({ (void)(hba); BUILD_BUG_ON(sg_entry_size != sizeof(struct ufshcd_sg_entry)); }) +#endif + +static inline size_t sizeof_utp_transfer_cmd_desc (const struct ufs_hba *hba) +{ +return sizeof (struct utp_transfer_cmd_desc ) +SG_ALL *ufshcd_sg_entry_size (hba); +} +/* Returns true if clocks can be gated. Otherwise false */ static inline bool ufshcd_is_clkgating_allowed (struct ufs_hba *hba) {
include/ufs/ufshci.h +7 -2
@@ -426,18 +426,23 @@ struct ufshcd_sg_entry {__le64 addr;__le32 reserved;__le32 size; +/* +* followed by variant-specific fields if +* CONFIG_SCSI_UFS_VARIABLE_SG_ENTRY_SIZE has been defined. +*/ };/** * struct utp_transfer_cmd_desc - UTP Command Descriptor (UCD) * @command_upiu: Command UPIU Frame address * @response_upiu: Response UPIU Frame address -* @prd_table: Physical Region Descriptor +* @prd_table: Physical Region Descriptor: an array of SG_ALL struct +* ufshcd_sg_entry's. Variant-specific fields may be present after each. */ struct utp_transfer_cmd_desc {u8 command_upiu[ALIGNED_UPIU_SIZE ];u8 response_upiu[ALIGNED_UPIU_SIZE ]; -struct ufshcd_sg_entry prd_table[SG_ALL ]; +u8 prd_table[]; };/**
Avri Altman
reviewerRE: [PATCH v2 5/5] scsi: ufs: Allow UFS host drivers to override the sg entry size
Quoted 104 lines
From: Eric Biggers <[email protected]>
Modify the UFSHCD core to allow 'struct
ufshcd_sg_entry' to be variable-length. The default is the standard length, but variants can overrideufs_hba::sg_entry_sizewith a larger value if there are vendor-specific fields following the standard ones.This is needed to support inline encryption with ufs-exynos (FMP).
Cc Eric Biggers <[email protected]>Signed-off-by Eric Biggers <[email protected]>[ bvanassche: edited commit message and introduced
CONFIG_SCSI_UFS_VARIABLE_SG_ENTRY_SIZE]Signed-off-by Bart Van Assche <[email protected]>--- drivers/ufs/core/ufshcd.c | 39 ++++++++++++++++++--------------------- drivers/ufs/host/Kconfig | 10 ++++++++++ include/ufs/ufshcd.h | 32 ++++++++++++++++++++++++++++++++ include/ufs/ufshci.h | 9 +++++++-- 4 files changed, 67 insertions(+), 23 deletions(-)drivers/ufs/core/ufshcd.c +1
-1@@ -523,7 +523,7 @@ void ufshcd_print_trs(struct ufs_hba *hba, unsigned long bitmap, bool pr_prdt) prdt_length =le16_to_cpu ( lrbp->utr_descriptor_ptr->prd_table_length);if (hba->quirks &UFSHCD_QUIRK_PRDT_BYTE_GRAN ) - prdt_length /=sizeof (struct ufshcd_sg_entry ); + prdt_length /=ufshcd_sg_entry_size (hba); dev_err(hba->dev,"UPIU[%d] - PRDT - %d entries phys@0x%llx\n",drivers/ufs/core/ufshcd.c +2
-2@@ -532,7 +532,7 @@ void ufshcd_print_trs(struct ufs_hba *hba, unsigned long bitmap, bool pr_prdt) if (pr_prdt) ufshcd_hex_dump("UPIU PRDT: ", lrbp->ucd_prdt_ptr, - sizeof(struct ufshcd_sg_entry) * prdt_length); + ufshcd_sg_entry_size(hba) * prdt_length); } }@@ -2437,7 +2437,7 @@ int ufshcd_send_uic_cmd(struct ufs_hba *hba,struct uic_command *uic_cmd) */ static int ufshcd_map_sg (struct ufs_hba *hba,struct ufshcd_lrb *lrbp) { -struct ufshcd_sg_entry *prd_table; +struct ufshcd_sg_entry *prd;struct scatterlist *sg;struct scsi_cmnd *cmd;int sg_segments;drivers/ufs/core/ufshcd.c +2
-3@@ -2452,13 +2452,12 @@ static int ufshcd_map_sg(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)if (hba->quirks &UFSHCD_QUIRK_PRDT_BYTE_GRAN ) lrbp->utr_descriptor_ptr->prd_table_length = -cpu_to_le16 ((sg_segments * -sizeof (struct ufshcd_sg_entry ))); +cpu_to_le16 (sg_segments *ufshcd_sg_entry_size (hba));else lrbp->utr_descriptor_ptr->prd_table_length =cpu_to_le16 (sg_segments); - prd_table = lrbp->ucd_prdt_ptr; + prd = lrbp->ucd_prdt_ptr;scsi_for_each_sg (cmd, sg, sg_segments, i) {const unsigned int len = sg_dma_len(sg);drivers/ufs/core/ufshcd.c +4
-3@@ -2472,9 +2471,10 @@ static int ufshcd_map_sg(struct ufs_hba *hba,struct ufshcd_lrb *lrbp) * indicates 4 bytes, '7' indicates 8 bytes, etc." */ WARN_ONCE (len >256 *1024 ,"len = %#x\n" , len); - prd_table[i].size =cpu_to_le32 (len -1 ); - prd_table[i].addr =cpu_to_le64 (sg->dma_address); - prd_table[i].reserved =0 ; + prd->size =cpu_to_le32 (len -1 ); + prd->addr =cpu_to_le64 (sg->dma_address); + prd->reserved =0 ; + prd = (void *)prd +ufshcd_sg_entry_size (hba); } }else {lrbp->utr_descriptor_ptr->prd_table_length = 0;drivers/ufs/core/ufshcd.c +1
-1@@ -2767,10 +2767,11 @@ static int ufshcd_map_queues(struct Scsi_Host *shost)static void ufshcd_init_lrb (struct ufs_hba *hba,struct ufshcd_lrb *lrb,int i) { -struct utp_transfer_cmd_desc *cmd_descp = hba->ucdl_base_addr; + struct utp_transfer_cmd_desc *cmd_descp = (void *)hba-Quoted 1 line
ucdl_base_addr+drivers/ufs/core/ufshcd.c +2
-1+i *sizeof_utp_transfer_cmd_desc (hba );struct utp_transfer_req_desc *utrdlp = hba->utrdl_base_addr;dma_addr_t cmd_desc_element_addr = hba->ucdl_dma_addr + - i *sizeof (struct utp_transfer_cmd_desc ); + i *sizeof_utp_transfer_cmd_desc (hba);u16 response_offset = offsetof(struct utp_transfer_cmd_desc , response_upiu);u16 prdt_offset = offsetof(struct utp_transfer_cmd_desc , prd_table);
drivers/ufs/core/ufshcd.c static void ufshcd_init_lrb(struct ufs_hba *hba, · line 2783
Quoted 6 lines
drivers/ufs/core/ufshcd.c +1
-1@@ -2778,11 +2779,11 @@ static void ufshcd_init_lrb(struct ufs_hba *hba, struct ufshcd_lrb *lrb, int i) lrb->utr_descriptor_ptr = utrdlp + i; lrb->utrd_dma_addr = hba->utrdl_dma_addr + i *sizeof (struct utp_transfer_req_desc ); - lrb->ucd_req_ptr = (struct utp_upiu_req *)(cmd_descp + i); + lrb->ucd_req_ptr = (struct utp_upiu_req *)cmd_descp;
Maybe here cmd_descp->command_upiu ?
Quoted 30 lines
drivers/ufs/core/ufshcd.c +3
-3lrb->ucd_req_dma_addr = cmd_desc_element_addr; - lrb->ucd_rsp_ptr = (struct utp_upiu_rsp *)cmd_descp[i].response_upiu; + lrb->ucd_rsp_ptr = (struct utp_upiu_rsp *)cmd_descp->response_upiu; lrb->ucd_rsp_dma_addr = cmd_desc_element_addr + response_offset; - lrb->ucd_prdt_ptr = cmd_descp[i].prd_table; + lrb->ucd_prdt_ptr = (struct ufshcd_sg_entry *)cmd_descp->prd_table; lrb->ucd_prdt_dma_addr = cmd_desc_element_addr + prdt_offset; }@@ -3669,7 +3670,7 @@ static int ufshcd_memory_alloc(struct ufs_hba *hba)size_t utmrdl_size, utrdl_size, ucdl_size;/* Allocate memory for UTP command descriptors */ - ucdl_size = (sizeof (struct utp_transfer_cmd_desc ) * hba->nutrs); + ucdl_size =sizeof_utp_transfer_cmd_desc (hba) * hba->nutrs; hba->ucdl_base_addr = dmam_alloc_coherent(hba->dev, ucdl_size,&hba->ucdl_dma_addr,drivers/ufs/core/ufshcd.c +1
-1@@ -3763,7 +3764,7 @@ static void ufshcd_host_memory_configure(struct ufs_hba *hba) prdt_offset = offsetof(struct utp_transfer_cmd_desc , prd_table); - cmd_desc_size =sizeof (struct utp_transfer_cmd_desc ); + cmd_desc_size =sizeof_utp_transfer_cmd_desc (hba); cmd_desc_dma_addr = hba->ucdl_dma_addr;for (i = 0; i < hba->nutrs; i++) {
drivers/ufs/core/ufshcd.c int ufshcd_alloc_host(struct device *dev, struct · line 9606
Quoted 5 lines
drivers/ufs/core/ufshcd.c +1
-0@@ -9601,6 +9602,7 @@ int ufshcd_alloc_host(struct device *dev, struct ufs_hba **hba_handle) hba->dev = dev; hba->dev_ref_clk_freq =REF_CLK_FREQ_INVAL ; hba->nop_out_timeout =NOP_OUT_TIMEOUT ; +ufshcd_set_sg_entry_size (hba,sizeof (struct ufshcd_sg_entry ));
Where are you setting this variant for ufs-exynos? I would expect here a set_sg_entry_size vop.
Quoted 23 lines
drivers/ufs/core/ufshcd.c +0
-5INIT_LIST_HEAD (&hba->clk_list_head);spin_lock_init (&hba->outstanding_lock);@@ -9979,11 +9981,6 @@ static int __init ufshcd_core_init(void) {int ret; -/* Verify that there are no gaps in struct utp_transfer_cmd_desc. */ -static_assert (sizeof (struct utp_transfer_cmd_desc ) == -2 *ALIGNED_UPIU_SIZE + -SG_ALL *sizeof (struct ufshcd_sg_entry )); -ufs_debugfs_init (); ret =scsi_register_driver (&ufs_dev_wlun_template.gendrv);drivers/ufs/host/Kconfig +0
-0@@ -124,3 +124,13 @@ config SCSI_UFS_EXYNOS Select this if you have UFS host controller on Samsung Exynos SoC. If unsure, say N.
drivers/ufs/host/Kconfig config SCSI_UFS_EXYNOS · line 136
Quoted 10 lines
drivers/ufs/host/Kconfig +10
-0+ +config SCSI_UFS_VARIABLE_SG_ENTRY_SIZE + bool "Variable size UTP physical region descriptor" + help + In the UFSHCI 3.0 standard the Physical Region Descriptor (PRD) is a + data structure used for transferring data between host and UFS + device. This data structure describes a single region in physical + memory. Although the standard requires that this data structure has a + size of 16 bytes, for some controllers this data structure has a + different size. Enable this option for UFS controllers that need it.
Then this change should take the form of a quirk, AKA "opts" in exynos_ufs_drv_data.
Quoted 40 lines
include/ufs/ufshcd.h +3
-0@@ -744,6 +744,9 @@ struct ufs_hba_monitor {* @vops: pointer to variant specific operations * @vps: pointer to variant specific parameters * @priv: pointer to variant specific private data +#ifdef CONFIG_SCSI_UFS_VARIABLE_SG_ENTRY_SIZE +* @sg_entry_size: size of struct ufshcd_sg_entry (may include variant fields) +#endif * @irq: Irq number of the controller * @is_irq_enabled: whether or not the UFS controller interrupt is enabled. * @dev_ref_clk_freq: reference clock frequency
include/ufs/ufshcd.h +13
-0@@ -865,6 +868,9 @@ struct ufs_hba {const struct ufs_hba_variant_ops *vops;struct ufs_hba_variant_params *vps;void *priv; +#ifdef CONFIG_SCSI_UFS_VARIABLE_SG_ENTRY_SIZE +size_t sg_entry_size; +#endif unsigned int irq;bool is_irq_enabled;enum ufs_ref_clk_freq dev_ref_clk_freq;@@ -967,6 +973,32 @@ struct ufs_hba {bool complete_put; }; +#ifdef CONFIG_SCSI_UFS_VARIABLE_SG_ENTRY_SIZE +static inline size_t ufshcd_sg_entry_size (const struct ufs_hba *hba) +{ +return hba->sg_entry_size; +} + +static inline void ufshcd_set_sg_entry_size (struct ufs_hba *hba,size_t sg_entry_size) +{ +WARN_ON_ONCE (sg_entry_size <sizeof (struct ufshcd_sg_entry )); + hba->sg_entry_size = sg_entry_size;
include/ufs/ufshcd.h struct ufs_hba { · line 996
Quoted 10 lines
include/ufs/ufshcd.h +9
-0+} +#else +static inline size_t ufshcd_sg_entry_size (const struct ufs_hba *hba) +{ +return sizeof (struct ufshcd_sg_entry ); +} + +#define ufshcd_set_sg_entry_size (hba, sg_entry_size) \ + ({ (void)(hba); BUILD_BUG_ON(sg_entry_size != sizeof(struct ufshcd_sg_entry)); })
Why not static inline void?
Quoted 30 lines
include/ufs/ufshcd.h +5
-0+#endif + +static inline size_t sizeof_utp_transfer_cmd_desc (const struct ufs_hba *hba) +{ + return sizeof(struct utp_transfer_cmd_desc) + SG_ALL *
ufshcd_sg_entry_size(hba);include/ufs/ufshcd.h +2
-0+} +/* Returns true if clocks can be gated. Otherwise false */ static inline bool ufshcd_is_clkgating_allowed (struct ufs_hba *hba) {include/ufs/ufshci.h +4
-0@@ -426,18 +426,23 @@ struct ufshcd_sg_entry {__le64 addr;__le32 reserved;__le32 size; +/* +* followed by variant-specific fields if +* CONFIG_SCSI_UFS_VARIABLE_SG_ENTRY_SIZE has been defined. +*/ };/** * struct utp_transfer_cmd_desc - UTP Command Descriptor (UCD) * @command_upiu: Command UPIU Frame address * @response_upiu: Response UPIU Frame address
include/ufs/ufshci.h struct ufshcd_sg_entry { · line 446
Quoted 10 lines
include/ufs/ufshci.h +3
-2-* @prd_table: Physical Region Descriptor +* @prd_table: Physical Region Descriptor: an array of SG_ALL struct +* ufshcd_sg_entry's. Variant-specific fields may be present after each. */ struct utp_transfer_cmd_desc {u8 command_upiu[ALIGNED_UPIU_SIZE ];u8 response_upiu[ALIGNED_UPIU_SIZE ]; -struct ufshcd_sg_entry prd_table[SG_ALL ]; +u8 prd_table[]; };
Thanks,
Avri
Quoted 2 lines
include/ufs/ufshci.h +0
-0/**
Bart Van Assche
reviewerRe: [PATCH v2 5/5] scsi: ufs: Allow UFS host drivers to override the sg entry size
Quoted 9 lines · Avri Altman
drivers/ufs/core/ufshcd.c int ufshcd_alloc_host(struct device *dev, struct · line 9606
Quoted 6 lines
drivers/ufs/core/ufshcd.c +1
-0@@ -9601,6 +9602,7 @@ int ufshcd_alloc_host(struct device *dev, struct ufs_hba **hba_handle) hba->dev = dev; hba->dev_ref_clk_freq =REF_CLK_FREQ_INVAL ; hba->nop_out_timeout =NOP_OUT_TIMEOUT ; +ufshcd_set_sg_entry_size (hba,sizeof (struct ufshcd_sg_entry ));Where are you setting this variant for ufs-exynos? I would expect here a
set_sg_entry_sizevop.
The Exynos code used in Pixel 6 phones is available for download from a Google server but is not yet upstream unfortunately. This is why no ufshcd_set_sg_entry_size() call has been added in the ufs-exynos driver.
Quoted 5 lines
Quoted 3 lines
(quoted diff) +2
-0+#define ufshcd_set_sg_entry_size(hba, sg_entry_size) \ + ({ (void)(hba); BUILD_BUG_ON(sg_entry_size != sizeof(struct
ufshcd_sg_entry)); })Why not static inline void?
Because sg_entry_size is used inside BUILD_BUG_ON(). The resulting code does not compile if the above macro is changed into a function.
Thanks,
Bart.
Bart Van Assche
reviewerRe: [PATCH v2 5/5] scsi: ufs: Allow UFS host drivers to override the sg entry size
In reply to Avri AltmanQuoted 9 lines · Avri Altman
drivers/ufs/core/ufshcd.c static void ufshcd_init_lrb(struct ufs_hba *hba, · line 2783
Quoted 7 lines
drivers/ufs/core/ufshcd.c +1
-1@@ -2778,11 +2779,11 @@ static void ufshcd_init_lrb(struct ufs_hba *hba, struct ufshcd_lrb *lrb, int i) lrb->utr_descriptor_ptr = utrdlp + i; lrb->utrd_dma_addr = hba->utrdl_dma_addr + i *sizeof (struct utp_transfer_req_desc ); - lrb->ucd_req_ptr = (struct utp_upiu_req *)(cmd_descp + i); + lrb->ucd_req_ptr = (struct utp_upiu_req *)cmd_descp;Maybe here
cmd_descp->command_upiu?
I like that idea. I will make this change.
Thanks,
Bart.
Bart Van Assche
reviewerRe: [PATCH v2 5/5] scsi: ufs: Allow UFS host drivers to override the sg entry size
In reply to Avri AltmanQuoted 11 lines · Avri Altman
drivers/ufs/host/Kconfig config SCSI_UFS_EXYNOS · line 136
Quoted 10 lines
From: Eric Biggers <[email protected]>
drivers/ufs/host/Kconfig +9
-0+config SCSI_UFS_VARIABLE_SG_ENTRY_SIZE + bool "Variable size UTP physical region descriptor" + help + In the UFSHCI 3.0 standard the Physical Region Descriptor (PRD) is a + data structure used for transferring data between host and UFS + device. This data structure describes a single region in physical + memory. Although the standard requires that this data structure has a + size of 16 bytes, for some controllers this data structure has a + different size. Enable this option for UFS controllers that need it.
Quoted 1 line
Then this change should take the form of a quirk, AKA "opts" in
exynos_ufs_drv_data.
Hi Avri,
I prefer a new CONFIG variable in combination with the ufshcd_set_sg_entry_size() function because:
* Without the new CONFIG variable, the hot path would become slightly slower for controllers that comply with the UFSHCI standard.
* I prefer to keep the details of the Exynos Physical Region Descriptor in the Exynos driver instead of in the UFSHCI core.
Thanks,
Bart.