DMA mapping changes for SCSI core
Subsystems (6)
v6 cover DMA mapping changes for SCSI core
As reported in [0], DMA mappings whose size exceeds the IOMMU IOVA caching limit may see a big performance hit.
This series introduces a new DMA mapping API, dma_opt_mapping_size(), so that drivers may know this limit when performance is a factor in the mapping.
The SCSI SAS transport code is modified only to use this limit. For now I did not want to touch other hosts as I have a concern that this change could cause a performance regression.
I also added a patch for libata-scsi as it does not currently honour the shost max_sectors limit.
[0] https://lore.kernel.org/linux-iommu/[email protected]/
Changes since v5:
- Rebase to Linux 5.19-rc6
- Add Martin's tags to unmodified patches (thanks!)
- Apply DMA opt limit to max_sectors in sd driver, and not max_hw_sectors
Changes since v4:
- tweak libata and other patch titles
- Add Robin's tag (thanks!)
- Clarify description of new DMA mapping API
Changes since v3:
- Apply max DMA optimial limit to SAS hosts only
Note: Even though "scsi: core: Cap shost max_sectors only once when adding" is a subset of a previous patch I did not transfer the RB tags
- Rebase on v5.19-rc4
John Garry (6):
dma-mapping: Add dma_opt_mapping_size()
dma-iommu: Add iommu_dma_opt_mapping_size()
scsi: core: Cap shost max_sectors according to DMA limits only once
scsi: sd: Allow max_sectors be capped at DMA optimal size limit
scsi: scsi_transport_sas: Cap shost opt_sectors according to DMA
optimal limit
ata: libata-scsi: Cap ata_device->max_sectors according to
shost->max_sectors
Documentation/core-api/dma-api.rst | 14 ++++++++++++++ drivers/ata/libata-scsi.c | 1 + drivers/iommu/dma-iommu.c | 6 ++++++ drivers/iommu/iova.c | 5 +++++ drivers/scsi/hosts.c | 5 +++++ drivers/scsi/scsi_lib.c | 4 ---- drivers/scsi/scsi_transport_sas.c | 6 ++++++ drivers/scsi/sd.c | 2 ++ include/linux/dma-map-ops.h | 1 + include/linux/dma-mapping.h | 5 +++++ include/linux/iova.h | 2 ++ include/scsi/scsi_host.h | 1 + kernel/dma/mapping.c | 12 ++++++++++++ 13 files changed, 60 insertions(+), 4 deletions(-)
Signature · 1 line
2.35.3
PATCH v6 1/6 dma-mapping: Add dma_opt_mapping_size()
Streaming DMA mapping involving an IOMMU may be much slower for larger total mapping size. This is because every IOMMU DMA mapping requires an IOVA to be allocated and freed. IOVA sizes above a certain limit are not cached, which can have a big impact on DMA mapping performance.
Provide an API for device drivers to know this "optimal" limit, such that they may try to produce mapping which don't exceed it.
--- Documentation/core-api/dma-api.rst | 14 ++++++++++++++ include/linux/dma-map-ops.h | 1 + include/linux/dma-mapping.h | 5 +++++ kernel/dma/mapping.c | 12 ++++++++++++ 4 files changed, 32 insertions(+)
Documentation/core-api/dma-api.rst +14 -0
@@ -204,6 +204,20 @@ Returns the maximum size of a mapping for the device. The size parameter of the mapping functions like dma_map_single(), dma_map_page() and others should not be larger than the returned value. +:: + + size_t + dma_opt_mapping_size(struct device *dev); + +Returns the maximum optimal size of a mapping for the device. + +Mapping larger buffers may take much longer in certain scenarios. In +addition, for high-rate short-lived streaming mappings, the upfront time +spent on the mapping may account for an appreciable part of the total +request lifetime. As such, if splitting larger requests incurs no +significant performance penalty, then device drivers are advised to +limit total DMA streaming mappings length to the returned value. + :: bool
include/linux/dma-map-ops.h +1 -0
@@ -69,6 +69,7 @@ struct dma_map_ops {int (*dma_supported)(struct device *dev,u64 mask);u64 (*get_required_mask)(struct device *dev);size_t (*max_mapping_size)(struct device *dev); +size_t (*opt_mapping_size)(void );unsigned long (*get_merge_boundary)(struct device *dev); };
include/linux/dma-mapping.h +5 -0
@@ -144,6 +144,7 @@ int dma_set_mask(struct device *dev, u64 mask);int dma_set_coherent_mask (struct device *dev,u64 mask);u64 dma_get_required_mask (struct device *dev);size_t dma_max_mapping_size (struct device *dev); +size_t dma_opt_mapping_size (struct device *dev);bool dma_need_sync (struct device *dev,dma_addr_t dma_addr);unsigned long dma_get_merge_boundary (struct device *dev); struct sg_table *dma_alloc_noncontiguous(struct device *dev, size_t size,
@@ -266,6 +267,10 @@ static inline size_t dma_max_mapping_size(struct device *dev) {return 0 ; } +static inline size_t dma_opt_mapping_size (struct device *dev) +{ +return 0 ; +}static inline bool dma_need_sync (struct device *dev,dma_addr_t dma_addr) {return false;
kernel/dma/mapping.c +12 -0
@@ -773,6 +773,18 @@ size_t dma_max_mapping_size(struct device *dev) }EXPORT_SYMBOL_GPL (dma_max_mapping_size); +size_t dma_opt_mapping_size (struct device *dev) +{ +const struct dma_map_ops *ops =get_dma_ops (dev); +size_t size =SIZE_MAX ; + +if (ops && ops->opt_mapping_size) + size = ops->opt_mapping_size (); + +return min (dma_max_mapping_size (dev), size); +} +EXPORT_SYMBOL_GPL (dma_opt_mapping_size); +bool dma_need_sync (struct device *dev,dma_addr_t dma_addr) {const struct dma_map_ops *ops =get_dma_ops (dev);
Signature · 1 line
2.35.3
PATCH v6 2/6 dma-iommu: Add iommu_dma_opt_mapping_size()
In reply to John GarryAdd the IOMMU callback for DMA mapping API dma_opt_mapping_size(), which allows the drivers to know the optimal mapping limit and thus limit the requested IOVA lengths.
This value is based on the IOVA rcache range limit, as IOVAs allocated above this limit must always be newly allocated, which may be quite slow.
--- drivers/iommu/dma-iommu.c | 6 ++++++ drivers/iommu/iova.c | 5 +++++ include/linux/iova.h | 2 ++ 3 files changed, 13 insertions(+)
drivers/iommu/dma-iommu.c +6 -0
@@ -1459,6 +1459,11 @@ static unsigned long iommu_dma_get_merge_boundary(struct device *dev)return (1UL <<__ffs (domain->pgsize_bitmap)) -1 ; } +static size_t iommu_dma_opt_mapping_size (void ) +{ +return iova_rcache_range (); +} +static const struct dma_map_ops iommu_dma_ops = { .alloc = iommu_dma_alloc, .free = iommu_dma_free,
@@ -1479,6 +1484,7 @@ static const struct dma_map_ops iommu_dma_ops = { .map_resource = iommu_dma_map_resource, .unmap_resource = iommu_dma_unmap_resource, .get_merge_boundary = iommu_dma_get_merge_boundary, + .opt_mapping_size = iommu_dma_opt_mapping_size, };/*
drivers/iommu/iova.c +5 -0
@@ -26,6 +26,11 @@ static unsigned long iova_rcache_get(struct iova_domain *iovad,static void free_cpu_cached_iovas (unsigned int cpu,struct iova_domain *iovad);static void free_iova_rcaches (struct iova_domain *iovad); +unsigned long iova_rcache_range (void ) +{ +return PAGE_SIZE << (IOVA_RANGE_CACHE_MAX_SIZE -1 ); +} +static int iova_cpuhp_dead (unsigned int cpu,struct hlist_node *node) {struct iova_domain *iovad;
include/linux/iova.h +2 -0
@@ -79,6 +79,8 @@ static inline unsigned long iova_pfn(struct iova_domain *iovad, dma_addr_t iova)int iova_cache_get (void );void iova_cache_put (void ); +unsigned long iova_rcache_range (void ); +void free_iova (struct iova_domain *iovad,unsigned long pfn);void __free_iova (struct iova_domain *iovad,struct iova *iova); struct iova *alloc_iova(struct iova_domain *iovad, unsigned long size,
Signature · 1 line
2.35.3
PATCH v6 3/6 scsi: core: Cap shost max_sectors according to DMA limits only once
In reply to John GarryThe shost->max_sectors is repeatedly capped according to the host DMA mapping limit for each sdev in __scsi_init_queue(). This is unnecessary, so set only once when adding the host.
--- drivers/scsi/hosts.c | 5 +++++ drivers/scsi/scsi_lib.c | 4 ---- 2 files changed, 5 insertions(+), 4 deletions(-)
drivers/scsi/hosts.c +5 -0
@@ -236,6 +236,11 @@ int scsi_add_host_with_dma(struct Scsi_Host *shost, struct device *dev, shost->dma_dev = dma_dev; +if (dma_dev->dma_mask) { + shost->max_sectors = min_t(unsigned int, shost->max_sectors, +dma_max_mapping_size (dma_dev) >>SECTOR_SHIFT ); + } + error =scsi_mq_setup_tags (shost);if (error) goto fail;
drivers/scsi/scsi_lib.c +0 -4
@@ -1884,10 +1884,6 @@ void __scsi_init_queue(struct Scsi_Host *shost, struct request_queue *q)blk_queue_max_integrity_segments (q, shost->sg_prot_tablesize); } -if (dev->dma_mask) { - shost->max_sectors = min_t(unsigned int, shost->max_sectors, -dma_max_mapping_size (dev) >>SECTOR_SHIFT ); - }blk_queue_max_hw_sectors (q, shost->max_sectors);blk_queue_segment_boundary (q, shost->dma_boundary);dma_set_seg_boundary (dev, shost->dma_boundary);
Signature · 1 line
2.35.3
PATCH v6 4/6 scsi: sd: Allow max_sectors be capped at DMA optimal size limit
In reply to John GarryStreaming DMA mappings may be considerably slower when mappings go through an IOMMU and the total mapping length is somewhat long. This is because the IOMMU IOVA code allocates and free an IOVA for each mapping, which may affect performance.
New member Scsi_Host.opt_sectors is added, which is the optimal host max_sectors, and use this value to cap the request queue max_sectors when set.
It could be considered to have request queues io_opt value initially set at Scsi_Host.opt_sectors in __scsi_init_queue(), but that is not really the purpose of io_opt.
Finally, even though Scsi_Host.opt_sectors value should never be greater than the request queue max_hw_sectors value, continue to limit to this value for safety.
--- drivers/scsi/sd.c | 2 ++ include/scsi/scsi_host.h | 1 + 2 files changed, 3 insertions(+)
drivers/scsi/sd.c +2 -0
@@ -3296,6 +3296,8 @@ static int sd_revalidate_disk(struct gendisk *disk) (sector_t)BLK_DEF_MAX_SECTORS); } + rw_max =min_not_zero (rw_max, sdp->host->opt_sectors); +/* Do not exceed controller limit */ rw_max =min (rw_max,queue_max_hw_sectors (q));
include/scsi/scsi_host.h +1 -0
@@ -607,6 +607,7 @@ struct Scsi_Host {short unsigned int sg_tablesize;short unsigned int sg_prot_tablesize;unsigned int max_sectors; +unsigned int opt_sectors;unsigned int max_segment_size;unsigned long dma_boundary;unsigned long virt_boundary_mask;
Signature · 1 line
2.35.3
Re: [PATCH v6 4/6] scsi: sd: Allow max_sectors be capped at DMA optimal size limit
Quoted 28 lines · John Garry
Streaming DMA mappings may be considerably slower when mappings go through an IOMMU and the total mapping length is somewhat long. This is because the IOMMU IOVA code allocates and free an IOVA for each mapping, which may affect performance.
New member Scsi_Host.opt_sectors is added, which is the optimal host
max_sectors, and use this value to cap the request queuemax_sectorswhen set.It could be considered to have request queues
io_optvalue initially set at Scsi_Host.opt_sectors in__scsi_init_queue(), but that is not really the purpose ofio_opt.Finally, even though Scsi_Host.opt_sectors value should never be greater than the request queue
max_hw_sectorsvalue, continue to limit to this value for safety.Signed-off-by John Garry <[email protected]>--- drivers/scsi/sd.c | 2 ++ include/scsi/scsi_host.h | 1 + 2 files changed, 3 insertions(+)drivers/scsi/sd.c +0
-0
drivers/scsi/sd.c static int sd_revalidate_disk(struct gendisk *disk) · line 3300
Quoted 5 lines
drivers/scsi/sd.c +2
-0@@ -3296,6 +3296,8 @@ static int sd_revalidate_disk(struct gendisk *disk) (sector_t)BLK_DEF_MAX_SECTORS); } + rw_max =min_not_zero (rw_max, sdp->host->opt_sectors); +
Adding a comment explaining what the cap is would be nice.
Quoted 8 lines
drivers/scsi/sd.c +0
-0/* Do not exceed controller limit */ rw_max =min (rw_max,queue_max_hw_sectors (q));include/scsi/scsi_host.h +0
-0
include/scsi/scsi_host.h struct Scsi_Host { · line 613
Quoted 7 lines
include/scsi/scsi_host.h +1
-0@@ -607,6 +607,7 @@ struct Scsi_Host {short unsigned int sg_tablesize;short unsigned int sg_prot_tablesize;unsigned int max_sectors; +unsigned int opt_sectors;unsigned int max_segment_size;unsigned long dma_boundary;unsigned long virt_boundary_mask;
Otherwise, looks good.
Signature · 2 lines
Damien Le Moal Western Digital Research
Re: [PATCH v6 4/6] scsi: sd: Allow max_sectors be capped at DMA optimal size limit
Quoted 7 lines · Damien Le Moal
drivers/scsi/sd.c static int sd_revalidate_disk(struct gendisk *disk) · line 3297
Quoted 7 lines
drivers/scsi/sd.c +0
-0@@ -3296,6 +3296,8 @@ static int sd_revalidate_disk(struct gendisk *disk) (sector_t)BLK_DEF_MAX_SECTORS); }
Hi Damien,
Quoted 5 lines
drivers/scsi/sd.c static int sd_revalidate_disk(struct gendisk *disk) · line 3300
Quoted 3 lines
drivers/scsi/sd.c +2
-0+ rw_max =min_not_zero (rw_max, sdp->host->opt_sectors); +Adding a comment explaining what the cap is would be nice.
Christoph has now applied this (thanks, BTW), so would you like me to follow up with a patch on top with a comment?
Thanks,
John
Re: [PATCH v6 4/6] scsi: sd: Allow max_sectors be capped at DMA optimal size limit
Quoted 2 lines · John Garry
Christoph has now applied this (thanks, BTW), so would you like me to follow up with a patch on top with a comment?
Please do.
Re: [PATCH v6 4/6] scsi: sd: Allow max_sectors be capped at DMA optimal size limit
Quoted 4 lines · Christoph Hellwig
Quoted 2 lines · John Garry
Christoph has now applied this (thanks, BTW), so would you like me to follow up with a patch on top with a comment?
Please do.
ok, fine, I'll do it now.
Just saying in case it's an issue - I was looking at http://git.infradead.org/users/hch/dma-mapping.git/log/refs/heads/for-next and the order is not the same as this series and would cause an intermediate build breakage at 9f5ec52ae501 ("scsi: scsi_transport_sas: cap shost opt_sectors according to DMA optimal limit")
Cheers,
John
Re: [PATCH v6 4/6] scsi: sd: Allow max_sectors be capped at DMA optimal size limit
Quoted 5 lines · John Garry
Just saying in case it's an issue - I was looking at
http://git.infradead.org/users/hch/dma-mapping.git/log/refs/heads/for-next
and the order is not the same as this series and would cause an intermediate build breakage at9f5ec52ae501("scsi:scsi_transport_sas: cap shostopt_sectorsaccording to DMA optimal limit")
No idea what git-am did there, I've fixed it up now.
Re: [PATCH v6 4/6] scsi: sd: Allow max_sectors be capped at DMA optimal size limit
In reply to John GarryJohn,
drivers/scsi/sd.c static int sd_revalidate_disk(struct gendisk *disk) · line 3302
Quoted 12 lines
drivers/scsi/sd.c +2
-0@@ -3296,6 +3296,8 @@ static int sd_revalidate_disk(struct gendisk *disk) (sector_t)BLK_DEF_MAX_SECTORS); } + rw_max =min_not_zero (rw_max, sdp->host->opt_sectors); +/* Do not exceed controller limit */ rw_max =min (rw_max,queue_max_hw_sectors (q));
I'm OK with this approach.
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
PATCH v6 5/6 scsi: scsi_transport_sas: Cap shost opt_sectors according to DMA optimal limit
In reply to John GarryStreaming DMA mappings may be considerably slower when mappings go through an IOMMU and the total mapping length is somewhat long. This is because the IOMMU IOVA code allocates and free an IOVA for each mapping, which may affect performance.
For performance reasons set the request queue max_sectors from dma_opt_mapping_size(), which knows this mapping limit.
--- drivers/scsi/scsi_transport_sas.c | 6 ++++++ 1 file changed, 6 insertions(+)
drivers/scsi/scsi_transport_sas.c +6 -0
@@ -225,6 +225,7 @@ static int sas_host_setup(struct transport_container *tc, struct device *dev, {struct Scsi_Host *shost =dev_to_shost (dev);struct sas_host_attrs *sas_host =to_sas_host_attrs (shost); +struct device *dma_dev = shost->dma_dev;INIT_LIST_HEAD (&sas_host->rphy_list);mutex_init (&sas_host->lock);
@@ -236,6 +237,11 @@ static int sas_host_setup(struct transport_container *tc, struct device *dev,dev_printk (KERN_ERR , dev,"fail to a bsg device %d\n" , shost->host_no); +if (dma_dev->dma_mask) { + shost->opt_sectors = min_t(unsigned int, shost->max_sectors, +dma_opt_mapping_size (dma_dev) >>SECTOR_SHIFT ); + } +return 0 ; }
Signature · 1 line
2.35.3
Re: [PATCH v6 5/6] scsi: scsi_transport_sas: Cap shost opt_sectors according to DMA optimal limit
Quoted 27 lines · John Garry
Streaming DMA mappings may be considerably slower when mappings go through an IOMMU and the total mapping length is somewhat long. This is because the IOMMU IOVA code allocates and free an IOVA for each mapping, which may affect performance.
For performance reasons set the request queue
max_sectorsfromdma_opt_mapping_size(), which knows this mapping limit.Signed-off-by John Garry <[email protected]>--- drivers/scsi/scsi_transport_sas.c | 6 ++++++ 1 file changed, 6 insertions(+)drivers/scsi/scsi_transport_sas.c +1
-0@@ -225,6 +225,7 @@ static int sas_host_setup(struct transport_container *tc, struct device *dev, {struct Scsi_Host *shost =dev_to_shost (dev);struct sas_host_attrs *sas_host =to_sas_host_attrs (shost); +struct device *dma_dev = shost->dma_dev;INIT_LIST_HEAD (&sas_host->rphy_list);mutex_init (&sas_host->lock);@@ -236,6 +237,11 @@ static int sas_host_setup(struct transport_container *tc, struct device *dev, dev_printk(KERN_ERR, dev, "fail to a bsg device %d\n",
drivers/scsi/scsi_transport_sas.c static int sas_host_setup(struct transport_container *tc, struct device *dev, · line 247
Quoted 10 lines
drivers/scsi/scsi_transport_sas.c +5
-0shost->host_no); +if (dma_dev->dma_mask) { + shost->opt_sectors = min_t(unsigned int, shost->max_sectors, +dma_opt_mapping_size (dma_dev) >>SECTOR_SHIFT ); + } +return 0 ; }
Signature · 2 lines
Damien Le Moal Western Digital Research
PATCH v6 6/6 ata: libata-scsi: Cap ata_device->max_sectors according to shost->max_sectors
In reply to John GarryATA devices (struct ata_device) have a max_sectors field which is configured internally in libata. This is then used to (re)configure the associated sdev request queue max_sectors value from how it is earlier set in __scsi_init_queue(). In __scsi_init_queue() the max_sectors value is set according to shost limits, which includes host DMA mapping limits.
Cap the ata_device max_sectors according to shost->max_sectors to respect this shost limit.
--- drivers/ata/libata-scsi.c | 1 + 1 file changed, 1 insertion(+)
drivers/ata/libata-scsi.c +1 -0
@@ -1060,6 +1060,7 @@ int ata_scsi_dev_config(struct scsi_device *sdev, struct ata_device *dev) dev->flags |=ATA_DFLAG_NO_UNLOAD ;/* configure max sectors */ + dev->max_sectors =min (dev->max_sectors, sdev->host->max_sectors);blk_queue_max_hw_sectors (q, dev->max_sectors);if (dev->class ==ATA_DEV_ATAPI ) {
Signature · 1 line
2.35.3
Christoph Hellwig
AppliedThanks,
applied to the dma-mapping tree.