Skip to content

iommu: Allow IOVA rcache range be configured

John Garry linux-iommu lkml virtualization 6 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v5 Latest posting 2/25 patches 11 repliesIn review
  2. v5 posting 1/25 patches 1 replySuperseded
  3. v46 patches 13 repliesSuperseded
Earlier versions · 3
VersionPostedPatchesRepliesStatus
v37 patches4 repliesSuperseded
v215 patches1 replySuperseded
v16 patches16 repliesSuperseded

Patches in v5 posting 2/2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/5iommu: Refactor iommu_group_store_type()drivers/iommu/iommu.c+62 -344 replies
2/5iova: Allow rcache range upper limit to be flexibledrivers/iommu/iova.c include/linux/iova.h+13 -103 replies
3/5iommu: Allow iommu_change_dev_def_domain() realloc same default domain typedrivers/iommu/iommu.c include/linux/iommu.h+26 -240 replies
4/5iommu: Allow max opt DMA len be set for a group via sysfsDocumentation/ABI/testing/sysfs-kernel-iommu_groups drivers/iommu/iommu.c include/linux/iommu.h+79 -22 replies
5/5iova: Add iova_len argument to iova_domain_init_rcaches()drivers/iommu/dma-iommu.c drivers/iommu/iova.c drivers/vdpa/vdpa_user/iova_domain.c +1 more+34 -72 replies

Reviews trailers per patch

2 of 5 patches have a Reviewed-by or Acked-by.

PatchZhen LeiWill Deacon
1/5R-b A-b
2/5A-b
3/5
4/5
5/5

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v5 thread

Cover letter [PATCH RESEND v5 0/5] iommu: Allow IOVA rcache range be configured

For streaming DMA mappings involving an IOMMU and whose IOVA len regularly exceeds the IOVA rcache upper limit (meaning that they are not cached), performance can be reduced.

This may be much more pronounced from commit 4e89dce72521 ("iommu/iova:
Retry from last rb tree node if iova search fails"), as discussed at [0].

IOVAs which cannot be cached are highly involved in the IOVA ageing issue, as discussed at [1].

This series allows the IOVA rcache range be configured, so that we may cache all IOVAs per domain, thus improving performance.

A new IOMMU group sysfs file is added - max_opt_dma_size - which is used indirectly to configure the IOVA rcache range:
/sys/kernel/iommu_groups/X/max_opt_dma_size

This file is updated same as how the IOMMU group default domain type is updated, i.e. must unbind the only device in the group first.

The inspiration here comes from block layer request queue sysfs "optimal_io_size" file, in /sys/block/sdX/queue/optimal_io_size

Some old figures* for storage scenario (when increasing IOVA rcache range to cover all DMA mapping sizes from the LLD):
v5.13-rc1 baseline: 1200K IOPS
With series: 1800K IOPS

All above are for IOMMU strict mode. Non-strict mode gives ~1800K IOPS in all scenarios.

Based on v5.18-rc1
* I lost my high data throughout test setup

Differences to v4:
https://lore.kernel.org/linux-iommu/[email protected]/
- Major rebase
- Change the "Refactor iommu_group_store_type()" to not use a callback and an op type enum instead
- I didn't pick up Will's Ack as it has changed so much
- Use a domain feature flag to keep same default group type
- Add wrapper for default IOVA rcache range
- Combine last 2x patches

[0] https://lore.kernel.org/linux-iommu/[email protected]/
[1] https://lore.kernel.org/linux-iommu/[email protected]/

John Garry (5):
iommu: Refactor iommu_group_store_type()
iova: Allow rcache range upper limit to be flexible
iommu: Allow iommu_change_dev_def_domain() realloc same default domain
type
iommu: Allow max opt DMA len be set for a group via sysfs
iova: Add iova_len argument to iova_domain_init_rcaches()

 .../ABI/testing/sysfs-kernel-iommu_groups     |  16 ++
 drivers/iommu/dma-iommu.c                     |  15 +-
 drivers/iommu/iommu.c                         | 202 +++++++++++++-----
 drivers/iommu/iova.c                          |  37 ++--
 drivers/vdpa/vdpa_user/iova_domain.c          |   4 +-
 include/linux/iommu.h                         |   7 +
 include/linux/iova.h                          |   6 +-
 7 files changed, 212 insertions(+), 75 deletions(-)
Signature · 1 line
2.26.2