Skip to contentSkip to conversation

dma_opt_mapping_size returns way too low sizes when using IOMMU

DISCUSSION Christoph Hellwig 3 participants 7 messages Aug 17 – Aug 19 NVMe

lore
Share & tools
lore

Christoph Hellwig

author maintainer
#0 lore raw

Hi all,

I got reports that NVMe devices were arbitrarily limited to 128kiB transfers in recent kernel. It turns out that this only happens when using an IOMMU and is caused by iommu_dma_opt_mapping_size returning iova_rcache_range(), which is defined as follows:

unsigned long iova_rcache_range(void)

{
	return PAGE_SIZE << (IOVA_RANGE_CACHE_MAX_SIZE - 1);
}

and thus indeed hardcodes a 128kiB return on 4k page size architectures.

Both the NVMe performance numbers and common sense suggest that this is NOT the optimal DMA mapping granularity. Can we pick a saner value for iommu_dma_opt_mapping_size that does not restrict common I/O sizes?

#1 lore raw
Quoted 4 lines · Christoph Hellwig

Hi all,

I got reports that NVMe devices were arbitrarily limited to 128kiB transfers in recent kernel.

How recent a kernel? This NVMe and DMA mapping code has not changed in years as far as I know.

Quoted 9 lines

It turns out that this only happens when
using an IOMMU and is caused by iommu_dma_opt_mapping_size returning iova_rcache_range(), which is defined as follows:

unsigned long iova_rcache_range(void)

{
	return PAGE_SIZE << (IOVA_RANGE_CACHE_MAX_SIZE - 1);
}
Quoted 5 lines

and thus indeed hardcodes a 128kiB return on 4k page size architectures.

Both the NVMe performance numbers and common sense suggest that this is NOT the optimal DMA mapping granularity. Can we pick a saner value for iommu_dma_opt_mapping_size that does not restrict common I/O sizes?

Note that SCSI does not use iommu_dma_opt_mapping_size() for clamping max HW sectors, but instead sets opt size / max sectors from this value (so it is not a hard limit there). Could we consider similar for NVMe?

The reason for which we have iommu_dma_opt_mapping_size() is that performance can go through the floor we can't use the rcache for getting the IOVA, i.e. we need to always alloc and dealloc an IOVA from the RB tree for each mapping, and this can greatly reduce performance when the IOVA space fills.

If we increase IOVA_RANGE_CACHE_MAX_SIZE, then we just get caching of larger IOVAs and I am not sure that is a great idea.

author maintainer
#2 lore raw
Quoted 8 lines · John Garry
Quoted 4 lines · Christoph Hellwig

Hi all,

I got reports that NVMe devices were arbitrarily limited to 128kiB transfers in recent kernel.

How recent a kernel? This NVMe and DMA mapping code has not changed in years as far as I know.

This was hardware QA moving from an old distro kernel to a "recent" (aka
still old) one. But I've actually reproduced it locally on an
upstream kernel. I guess most kernel developers or power users simply do not run with IOMMU enabled.

Quoted 7 lines
Quoted 3 lines

Both the NVMe performance numbers and common sense suggest that this is NOT the optimal DMA mapping granularity. Can we pick a saner value for iommu_dma_opt_mapping_size that does not restrict common I/O sizes?

Note that SCSI does not use iommu_dma_opt_mapping_size() for clamping max HW sectors, but instead sets opt size / max sectors from this value (so it is not a hard limit there). Could we consider similar for NVMe?

We could consider that, but it would still reduce performane..

Quoted 8 lines

The reason for which we have iommu_dma_opt_mapping_size() is that performance can go through the floor we can't use the rcache for getting the IOVA, i.e. we need to always alloc and dealloc an IOVA from the RB tree for each mapping, and this can greatly reduce performance when the IOVA space fills.

If we increase IOVA_RANGE_CACHE_MAX_SIZE, then we just get caching of larger IOVAs and I am not sure that is a great idea.

At least on the four different nvme devices I tested, the larger I/O sizes made up for this. But maybe the details depend on other factors as well.

#3 lore raw
Quoted 7 lines · Christoph Hellwig
Quoted 6 lines · John Garry
Quoted 4 lines · Christoph Hellwig

Hi all,

I got reports that NVMe devices were arbitrarily limited to 128kiB transfers in recent kernel.

Quoted 6 lines
Quoted 1 line

How recent a kernel? This NVMe and DMA mapping code has not changed in years as far as I know.

This was hardware QA moving from an old distro kernel to a "recent" (aka
still old) one. But I've actually reproduced it locally on an
upstream kernel. I guess most kernel developers or power users simply do not run with IOMMU enabled.

Yeah, they don't like any performance hit.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

Both the NVMe performance numbers and common sense suggest that this is NOT the optimal DMA mapping granularity. Can we pick a saner value for iommu_dma_opt_mapping_size that does not restrict common I/O sizes?

Note that SCSI does not use iommu_dma_opt_mapping_size() for clamping max HW sectors, but instead sets opt size / max sectors from this value (so it is not a hard limit there). Could we consider similar for NVMe?

We could consider that, but it would still reduce performane..

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

The reason for which we have iommu_dma_opt_mapping_size() is that

performance can go through the floor we can't use the rcache for getting the IOVA, i.e. we need to always alloc and dealloc an IOVA from the RB tree for each mapping, and this can greatly reduce performance when the IOVA space fills.

If we increase IOVA_RANGE_CACHE_MAX_SIZE, then we just get caching of larger IOVAs and I am not sure that is a great idea.

At least on the four different nvme devices I tested, the larger I/O sizes made up for this. But maybe the details depend on other factors as well.

Are you saying that you tried increasing IOVA_RANGE_CACHE_MAX_SIZE and got better performance?

As I remember, I was told that the value of 6 for IOVA_RANGE_CACHE_MAX_SIZE was originally chosen from the value then in max page order, i.e. the idea was that we should not be getting streaming IOs larger than that value. But in looking at lore, 8 was very originally proposed, but I can't see any discussion on why that changed or any relation to page max order.

Some time ago I did try some work to allow IOVA_RANGE_CACHE_MAX_SIZE be set per IOMMU domain, but it was not merged. We went with dma_opt_mapping_size() solution instead.

https://lore.kernel.org/linux-scsi/[email protected]/

#4 lore raw
Quoted 28 lines · John Garry
Quoted 7 lines · Christoph Hellwig
Quoted 6 lines · John Garry
Quoted 4 lines · Christoph Hellwig

Hi all,

I got reports that NVMe devices were arbitrarily limited to 128kiB transfers in recent kernel.

Quoted 6 lines
Quoted 1 line

How recent a kernel? This NVMe and DMA mapping code has not changed in years as far as I know.

This was hardware QA moving from an old distro kernel to a "recent" (aka still old) one.   But I've actually reproduced it locally on an upstream kernel.  I guess most kernel developers or power users simply do not run with IOMMU enabled.

Yeah, they don't like any performance hit.

Quoted 11 lines
Quoted 6 lines
Quoted 4 lines

Both the NVMe performance numbers and common sense suggest that this is NOT the optimal DMA mapping granularity.  Can we pick a saner value for iommu_dma_opt_mapping_size that does not restrict common I/O sizes?

Note that SCSI does not use iommu_dma_opt_mapping_size() for clamping max HW sectors, but instead sets opt size / max sectors from this value (so it is not a hard limit there). Could we consider similar for NVMe?

We could consider that, but it would still reduce performane..

Quoted 3 lines
Quoted 1 line

The reason for which we have iommu_dma_opt_mapping_size() is that

performance can go through the floor we can't use the rcache for getting the IOVA, i.e. we need to always alloc and dealloc an IOVA from the RB tree for each mapping, and this can greatly reduce performance when the IOVA space fills.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

If we increase IOVA_RANGE_CACHE_MAX_SIZE, then we just get caching of larger IOVAs and I am not sure that is a great idea.

At least on the four different nvme devices I tested, the larger I/O sizes made up for this.  But maybe the details depend on other factors as well.

Are you saying that you tried increasing IOVA_RANGE_CACHE_MAX_SIZE and got better performance?

As I remember, I was told that the value of 6 for IOVA_RANGE_CACHE_MAX_SIZE was originally chosen from the value then in max page order, i.e. the idea was that we should not be getting streaming IOs larger than that value. But in looking at lore, 8 was very originally proposed, but I can't see any discussion on why that changed or any relation to page max order.

From a quick skim I can't spot any specific mention in the original threads or accompanying paper, but I have a feeling it was as much about just balancing memory consumption with observable performance improvement, and larger sizes weren't significantly impactful to the original testing, which did seem to be focused on networking rather than storage.

Since these days we have the flexibly-sized depot, and now even lazy-allocation of the CPU magazines, we might be in a better position to consider scaling IOVA_RANGE_CACHE_MAX_SIZE up without too much inherent overhead (I guess the base cost per level will just be 2*NR_CPUS + 1 pointers now). The larger sizes we cache, though, the more it's also going to blow up IOVA consumption if freeing/reallocation isn't well balanced across CPUs, so for devices/systems with limited IOVA space, at some point the flush_rcache case is liable to start adding more overhead than the caching itself saves...

FWIW, my understanding of the idea we settled on for opt_mapping_size was that it was never meant to be an absolute target, just a boundary between where callers can expect "ideal" mapping performance, and where it may start scaling with size/contention/etc., and thus it's for the callers to judge whether larger sizes are still worth it for their particular use-case.

Thanks,
Robin.

Quoted 5 lines

Some time ago I did try some work to allow IOVA_RANGE_CACHE_MAX_SIZE be set per IOMMU domain, but it was not merged. We went with dma_opt_mapping_size() solution instead.

https://lore.kernel.org/linux-scsi/1620656249-68890-1-git-send-email- [email protected]/

author maintainer
#6 lore raw
In reply to John Garry
Quoted 5 lines · John Garry
Quoted 3 lines

At least on the four different nvme devices I tested, the larger I/O sizes made up for this. But maybe the details depend on other factors as well.

Are you saying that you tried increasing IOVA_RANGE_CACHE_MAX_SIZE and got better performance?

No, just disabling this hint.

Quoted 5 lines

As I remember, I was told that the value of 6 for IOVA_RANGE_CACHE_MAX_SIZE was originally chosen from the value then in max page order, i.e. the idea was that we should not be getting streaming IOs larger than that value. But in looking at lore, 8 was very originally proposed, but I can't see any discussion on why that changed or any relation to page max order.

But that's a bit weird. A bio already does up to 256 entries, and a request can have many more.

author maintainer
#5 lore raw
In reply to Robin Murphy

[adding Adrian who originall added the cap to NVMe to work around a softlockup, see commit 3710e2b056cb ("nvme-pci: clamp max_hw_sectors based on DMA optimized limitation")]

Quoted 10 lines · Robin Murphy
Quoted 4 lines

page order, i.e. the idea was that we should not be getting streaming IOs larger than that value. But in looking at lore, 8 was very originally proposed, but I can't see any discussion on why that changed or any relation to page max order.

From a quick skim I can't spot any specific mention in the original threads or accompanying paper, but I have a feeling it was as much about just balancing memory consumption with observable performance improvement, and larger sizes weren't significantly impactful to the original testing, which did seem to be focused on networking rather than storage.

Yeah, network transfer sizes tend to be much lower than those for storage.

Quoted 5 lines

Since these days we have the flexibly-sized depot, and now even lazy-allocation of the CPU magazines, we might be in a better position to consider scaling IOVA_RANGE_CACHE_MAX_SIZE up without too much inherent overhead (I guess the base cost per level will just be 2*NR_CPUS + 1 pointers now).

That would be useful. Is just incrementing it enough or do we need more fine tuning?

Quoted 5 lines

The larger sizes we cache, though, the more it's also going
to blow up IOVA consumption if freeing/reallocation isn't well balanced across CPUs, so for devices/systems with limited IOVA space, at some point the flush_rcache case is liable to start adding more overhead than the caching itself saves...

Storage drivers tend to steer I/O completions to the submitting CPU by default. There are some exceptions, though. For example polling with a dedicated polling thread will usually not do this. Then again the overlap of polled I/O and IOMMU usage should be about 0 right now. This could change with the dmabuf I/O path, but that would lock down the IOVA for the lifetime of the dmabuf and sidestep the entire issue here.

Quoted 6 lines

FWIW, my understanding of the idea we settled on for opt_mapping_size was that it was never meant to be an absolute target, just a boundary between where callers can expect "ideal" mapping performance, and where it may start scaling with size/contention/etc., and thus it's for the callers to judge whether larger sizes are still worth it for their particular use-case.

That would suggest the approach done in the SAS transport class makes more sense and NVMe should just cap to the max mapping size. Of course even that might be the wrong answer. For example for HDD we really do want much larger I/O sizes even if it slows down the IOMMU, but right now HDD on NVMe isn't really beyond the proof of concept stage, at least discounting things like smart NICs as an in-between layer.