Skip to contentSkip to conversation

Intel IOMMU scalability improvements

PATCH Superseded Adam Morrison 3 participants 5 messages Dec 28, 2015 – Apr 5, 2016 2 messages not in the archive

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

cover Intel IOMMU scalability improvements

This patchset improves the scalability of the Intel IOMMU code by resolving two spinlock bottlenecks, yielding up to ~10x performance improvement and approaching iommu=off performance.

For example, here's the throughput obtained by 16 memcached instances running on a 16-core Sandy Bridge system, accessed using memslap on another machine that has iommu=off, using the default memslap config (64-byte keys, 1024-byte values, and 10%/90% SET/GET ops):

    stock iommu=off:
       1,088,996 memcached transactions/sec (=100%, median of 10 runs).
    stock iommu=on:
       123,760 memcached transactions/sec (=11%).
       [perf: 43.56%    0.86%  memcached       [kernel.kallsyms]      [k] _raw_spin_lock_irqsave]
    patched iommu=on:
       1,067,586 memcached transactions/sec (=98%).
       [perf: 0.75%     0.75%  memcached       [kernel.kallsyms]      [k] _raw_spin_lock_irqsave]

The two resolved spinlocks:

- Deferred IOTLB invalidations are batched in a global data structure and serialized under a spinlock (add_unmap() & flush_unmaps()); this patchset batches IOTLB invalidations in a per-CPU data structure.

- IOVA management (alloc_iova() & __free_iova()) is serialized under the rbtree spinlock; this patchset adds per-CPU caches of allocated IOVAs so that the rbtree doesn't get accessed frequently. (Adding a cache above the existing IOVA allocator is less intrusive than dynamic identity mapping and helps keep IOMMU page table usage low; see Patch 7.)

The paper "Utilizing the IOMMU Scalably" (presented at the 2015 USENIX Annual Technical Conference) contains many more details and experiments:

https://www.usenix.org/system/files/conference/atc15/atc15-paper-peleg.pdf

Omer Peleg (7):
iommu: refactoring of deferred flush entries
iommu: per-cpu deferred invalidation queues
iommu: correct flush_unmaps pfn usage
iommu: only unmap mapped entries
iommu: avoid dev iotlb logic in intel-iommu for domains with no dev
iotlbs
iommu: change intel-iommu to use IOVA frame numbers
iommu: introduce per-cpu caching to iova allocation

 drivers/iommu/intel-iommu.c | 264 +++++++++++++++++++++-------------
 drivers/iommu/iova.c        | 334 +++++++++++++++++++++++++++++++++++++-------
 include/linux/iova.h        |  23 ++-
 3 files changed, 470 insertions(+), 151 deletions(-)
Signature · 1 line
1.9.1

A message that is not in the archive (sent off-list or not yet received): 20151228161421.GA27829-FrESSTt7Abv7r6psnUbsSmZHpeb/A1Y/@public.gmane.org

#2 lore raw

Hi Adam,

Quoted 9 lines · Adam Morrison

This patchset improves the scalability of the Intel IOMMU code by resolving two spinlock bottlenecks, yielding up to ~10x performance improvement and approaching iommu=off performance.

For example, here's the throughput obtained by 16 memcached instances running on a 16-core Sandy Bridge system, accessed using memslap on another machine that has iommu=off, using the default memslap config (64-byte keys, 1024-byte values, and 10%/90% SET/GET ops):

Quoted 9 lines
    stock iommu=off:
       1,088,996 memcached transactions/sec (=100%, median of 10 runs).
    stock iommu=on:
       123,760 memcached transactions/sec (=11%).
       [perf: 43.56%    0.86%  memcached       [kernel.kallsyms]      [k] _raw_spin_lock_irqsave]
    patched iommu=on:
       1,067,586 memcached transactions/sec (=98%).
       [perf: 0.75%     0.75%  memcached       [kernel.kallsyms]      [k] _raw_spin_lock_irqsave]

Thanks for the patches, the results look pretty impressive. I'll have a closer look at your changes this week.

Joerg

#3 lore raw
In reply to a message not in the archive

There are nice. Thanks very much for doing this work!

We have some preliminary results, looking at scaling to high core counts. We tested the patches on a 2-socket high core count SNB-EP server with a Broadcomm NIC. Our benchmark uses 200 threads of TCP_RR. We see similar performance for IOMMU disabled as we do for IOMMU enabled with this patchset, which is good news. We're working on getting a lab setup with Haswell servers so we can further test the scalability of the code.

We owe the scaling results and of course actual code reviews.

Quoted 60 lines · Adam Morrison

This patchset improves the scalability of the Intel IOMMU code by resolving two spinlock bottlenecks, yielding up to ~10x performance improvement and approaching iommu=off performance.

For example, here's the throughput obtained by 16 memcached instances running on a 16-core Sandy Bridge system, accessed using memslap on another machine that has iommu=off, using the default memslap config (64-byte keys, 1024-byte values, and 10%/90% SET/GET ops):

    stock iommu=off:
       1,088,996 memcached transactions/sec (=100%, median of 10 runs).
    stock iommu=on:
       123,760 memcached transactions/sec (=11%).
       [perf: 43.56%    0.86%  memcached       [kernel.kallsyms]      [k]
_raw_spin_lock_irqsave]
    patched iommu=on:
       1,067,586 memcached transactions/sec (=98%).
       [perf: 0.75%     0.75%  memcached       [kernel.kallsyms]      [k]

_raw_spin_lock_irqsave]

The two resolved spinlocks:

- Deferred IOTLB invalidations are batched in a global data structure and serialized under a spinlock (add_unmap() & flush_unmaps()); this patchset batches IOTLB invalidations in a per-CPU data structure.

- IOVA management (alloc_iova() & __free_iova()) is serialized under the rbtree spinlock; this patchset adds per-CPU caches of allocated IOVAs so that the rbtree doesn't get accessed frequently. (Adding a cache above the existing IOVA allocator is less intrusive than dynamic identity mapping and helps keep IOMMU page table usage low; see Patch 7.)

The paper "Utilizing the IOMMU Scalably" (presented at the 2015 USENIX Annual Technical Conference) contains many more details and experiments:

https://www.usenix.org/system/files/conference/atc15/atc15-paper-peleg.pdf

Omer Peleg (7):
iommu: refactoring of deferred flush entries
iommu: per-cpu deferred invalidation queues
iommu: correct flush_unmaps pfn usage
iommu: only unmap mapped entries
iommu: avoid dev iotlb logic in intel-iommu for domains with no dev
iotlbs
iommu: change intel-iommu to use IOVA frame numbers
iommu: introduce per-cpu caching to iova allocation

 drivers/iommu/intel-iommu.c | 264 +++++++++++++++++++++-------------
 drivers/iommu/iova.c        | 334
+++++++++++++++++++++++++++++++++++++-------
 include/linux/iova.h        |  23 ++-
 3 files changed, 470 insertions(+), 151 deletions(-)

--
1.9.1

A message that is not in the archive (sent off-list or not yet received): CAN+hb0Xt21CMmM7uE0rzjf5p9w-W+5y8at4v1J8+pYd8tamLpQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#5 lore raw
Quoted 10 lines · Benjamin Serebrin via iommu

There are nice.  Thanks very much for doing this work!

We have some preliminary results, looking at scaling to high core counts.  We tested the patches on a 2-socket high core count SNB-EP server with a Broadcomm NIC.  Our benchmark uses 200 threads of TCP_RR.  We see similar performance for IOMMU disabled as we do for IOMMU enabled with this patchset, which is good news.  We're working on getting a lab setup with Haswell servers so we can further test the scalability of the code.

We owe the scaling results and of course actual code reviews.

This sounds very promising. Could you guys rebase the patches to a recent upstream kernel and repost together with your performance results? I'd really like to see that merged.

Joerg

#6 lore raw
In reply to Adam Morrison

There are nice. Thanks very much for doing this work!

We have some preliminary results, looking at scaling to high core counts. We tested the patches on a 2-socket high core count SNB-EP server with a Broadcomm NIC. Our benchmark uses 200 threads of TCP_RR. We see similar performance for IOMMU disabled as we do for IOMMU enabled with this patchset, which is good news. We're working on getting a lab setup with Haswell servers so we can further test the scalability of the code.

We owe the scaling results and of course actual code reviews.