Intel IOMMU scalability improvements
cover Intel IOMMU scalability improvements
This patchset improves the scalability of the Intel IOMMU code by resolving two spinlock bottlenecks, yielding up to ~10x performance improvement and approaching iommu=off performance.
For example, here's the throughput obtained by 16 memcached instances running on a 16-core Sandy Bridge system, accessed using memslap on another machine that has iommu=off, using the default memslap config (64-byte keys, 1024-byte values, and 10%/90% SET/GET ops):
stock iommu=off:
1,088,996 memcached transactions/sec (=100%, median of 10 runs).
stock iommu=on:
123,760 memcached transactions/sec (=11%).
[perf: 43.56% 0.86% memcached [kernel.kallsyms] [k] _raw_spin_lock_irqsave]
patched iommu=on:
1,067,586 memcached transactions/sec (=98%).
[perf: 0.75% 0.75% memcached [kernel.kallsyms] [k] _raw_spin_lock_irqsave]The two resolved spinlocks:
- Deferred IOTLB invalidations are batched in a global data structure and serialized under a spinlock (add_unmap() & flush_unmaps()); this patchset batches IOTLB invalidations in a per-CPU data structure.
- IOVA management (alloc_iova() & __free_iova()) is serialized under the rbtree spinlock; this patchset adds per-CPU caches of allocated IOVAs so that the rbtree doesn't get accessed frequently. (Adding a cache above the existing IOVA allocator is less intrusive than dynamic identity mapping and helps keep IOMMU page table usage low; see Patch 7.)
The paper "Utilizing the IOMMU Scalably" (presented at the 2015 USENIX Annual Technical Conference) contains many more details and experiments:
https://www.usenix.org/system/files/conference/atc15/atc15-paper-peleg.pdf
Omer Peleg (7):
iommu: refactoring of deferred flush entries
iommu: per-cpu deferred invalidation queues
iommu: correct flush_unmaps pfn usage
iommu: only unmap mapped entries
iommu: avoid dev iotlb logic in intel-iommu for domains with no dev
iotlbs
iommu: change intel-iommu to use IOVA frame numbers
iommu: introduce per-cpu caching to iova allocation
drivers/iommu/intel-iommu.c | 264 +++++++++++++++++++++------------- drivers/iommu/iova.c | 334 +++++++++++++++++++++++++++++++++++++------- include/linux/iova.h | 23 ++- 3 files changed, 470 insertions(+), 151 deletions(-)
Signature · 1 line
1.9.1
Hi Adam,
Quoted 9 lines · Adam Morrison
This patchset improves the scalability of the Intel IOMMU code by resolving two spinlock bottlenecks, yielding up to ~10x performance improvement and approaching iommu=off performance.
For example, here's the throughput obtained by 16 memcached instances running on a 16-core Sandy Bridge system, accessed using memslap on another machine that has iommu=off, using the default memslap config (64-byte keys, 1024-byte values, and 10%/90% SET/GET ops):
Quoted 9 lines
stock iommu=off:
1,088,996 memcached transactions/sec (=100%, median of 10 runs).
stock iommu=on:
123,760 memcached transactions/sec (=11%).
[perf: 43.56% 0.86% memcached [kernel.kallsyms] [k] _raw_spin_lock_irqsave]
patched iommu=on:
1,067,586 memcached transactions/sec (=98%).
[perf: 0.75% 0.75% memcached [kernel.kallsyms] [k] _raw_spin_lock_irqsave]Thanks for the patches, the results look pretty impressive. I'll have a closer look at your changes this week.
Joerg
There are nice. Thanks very much for doing this work!
We have some preliminary results, looking at scaling to high core counts. We tested the patches on a 2-socket high core count SNB-EP server with a Broadcomm NIC. Our benchmark uses 200 threads of TCP_RR. We see similar performance for IOMMU disabled as we do for IOMMU enabled with this patchset, which is good news. We're working on getting a lab setup with Haswell servers so we can further test the scalability of the code.
We owe the scaling results and of course actual code reviews.
Quoted 60 lines · Adam Morrison
This patchset improves the scalability of the Intel IOMMU code by resolving two spinlock bottlenecks, yielding up to ~10x performance improvement and approaching iommu=off performance.
For example, here's the throughput obtained by 16 memcached instances running on a 16-core Sandy Bridge system, accessed using memslap on another machine that has iommu=off, using the default memslap config (64-byte keys, 1024-byte values, and 10%/90% SET/GET ops):
stock iommu=off: 1,088,996 memcached transactions/sec (=100%, median of 10 runs). stock iommu=on: 123,760 memcached transactions/sec (=11%). [perf: 43.56% 0.86% memcached [kernel.kallsyms] [k] _raw_spin_lock_irqsave] patched iommu=on: 1,067,586 memcached transactions/sec (=98%). [perf: 0.75% 0.75% memcached [kernel.kallsyms] [k]
_raw_spin_lock_irqsave]The two resolved spinlocks:
- Deferred IOTLB invalidations are batched in a global data structure and serialized under a spinlock (
add_unmap()&flush_unmaps()); this patchset batches IOTLB invalidations in a per-CPU data structure.- IOVA management (
alloc_iova()&__free_iova()) is serialized under the rbtree spinlock; this patchset adds per-CPU caches of allocated IOVAs so that the rbtree doesn't get accessed frequently. (Adding a cache above the existing IOVA allocator is less intrusive than dynamic identity mapping and helps keep IOMMU page table usage low; see Patch 7.)The paper "Utilizing the IOMMU Scalably" (presented at the 2015 USENIX Annual Technical Conference) contains many more details and experiments:
https://www.usenix.org/system/files/conference/atc15/atc15-paper-peleg.pdf
Omer Peleg (7):
iommu: refactoring of deferred flush entries
iommu: per-cpu deferred invalidation queues
iommu: correctflush_unmapspfn usage
iommu: only unmap mapped entries
iommu: avoid dev iotlb logic in intel-iommu for domains with no dev
iotlbs
iommu: change intel-iommu to use IOVA frame numbers
iommu: introduce per-cpu caching to iova allocationdrivers/iommu/intel-iommu.c | 264 +++++++++++++++++++++------------- drivers/iommu/iova.c | 334 +++++++++++++++++++++++++++++++++++++------- include/linux/iova.h | 23 ++- 3 files changed, 470 insertions(+), 151 deletions(-)--
1.9.1
There are nice. Thanks very much for doing this work!
We have some preliminary results, looking at scaling to high core counts. We tested the patches on a 2-socket high core count SNB-EP server with a Broadcomm NIC. Our benchmark uses 200 threads of TCP_RR. We see similar performance for IOMMU disabled as we do for IOMMU enabled with this patchset, which is good news. We're working on getting a lab setup with Haswell servers so we can further test the scalability of the code.
We owe the scaling results and of course actual code reviews.
Quoted 10 lines · Benjamin Serebrin via iommu
There are nice. Thanks very much for doing this work!
We have some preliminary results, looking at scaling to high core counts. We tested the patches on a 2-socket high core count SNB-EP server with a Broadcomm NIC. Our benchmark uses 200 threads of TCP_RR. We see similar performance for IOMMU disabled as we do for IOMMU enabled with this patchset, which is good news. We're working on getting a lab setup with Haswell servers so we can further test the scalability of the code.
We owe the scaling results and of course actual code reviews.
This sounds very promising. Could you guys rebase the patches to a recent upstream kernel and repost together with your performance results? I'd really like to see that merged.
Joerg