Revert "nvme-pci: clamp max_hw_sectors based on DMA optimized limitation"
Christoph Hellwig
maintainerPATCH 1/1 Revert "nvme-pci: clamp max_hw_sectors based on DMA optimized limitation"
This turns down to cause up to 15% performance regressions on standard large transfer workloads. Revert it and hope the IOVA allocator can eventually be fixed to remove the need for the original workaround.
This reverts commit 3710e2b056cb92ad816e4d79fa54a6a5b6ad8cbd.
--- drivers/nvme/host/pci.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/pci.c +1 -1
@@ -3747,7 +3747,7 @@ static struct nvme_dev *nvme_pci_alloc_dev(struct pci_dev *pdev,*/ dev->ctrl.max_hw_sectors =min_t (u32,NVME_MAX_BYTES >>SECTOR_SHIFT , -dma_opt_mapping_size (&pdev->dev) >>9 ); +dma_max_mapping_size (&pdev->dev) >>9 ); dev->ctrl.max_segments =NVME_MAX_SEGS ; dev->ctrl.max_integrity_segments =1 ;return dev;
Signature · 1 line
2.53.0
Quoted 6 lines · Christoph Hellwig
This turns down to cause up to 15% performance regressions on standard large transfer workloads. Revert it and hope the IOVA allocator can eventually be fixed to remove the need for the original workaround.
This reverts commit
3710e2b056cb92ad816e4d79fa54a6a5b6ad8cbd.
What was the block size and conditions for the original issue? The commit just mentions that mdts=10 and we get a 4096KB max HW sectors.
I can't find the original thread.
Quoted 18 lines
Signed-off-by Christoph Hellwig <[email protected]>--- drivers/nvme/host/pci.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/nvme/host/pci.c +1
-1@@ -3747,7 +3747,7 @@ static struct nvme_dev *nvme_pci_alloc_dev(struct pci_dev *pdev,*/ dev->ctrl.max_hw_sectors =min_t (u32,NVME_MAX_BYTES >>SECTOR_SHIFT , -dma_opt_mapping_size (&pdev->dev) >>9 ); +dma_max_mapping_size (&pdev->dev) >>9 ); dev->ctrl.max_segments =NVME_MAX_SEGS ; dev->ctrl.max_integrity_segments =1 ;return dev;
Christoph Hellwig
maintainerQuoted 10 lines · John Garry
Quoted 6 lines · Christoph Hellwig
This turns down to cause up to 15% performance regressions on standard large transfer workloads. Revert it and hope the IOVA allocator can eventually be fixed to remove the need for the original workaround.
This reverts commit
3710e2b056cb92ad816e4d79fa54a6a5b6ad8cbd.What was the block size and conditions for the original issue? The commit just mentions that mdts=10 and we get a 4096KB max HW sectors.
Good question. The original submitter and reporter are on Cc.
Quoted 17 lines
-----Original Message-----
From: Christoph Hellwig <[email protected]>
Sent: Thursday, October 8, 2026 8:37 PM
To: John Garry <[email protected]>
Cc: Christoph Hellwig <[email protected]>; [email protected]; Adrian Huang12 <[email protected]>; [email protected]; [email protected]; Jiwei JW10 Sun <[email protected]>; [email protected]
Subject: [External] Re: [PATCH] Revert "nvme-pci: clampmax_hw_sectorsbased on DMA optimized limitation"Quoted 4 lines · John Garry
Quoted 3 lines · Christoph Hellwig
This turns down to cause up to 15% performance regressions on standard large transfer workloads. Revert it and hope the IOVA allocator can eventually be fixed to remove the need for the original
workaround.
Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
This reverts commit
3710e2b056cb92ad816e4d79fa54a6a5b6ad8cbd.What was the block size and conditions for the original issue? The commit just mentions that mdts=10 and we get a 4096KB max HW sectors.
Good question. The original submitter and reporter are on Cc.
1. The block size depends on the configuration of the fio parameter 'bs'. See below for details.
2. Conditions for the original issue: The issue is triggered if the IO size > 128KB.
[Detail]
1. A NVME with mdts=10 (maximum data transfer size = 4096 KB)
1.1 fio with bs=128k or below => Passed
1.2 fio with bs > 128k => Failed
2. Another NVME with mdts=5 (maximum data transfer size = 128KB)
2.1 The issue does not reproduce regardless of the 'bs' configuration
3. The condition '128 KB' is identified because the IOMMU scalable IOVA mechanism supports a maximum transfer size of 128 KB.
4. IOMMU pass-through mode does not trigger the issue with the 'mdts=10' nvme disk.
5. The issue only appears with high-core-count servers
5.1 Reproduced servers: 448-core server and 384-core server
5.2 Non-reproduced server: 128-core server
BTW, with this reverted patch, I'm setting up the test environment to check whether the issue does not appear with a newer kernel that includes the updated IOMMU subsystem. I'll share the results by the end of next week.
-- Adrian
Keith Busch
maintainerQuoted 5 lines · On Fri, Oct 09, 2026 at 09:11:56AM +0000, Adrian Huang12 wrote:
BTW, with this reverted patch, I'm setting up the test environment to check whether the issue does not appear with a newer kernel that includes the updated IOMMU subsystem. I'll share the results by the end of next week.
It does sound like an IOMMU implementation issue if the revert regresses your testing. Let's say the application wants to write 1MB of data:
we're DMA mapping the same total amount either way, so I don't think it should have been more efficient for the driver to send eight 128k mapping requests vs a single 1MB.
Quoted 25 lines · On 10/9/26 10:11, Adrian Huang12 wrote:
Quoted 17 lines
-----Original Message-----
From: Christoph Hellwig <[email protected]>
Sent: Thursday, October 8, 2026 8:37 PM
To: John Garry <[email protected]>
Cc: Christoph Hellwig <[email protected]>; [email protected]; Adrian Huang12 <[email protected]>; [email protected]; [email protected]; Jiwei JW10 Sun <[email protected]>; [email protected]
Subject: [External] Re: [PATCH] Revert "nvme-pci: clampmax_hw_sectorsbased on DMA optimized limitation"Quoted 4 lines · John Garry
Quoted 3 lines · Christoph Hellwig
This turns down to cause up to 15% performance regressions on standard large transfer workloads. Revert it and hope the IOVA allocator can eventually be fixed to remove the need for the original
workaround.
Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
This reverts commit
3710e2b056cb92ad816e4d79fa54a6a5b6ad8cbd.What was the block size and conditions for the original issue? The commit just mentions that mdts=10 and we get a 4096KB max HW sectors.
Good question. The original submitter and reporter are on Cc.
Quoted 7 lines
1. The block size depends on the configuration of the fio parameter 'bs'. See below for details.
2. Conditions for the original issue: The issue is triggered if the IO size > 128KB.[Detail]
1. A NVME with mdts=10 (maximum data transfer size = 4096 KB) 1.1 fio with bs=128k or below => Passed
1.2 fio with bs > 128k => Failed
That block size limit is an important detail missing from 3710e2b056cb92ad816e4d79fa54a6a5b6ad8cbd commit message.
And now we seem to have a conflict, as Christoph wants block size > IOVA rcache limit but you find it may cause this soft lockup.
Quoted 7 lines
2. Another NVME with mdts=5 (maximum data transfer size = 128KB)
2.1 The issue does not reproduce regardless of the 'bs' configuration
3. The condition '128 KB' is identified because the IOMMU scalable IOVA mechanism supports a maximum transfer size of 128 KB.
4. IOMMU pass-through mode does not trigger the issue with the 'mdts=10' nvme disk.
5. The issue only appears with high-core-count servers
5.1 Reproduced servers: 448-core server and 384-core server
5.2 Non-reproduced server: 128-core server
How long does it take for the lockup to trigger? Pretty quickly, I guess.
Does changing the number of CPUs affect throughput (before the lockup)? Are you using just a single disk?
Quoted 3 lines
BTW, with this reverted patch, I'm setting up the test environment to check whether the issue does not appear with a newer kernel that includes the updated IOMMU subsystem. I'll share the results by the end of next week.
thanks
Christoph Hellwig
maintainerQuoted 11 lines · John Garry
And now we seem to have a conflict, as Christoph wants block size > IOVA rcache limit but you find it may cause this soft lockup.
Quoted 7 lines
2. Another NVME with mdts=5 (maximum data transfer size = 128KB)
2.1 The issue does not reproduce regardless of the 'bs' configuration
3. The condition '128 KB' is identified because the IOMMU scalable IOVA mechanism supports a maximum transfer size of 128 KB.
4. IOMMU pass-through mode does not trigger the issue with the 'mdts=10' nvme disk.
5. The issue only appears with high-core-count servers
5.1 Reproduced servers: 448-core server and 384-core server
5.2 Non-reproduced server: 128-core server
Quoted 4 lines
How long does it take for the lockup to trigger? Pretty quickly, I guess.
Does changing the number of CPUs affect throughput (before the lockup)? Are you using just a single disk?
If the iova cache locks up under a perfectly normal workloads it is broken. We really need to fix the code there.
Quoted 14 lines · Christoph Hellwig
Quoted 13 lines · John Garry
And now we seem to have a conflict, as Christoph wants block size > IOVA rcache limit but you find it may cause this soft lockup.
Quoted 7 lines
2. Another NVME with mdts=5 (maximum data transfer size = 128KB)
2.1 The issue does not reproduce regardless of the 'bs' configuration
3. The condition '128 KB' is identified because the IOMMU scalable IOVA mechanism supports a maximum transfer size of 128 KB.
4. IOMMU pass-through mode does not trigger the issue with the 'mdts=10' nvme disk.
5. The issue only appears with high-core-count servers
5.1 Reproduced servers: 448-core server and 384-core server
5.2 Non-reproduced server: 128-core serverHow long does it take for the lockup to trigger? Pretty quickly, I guess.
Quoted 8 lines
Quoted 2 lines
Does changing the number of CPUs affect throughput (before the lockup)? Are you using just a single disk?
If the iova cache locks up under a perfectly normal workloads it is broken.
We really need to fix the code there.
Sure, but it would help to first fully understand the conditions to fix it.
Quoted 18 lines · John Garry
Quoted 17 lines · Christoph Hellwig
Quoted 15 lines · John Garry
And now we seem to have a conflict, as Christoph wants block size > IOVA rcache limit but you find it may cause this soft lockup.
Quoted 7 lines
2. Another NVME with mdts=5 (maximum data transfer size = 128KB)
2.1 The issue does not reproduce regardless of the 'bs' configuration
3. The condition '128 KB' is identified because the IOMMU scalable IOVA mechanism supports a maximum transfer size of 128 KB.
4. IOMMU pass-through mode does not trigger the issue with the 'mdts=10' nvme disk.
5. The issue only appears with high-core-count servers
5.1 Reproduced servers: 448-core server and 384-core server
5.2 Non-reproduced server: 128-core serverHow long does it take for the lockup to trigger? Pretty quickly, I guess.
Does changing the number of CPUs affect throughput (before the lockup)? Are you using just a single disk?
Quoted 7 lines
Quoted 5 lines
If the iova cache locks up under a perfectly normal workloads it is broken.
We really need to fix the code there.
Sure, but it would help to first fully understand the conditions to fix it.
BTW, @Adrian, it would be good to run your test with the IOMMU in strict mode as well, just to give an idea if there is a problem there as well.
Adrian Huang12
Tested-byQuoted 37 lines
-----Original Message-----
From: John Garry <[email protected]>
Sent: Thursday, October 15, 2026 6:31 PM
To: Christoph Hellwig <[email protected]>; Adrian Huang12 <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Jiwei JW10 Sun <[email protected]>; [email protected]
Subject: Re: [External] Re: [PATCH] Revert "nvme-pci: clampmax_hw_sectorsbased on DMA optimized limitation"Quoted 26 lines · John Garry
Quoted 23 lines · Christoph Hellwig
Quoted 14 lines · John Garry
And now we seem to have a conflict, as Christoph wants block size > IOVA rcache limit but you find it may cause this soft lockup.
Quoted 10 lines
2. Another NVME with mdts=5 (maximum data transfer size = 128KB) 2.1 The issue does not reproduce regardless of the 'bs' configuration
3. The condition '128 KB' is identified because the IOMMU scalable IOVA mechanism supports a maximum transfer size of 128 KB.
4. IOMMU pass-through mode does not trigger the issue with the 'mdts=10' nvme disk.
5. The issue only appears with high-core-count servers 5.1 Reproduced servers: 448-core server and 384-core server 5.2 Non-reproduced server: 128-core serverQuoted 5 lines
How long does it take for the lockup to trigger? Pretty quickly, I guess.
Does changing the number of CPUs affect throughput (before the lockup)? Are you using just a single disk?
If the iova cache locks up under a perfectly normal workloads it is broken.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
We really need to fix the code there.
Sure, but it would help to first fully understand the conditions to fix it.
BTW, @Adrian, it would be good to run your test with the IOMMU in strict mode as well, just to give an idea if there is a problem there as well.
Sorry for the delayed update. The original test system used three years ago is no longer available, so I needed additional time to assemble two new test systems.
== Test Environment ==
1. 448-core AMD server (2-socket) with an NVMe device (mdts=10)
2. 960-core Intel server (8-socket) with an NVMe device (mdts=10)
3. The issue can be reproduced with a single NVMe device (mdts=10)
4. IOMMU lazy mode is enabled. We did not test strict mode three years ago (and now). @John, let me know if you need me to run the strict mode test.
5. stress-ng and fio used for reproduction
6. A 100-run test was performed for each tested kernel
== Summary ==
1. Since the original patch was merged in v6.4-rc3, I used v6.3 as the baseline to verify whether the issue could be reproduced on these two systems. The issue was successfully reproduced on both platforms.
2. On v6.7 and later kernels, after reverting commit 3710e2b056cb, the issue no longer occurs.
3. Detailed test information and logs can be found in [1].
== Detailed Results (Both Systems) ==
* v7.3.0-rc7 + revert 3710e2b056cb: Passed
* v6.12 + revert 3710e2b056cb: Passed
* v6.8 + revert 3710e2b056cb: Passed
* v6.7 + revert 3710e2b056cb: Passed
* v6.6 + revert 3710e2b056cb: Failed
* v6.5 + revert 3710e2b056cb: Failed
* Vanilla 6.3: Failed
Based on these results, I believe it is safe to apply the revert. Please feel free to add:
Also, my apologies for introducing the performance regression. The results show that changes in the IOMMU subsystem introduced around v6.7 significantly improved scalability, although I have not yet investigated which specific commit resolved the underlying issue.
[1] Detailed test info: https://hackmd.io/@lQBEPrYxRoOVV2MatUYusg/B14XRbZ1nGx
-- Adrian
Quoted 6 lines · On 10/16/26 15:07, Adrian Huang12 wrote:
Quoted 2 lines
BTW, @Adrian, it would be good to run your test with the IOMMU in strict mode as well, just to give an idea if there is a problem there as well.
Sorry for the delayed update. The original test system used three years ago is no longer available, so I needed additional time to assemble two new test systems.
Quoted 7 lines
== Test Environment ==
1. 448-core AMD server (2-socket) with an NVMe device (mdts=10)
2. 960-core Intel server (8-socket) with an NVMe device (mdts=10)
3. The issue can be reproduced with a single NVMe device (mdts=10)
4. IOMMU lazy mode is enabled. We did not test strict mode three years ago (and now). @John, let me know if you need me to run the strict mode test.
Not necessarily, unless you want to. strict mode generally shows a more severe performance degradation in terms of throughput, but strict mode would not be relevant to IOVA FQ timeouts which you were seeing.
Quoted 20 lines
5. stress-ng and fio used for reproduction
6. A 100-run test was performed for each tested kernel== Summary ==
1. Since the original patch was merged in v6.4-rc3, I used v6.3 as the baseline to verify whether the issue could be reproduced on these two systems. The issue was successfully reproduced on both platforms.
2. On v6.7 and later kernels, after reverting commit3710e2b056cb, the issue no longer occurs.
3. Detailed test information and logs can be found in [1].== Detailed Results (Both Systems) ==
* v7.3.0-rc7 + revert3710e2b056cb: Passed
* v6.12 + revert3710e2b056cb: Passed
* v6.8 + revert3710e2b056cb: Passed
* v6.7 + revert3710e2b056cb: Passed
* v6.6 + revert3710e2b056cb: Failed
* v6.5 + revert3710e2b056cb: Failed
* Vanilla 6.3: Failed
Quoted 2 lines
Based on these results, I believe it is safe to apply the revert. Please feel free to add:
thanks for testing
Quoted 6 lines
Tested-by Adrian Huang<[email protected]>Also, my apologies for introducing the performance regression. The results show that changes in the IOMMU subsystem introduced around v6.7 significantly improved scalability,
Nothing sticks out to me which could affect this. The iova rcache code changed in terms of removing the fixed depot size upper limit in v6.7, but this should not affect you (as your data block size exceeds that of the rcache limit). Having said that, you mention that stress-ng is being used also, and I don't know the profile of the storage data size used there.
Quoted 4 lines
although I have not yet investigated
which specific commit resolved the underlying issue.[1] Detailed test info:https://hackmd.io/@lQBEPrYxRoOVV2MatUYusg/B14XRbZ1nGx
Quoted 16 lines · Christoph Hellwig
This turns down to cause up to 15% performance regressions on standard large transfer workloads. Revert it and hope the IOVA allocator can eventually be fixed to remove the need for the original workaround.
This reverts commit
3710e2b056cb92ad816e4d79fa54a6a5b6ad8cbd.Signed-off-by Christoph Hellwig <[email protected]>--- drivers/nvme/host/pci.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/nvme/host/pci.c +0
-0
drivers/nvme/host/pci.c static struct nvme_dev *nvme_pci_alloc_dev(struct pci_dev *pdev, · line 3753
Quoted 8 lines
drivers/nvme/host/pci.c +1
-1@@ -3747,7 +3747,7 @@ static struct nvme_dev *nvme_pci_alloc_dev(struct pci_dev *pdev,*/ dev->ctrl.max_hw_sectors =min_t (u32,NVME_MAX_BYTES >>SECTOR_SHIFT , -dma_opt_mapping_size (&pdev->dev) >>9 ); +dma_max_mapping_size (&pdev->dev) >>9 ); dev->ctrl.max_segments =NVME_MAX_SEGS ; dev->ctrl.max_integrity_segments =1 ;return dev;
I saw a related problem with the limit capped to dma_opt_mapping_size - this can cause inflated number of ios in zoned xfs as the max zone append limit is capped by max_hw_sectors. It can also lead to more extents being generated.
In my case, the drive did not have any limits on appends (mdts: 0 , zasl: 0) but dma_opt_mapping_size capped that to 128kib which is very small. It would be great to fix this.