large DMA segments vs SWIOTLB
Hi all,
I'm currently looking at an issue with an NVMe device, which isn't working properly under some specific conditions.
The issue comes down to my platform having DMA addressing restrictions, with only 3 of the total 4GiB of RAM being device addressable, which means a bunch of DMA mappings are going through the SWIOTLB.
Now with this NVMe device I'm getting a request with ~520KiB data
payload. The system memory isn't heavily fragmented at that point yet, so the payload gets mapped a single dma segment in nvme_map_data(). Due to the addressing restrictions the request is passed to SWIOTLB, which is unable to satisfy the mapping request, despite plenty of TLB space being available due to the maximum segment size imposed by SWIOTLB. Currently a SWIOTLB slab is 2KiB (IO_TLB_SHIFT) in size, while the maximum segment size is IO_TLB_SEGSIZE = 128 slabs. This causes the dma mapping to fail, which means the blk layer will retry the request indefinitely.
Now I can work around the issue at hand simply by bumping IO_TLB_SEGSIZE to 512, but this doesn't seem like a very robust solution.
Do we need a SWIOTLB allocator that doesn't exhibit linear complexity with the maximum segment size? Some buddy scheme maybe? Splitting the dma segment doesn't seem to be an option, as the documentation states that dma_map_sg may return less segments as a result of the mapping operation, not more. I'm not sure how far this assumption is ingrained into the users of the API.
Regards,
Lucas
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Hi Lukas,
have you tried the latest 5.3-rc kernel, where we limited the NVMe I/O size based on the swiotlb buffer size?
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Hi Christoph,
Am Donnerstag, den 01.08.2019, 09:29 +0200 schrieb Christoph Hellwig:
Quoted 4 lines
Hi Lukas,
have you tried the latest 5.3-rc kernel, where we limited the NVMe I/O size based on the swiotlb buffer size?
Yes, the issue was reproduced on 5.3-rc2. I now see your commit limiting the request size, so I guess I need to dig in to see why I'm still getting requests larger than the SWIOTLB max segment size. Thanks for the pointer!
Regards,
Lucas
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Quoted 5 lines · Lucas Stach
Hi Christoph,
Am Donnerstag, den 01.08.2019, 09:29 +0200 schrieb Christoph Hellwig:
Quoted 2 lines
Hi Lukas,
Quoted 7 lines
Quoted 2 lines
have you tried the latest 5.3-rc kernel, where we limited the NVMe I/O size based on the swiotlb buffer size?
Yes, the issue was reproduced on 5.3-rc2. I now see your commit limiting the request size, so I guess I need to dig in to see why I'm still getting requests larger than the SWIOTLB max segment size. Thanks for the pointer!
a similar setup to yours thedma_addressing_limited doesn't work, but if we changed it to a <= it does. The result is counter to what I'd expect, but because I'm on vacation I didn't have time to look into why it works. This is his patch, let me know if this works for you:
include/linux/dma-mapping.h +1 -1
@@ -689,7 +689,7 @@ static inline int dma_coerce_mask_and_coherent(struct device *dev, u64 mask)*/ static inline bool dma_addressing_limited (struct device *dev) { -return min_not_zero (dma_get_mask (dev), dev->bus_dma_mask) < +return min_not_zero (dma_get_mask (dev), dev->bus_dma_mask) <=dma_get_required_mask (dev); }
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Hi Christoph,
Am Donnerstag, den 01.08.2019, 16:00 +0200 schrieb Christoph Hellwig:
Quoted 29 lines · Lucas Stach
Quoted 5 lines
Hi Christoph,
Am Donnerstag, den 01.08.2019, 09:29 +0200 schrieb Christoph Hellwig:
Quoted 2 lines
Hi Lukas,
Quoted 7 lines
Quoted 2 lines
have you tried the latest 5.3-rc kernel, where we limited the NVMe I/O size based on the swiotlb buffer size?
Yes, the issue was reproduced on 5.3-rc2. I now see your commit limiting the request size, so I guess I need to dig in to see why I'm still getting requests larger than the SWIOTLB max segment size. Thanks for the pointer!
a similar setup to yours the
dma_addressing_limiteddoesn't work, but if we changed it to a <= it does. The result is counter to what I'd expect, but because I'm on vacation I didn't have time to look into why it works. This is his patch, let me know if this works for you:include/linux/dma-mapping.h +0
-0@@ -689,7 +689,7 @@ static inline int dma_coerce_mask_and_coherent(struct device *dev, u64 mask)*/ static inline bool dma_addressing_limited(struct device *dev) {
Quoted 4 lines
Quoted 3 lines
include/linux/dma-mapping.h +1
-1- return min_not_zero(dma_get_mask(dev), dev->bus_dma_mask) < + return min_not_zero(dma_get_mask(dev), dev->bus_dma_mask) <=dma_get_required_mask(dev);
}
From the patch I just sent it should be clear why the above works. With my patch applied I can't reproduce any issues with this NVMe device anymore.
Thanks for pointing me into the right direction!
Regards,
Lucas
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu