Skip to contentSkip to conversation

x86 swiotlb: Verify we can perform the remapping requested.

PATCH In review Eric W. Biederman 4 participants 10 messages Oct 19, 2011 – Nov 12, 2011 x86

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 x86 swiotlb: Verify we can perform the remapping requested.

Recently I had a driver try with a peculiar 2G dma memory limit. It failed in weird and strange ways because my bounce buffers were being allocated above 2G where the driver could not reach, and no error was reported when the mappings were setup.

Use the swiotlb_dma_supported to avoid silent problems like this in the future.

---
 arch/x86/kernel/pci-swiotlb.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
arch/x86/kernel/pci-swiotlb.c +1 -1
index 8f972cb..6a802fa 100644
@@ -38,7 +38,7 @@ static struct dma_map_ops swiotlb_dma_ops = {
 	.unmap_sg = swiotlb_unmap_sg_attrs,
 	.map_page = swiotlb_map_page,
 	.unmap_page = swiotlb_unmap_page,
-	.dma_supported = NULL,
+	.dma_supported = swiotlb_dma_supported,
 };
 
 /*
Signature · 1 line
1.7.2.5
#1 lore raw

PATCH 2/2 x86 amd_gart_64: Verify we can perform the remapping requested

Recently I had a driver try with a peculiar 2G dma memory limit. The driver failed in weird and strange ways because the GART remapping apperture had been allocated above 2G where the driver cound not reach, and no error was reported when the mappings were setup.

Implement gart_dma_supported to test for this problem case and to return and error if we can not support the remapping.

---
 arch/x86/kernel/amd_gart_64.c |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)
arch/x86/kernel/amd_gart_64.c +9 -0
index 8a439d3..66279cb 100644
@@ -519,6 +519,14 @@ static int gart_mapping_error(struct device *dev, dma_addr_t dma_addr)
 	return (dma_addr == bad_dma_addr);
 }
 
+static int gart_dma_supported(struct device *dev, u64 mask)
+{
+	unsigned long iommu_max_addr = iommu_bus_base + iommu_size - 1;
+
+	/* Fail if the gart window is too high to fit in the devices dma mask */
+	return iommu_max_addr <= mask;
+}
+
 static int no_agp;
 
 static __init unsigned long check_iommu_size(unsigned long aper, u64 aper_size)
@@ -703,6 +711,7 @@ static struct dma_map_ops gart_dma_ops = {
 	.alloc_coherent			= gart_alloc_coherent,
 	.free_coherent			= gart_free_coherent,
 	.mapping_error			= gart_mapping_error,
+	.dma_supported			= gart_dma_supported,
 };
 
 static void gart_iommu_shutdown(void)
Signature · 1 line
1.7.2.5
Acked-by on v1 2/2
#2 lore raw

Re: [PATCH 2/2] x86 amd_gart_64: Verify we can perform the remapping requested

Quoted 8 lines · Eric W. Biederman

Recently I had a driver try with a peculiar 2G dma memory limit. The driver failed in weird and strange ways because the GART remapping apperture had been allocated above 2G where the driver cound not reach, and no error was reported when the mappings were setup.

Implement gart_dma_supported to test for this problem case and to return and error if we can not support the remapping.

You do basically the same as for swiotlb, so it must be good :)

Quoted 1 line

For both patches:

Quoted 39 lines
---
 arch/x86/kernel/amd_gart_64.c |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)
arch/x86/kernel/amd_gart_64.c +9 -0
index 8a439d3..66279cb 100644
@@ -519,6 +519,14 @@ static int gart_mapping_error(struct device *dev, dma_addr_t dma_addr)
 	return (dma_addr == bad_dma_addr);
 }
 
+static int gart_dma_supported(struct device *dev, u64 mask)
+{
+	unsigned long iommu_max_addr = iommu_bus_base + iommu_size - 1;
+
+	/* Fail if the gart window is too high to fit in the devices dma mask */
+	return iommu_max_addr <= mask;
+}
+
 static int no_agp;
 
 static __init unsigned long check_iommu_size(unsigned long aper, u64 aper_size)
@@ -703,6 +711,7 @@ static struct dma_map_ops gart_dma_ops = {
 	.alloc_coherent			= gart_alloc_coherent,
 	.free_coherent			= gart_free_coherent,
 	.mapping_error			= gart_mapping_error,
+	.dma_supported			= gart_dma_supported,
 };
 
 static void gart_iommu_shutdown(void)
Signature · 7 lines
1.7.2.5

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
Signature · 1 line
Nothing great was ever achieved without enthusiasm.
#6 lore raw
In reply to Eric W. Biederman
Quoted 5 lines · Eric W. Biederman

Recently I had a driver try with a peculiar 2G dma memory limit. It failed in weird and strange ways because my bounce buffers were being allocated above 2G where the driver could not reach, and no error was reported when the mappings were setup.

OK, so the overflow buffer was used instead.. which presumarily also was allocated above the 2G? That seems to point that alloc_bootmem_low_pages is not doing its job?

Quoted 3 lines

Use the swiotlb_dma_supported to avoid silent problems like this in the future.

Which driver was it that had this limit?

Quoted 2 lines

I also CC-ed Fujita on this as he is the swiotlb maintainer.

Quoted 25 lines
---
 arch/x86/kernel/pci-swiotlb.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
arch/x86/kernel/pci-swiotlb.c +1 -1
index 8f972cb..6a802fa 100644
@@ -38,7 +38,7 @@ static struct dma_map_ops swiotlb_dma_ops = {
 	.unmap_sg = swiotlb_unmap_sg_attrs,
 	.map_page = swiotlb_map_page,
 	.unmap_page = swiotlb_unmap_page,
-	.dma_supported = NULL,
+	.dma_supported = swiotlb_dma_supported,
 };
 
 /*
Signature · 7 lines
1.7.2.5

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
#7 lore raw
Quoted 10 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Eric W. Biederman

Recently I had a driver try with a peculiar 2G dma memory limit. It failed in weird and strange ways because my bounce buffers were being allocated above 2G where the driver could not reach, and no error was reported when the mappings were setup.

OK, so the overflow buffer was used instead.. which presumarily also was allocated above the 2G? That seems to point that alloc_bootmem_low_pages is not doing its job?

I just looked alloc_bootmem_low_pages allocates memory below ARCH_ADDRESS_LIMIT which for everything except s390 is 4G.

I know I was mostly using the amd gart driver. So I may be mistaken that the swiotlb driver had the same issue. However my only solution at the time was to boot with mem=2G. So I believe the swiotlb did have this issue.

Mostly the patch was. Hmm. That looks stupid not wiring up the swiotlb address space limit check when someone has already written it.

Quoted 4 lines
Quoted 2 lines

Use the swiotlb_dma_supported to avoid silent problems like this in the future.

Which driver was it that had this limit?

An unmerged driver. The driver has since been fixed to use it's hardware in a way that doesn't hit this limit.

At the time that I looked drivers in the kernel with a dma limit above 16MB and blow 4GB only existed on exotic architectures in very special cases.

Unfortunately I no longer have access to this hardware to test with.

Eric

#8 lore raw
Quoted 16 lines · On Mon, 24 Oct 2011 08:07:31 -0700 [email protected] (Eric W. Biederman)…
Quoted 10 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Eric W. Biederman

Recently I had a driver try with a peculiar 2G dma memory limit. It failed in weird and strange ways because my bounce buffers were being allocated above 2G where the driver could not reach, and no error was reported when the mappings were setup.

OK, so the overflow buffer was used instead.. which presumarily also was allocated above the 2G? That seems to point that alloc_bootmem_low_pages is not doing its job?

I just looked alloc_bootmem_low_pages allocates memory below ARCH_ADDRESS_LIMIT which for everything except s390 is 4G.

Quoted 7 lines

I know I was mostly using the amd gart driver. So I may be mistaken that the swiotlb driver had the same issue. However my only solution at the time was to boot with mem=2G. So I believe the swiotlb did have this issue.

Mostly the patch was. Hmm. That looks stupid not wiring up the swiotlb address space limit check when someone has already written it.

http://www.gossamer-threads.com/lists/linux/kernel/1333685?do=post_view_threaded#1333685

swiotlb isn't designed to handle hardware having odd dma mask (e.g. 2G or something). Such device needs the own bouncing mechanism.

#9 lore raw
Quoted 24 lines · FUJITA Tomonori
Quoted 21 lines · On Mon, 24 Oct 2011 08:07:31 -0700 [email protected] (Eric W. Biederman)…
Quoted 10 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Eric W. Biederman

Recently I had a driver try with a peculiar 2G dma memory limit. It failed in weird and strange ways because my bounce buffers were being allocated above 2G where the driver could not reach, and no error was reported when the mappings were setup.

OK, so the overflow buffer was used instead.. which presumarily also was allocated above the 2G? That seems to point that alloc_bootmem_low_pages is not doing its job?

I just looked alloc_bootmem_low_pages allocates memory below ARCH_ADDRESS_LIMIT which for everything except s390 is 4G.

I know I was mostly using the amd gart driver. So I may be mistaken that the swiotlb driver had the same issue. However my only solution at the time was to boot with mem=2G. So I believe the swiotlb did have this issue.

Quoted 7 lines
Quoted 2 lines

Mostly the patch was. Hmm. That looks stupid not wiring up the swiotlb address space limit check when someone has already written it.

http://www.gossamer-threads.com/lists/linux/kernel/1333685?do=post_view_threaded#1333685

swiotlb isn't designed to handle hardware having odd dma mask (e.g. 2G or something). Such device needs the own bouncing mechanism.

I have no problem with that with not supporting weird dma masks.

There is a perfectly fine method to report an unsupported dma mask programmatically that just needs a trivial code change to be wired up. I just want the method to be wired up on x86 so that we fail gracefully, instead of mysteriously.

Eric

#3 lore raw

Re: [PATCH 2/2] x86 amd_gart_64: Verify we can perform the remapping requested

In reply to Joerg Roedel
Quoted 11 lines · Joerg Roedel
Quoted 8 lines · Eric W. Biederman

Recently I had a driver try with a peculiar 2G dma memory limit. The driver failed in weird and strange ways because the GART remapping apperture had been allocated above 2G where the driver cound not reach, and no error was reported when the mappings were setup.

Implement gart_dma_supported to test for this problem case and to return and error if we can not support the remapping.

You do basically the same as for swiotlb, so it must be good :)

Okay, thinking more about this, I am not so sure anymore it is a really good idea to fix it this way. It basically signals the driver that DMA is not possible with the device if the aperture does not fit into the dma_mask. But DMA within the ZONE_DMA might still work, no?

So I think the right way to fix it is to return a dma-mapping error if and only if the address needs to be remapped by the aperture (of swiotlb or gart).

This still allows the driver to use GFP_DMA allocations.

Joerg

#4 lore raw

Re: [PATCH 2/2] x86 amd_gart_64: Verify we can perform the remapping requested

Quoted 13 lines · Joerg Roedel
Quoted 11 lines · Joerg Roedel
Quoted 8 lines · Eric W. Biederman

Recently I had a driver try with a peculiar 2G dma memory limit. The driver failed in weird and strange ways because the GART remapping apperture had been allocated above 2G where the driver cound not reach, and no error was reported when the mappings were setup.

Implement gart_dma_supported to test for this problem case and to return and error if we can not support the remapping.

You do basically the same as for swiotlb, so it must be good :)

Quoted 8 lines

Okay, thinking more about this, I am not so sure anymore it is a really good idea to fix it this way. It basically signals the driver that DMA is not possible with the device if the aperture does not fit into the dma_mask. But DMA within the ZONE_DMA might still work, no?

So I think the right way to fix it is to return a dma-mapping error if and only if the address needs to be remapped by the aperture (of swiotlb or gart).

Perhaps I am mistaken but since the method is iommu type specific if we don't actually use the iommu we should not call the method so I think this is patch is already implementing what you want.

Quoted 1 line

This still allows the driver to use GFP_DMA allocations.

I definitely agree on that.

Eric

#5 lore raw

Re: [PATCH 2/2] x86 amd_gart_64: Verify we can perform the remapping requested

Quoted 6 lines · Eric W. Biederman
Quoted 5 lines · Joerg Roedel

Okay, thinking more about this, I am not so sure anymore it is a really good idea to fix it this way. It basically signals the driver that DMA is not possible with the device if the aperture does not fit into the dma_mask. But DMA within the ZONE_DMA might still work, no?

Quoted 7 lines
Quoted 3 lines

So I think the right way to fix it is to return a dma-mapping error if and only if the address needs to be remapped by the aperture (of swiotlb or gart).

Perhaps I am mistaken but since the method is iommu type specific if we don't actually use the iommu we should not call the method so I think this is patch is already implementing what you want.

No it does not. With GART or SWIOTLB all DMA-mapping functions call directly into the IOMMU driver. The driver then checks whether it needs to remap or not (see dma_capable() in SWIOTLB or need_iommu() in GART).

I think the check you want to add needs to happen after these checks in each of the IOMMU drivers.

Joerg