Skip to contentSkip to conversation

dma-debug: Check for drivers mapping vmalloc addresses

PATCH Superseded Stephen Boyd 3 participants 7 messages Sep 20, 2018 – Sep 28, 2018 DMA mapping helpers

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 dma-debug: Check for drivers mapping vmalloc addresses

I recently debugged a DMA mapping oops where a driver was trying to map a buffer returned from request_firmware() with dma_map_single(). Memory returned from request_firmware() is mapped into the vmalloc region and this isn't a valid region to map with dma_map_single() per the DMA documentation's "What memory is DMA'able?" section.

Unfortunately, we don't really check that in the DMA debugging code, so enabling DMA debugging doesn't help catch this problem. Let's add a new DMA debug function to check for a vmalloc address and print a warning if this happens. This makes it a little easier to debug these sorts of problems, instead of seeing odd behavior or crashes when drivers attempt to map the vmalloc space for DMA.

Cc Marek Szyprowski <[email protected]>
Cc Robin Murphy <[email protected]>
---
 include/linux/dma-debug.h   |  8 ++++++++
 include/linux/dma-mapping.h |  1 +
 kernel/dma/debug.c          | 12 ++++++++++++
 3 files changed, 21 insertions(+)
include/linux/dma-debug.h +8 -0
index a785f2507159..5aec2ca8a426 100644
@@ -32,6 +32,9 @@ extern void dma_debug_add_bus(struct bus_type *bus);
 
 extern int dma_debug_resize_entries(u32 num_entries);
 
+extern void debug_dma_check_vmalloc(struct device *dev, const void *addr,
+				    unsigned long len);
+
 extern void debug_dma_map_page(struct device *dev, struct page *page,
 			       size_t offset, size_t size,
 			       int direction, dma_addr_t dma_addr,
@@ -103,6 +106,11 @@ static inline int dma_debug_resize_entries(u32 num_entries)
 	return 0;
 }
 
+static inline void debug_dma_check_vmalloc(struct device *dev, const void *addr,
+					   unsigned long len)
+{
+}
+
 static inline void debug_dma_map_page(struct device *dev, struct page *page,
 				      size_t offset, size_t size,
 				      int direction, dma_addr_t dma_addr,
include/linux/dma-mapping.h +1 -0
index eafd6f318e78..a0e67fd5433c 100644
@@ -232,6 +232,7 @@ static inline dma_addr_t dma_map_single_attrs(struct device *dev, void *ptr,
 	dma_addr_t addr;
 
 	BUG_ON(!valid_dma_direction(dir));
+	debug_dma_check_vmalloc(dev, ptr, size);
 	addr = ops->map_page(dev, virt_to_page(ptr),
 			     offset_in_page(ptr), size,
 			     dir, attrs);
kernel/dma/debug.c +12 -0
index c007d25bee09..7ee7978868d4 100644
@@ -1312,6 +1312,18 @@ static void check_sg_segment(struct device *dev, struct scatterlist *sg)
 #endif
 }
 
+void debug_dma_check_vmalloc(struct device *dev, const void *addr,
+		unsigned long len)
+{
+	if (unlikely(dma_debug_disabled()))
+		return;
+
+	if (is_vmalloc_addr(addr))
+		err_printk(dev, NULL, "DMA-API: device driver maps memory from vmalloc area [addr=%p] [len=%lu]\n",
+			   addr, len);
+}
+EXPORT_SYMBOL(debug_dma_check_vmalloc);
+
 void debug_dma_map_page(struct device *dev, struct page *page, size_t offset,
 			size_t size, int direction, dma_addr_t dma_addr,
 			bool map_single)
Signature · 1 line
Sent by a computer through tubes
#1 lore raw
Quoted 6 lines · Stephen Boyd

I recently debugged a DMA mapping oops where a driver was trying to map a buffer returned from request_firmware() with dma_map_single(). Memory returned from request_firmware() is mapped into the vmalloc region and this isn't a valid region to map with dma_map_single() per the DMA documentation's "What memory is DMA'able?" section.

Quoted 6 lines

Unfortunately, we don't really check that in the DMA debugging code, so enabling DMA debugging doesn't help catch this problem. Let's add a new DMA debug function to check for a vmalloc address and print a warning if this happens. This makes it a little easier to debug these sorts of problems, instead of seeing odd behavior or crashes when drivers attempt to map the vmalloc space for DMA.

This looks sensible to me. I wonder if we shouldn't just throw in an unconditional

	WARN_ON_ONCE(is_vmalloc_addr(ptr)));

into dma_map_single_attrs, though.

#2 lore raw

Quoting Christoph Hellwig (2018-09-20 22:25:37)

Quoted 14 lines · Stephen Boyd
Quoted 6 lines

I recently debugged a DMA mapping oops where a driver was trying to map a buffer returned from request_firmware() with dma_map_single(). Memory returned from request_firmware() is mapped into the vmalloc region and this isn't a valid region to map with dma_map_single() per the DMA documentation's "What memory is DMA'able?" section.

Quoted 6 lines

Unfortunately, we don't really check that in the DMA debugging code, so enabling DMA debugging doesn't help catch this problem. Let's add a new DMA debug function to check for a vmalloc address and print a warning if this happens. This makes it a little easier to debug these sorts of problems, instead of seeing odd behavior or crashes when drivers attempt to map the vmalloc space for DMA.

Quoted 6 lines

This looks sensible to me. I wonder if we shouldn't just throw in an unconditional

        WARN_ON_ONCE(is_vmalloc_addr(ptr)));

into dma_map_single_attrs, though.

Yes that works just as well if we're willing to take the overhead of the check all the time. This isn't a hotpath?

reviewer
#3 lore raw
In reply to Stephen Boyd

Hi Stephen,

Quoted 6 lines · Stephen Boyd

I recently debugged a DMA mapping oops where a driver was trying to map a buffer returned from request_firmware() with dma_map_single(). Memory returned from request_firmware() is mapped into the vmalloc region and this isn't a valid region to map with dma_map_single() per the DMA documentation's "What memory is DMA'able?" section.

Quoted 6 lines

Unfortunately, we don't really check that in the DMA debugging code, so enabling DMA debugging doesn't help catch this problem. Let's add a new DMA debug function to check for a vmalloc address and print a warning if this happens. This makes it a little easier to debug these sorts of problems, instead of seeing odd behavior or crashes when drivers attempt to map the vmalloc space for DMA.

Good idea!

Quoted 8 lines
Cc Marek Szyprowski <[email protected]>
Cc Robin Murphy <[email protected]>
---
  include/linux/dma-debug.h   |  8 ++++++++
  include/linux/dma-mapping.h |  1 +
  kernel/dma/debug.c          | 12 ++++++++++++
  3 files changed, 21 insertions(+)

However I can't help thinking this looks a little heavyweight for a single specific check. It seems like it would be enough to simply pass the VA as an extra argument to debug_dma_map_page(), since we already have the map_single argument which would indicate when it's valid. What do you reckon?

Robin.

Quoted 62 lines
include/linux/dma-debug.h +8 -0
index a785f2507159..5aec2ca8a426 100644
@@ -32,6 +32,9 @@ extern void dma_debug_add_bus(struct bus_type *bus);
 
  extern int dma_debug_resize_entries(u32 num_entries);
 
+extern void debug_dma_check_vmalloc(struct device *dev, const void *addr,
+				    unsigned long len);
+
  extern void debug_dma_map_page(struct device *dev, struct page *page,
  			       size_t offset, size_t size,
  			       int direction, dma_addr_t dma_addr,
@@ -103,6 +106,11 @@ static inline int dma_debug_resize_entries(u32 num_entries)
  	return 0;
  }
 
+static inline void debug_dma_check_vmalloc(struct device *dev, const void *addr,
+					   unsigned long len)
+{
+}
+
  static inline void debug_dma_map_page(struct device *dev, struct page *page,
  				      size_t offset, size_t size,
  				      int direction, dma_addr_t dma_addr,
include/linux/dma-mapping.h +1 -0
index eafd6f318e78..a0e67fd5433c 100644
@@ -232,6 +232,7 @@ static inline dma_addr_t dma_map_single_attrs(struct device *dev, void *ptr,
  	dma_addr_t addr;
 
  	BUG_ON(!valid_dma_direction(dir));
+	debug_dma_check_vmalloc(dev, ptr, size);
  	addr = ops->map_page(dev, virt_to_page(ptr),
  			     offset_in_page(ptr), size,
  			     dir, attrs);
kernel/dma/debug.c +12 -0
index c007d25bee09..7ee7978868d4 100644
@@ -1312,6 +1312,18 @@ static void check_sg_segment(struct device *dev, struct scatterlist *sg)
  #endif
  }
 
+void debug_dma_check_vmalloc(struct device *dev, const void *addr,
+		unsigned long len)
+{
+	if (unlikely(dma_debug_disabled()))
+		return;
+
+	if (is_vmalloc_addr(addr))
+		err_printk(dev, NULL, "DMA-API: device driver maps memory from vmalloc area [addr=%p] [len=%lu]\n",
+			   addr, len);
+}
+EXPORT_SYMBOL(debug_dma_check_vmalloc);
+
  void debug_dma_map_page(struct device *dev, struct page *page, size_t offset,
  			size_t size, int direction, dma_addr_t dma_addr,
  			bool map_single)
#4 lore raw

Quoting Robin Murphy (2018-09-21 04:09:50)

Quoted 27 lines

Hi Stephen,

Quoted 6 lines · Stephen Boyd

I recently debugged a DMA mapping oops where a driver was trying to map a buffer returned from request_firmware() with dma_map_single(). Memory returned from request_firmware() is mapped into the vmalloc region and this isn't a valid region to map with dma_map_single() per the DMA documentation's "What memory is DMA'able?" section.

Quoted 6 lines

Unfortunately, we don't really check that in the DMA debugging code, so enabling DMA debugging doesn't help catch this problem. Let's add a new DMA debug function to check for a vmalloc address and print a warning if this happens. This makes it a little easier to debug these sorts of problems, instead of seeing odd behavior or crashes when drivers attempt to map the vmalloc space for DMA.

Good idea!

Quoted 8 lines
Cc Marek Szyprowski <[email protected]>
Cc Robin Murphy <[email protected]>
---
  include/linux/dma-debug.h   |  8 ++++++++
  include/linux/dma-mapping.h |  1 +
  kernel/dma/debug.c          | 12 ++++++++++++
  3 files changed, 21 insertions(+)
Quoted 5 lines

However I can't help thinking this looks a little heavyweight for a single specific check. It seems like it would be enough to simply pass the VA as an extra argument to debug_dma_map_page(), since we already have the map_single argument which would indicate when it's valid. What do you reckon?

I thought about augmenting debug_dma_map_page() but that has another problem where those checks are done after the page has been mapped. The kernel can oops when it's operating on the incorrect page so we need to check things before calling the dma ops.

So I split that check up into pre_map and post_map parts but then that seemed like overkill just to check for this vmalloc problem. Here's the patch if you're interested. I can make it a little nicer if you think it's also worthwhile.

---8<----

 include/linux/dma-debug.h   |  8 ++++++++
 include/linux/dma-mapping.h |  3 +++
 kernel/dma/debug.c          | 24 ++++++++++++++++--------
 3 files changed, 27 insertions(+), 8 deletions(-)
include/linux/dma-debug.h +8 -0
index 5aec2ca8a426..3287240a474c 100644
@@ -35,6 +35,9 @@ extern int dma_debug_resize_entries(u32 num_entries);
 extern void debug_dma_check_vmalloc(struct device *dev, const void *addr,
 				    unsigned long len);
 
+extern void debug_dma_pre_map_page(struct device *dev, struct page *page,
+				   size_t offset, size_t size);
+
 extern void debug_dma_map_page(struct device *dev, struct page *page,
 			       size_t offset, size_t size,
 			       int direction, dma_addr_t dma_addr,
@@ -111,6 +114,11 @@ static inline void debug_dma_check_vmalloc(struct device *dev, const void *addr,
 {
 }
 
+static inline void debug_dma_pre_map_page(struct device *dev, struct page *page,
+					  size_t offset, size_t size)
+{
+}
+
 static inline void debug_dma_map_page(struct device *dev, struct page *page,
 				      size_t offset, size_t size,
 				      int direction, dma_addr_t dma_addr,
include/linux/dma-mapping.h +3 -0
index a0e67fd5433c..4839bbb4fdc7 100644
@@ -233,6 +233,8 @@ static inline dma_addr_t dma_map_single_attrs(struct device *dev, void *ptr,
 
 	BUG_ON(!valid_dma_direction(dir));
 	debug_dma_check_vmalloc(dev, ptr, size);
+	debug_dma_pre_map_page(dev, virt_to_page(ptr),
+			       offset_in_page(ptr), size);
 	addr = ops->map_page(dev, virt_to_page(ptr),
 			     offset_in_page(ptr), size,
 			     dir, attrs);
@@ -296,6 +298,7 @@ static inline dma_addr_t dma_map_page_attrs(struct device *dev,
 	dma_addr_t addr;
 
 	BUG_ON(!valid_dma_direction(dir));
+	debug_dma_pre_map_page(dev, page, offset, size);
 	addr = ops->map_page(dev, page, offset, size, dir, attrs);
 	debug_dma_map_page(dev, page, offset, size, dir, addr, false);
 
kernel/dma/debug.c +16 -8
index 7ee7978868d4..dd1ff9f7d783 100644
@@ -1324,6 +1324,22 @@ void debug_dma_check_vmalloc(struct device *dev, const void *addr,
 }
 EXPORT_SYMBOL(debug_dma_check_vmalloc);
 
+void debug_dma_pre_map_page(struct device *dev, struct page *page, size_t offset,
+			    size_t size)
+{
+	if (unlikely(dma_debug_disabled()))
+		return;
+
+	check_for_stack(dev, page, offset);
+
+	if (!PageHighMem(page)) {
+		void *addr = page_address(page) + offset;
+
+		check_for_illegal_area(dev, addr, size);
+	}
+}
+EXPORT_SYMBOL(debug_dma_pre_map_page);
+
 void debug_dma_map_page(struct device *dev, struct page *page, size_t offset,
 			size_t size, int direction, dma_addr_t dma_addr,
 			bool map_single)
@@ -1352,14 +1368,6 @@ void debug_dma_map_page(struct device *dev, struct page *page, size_t offset,
 	if (map_single)
 		entry->type = dma_debug_single;
 
-	check_for_stack(dev, page, offset);
-
-	if (!PageHighMem(page)) {
-		void *addr = page_address(page) + offset;
-
-		check_for_illegal_area(dev, addr, size);
-	}
-
 	add_dma_entry(entry);
 }
 EXPORT_SYMBOL(debug_dma_map_page);
Signature · 1 line
Sent by a computer through tubes
reviewer
#5 lore raw
Quoted 34 lines · Stephen Boyd

Quoting Robin Murphy (2018-09-21 04:09:50)

Quoted 27 lines

Hi Stephen,

Quoted 6 lines · Stephen Boyd

I recently debugged a DMA mapping oops where a driver was trying to map a buffer returned from request_firmware() with dma_map_single(). Memory returned from request_firmware() is mapped into the vmalloc region and this isn't a valid region to map with dma_map_single() per the DMA documentation's "What memory is DMA'able?" section.

Quoted 6 lines

Unfortunately, we don't really check that in the DMA debugging code, so enabling DMA debugging doesn't help catch this problem. Let's add a new DMA debug function to check for a vmalloc address and print a warning if this happens. This makes it a little easier to debug these sorts of problems, instead of seeing odd behavior or crashes when drivers attempt to map the vmalloc space for DMA.

Good idea!

Quoted 8 lines
Cc Marek Szyprowski <[email protected]>
Cc Robin Murphy <[email protected]>
---
   include/linux/dma-debug.h   |  8 ++++++++
   include/linux/dma-mapping.h |  1 +
   kernel/dma/debug.c          | 12 ++++++++++++
   3 files changed, 21 insertions(+)
Quoted 5 lines

However I can't help thinking this looks a little heavyweight for a single specific check. It seems like it would be enough to simply pass the VA as an extra argument to debug_dma_map_page(), since we already have the map_single argument which would indicate when it's valid. What do you reckon?

Quoted 4 lines

I thought about augmenting debug_dma_map_page() but that has another problem where those checks are done after the page has been mapped. The kernel can oops when it's operating on the incorrect page so we need to check things before calling the dma ops.

Oh, right, as usual I've managed to overlook a subtlety :)

In fact, with memory now jogged, I think the virt_to_page() itself is allowed to blow up if !virt_addr_valid(), so we probably do need something like your original idea to be properly robust. I guess it probably makes sense to implement that as debug_dma_map_single() then, since the purpose is really to cover any VA checks specific to that case which debug_dma_map_page() can't be expected to do after the fact.

Thanks,
Robin.

Quoted 101 lines

So I split that check up into pre_map and post_map parts but then that seemed like overkill just to check for this vmalloc problem. Here's the patch if you're interested. I can make it a little nicer if you think it's also worthwhile.

---8<----

  include/linux/dma-debug.h   |  8 ++++++++
  include/linux/dma-mapping.h |  3 +++
  kernel/dma/debug.c          | 24 ++++++++++++++++--------
  3 files changed, 27 insertions(+), 8 deletions(-)
include/linux/dma-debug.h +8 -0
index 5aec2ca8a426..3287240a474c 100644
@@ -35,6 +35,9 @@ extern int dma_debug_resize_entries(u32 num_entries);
  extern void debug_dma_check_vmalloc(struct device *dev, const void *addr,
  				    unsigned long len);
 
+extern void debug_dma_pre_map_page(struct device *dev, struct page *page,
+				   size_t offset, size_t size);
+
  extern void debug_dma_map_page(struct device *dev, struct page *page,
  			       size_t offset, size_t size,
  			       int direction, dma_addr_t dma_addr,
@@ -111,6 +114,11 @@ static inline void debug_dma_check_vmalloc(struct device *dev, const void *addr,
  {
  }
 
+static inline void debug_dma_pre_map_page(struct device *dev, struct page *page,
+					  size_t offset, size_t size)
+{
+}
+
  static inline void debug_dma_map_page(struct device *dev, struct page *page,
  				      size_t offset, size_t size,
  				      int direction, dma_addr_t dma_addr,
include/linux/dma-mapping.h +3 -0
index a0e67fd5433c..4839bbb4fdc7 100644
@@ -233,6 +233,8 @@ static inline dma_addr_t dma_map_single_attrs(struct device *dev, void *ptr,
 
  	BUG_ON(!valid_dma_direction(dir));
  	debug_dma_check_vmalloc(dev, ptr, size);
+	debug_dma_pre_map_page(dev, virt_to_page(ptr),
+			       offset_in_page(ptr), size);
  	addr = ops->map_page(dev, virt_to_page(ptr),
  			     offset_in_page(ptr), size,
  			     dir, attrs);
@@ -296,6 +298,7 @@ static inline dma_addr_t dma_map_page_attrs(struct device *dev,
  	dma_addr_t addr;
 
  	BUG_ON(!valid_dma_direction(dir));
+	debug_dma_pre_map_page(dev, page, offset, size);
  	addr = ops->map_page(dev, page, offset, size, dir, attrs);
  	debug_dma_map_page(dev, page, offset, size, dir, addr, false);
 
kernel/dma/debug.c +16 -8
index 7ee7978868d4..dd1ff9f7d783 100644
@@ -1324,6 +1324,22 @@ void debug_dma_check_vmalloc(struct device *dev, const void *addr,
  }
  EXPORT_SYMBOL(debug_dma_check_vmalloc);
 
+void debug_dma_pre_map_page(struct device *dev, struct page *page, size_t offset,
+			    size_t size)
+{
+	if (unlikely(dma_debug_disabled()))
+		return;
+
+	check_for_stack(dev, page, offset);
+
+	if (!PageHighMem(page)) {
+		void *addr = page_address(page) + offset;
+
+		check_for_illegal_area(dev, addr, size);
+	}
+}
+EXPORT_SYMBOL(debug_dma_pre_map_page);
+
  void debug_dma_map_page(struct device *dev, struct page *page, size_t offset,
  			size_t size, int direction, dma_addr_t dma_addr,
  			bool map_single)
@@ -1352,14 +1368,6 @@ void debug_dma_map_page(struct device *dev, struct page *page, size_t offset,
  	if (map_single)
  		entry->type = dma_debug_single;
 
-	check_for_stack(dev, page, offset);
-
-	if (!PageHighMem(page)) {
-		void *addr = page_address(page) + offset;
-
-		check_for_illegal_area(dev, addr, size);
-	}
-
  	add_dma_entry(entry);
  }
  EXPORT_SYMBOL(debug_dma_map_page);
#6 lore raw

Quoting Robin Murphy (2018-09-26 06:24:42)

Quoted 40 lines · Stephen Boyd
Quoted 34 lines

Quoting Robin Murphy (2018-09-21 04:09:50)

Quoted 27 lines

Hi Stephen,

Quoted 6 lines · Stephen Boyd

I recently debugged a DMA mapping oops where a driver was trying to map a buffer returned from request_firmware() with dma_map_single(). Memory returned from request_firmware() is mapped into the vmalloc region and this isn't a valid region to map with dma_map_single() per the DMA documentation's "What memory is DMA'able?" section.

Quoted 6 lines

Unfortunately, we don't really check that in the DMA debugging code, so enabling DMA debugging doesn't help catch this problem. Let's add a new DMA debug function to check for a vmalloc address and print a warning if this happens. This makes it a little easier to debug these sorts of problems, instead of seeing odd behavior or crashes when drivers attempt to map the vmalloc space for DMA.

Good idea!

Quoted 8 lines
Cc Marek Szyprowski <[email protected]>
Cc Robin Murphy <[email protected]>
---
   include/linux/dma-debug.h   |  8 ++++++++
   include/linux/dma-mapping.h |  1 +
   kernel/dma/debug.c          | 12 ++++++++++++
   3 files changed, 21 insertions(+)
Quoted 5 lines

However I can't help thinking this looks a little heavyweight for a single specific check. It seems like it would be enough to simply pass the VA as an extra argument to debug_dma_map_page(), since we already have the map_single argument which would indicate when it's valid. What do you reckon?

Quoted 4 lines

I thought about augmenting debug_dma_map_page() but that has another problem where those checks are done after the page has been mapped. The kernel can oops when it's operating on the incorrect page so we need to check things before calling the dma ops.

Quoted 9 lines

Oh, right, as usual I've managed to overlook a subtlety :)

In fact, with memory now jogged, I think the virt_to_page() itself is allowed to blow up if !virt_addr_valid(), so we probably do need something like your original idea to be properly robust. I guess it probably makes sense to implement that as debug_dma_map_single() then, since the purpose is really to cover any VA checks specific to that case which debug_dma_map_page() can't be expected to do after the fact.

Ok. I can rename it to dma_debug_map_single() and throw in a virt_addr_valid() check too. Thanks for the suggestion.