Skip to contentSkip to conversation

iommu: Add IOMMU_DEBUG_PAGEALLOC sanitizer

PATCH Superseded Mostafa Saleh 2 participants 14 messages Dec 11, 2025 – Jan 6

Subsystems (5)

v4 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v4 cover iommu: Add IOMMU_DEBUG_PAGEALLOC sanitizer

Overview
--------
This patch series introduces a new debugging feature, IOMMU_DEBUG_PAGEALLOC, designed to catch DMA use-after-free bugs and IOMMU mapping leaks from buggy drivers.

The kernel has powerful sanitizers like KASAN and DEBUG_PAGEALLOC for catching CPU-side memory corruption. However, there is limited runtime sanitization for DMA mappings managed by the IOMMU. A buggy driver can free a page while it is still mapped for DMA, leading to memory corruption or use-after-free vulnerabilities when that page is reallocated and used for a different purpose.

Inspired by DEBUG_PAGEALLOC, this sanitizer tracks IOMMU mappings on a per-page basis, as it’s not possible to unmap the pages, because it requires to lock and walk all domains on every kernel free, instead we rely on page_ext to add an IOMMU-specific mapping reference count for each page.
And on each page allocated/freed from the kernel we simply check the count and WARN if it is not zero, and dumping page owner information if enabled.

Concurrency
-----------
By design this check is racy where one caller can map pages just after the check, which can lead to false negatives.
In my opinion this is acceptable for sanitizers (for ex KCSAN have that property).
Otherwise we have to implement locks in iommu_map/unmap for all domains which is not favourable even for a debug feature.
The sanitizer only guarantees that the refcount itself doesn’t get corrupted using atomics. And there are no false positives.

CPU vs IOMMU Page Size
----------------------
IOMMUs can use different page sizes and which can be non-homogeneous; not even all of them have the same page size.

To solve this, the refcount is always incremented and decremented in units of the smallest page size supported by the IOMMU domain. This ensures the accounting remains consistent regardless of the size of the map or unmap operation, otherwise double counting can happen.

Testing & Performance
---------------------
This was tested on Morello with Arm64 + SMMUv3
Did some tests on Qemu including different SMMUv3/CPU page size (arm64).

I also ran dma_map_benchmark on Morello:

echo dma_map_benchmark > /sys/bus/pci/devices/0000\:06\:00.0/driver_override echo 0000:06:00.0 > /sys/bus/pci/devices/0000\:06\:00.0/driver/unbind echo 0000:06:00.0 > /sys/bus/pci/drivers/dma_map_benchmark/bind ./dma_map_benchmark -t $threads -g $nr_pages

CONFIG refers to "CONFIG_IOMMU_DEBUG_PAGEALLOC" cmdline refers to "iommu.debug_pagealloc"
Numbers are (map latency)/(unmap latency), lower is better.

			CONFIG=n    CONFIG=y    CONFIG=y
			            cmdline=0   cmdline=1
4K - 1 thread		0.1/0.6     0.1/0.6     0.1/0.7

4K - 4 threads 0.1/1.1 0.1/1.0 0.2/1.1
1M - 1 thread 0.8/21.2 0.7/21.2 5.4/42.3
1M - 4 threads 1.1/45.9 1.1/46.0 5.9/45.1

Main changes in v4:
v3: https://lore.kernel.org/all/[email protected]/
- Update the kernel parameter format in docs based on Randy feedback
- Update commit subjects
- Add IOMMU only functions in iommu-priv.h based on Baolu feedback

Main changes in v3: (Most of them addressing Will comments)
v2: https://lore.kernel.org/linux-iommu/[email protected]/
- Reword the Kconfig help
- Use unmap_begin/end instead of unmap/remap
- Use relaxed accessors when refcounting
- Fix a bug with checking the returned address from iova_to_phys
- Add more hardening checks (overflow)
- Add more debug info on assertions (dump_page_owner())
- Handle cases where unmap returns larger size as the core code seems to tolerate that.
- Drop Tested-by tags from Qinxin as the code logic changed

Main changes in v2:
v1: https://lore.kernel.org/linux-iommu/[email protected]/
- Address Jörg comments about #ifdefs and static keys
- Reword the Kconfig help
- Drop RFC
- Collect t-b from Qinxin
- Minor cleanups

Mostafa Saleh (4):
iommu: Add page_ext for IOMMU_DEBUG_PAGEALLOC
iommu: Add calls for IOMMU_DEBUG_PAGEALLOC
iommu: debug-pagealloc: Track IOMMU pages
iommu: debug-pagealloc: Check mapped/unmapped kernel memory

 .../admin-guide/kernel-parameters.txt         |   9 +
 drivers/iommu/Kconfig                         |  19 ++
 drivers/iommu/Makefile                        |   1 +
 drivers/iommu/iommu-debug-pagealloc.c         | 174 ++++++++++++++++++
 drivers/iommu/iommu-priv.h                    |  58 ++++++
 drivers/iommu/iommu.c                         |  11 +-
 include/linux/iommu-debug-pagealloc.h         |  32 ++++
 include/linux/mm.h                            |   5 +
 mm/page_ext.c                                 |   4 +
 9 files changed, 311 insertions(+), 2 deletions(-)
 create mode 100644 drivers/iommu/iommu-debug-pagealloc.c
 create mode 100644 include/linux/iommu-debug-pagealloc.h
Signature · 1 line
2.52.0.223.gf5cc29aaa4-goog
#1 lore raw

PATCH v4 1/4 iommu: Add page_ext for IOMMU_DEBUG_PAGEALLOC

Add a new config IOMMU_DEBUG_PAGEALLOC, which registers new data to page_ext.

This config will be used by the IOMMU API to track pages mapped in the IOMMU to catch drivers trying to free kernel memory that they still map in their domains, causing all types of memory corruption.

This behaviour is disabled by default and can be enabled using kernel cmdline iommu.debug_pagealloc.

---
 .../admin-guide/kernel-parameters.txt         |  9 ++++++
 drivers/iommu/Kconfig                         | 19 +++++++++++
 drivers/iommu/Makefile                        |  1 +
 drivers/iommu/iommu-debug-pagealloc.c         | 32 +++++++++++++++++++
 include/linux/iommu-debug-pagealloc.h         | 17 ++++++++++
 mm/page_ext.c                                 |  4 +++
 6 files changed, 82 insertions(+)
 create mode 100644 drivers/iommu/iommu-debug-pagealloc.c
 create mode 100644 include/linux/iommu-debug-pagealloc.h
Documentation/admin-guide/kernel-parameters.txt +9 -0
index a8d0afde7f85..d484d9d8d0a4 100644
@@ -2675,6 +2675,15 @@ Kernel parameters
 			1 - Bypass the IOMMU for DMA.
 			unset - Use value of CONFIG_IOMMU_DEFAULT_PASSTHROUGH.
 
+	iommu.debug_pagealloc=
+			[KNL,EARLY] When CONFIG_IOMMU_DEBUG_PAGEALLOC is set, this
+			parameter enables the feature at boot time. By default, it
+			is disabled and the system behaves the same way as a kernel
+			built without CONFIG_IOMMU_DEBUG_PAGEALLOC.
+			Format: { "0" | "1" }
+			0 - Sanitizer disabled.
+			1 - Sanitizer enabled, expect runtime overhead.
+
 	io7=		[HW] IO7 for Marvel-based Alpha systems
 			See comment before marvel_specify_io7 in
 			arch/alpha/kernel/core_marvel.c.
drivers/iommu/Kconfig +19 -0
index 99095645134f..f86262b11416 100644
@@ -384,6 +384,25 @@ config SPRD_IOMMU
 
 	  Say Y here if you want to use the multimedia devices listed above.
 
+config IOMMU_DEBUG_PAGEALLOC
+	bool "Debug IOMMU mappings against page allocations"
+	depends on DEBUG_PAGEALLOC && IOMMU_API && PAGE_EXTENSION
+	help
+	  This enables a consistency check between the kernel page allocator and
+	  the IOMMU subsystem. It verifies that pages being allocated or freed
+	  are not currently mapped in any IOMMU domain.
+
+	  This helps detect DMA use-after-free bugs where a driver frees a page
+	  but forgets to unmap it from the IOMMU, potentially allowing a device
+	  to overwrite memory that the kernel has repurposed.
+
+	  These checks are best-effort and may not detect all problems.
+
+	  Due to performance overhead, this feature is disabled by default.
+	  You must enable "iommu.debug_pagealloc" from the kernel command
+	  line to activate the runtime checks.
+
+	  If unsure, say N.
 endif # IOMMU_SUPPORT
 
 source "drivers/iommu/generic_pt/Kconfig"
drivers/iommu/Makefile +1 -0
index 8e8843316c4b..0275821f4ef9 100644
@@ -36,3 +36,4 @@ obj-$(CONFIG_IOMMU_SVA) += iommu-sva.o
 obj-$(CONFIG_IOMMU_IOPF) += io-pgfault.o
 obj-$(CONFIG_SPRD_IOMMU) += sprd-iommu.o
 obj-$(CONFIG_APPLE_DART) += apple-dart.o
+obj-$(CONFIG_IOMMU_DEBUG_PAGEALLOC) += iommu-debug-pagealloc.o
drivers/iommu/iommu-debug-pagealloc.c new file +32 -0
new file mode 100644
index 000000000000..4022e9af7f27
@@ -0,0 +1,32 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2025 - Google Inc
+ * Author: Mostafa Saleh <[email protected]>
+ * IOMMU API debug page alloc sanitizer
+ */
+#include <linux/atomic.h>
+#include <linux/iommu-debug-pagealloc.h>
+#include <linux/kernel.h>
+#include <linux/page_ext.h>
+
+static bool needed;
+
+struct iommu_debug_metadata {
+	atomic_t ref;
+};
+
+static __init bool need_iommu_debug(void)
+{
+	return needed;
+}
+
+struct page_ext_operations page_iommu_debug_ops = {
+	.size = sizeof(struct iommu_debug_metadata),
+	.need = need_iommu_debug,
+};
+
+static int __init iommu_debug_pagealloc(char *str)
+{
+	return kstrtobool(str, &needed);
+}
+early_param("iommu.debug_pagealloc", iommu_debug_pagealloc);
include/linux/iommu-debug-pagealloc.h new file +17 -0
new file mode 100644
index 000000000000..83e64d70bf6c
@@ -0,0 +1,17 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2025 - Google Inc
+ * Author: Mostafa Saleh <[email protected]>
+ * IOMMU API debug page alloc sanitizer
+ */
+
+#ifndef __LINUX_IOMMU_DEBUG_PAGEALLOC_H
+#define __LINUX_IOMMU_DEBUG_PAGEALLOC_H
+
+#ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC
+
+extern struct page_ext_operations page_iommu_debug_ops;
+
+#endif /* CONFIG_IOMMU_DEBUG_PAGEALLOC */
+
+#endif /* __LINUX_IOMMU_DEBUG_PAGEALLOC_H */
mm/page_ext.c +4 -0
index d7396a8970e5..297e4cd8ce90 100644
@@ -11,6 +11,7 @@
 #include <linux/page_table_check.h>
 #include <linux/rcupdate.h>
 #include <linux/pgalloc_tag.h>
+#include <linux/iommu-debug-pagealloc.h>
 
 /*
  * struct page extension
@@ -89,6 +90,9 @@ static struct page_ext_operations *page_ext_ops[] __initdata = {
 #ifdef CONFIG_PAGE_TABLE_CHECK
 	&page_table_check_ops,
 #endif
+#ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC
+	&page_iommu_debug_ops,
+#endif
 };
 
 unsigned long page_ext_size;
Signature · 1 line
2.52.0.223.gf5cc29aaa4-goog
Reviewed-by on v4 1/4
#2 lore raw

Re: [PATCH v4 1/4] iommu: Add page_ext for IOMMU_DEBUG_PAGEALLOC

Quoted 19 lines · Mostafa Saleh

Add a new config IOMMU_DEBUG_PAGEALLOC, which registers new data to page_ext.

This config will be used by the IOMMU API to track pages mapped in the IOMMU to catch drivers trying to free kernel memory that they still map in their domains, causing all types of memory corruption.

This behaviour is disabled by default and can be enabled using kernel cmdline iommu.debug_pagealloc.

---
  .../admin-guide/kernel-parameters.txt         |  9 ++++++
  drivers/iommu/Kconfig                         | 19 +++++++++++
  drivers/iommu/Makefile                        |  1 +
  drivers/iommu/iommu-debug-pagealloc.c         | 32 +++++++++++++++++++
  include/linux/iommu-debug-pagealloc.h         | 17 ++++++++++
  mm/page_ext.c                                 |  4 +++
  6 files changed, 82 insertions(+)
Quoted 2 lines

create mode 100644 drivers/iommu/iommu-debug-pagealloc.c create mode 100644 include/linux/iommu-debug-pagealloc.h

Do you need to update the MAINTAINERS file, since a new file (iommu- debug-pagealloc.h) is being added?

Quoted 7 lines
Documentation/admin-guide/kernel-parameters.txt +0 -0
index a8d0afde7f85..d484d9d8d0a4 100644
@@ -2675,6 +2675,15 @@ Kernel parameters
  			1 - Bypass the IOMMU for DMA.

Documentation/admin-guide/kernel-parameters.txt Kernel parameters · line 2685

Quoted 10 lines
Documentation/admin-guide/kernel-parameters.txt +8 -0
  			unset - Use value of CONFIG_IOMMU_DEFAULT_PASSTHROUGH.
 
+	iommu.debug_pagealloc=
+			[KNL,EARLY] When CONFIG_IOMMU_DEBUG_PAGEALLOC is set, this
+			parameter enables the feature at boot time. By default, it
+			is disabled and the system behaves the same way as a kernel
+			built without CONFIG_IOMMU_DEBUG_PAGEALLOC.
+			Format: { "0" | "1" }
+			0 - Sanitizer disabled.
+			1 - Sanitizer enabled, expect runtime overhead.

"iommu.debug_pagealloc=0" has the same meaning as "unset". Would "iommu=debug_pagealloc" look more appropriate?

Documentation/admin-guide/kernel-parameters.txt Kernel parameters · line 2689

Quoted 4 lines
Documentation/admin-guide/kernel-parameters.txt +1 -0
+
  	io7=		[HW] IO7 for Marvel-based Alpha systems
  			See comment before marvel_specify_io7 in
  			arch/alpha/kernel/core_marvel.c.

[ ... ]

Other changes look good to me.

#3 lore raw

Re: [PATCH v4 1/4] iommu: Add page_ext for IOMMU_DEBUG_PAGEALLOC

Quoted 37 lines · Baolu Lu
Quoted 19 lines · Mostafa Saleh

Add a new config IOMMU_DEBUG_PAGEALLOC, which registers new data to page_ext.

This config will be used by the IOMMU API to track pages mapped in the IOMMU to catch drivers trying to free kernel memory that they still map in their domains, causing all types of memory corruption.

This behaviour is disabled by default and can be enabled using kernel cmdline iommu.debug_pagealloc.

---
  .../admin-guide/kernel-parameters.txt         |  9 ++++++
  drivers/iommu/Kconfig                         | 19 +++++++++++
  drivers/iommu/Makefile                        |  1 +
  drivers/iommu/iommu-debug-pagealloc.c         | 32 +++++++++++++++++++
  include/linux/iommu-debug-pagealloc.h         | 17 ++++++++++
  mm/page_ext.c                                 |  4 +++
  6 files changed, 82 insertions(+)
Quoted 2 lines

create mode 100644 drivers/iommu/iommu-debug-pagealloc.c create mode 100644 include/linux/iommu-debug-pagealloc.h

Do you need to update the MAINTAINERS file, since a new file (iommu- debug-pagealloc.h) is being added?

Quoted 11 lines
Documentation/admin-guide/kernel-parameters.txt +3 -0
index a8d0afde7f85..d484d9d8d0a4 100644
@@ -2675,6 +2675,15 @@ Kernel parameters
  			1 - Bypass the IOMMU for DMA.
  			unset - Use value of CONFIG_IOMMU_DEFAULT_PASSTHROUGH.
+	iommu.debug_pagealloc=
+			[KNL,EARLY] When CONFIG_IOMMU_DEBUG_PAGEALLOC is set, this
+			parameter enables the feature at boot time. By default, it
Quoted 8 lines

Documentation/admin-guide/kernel-parameters.txt Kernel parameters · line 2685

Quoted 5 lines
Documentation/admin-guide/kernel-parameters.txt +5 -0
+			is disabled and the system behaves the same way as a kernel
+			built without CONFIG_IOMMU_DEBUG_PAGEALLOC.
+			Format: { "0" | "1" }
+			0 - Sanitizer disabled.
+			1 - Sanitizer enabled, expect runtime overhead.

"iommu.debug_pagealloc=0" has the same meaning as "unset". Would "iommu=debug_pagealloc" look more appropriate?

"iommu=" seems to be x86 specific. I was following the pattern for "iommu.passthrough=", "iommu.strict=" and also as "debug_pagealloc=" has it's own command line, it made sense to design this way.

Thanks,
Mostafa

Quoted 11 lines

Documentation/admin-guide/kernel-parameters.txt Kernel parameters · line 2689

Quoted 5 lines
Documentation/admin-guide/kernel-parameters.txt +1 -0
+
  	io7=		[HW] IO7 for Marvel-based Alpha systems
  			See comment before marvel_specify_io7 in
  			arch/alpha/kernel/core_marvel.c.

[ ... ]

Other changes look good to me.

#4 lore raw

Re: [PATCH v4 1/4] iommu: Add page_ext for IOMMU_DEBUG_PAGEALLOC

In reply to Baolu Lu
Quoted 24 lines · Baolu Lu
Quoted 19 lines · Mostafa Saleh

Add a new config IOMMU_DEBUG_PAGEALLOC, which registers new data to page_ext.

This config will be used by the IOMMU API to track pages mapped in the IOMMU to catch drivers trying to free kernel memory that they still map in their domains, causing all types of memory corruption.

This behaviour is disabled by default and can be enabled using kernel cmdline iommu.debug_pagealloc.

---
  .../admin-guide/kernel-parameters.txt         |  9 ++++++
  drivers/iommu/Kconfig                         | 19 +++++++++++
  drivers/iommu/Makefile                        |  1 +
  drivers/iommu/iommu-debug-pagealloc.c         | 32 +++++++++++++++++++
  include/linux/iommu-debug-pagealloc.h         | 17 ++++++++++
  mm/page_ext.c                                 |  4 +++
  6 files changed, 82 insertions(+)
Quoted 2 lines

create mode 100644 drivers/iommu/iommu-debug-pagealloc.c create mode 100644 include/linux/iommu-debug-pagealloc.h

Quoted 3 lines

Do you need to update the MAINTAINERS file, since a new file (iommu- debug-pagealloc.h) is being added?

Sorry I missed this last time, that should be under the path of "IOMMU SUBSYSTEM",
I am happy to maintain it, but it's up to Joerg and Will.

Thanks,
Mostafa

Quoted 31 lines
Quoted 7 lines
Documentation/admin-guide/kernel-parameters.txt +0 -0
index a8d0afde7f85..d484d9d8d0a4 100644
@@ -2675,6 +2675,15 @@ Kernel parameters
                      1 - Bypass the IOMMU for DMA.

Documentation/admin-guide/kernel-parameters.txt Kernel parameters · line 2685

Quoted 10 lines
Documentation/admin-guide/kernel-parameters.txt +8 -0
                      unset - Use value of CONFIG_IOMMU_DEFAULT_PASSTHROUGH.
 
+     iommu.debug_pagealloc=
+                     [KNL,EARLY] When CONFIG_IOMMU_DEBUG_PAGEALLOC is set, this
+                     parameter enables the feature at boot time. By default, it
+                     is disabled and the system behaves the same way as a kernel
+                     built without CONFIG_IOMMU_DEBUG_PAGEALLOC.
+                     Format: { "0" | "1" }
+                     0 - Sanitizer disabled.
+                     1 - Sanitizer enabled, expect runtime overhead.

"iommu.debug_pagealloc=0" has the same meaning as "unset". Would "iommu=debug_pagealloc" look more appropriate?

Documentation/admin-guide/kernel-parameters.txt Kernel parameters · line 2689

Quoted 4 lines
Documentation/admin-guide/kernel-parameters.txt +1 -0
+
      io7=            [HW] IO7 for Marvel-based Alpha systems
                      See comment before marvel_specify_io7 in
                      arch/alpha/kernel/core_marvel.c.

[ ... ]

Other changes look good to me.

#5 lore raw

PATCH v4 2/4 iommu: Add calls for IOMMU_DEBUG_PAGEALLOC

In reply to Mostafa Saleh

Add calls for the new iommu debug config IOMMU_DEBUG_PAGEALLOC:
- iommu_debug_init: Enable the debug mode if configured by the user.
- iommu_debug_map: Track iommu pages mapped, using physical address.
- iommu_debug_unmap_begin: Track start of iommu unmap operation, with IOVA and size.
- iommu_debug_unmap_end: Track the end of unmap operation, passing the actual unmapped size versus the tracked one at unmap_begin.

We have to do the unmap_begin/end as once pages are unmapped we lose the information of the physical address.
This is racy, but the API is racy by construction as it uses refcounts and doesn't attempt to lock/synchronize with the IOMMU API as that will be costly, meaning that possibility of false negative exists.

---
 drivers/iommu/iommu-debug-pagealloc.c | 28 +++++++++++++
 drivers/iommu/iommu-priv.h            | 58 +++++++++++++++++++++++++++
 drivers/iommu/iommu.c                 | 11 ++++-
 include/linux/iommu-debug-pagealloc.h |  1 +
 4 files changed, 96 insertions(+), 2 deletions(-)
drivers/iommu/iommu-debug-pagealloc.c +28 -0
index 4022e9af7f27..1d343421da98 100644
@@ -5,11 +5,15 @@
  * IOMMU API debug page alloc sanitizer
  */
 #include <linux/atomic.h>
+#include <linux/iommu.h>
 #include <linux/iommu-debug-pagealloc.h>
 #include <linux/kernel.h>
 #include <linux/page_ext.h>
 
+#include "iommu-priv.h"
+
 static bool needed;
+DEFINE_STATIC_KEY_FALSE(iommu_debug_initialized);
 
 struct iommu_debug_metadata {
 	atomic_t ref;
@@ -25,6 +29,30 @@ struct page_ext_operations page_iommu_debug_ops = {
 	.need = need_iommu_debug,
 };
 
+void __iommu_debug_map(struct iommu_domain *domain, phys_addr_t phys, size_t size)
+{
+}
+
+void __iommu_debug_unmap_begin(struct iommu_domain *domain,
+			       unsigned long iova, size_t size)
+{
+}
+
+void __iommu_debug_unmap_end(struct iommu_domain *domain,
+			     unsigned long iova, size_t size,
+			     size_t unmapped)
+{
+}
+
+void iommu_debug_init(void)
+{
+	if (!needed)
+		return;
+
+	pr_info("iommu: Debugging page allocations, expect overhead or disable iommu.debug_pagealloc");
+	static_branch_enable(&iommu_debug_initialized);
+}
+
 static int __init iommu_debug_pagealloc(char *str)
 {
 	return kstrtobool(str, &needed);
drivers/iommu/iommu-priv.h +58 -0

68 lines not on this page · raw message

drivers/iommu/iommu.c +9 -2
index 2ca990dfbb88..01b062575519 100644
@@ -232,6 +232,8 @@ static int __init iommu_subsys_init(void)
 	if (!nb)
 		return -ENOMEM;
 
+	iommu_debug_init();
+
 	for (int i = 0; i < ARRAY_SIZE(iommu_buses); i++) {
 		nb[i].notifier_call = iommu_bus_notifier;
 		bus_register_notifier(iommu_buses[i], &nb[i]);
@@ -2562,10 +2564,12 @@ int iommu_map_nosync(struct iommu_domain *domain, unsigned long iova,
 	}
 
 	/* unroll mapping in case something went wrong */
-	if (ret)
+	if (ret) {
 		iommu_unmap(domain, orig_iova, orig_size - size);
-	else
+	} else {
 		trace_map(orig_iova, orig_paddr, orig_size);
+		iommu_debug_map(domain, orig_paddr, orig_size);
+	}
 
 	return ret;
 }
@@ -2627,6 +2631,8 @@ static size_t __iommu_unmap(struct iommu_domain *domain,
 
 	pr_debug("unmap this: iova 0x%lx size 0x%zx\n", iova, size);
 
+	iommu_debug_unmap_begin(domain, iova, size);
+
 	/*
 	 * Keep iterating until we either unmap 'size' bytes (or more)
 	 * or we hit an area that isn't mapped.
@@ -2647,6 +2653,7 @@ static size_t __iommu_unmap(struct iommu_domain *domain,
 	}
 
 	trace_unmap(orig_iova, size, unmapped);
+	iommu_debug_unmap_end(domain, orig_iova, size, unmapped);
 	return unmapped;
 }
 
include/linux/iommu-debug-pagealloc.h +1 -0
index 83e64d70bf6c..a439d6815ca1 100644
@@ -9,6 +9,7 @@
 #define __LINUX_IOMMU_DEBUG_PAGEALLOC_H
 
 #ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC
+DECLARE_STATIC_KEY_FALSE(iommu_debug_initialized);
 
 extern struct page_ext_operations page_iommu_debug_ops;
 
Signature · 1 line
2.52.0.223.gf5cc29aaa4-goog
#6 lore raw

Re: [PATCH v4 2/4] iommu: Add calls for IOMMU_DEBUG_PAGEALLOC

Quoted 110 lines · Mostafa Saleh

Add calls for the new iommu debug config IOMMU_DEBUG_PAGEALLOC:
- iommu_debug_init: Enable the debug mode if configured by the user.
- iommu_debug_map: Track iommu pages mapped, using physical address.
- iommu_debug_unmap_begin: Track start of iommu unmap operation, with IOVA and size.
- iommu_debug_unmap_end: Track the end of unmap operation, passing the actual unmapped size versus the tracked one at unmap_begin.

We have to do the unmap_begin/end as once pages are unmapped we lose the information of the physical address.
This is racy, but the API is racy by construction as it uses refcounts and doesn't attempt to lock/synchronize with the IOMMU API as that will be costly, meaning that possibility of false negative exists.

---
  drivers/iommu/iommu-debug-pagealloc.c | 28 +++++++++++++
  drivers/iommu/iommu-priv.h            | 58 +++++++++++++++++++++++++++
  drivers/iommu/iommu.c                 | 11 ++++-
  include/linux/iommu-debug-pagealloc.h |  1 +
  4 files changed, 96 insertions(+), 2 deletions(-)
drivers/iommu/iommu-debug-pagealloc.c +28 -0
index 4022e9af7f27..1d343421da98 100644
@@ -5,11 +5,15 @@
   * IOMMU API debug page alloc sanitizer
   */
  #include <linux/atomic.h>
+#include <linux/iommu.h>
  #include <linux/iommu-debug-pagealloc.h>
  #include <linux/kernel.h>
  #include <linux/page_ext.h>
 
+#include "iommu-priv.h"
+
  static bool needed;
+DEFINE_STATIC_KEY_FALSE(iommu_debug_initialized);
 
  struct iommu_debug_metadata {
  	atomic_t ref;
@@ -25,6 +29,30 @@ struct page_ext_operations page_iommu_debug_ops = {
  	.need = need_iommu_debug,
  };
 
+void __iommu_debug_map(struct iommu_domain *domain, phys_addr_t phys, size_t size)
+{
+}
+
+void __iommu_debug_unmap_begin(struct iommu_domain *domain,
+			       unsigned long iova, size_t size)
+{
+}
+
+void __iommu_debug_unmap_end(struct iommu_domain *domain,
+			     unsigned long iova, size_t size,
+			     size_t unmapped)
+{
+}
+
+void iommu_debug_init(void)
+{
+	if (!needed)
+		return;
+
+	pr_info("iommu: Debugging page allocations, expect overhead or disable iommu.debug_pagealloc");
+	static_branch_enable(&iommu_debug_initialized);
+}
+
  static int __init iommu_debug_pagealloc(char *str)
  {
  	return kstrtobool(str, &needed);
drivers/iommu/iommu-priv.h +22 -0
index c95394cd03a7..aaffad5854fc 100644
@@ -5,6 +5,7 @@
  #define __LINUX_IOMMU_PRIV_H
 
  #include <linux/iommu.h>
+#include <linux/iommu-debug-pagealloc.h>
  #include <linux/msi.h>
 
  static inline const struct iommu_ops *dev_iommu_ops(struct device *dev)
@@ -65,4 +66,61 @@ static inline int iommufd_sw_msi(struct iommu_domain *domain,
  int iommu_replace_device_pasid(struct iommu_domain *domain,
  			       struct device *dev, ioasid_t pasid,
  			       struct iommu_attach_handle *handle);
+
+#ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC
+
+void __iommu_debug_map(struct iommu_domain *domain, phys_addr_t phys,
+		       size_t size);
+void __iommu_debug_unmap_begin(struct iommu_domain *domain,
+			       unsigned long iova, size_t size);
+void __iommu_debug_unmap_end(struct iommu_domain *domain,
+			     unsigned long iova, size_t size, size_t unmapped);
+
+static inline void iommu_debug_map(struct iommu_domain *domain,
+				   phys_addr_t phys, size_t size)
+{
+	if (static_branch_unlikely(&iommu_debug_initialized))
+		__iommu_debug_map(domain, phys, size);
+}
+
+static inline void iommu_debug_unmap_begin(struct iommu_domain *domain,
+					   unsigned long iova, size_t size)
+{
+	if (static_branch_unlikely(&iommu_debug_initialized))

drivers/iommu/iommu-priv.h static inline int iommufd_sw_msi(struct iommu_domain *domain, · line 99

Quoted 10 lines
drivers/iommu/iommu-priv.h +10 -0
+		__iommu_debug_unmap_begin(domain, iova, size);
+}
+
+static inline void iommu_debug_unmap_end(struct iommu_domain *domain,
+					 unsigned long iova, size_t size,
+					 size_t unmapped)
+{
+	if (static_branch_unlikely(&iommu_debug_initialized))
+		__iommu_debug_unmap_end(domain, iova, size, unmapped);
+}

I am wondering whether it would be better if we move iommu_debug_map() to iommu-debug-pagealloc.c,

void iommu_debug_map(struct iommu_domain *domain,
phys_addr_t phys, size_t size)

{
	if (static_branch_likely(&iommu_debug_initialized))
		__iommu_debug_map(domain, phys, size);
}

(Does it make sense to use static_branch_likely() here? Normally, people
who enable CONFIG_IOMMU_DEBUG_PAGEALLOC would want to use this debugging feature. Or not?)

So that ...

Quoted 77 lines
drivers/iommu/iommu-priv.h +26 -0
+
+void iommu_debug_init(void);
+
+#else
+static inline void iommu_debug_map(struct iommu_domain *domain,
+				   phys_addr_t phys, size_t size)
+{
+}
+
+static inline void iommu_debug_unmap_begin(struct iommu_domain *domain,
+					   unsigned long iova, size_t size)
+{
+}
+
+static inline void iommu_debug_unmap_end(struct iommu_domain *domain,
+					 unsigned long iova, size_t size,
+					 size_t unmapped)
+{
+}
+
+static inline void iommu_debug_init(void)
+{
+}
+
+#endif /* CONFIG_IOMMU_DEBUG_PAGEALLOC */
+
  #endif /* __LINUX_IOMMU_PRIV_H */
drivers/iommu/iommu.c +9 -2
index 2ca990dfbb88..01b062575519 100644
@@ -232,6 +232,8 @@ static int __init iommu_subsys_init(void)
  	if (!nb)
  		return -ENOMEM;
 
+	iommu_debug_init();
+
  	for (int i = 0; i < ARRAY_SIZE(iommu_buses); i++) {
  		nb[i].notifier_call = iommu_bus_notifier;
  		bus_register_notifier(iommu_buses[i], &nb[i]);
@@ -2562,10 +2564,12 @@ int iommu_map_nosync(struct iommu_domain *domain, unsigned long iova,
  	}
 
  	/* unroll mapping in case something went wrong */
-	if (ret)
+	if (ret) {
  		iommu_unmap(domain, orig_iova, orig_size - size);
-	else
+	} else {
  		trace_map(orig_iova, orig_paddr, orig_size);
+		iommu_debug_map(domain, orig_paddr, orig_size);
+	}
 
  	return ret;
  }
@@ -2627,6 +2631,8 @@ static size_t __iommu_unmap(struct iommu_domain *domain,
 
  	pr_debug("unmap this: iova 0x%lx size 0x%zx\n", iova, size);
 
+	iommu_debug_unmap_begin(domain, iova, size);
+
  	/*
  	 * Keep iterating until we either unmap 'size' bytes (or more)
  	 * or we hit an area that isn't mapped.
@@ -2647,6 +2653,7 @@ static size_t __iommu_unmap(struct iommu_domain *domain,
  	}
 
  	trace_unmap(orig_iova, size, unmapped);
+	iommu_debug_unmap_end(domain, orig_iova, size, unmapped);
  	return unmapped;
  }
 
include/linux/iommu-debug-pagealloc.h +0 -0
index 83e64d70bf6c..a439d6815ca1 100644

include/linux/iommu-debug-pagealloc.h line 12

Quoted 4 lines
include/linux/iommu-debug-pagealloc.h +1 -0
@@ -9,6 +9,7 @@
  #define __LINUX_IOMMU_DEBUG_PAGEALLOC_H
 
  #ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC
+DECLARE_STATIC_KEY_FALSE(iommu_debug_initialized);

... we could make this static?

include/linux/iommu-debug-pagealloc.h line 15

Quoted 3 lines
include/linux/iommu-debug-pagealloc.h +0 -0
  extern struct page_ext_operations page_iommu_debug_ops;
 

Thanks,
baolu

#7 lore raw

Re: [PATCH v4 2/4] iommu: Add calls for IOMMU_DEBUG_PAGEALLOC

Quoted 127 lines · Baolu Lu
Quoted 105 lines · Mostafa Saleh

Add calls for the new iommu debug config IOMMU_DEBUG_PAGEALLOC:
- iommu_debug_init: Enable the debug mode if configured by the user.
- iommu_debug_map: Track iommu pages mapped, using physical address.
- iommu_debug_unmap_begin: Track start of iommu unmap operation, with IOVA and size.
- iommu_debug_unmap_end: Track the end of unmap operation, passing the actual unmapped size versus the tracked one at unmap_begin.

We have to do the unmap_begin/end as once pages are unmapped we lose the information of the physical address.
This is racy, but the API is racy by construction as it uses refcounts and doesn't attempt to lock/synchronize with the IOMMU API as that will be costly, meaning that possibility of false negative exists.

---
  drivers/iommu/iommu-debug-pagealloc.c | 28 +++++++++++++
  drivers/iommu/iommu-priv.h            | 58 +++++++++++++++++++++++++++
  drivers/iommu/iommu.c                 | 11 ++++-
  include/linux/iommu-debug-pagealloc.h |  1 +
  4 files changed, 96 insertions(+), 2 deletions(-)
drivers/iommu/iommu-debug-pagealloc.c +28 -0
index 4022e9af7f27..1d343421da98 100644
@@ -5,11 +5,15 @@
   * IOMMU API debug page alloc sanitizer
   */
  #include <linux/atomic.h>
+#include <linux/iommu.h>
  #include <linux/iommu-debug-pagealloc.h>
  #include <linux/kernel.h>
  #include <linux/page_ext.h>
+#include "iommu-priv.h"
+
  static bool needed;
+DEFINE_STATIC_KEY_FALSE(iommu_debug_initialized);
  struct iommu_debug_metadata {
  	atomic_t ref;
@@ -25,6 +29,30 @@ struct page_ext_operations page_iommu_debug_ops = {
  	.need = need_iommu_debug,
  };
+void __iommu_debug_map(struct iommu_domain *domain, phys_addr_t phys, size_t size)
+{
+}
+
+void __iommu_debug_unmap_begin(struct iommu_domain *domain,
+			       unsigned long iova, size_t size)
+{
+}
+
+void __iommu_debug_unmap_end(struct iommu_domain *domain,
+			     unsigned long iova, size_t size,
+			     size_t unmapped)
+{
+}
+
+void iommu_debug_init(void)
+{
+	if (!needed)
+		return;
+
+	pr_info("iommu: Debugging page allocations, expect overhead or disable iommu.debug_pagealloc");
+	static_branch_enable(&iommu_debug_initialized);
+}
+
  static int __init iommu_debug_pagealloc(char *str)
  {
  	return kstrtobool(str, &needed);
drivers/iommu/iommu-priv.h +22 -0
index c95394cd03a7..aaffad5854fc 100644
@@ -5,6 +5,7 @@
  #define __LINUX_IOMMU_PRIV_H
  #include <linux/iommu.h>
+#include <linux/iommu-debug-pagealloc.h>
  #include <linux/msi.h>
  static inline const struct iommu_ops *dev_iommu_ops(struct device *dev)
@@ -65,4 +66,61 @@ static inline int iommufd_sw_msi(struct iommu_domain *domain,
  int iommu_replace_device_pasid(struct iommu_domain *domain,
  			       struct device *dev, ioasid_t pasid,
  			       struct iommu_attach_handle *handle);
+
+#ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC
+
+void __iommu_debug_map(struct iommu_domain *domain, phys_addr_t phys,
+		       size_t size);
+void __iommu_debug_unmap_begin(struct iommu_domain *domain,
+			       unsigned long iova, size_t size);
+void __iommu_debug_unmap_end(struct iommu_domain *domain,
+			     unsigned long iova, size_t size, size_t unmapped);
+
+static inline void iommu_debug_map(struct iommu_domain *domain,
+				   phys_addr_t phys, size_t size)
+{
+	if (static_branch_unlikely(&iommu_debug_initialized))
+		__iommu_debug_map(domain, phys, size);
+}
+
+static inline void iommu_debug_unmap_begin(struct iommu_domain *domain,
+					   unsigned long iova, size_t size)
+{
+	if (static_branch_unlikely(&iommu_debug_initialized))

drivers/iommu/iommu-priv.h static inline int iommufd_sw_msi(struct iommu_domain *domain, · line 99

Quoted 10 lines
drivers/iommu/iommu-priv.h +10 -0
+		__iommu_debug_unmap_begin(domain, iova, size);
+}
+
+static inline void iommu_debug_unmap_end(struct iommu_domain *domain,
+					 unsigned long iova, size_t size,
+					 size_t unmapped)
+{
+	if (static_branch_unlikely(&iommu_debug_initialized))
+		__iommu_debug_unmap_end(domain, iova, size, unmapped);
+}

I am wondering whether it would be better if we move iommu_debug_map() to iommu-debug-pagealloc.c,

void iommu_debug_map(struct iommu_domain *domain,
phys_addr_t phys, size_t size)

{
	if (static_branch_likely(&iommu_debug_initialized))
		__iommu_debug_map(domain, phys, size);
}
Quoted 5 lines

(Does it make sense to use static_branch_likely() here? Normally, people
who enable CONFIG_IOMMU_DEBUG_PAGEALLOC would want to use this debugging feature. Or not?)

So that ...

This actually was the v1 implementation [1], but Jörg suggested to move it to a header file as a function call would have an overhead if this feautre is disabled.

I believe the priority would be to keep the performance overhead minimal with CONFIG_IOMMU_DEBUG_PAGEALLOC and the commandline disabled, so people can run with the config in production and only enable the commandline it to debug problems, without having overhead on the typical case.

Quoted 68 lines
Quoted 68 lines
drivers/iommu/iommu-priv.h +26 -0
+
+void iommu_debug_init(void);
+
+#else
+static inline void iommu_debug_map(struct iommu_domain *domain,
+				   phys_addr_t phys, size_t size)
+{
+}
+
+static inline void iommu_debug_unmap_begin(struct iommu_domain *domain,
+					   unsigned long iova, size_t size)
+{
+}
+
+static inline void iommu_debug_unmap_end(struct iommu_domain *domain,
+					 unsigned long iova, size_t size,
+					 size_t unmapped)
+{
+}
+
+static inline void iommu_debug_init(void)
+{
+}
+
+#endif /* CONFIG_IOMMU_DEBUG_PAGEALLOC */
+
  #endif /* __LINUX_IOMMU_PRIV_H */
drivers/iommu/iommu.c +9 -2
index 2ca990dfbb88..01b062575519 100644
@@ -232,6 +232,8 @@ static int __init iommu_subsys_init(void)
  	if (!nb)
  		return -ENOMEM;
+	iommu_debug_init();
+
  	for (int i = 0; i < ARRAY_SIZE(iommu_buses); i++) {
  		nb[i].notifier_call = iommu_bus_notifier;
  		bus_register_notifier(iommu_buses[i], &nb[i]);
@@ -2562,10 +2564,12 @@ int iommu_map_nosync(struct iommu_domain *domain, unsigned long iova,
  	}
  	/* unroll mapping in case something went wrong */
-	if (ret)
+	if (ret) {
  		iommu_unmap(domain, orig_iova, orig_size - size);
-	else
+	} else {
  		trace_map(orig_iova, orig_paddr, orig_size);
+		iommu_debug_map(domain, orig_paddr, orig_size);
+	}
  	return ret;
  }
@@ -2627,6 +2631,8 @@ static size_t __iommu_unmap(struct iommu_domain *domain,
  	pr_debug("unmap this: iova 0x%lx size 0x%zx\n", iova, size);
+	iommu_debug_unmap_begin(domain, iova, size);
+
  	/*
  	 * Keep iterating until we either unmap 'size' bytes (or more)
  	 * or we hit an area that isn't mapped.
@@ -2647,6 +2653,7 @@ static size_t __iommu_unmap(struct iommu_domain *domain,
  	}
  	trace_unmap(orig_iova, size, unmapped);
+	iommu_debug_unmap_end(domain, orig_iova, size, unmapped);
  	return unmapped;
  }
include/linux/iommu-debug-pagealloc.h +0 -0
index 83e64d70bf6c..a439d6815ca1 100644
Quoted 9 lines

include/linux/iommu-debug-pagealloc.h line 11

Quoted 6 lines
include/linux/iommu-debug-pagealloc.h +1 -0
@@ -9,6 +9,7 @@
  #define __LINUX_IOMMU_DEBUG_PAGEALLOC_H
  #ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC
+DECLARE_STATIC_KEY_FALSE(iommu_debug_initialized);

... we could make this static?

This is not static because of the header usage as mentioned above.

Thanks,
Mostafa

Quoted 1 line

include/linux/iommu-debug-pagealloc.h line 14

Quoted 1 line
include/linux/iommu-debug-pagealloc.h +0 -0
  extern struct page_ext_operations page_iommu_debug_ops;

[1] https://lore.kernel.org/linux-iommu/[email protected]/

Quoted 4 lines

Thanks,
baolu

#8 lore raw

Re: [PATCH v4 2/4] iommu: Add calls for IOMMU_DEBUG_PAGEALLOC

Quoted 132 lines · Mostafa Saleh
Quoted 126 lines · Baolu Lu
Quoted 105 lines · Mostafa Saleh

Add calls for the new iommu debug config IOMMU_DEBUG_PAGEALLOC:
- iommu_debug_init: Enable the debug mode if configured by the user.
- iommu_debug_map: Track iommu pages mapped, using physical address.
- iommu_debug_unmap_begin: Track start of iommu unmap operation, with IOVA and size.
- iommu_debug_unmap_end: Track the end of unmap operation, passing the actual unmapped size versus the tracked one at unmap_begin.

We have to do the unmap_begin/end as once pages are unmapped we lose the information of the physical address.
This is racy, but the API is racy by construction as it uses refcounts and doesn't attempt to lock/synchronize with the IOMMU API as that will be costly, meaning that possibility of false negative exists.

---
   drivers/iommu/iommu-debug-pagealloc.c | 28 +++++++++++++
   drivers/iommu/iommu-priv.h            | 58 +++++++++++++++++++++++++++
   drivers/iommu/iommu.c                 | 11 ++++-
   include/linux/iommu-debug-pagealloc.h |  1 +
   4 files changed, 96 insertions(+), 2 deletions(-)
drivers/iommu/iommu-debug-pagealloc.c +28 -0
index 4022e9af7f27..1d343421da98 100644
@@ -5,11 +5,15 @@
    * IOMMU API debug page alloc sanitizer
    */
   #include <linux/atomic.h>
+#include <linux/iommu.h>
   #include <linux/iommu-debug-pagealloc.h>
   #include <linux/kernel.h>
   #include <linux/page_ext.h>
+#include "iommu-priv.h"
+
   static bool needed;
+DEFINE_STATIC_KEY_FALSE(iommu_debug_initialized);
   struct iommu_debug_metadata {
   	atomic_t ref;
@@ -25,6 +29,30 @@ struct page_ext_operations page_iommu_debug_ops = {
   	.need = need_iommu_debug,
   };
+void __iommu_debug_map(struct iommu_domain *domain, phys_addr_t phys, size_t size)
+{
+}
+
+void __iommu_debug_unmap_begin(struct iommu_domain *domain,
+			       unsigned long iova, size_t size)
+{
+}
+
+void __iommu_debug_unmap_end(struct iommu_domain *domain,
+			     unsigned long iova, size_t size,
+			     size_t unmapped)
+{
+}
+
+void iommu_debug_init(void)
+{
+	if (!needed)
+		return;
+
+	pr_info("iommu: Debugging page allocations, expect overhead or disable iommu.debug_pagealloc");
+	static_branch_enable(&iommu_debug_initialized);
+}
+
   static int __init iommu_debug_pagealloc(char *str)
   {
   	return kstrtobool(str, &needed);
drivers/iommu/iommu-priv.h +22 -0
index c95394cd03a7..aaffad5854fc 100644
@@ -5,6 +5,7 @@
   #define __LINUX_IOMMU_PRIV_H
   #include <linux/iommu.h>
+#include <linux/iommu-debug-pagealloc.h>
   #include <linux/msi.h>
   static inline const struct iommu_ops *dev_iommu_ops(struct device *dev)
@@ -65,4 +66,61 @@ static inline int iommufd_sw_msi(struct iommu_domain *domain,
   int iommu_replace_device_pasid(struct iommu_domain *domain,
   			       struct device *dev, ioasid_t pasid,
   			       struct iommu_attach_handle *handle);
+
+#ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC
+
+void __iommu_debug_map(struct iommu_domain *domain, phys_addr_t phys,
+		       size_t size);
+void __iommu_debug_unmap_begin(struct iommu_domain *domain,
+			       unsigned long iova, size_t size);
+void __iommu_debug_unmap_end(struct iommu_domain *domain,
+			     unsigned long iova, size_t size, size_t unmapped);
+
+static inline void iommu_debug_map(struct iommu_domain *domain,
+				   phys_addr_t phys, size_t size)
+{
+	if (static_branch_unlikely(&iommu_debug_initialized))
+		__iommu_debug_map(domain, phys, size);
+}
+
+static inline void iommu_debug_unmap_begin(struct iommu_domain *domain,
+					   unsigned long iova, size_t size)
+{
+	if (static_branch_unlikely(&iommu_debug_initialized))

drivers/iommu/iommu-priv.h static inline int iommufd_sw_msi(struct iommu_domain *domain, · line 99

Quoted 10 lines
drivers/iommu/iommu-priv.h +10 -0
+		__iommu_debug_unmap_begin(domain, iova, size);
+}
+
+static inline void iommu_debug_unmap_end(struct iommu_domain *domain,
+					 unsigned long iova, size_t size,
+					 size_t unmapped)
+{
+	if (static_branch_unlikely(&iommu_debug_initialized))
+		__iommu_debug_unmap_end(domain, iova, size, unmapped);
+}

I am wondering whether it would be better if we move iommu_debug_map() to iommu-debug-pagealloc.c,

void iommu_debug_map(struct iommu_domain *domain,
phys_addr_t phys, size_t size)

{
	if (static_branch_likely(&iommu_debug_initialized))
		__iommu_debug_map(domain, phys, size);
}
Quoted 5 lines

(Does it make sense to use static_branch_likely() here? Normally, people
who enable CONFIG_IOMMU_DEBUG_PAGEALLOC would want to use this debugging feature. Or not?)

So that ...

Quoted 8 lines

This actually was the v1 implementation [1], but Jörg suggested to move it to a header file as a function call would have an overhead if this feautre is disabled.

I believe the priority would be to keep the performance overhead minimal with CONFIG_IOMMU_DEBUG_PAGEALLOC and the commandline disabled, so people can run with the config in production and only enable the commandline it to debug problems, without having overhead on the typical case.

Okay, fair enough.

Thanks,
baolu

#9 lore raw

PATCH v4 3/4 iommu: debug-pagealloc: Track IOMMU pages

In reply to Mostafa Saleh

Using the new calls, use an atomic refcount to track how many times a page is mapped in any of the IOMMUs.

For unmap we need to use iova_to_phys() to get the physical address of the pages.

We use the smallest supported page size as the granularity of tracking per domain.
This is important as it is possible to map pages and unmap them with larger sizes (as in map_sg()) cases.

---
 drivers/iommu/iommu-debug-pagealloc.c | 91 +++++++++++++++++++++++++++
 1 file changed, 91 insertions(+)
drivers/iommu/iommu-debug-pagealloc.c +91 -0
index 1d343421da98..4639cf9518e6 100644
@@ -29,19 +29,110 @@ struct page_ext_operations page_iommu_debug_ops = {
 	.need = need_iommu_debug,
 };
 
+static struct page_ext *get_iommu_page_ext(phys_addr_t phys)
+{
+	struct page *page = phys_to_page(phys);
+	struct page_ext *page_ext = page_ext_get(page);
+
+	return page_ext;
+}
+
+static struct iommu_debug_metadata *get_iommu_data(struct page_ext *page_ext)
+{
+	return page_ext_data(page_ext, &page_iommu_debug_ops);
+}
+
+static void iommu_debug_inc_page(phys_addr_t phys)
+{
+	struct page_ext *page_ext = get_iommu_page_ext(phys);
+	struct iommu_debug_metadata *d = get_iommu_data(page_ext);
+
+	WARN_ON(atomic_inc_return_relaxed(&d->ref) <= 0);
+	page_ext_put(page_ext);
+}
+
+static void iommu_debug_dec_page(phys_addr_t phys)
+{
+	struct page_ext *page_ext = get_iommu_page_ext(phys);
+	struct iommu_debug_metadata *d = get_iommu_data(page_ext);
+
+	WARN_ON(atomic_dec_return_relaxed(&d->ref) < 0);
+	page_ext_put(page_ext);
+}
+
+/*
+ * IOMMU page size doesn't have tomatch the CPU page size. So, we use
+ * the smallest IOMMU page size to refcount the pages in the vmemmap.
+ * That is important as both map and unmap has to use the same page size
+ * to update the refcount to avoid double counting the same page.
+ * And as we can't know from iommu_unmap() what was the original page size
+ * used for map, we just use the minimum supported one for both.
+ */
+static size_t iommu_debug_page_size(struct iommu_domain *domain)
+{
+	return 1UL << __ffs(domain->pgsize_bitmap);
+}
+
 void __iommu_debug_map(struct iommu_domain *domain, phys_addr_t phys, size_t size)
 {
+	size_t off, end;
+	size_t page_size = iommu_debug_page_size(domain);
+
+	if (WARN_ON(!phys || check_add_overflow(phys, size, &end)))
+		return;
+
+	for (off = 0 ; off < size ; off += page_size) {
+		if (!pfn_valid(__phys_to_pfn(phys + off)))
+			continue;
+		iommu_debug_inc_page(phys + off);
+	}
+}
+
+static void __iommu_debug_update_iova(struct iommu_domain *domain,
+				      unsigned long iova, size_t size, bool inc)
+{
+	size_t off, end;
+	size_t page_size = iommu_debug_page_size(domain);
+
+	if (WARN_ON(check_add_overflow(iova, size, &end)))
+		return;
+
+	for (off = 0 ; off < size ; off += page_size) {
+		phys_addr_t phys = iommu_iova_to_phys(domain, iova + off);
+
+		if (!phys || !pfn_valid(__phys_to_pfn(phys)))
+			continue;
+
+		if (inc)
+			iommu_debug_inc_page(phys);
+		else
+			iommu_debug_dec_page(phys);
+	}
 }
 
 void __iommu_debug_unmap_begin(struct iommu_domain *domain,
 			       unsigned long iova, size_t size)
 {
+	__iommu_debug_update_iova(domain, iova, size, false);
 }
 
 void __iommu_debug_unmap_end(struct iommu_domain *domain,
 			     unsigned long iova, size_t size,
 			     size_t unmapped)
 {
+	if (unmapped == size)
+		return;
+
+	/*
+	 * If unmap failed, re-increment the refcount, but if it unmapped
+	 * larger size, decrement the extra part.
+	 */
+	if (unmapped < size)
+		__iommu_debug_update_iova(domain, iova + unmapped,
+					  size - unmapped, true);
+	else
+		__iommu_debug_update_iova(domain, iova + size,
+					  unmapped - size, false);
 }
 
 void iommu_debug_init(void)
Signature · 1 line
2.52.0.223.gf5cc29aaa4-goog
Reviewed-by on v4 3/4
#10 lore raw

Re: [PATCH v4 3/4] iommu: debug-pagealloc: Track IOMMU pages

Quoted 47 lines · Mostafa Saleh

Using the new calls, use an atomic refcount to track how many times a page is mapped in any of the IOMMUs.

For unmap we need to use iova_to_phys() to get the physical address of the pages.

We use the smallest supported page size as the granularity of tracking per domain.
This is important as it is possible to map pages and unmap them with larger sizes (as in map_sg()) cases.

---
  drivers/iommu/iommu-debug-pagealloc.c | 91 +++++++++++++++++++++++++++
  1 file changed, 91 insertions(+)
drivers/iommu/iommu-debug-pagealloc.c +23 -0
index 1d343421da98..4639cf9518e6 100644
@@ -29,19 +29,110 @@ struct page_ext_operations page_iommu_debug_ops = {
  	.need = need_iommu_debug,
  };
 
+static struct page_ext *get_iommu_page_ext(phys_addr_t phys)
+{
+	struct page *page = phys_to_page(phys);
+	struct page_ext *page_ext = page_ext_get(page);
+
+	return page_ext;
+}
+
+static struct iommu_debug_metadata *get_iommu_data(struct page_ext *page_ext)
+{
+	return page_ext_data(page_ext, &page_iommu_debug_ops);
+}
+
+static void iommu_debug_inc_page(phys_addr_t phys)
+{
+	struct page_ext *page_ext = get_iommu_page_ext(phys);
+	struct iommu_debug_metadata *d = get_iommu_data(page_ext);
+
+	WARN_ON(atomic_inc_return_relaxed(&d->ref) <= 0);
+	page_ext_put(page_ext);
+}
+
+static void iommu_debug_dec_page(phys_addr_t phys)

drivers/iommu/iommu-debug-pagealloc.c struct page_ext_operations page_iommu_debug_ops = { · line 64

Quoted 10 lines
drivers/iommu/iommu-debug-pagealloc.c +10 -0
+{
+	struct page_ext *page_ext = get_iommu_page_ext(phys);
+	struct iommu_debug_metadata *d = get_iommu_data(page_ext);
+
+	WARN_ON(atomic_dec_return_relaxed(&d->ref) < 0);
+	page_ext_put(page_ext);
+}
+
+/*
+ * IOMMU page size doesn't have tomatch the CPU page size. So, we use

s/have tomatch/have to match/

drivers/iommu/iommu-debug-pagealloc.c struct page_ext_operations page_iommu_debug_ops = { · line 74

Quoted 10 lines
drivers/iommu/iommu-debug-pagealloc.c +10 -0
+ * the smallest IOMMU page size to refcount the pages in the vmemmap.
+ * That is important as both map and unmap has to use the same page size
+ * to update the refcount to avoid double counting the same page.
+ * And as we can't know from iommu_unmap() what was the original page size
+ * used for map, we just use the minimum supported one for both.
+ */
+static size_t iommu_debug_page_size(struct iommu_domain *domain)
+{
+	return 1UL << __ffs(domain->pgsize_bitmap);
+}

The changes look good to me,

#11 lore raw

PATCH v4 4/4 iommu: debug-pagealloc: Check mapped/unmapped kernel memory

In reply to Mostafa Saleh

Now, as the page_ext holds count of IOMMU mappings, we can use it to assert that any page allocated/freed is indeed not in the IOMMU.

The sanitizer doesn’t protect against mapping/unmapping during this period. However, that’s less harmful as the page is not used by the kernel.

---
 drivers/iommu/iommu-debug-pagealloc.c | 23 +++++++++++++++++++++++
 include/linux/iommu-debug-pagealloc.h | 14 ++++++++++++++
 include/linux/mm.h                    |  5 +++++
 3 files changed, 42 insertions(+)
drivers/iommu/iommu-debug-pagealloc.c +23 -0
index 4639cf9518e6..424e00cd103b 100644
@@ -9,6 +9,7 @@
 #include <linux/iommu-debug-pagealloc.h>
 #include <linux/kernel.h>
 #include <linux/page_ext.h>
+#include <linux/page_owner.h>
 
 #include "iommu-priv.h"
 
@@ -73,6 +74,28 @@ static size_t iommu_debug_page_size(struct iommu_domain *domain)
 	return 1UL << __ffs(domain->pgsize_bitmap);
 }
 
+static bool iommu_debug_page_count(const struct page *page)
+{
+	unsigned int ref;
+	struct page_ext *page_ext = page_ext_get(page);
+	struct iommu_debug_metadata *d = get_iommu_data(page_ext);
+
+	ref = atomic_read(&d->ref);
+	page_ext_put(page_ext);
+	return ref != 0;
+}
+
+void __iommu_debug_check_unmapped(const struct page *page, int numpages)
+{
+	while (numpages--) {
+		if (WARN_ON(iommu_debug_page_count(page))) {
+			pr_warn("iommu: Detected page leak!\n");
+			dump_page_owner(page);
+		}
+		page++;
+	}
+}
+
 void __iommu_debug_map(struct iommu_domain *domain, phys_addr_t phys, size_t size)
 {
 	size_t off, end;
include/linux/iommu-debug-pagealloc.h +14 -0
index a439d6815ca1..46c3c1f70150 100644
@@ -13,6 +13,20 @@ DECLARE_STATIC_KEY_FALSE(iommu_debug_initialized);
 
 extern struct page_ext_operations page_iommu_debug_ops;
 
+void __iommu_debug_check_unmapped(const struct page *page, int numpages);
+
+static inline void iommu_debug_check_unmapped(const struct page *page, int numpages)
+{
+	if (static_branch_unlikely(&iommu_debug_initialized))
+		__iommu_debug_check_unmapped(page, numpages);
+}
+
+#else
+static inline void iommu_debug_check_unmapped(const struct page *page,
+					      int numpages)
+{
+}
+
 #endif /* CONFIG_IOMMU_DEBUG_PAGEALLOC */
 
 #endif /* __LINUX_IOMMU_DEBUG_PAGEALLOC_H */
include/linux/mm.h +5 -0
index 7a1819c20643..3763b71a7d3e 100644
@@ -36,6 +36,7 @@
 #include <linux/rcuwait.h>
 #include <linux/bitmap.h>
 #include <linux/bitops.h>
+#include <linux/iommu-debug-pagealloc.h>
 
 struct mempolicy;
 struct anon_vma;
@@ -4133,12 +4134,16 @@ extern void __kernel_map_pages(struct page *page, int numpages, int enable);
 #ifdef CONFIG_DEBUG_PAGEALLOC
 static inline void debug_pagealloc_map_pages(struct page *page, int numpages)
 {
+	iommu_debug_check_unmapped(page, numpages);
+
 	if (debug_pagealloc_enabled_static())
 		__kernel_map_pages(page, numpages, 1);
 }
 
 static inline void debug_pagealloc_unmap_pages(struct page *page, int numpages)
 {
+	iommu_debug_check_unmapped(page, numpages);
+
 	if (debug_pagealloc_enabled_static())
 		__kernel_map_pages(page, numpages, 0);
 }
Signature · 1 line
2.52.0.223.gf5cc29aaa4-goog
Reviewed-by on v4 4/4
#12 lore raw

Re: [PATCH v4 4/4] iommu: debug-pagealloc: Check mapped/unmapped kernel memory

Quoted 13 lines · Mostafa Saleh

Now, as the page_ext holds count of IOMMU mappings, we can use it to assert that any page allocated/freed is indeed not in the IOMMU.

The sanitizer doesn’t protect against mapping/unmapping during this period. However, that’s less harmful as the page is not used by the kernel.

---
  drivers/iommu/iommu-debug-pagealloc.c | 23 +++++++++++++++++++++++
  include/linux/iommu-debug-pagealloc.h | 14 ++++++++++++++
  include/linux/mm.h                    |  5 +++++
  3 files changed, 42 insertions(+)

[...]

Quoted 5 lines
include/linux/iommu-debug-pagealloc.h +0 -0
index a439d6815ca1..46c3c1f70150 100644

include/linux/iommu-debug-pagealloc.h DECLARE_STATIC_KEY_FALSE(iommu_debug_initialized); · line 22

Quoted 10 lines
include/linux/iommu-debug-pagealloc.h +7 -0
@@ -13,6 +13,20 @@ DECLARE_STATIC_KEY_FALSE(iommu_debug_initialized);
 
  extern struct page_ext_operations page_iommu_debug_ops;
 
+void __iommu_debug_check_unmapped(const struct page *page, int numpages);
+
+static inline void iommu_debug_check_unmapped(const struct page *page, int numpages)
+{
+	if (static_branch_unlikely(&iommu_debug_initialized))
+		__iommu_debug_check_unmapped(page, numpages);
+}

Same here, possible to make iommu_debug_initialized static?

Quoted 28 lines
include/linux/iommu-debug-pagealloc.h +6 -0
+#else
+static inline void iommu_debug_check_unmapped(const struct page *page,
+					      int numpages)
+{
+}
+
  #endif /* CONFIG_IOMMU_DEBUG_PAGEALLOC */
 
  #endif /* __LINUX_IOMMU_DEBUG_PAGEALLOC_H */
include/linux/mm.h +3 -0
index 7a1819c20643..3763b71a7d3e 100644
@@ -36,6 +36,7 @@
  #include <linux/rcuwait.h>
  #include <linux/bitmap.h>
  #include <linux/bitops.h>
+#include <linux/iommu-debug-pagealloc.h>
 
  struct mempolicy;
  struct anon_vma;
@@ -4133,12 +4134,16 @@ extern void __kernel_map_pages(struct page *page, int numpages, int enable);
  #ifdef CONFIG_DEBUG_PAGEALLOC
  static inline void debug_pagealloc_map_pages(struct page *page, int numpages)
  {
+	iommu_debug_check_unmapped(page, numpages);
+
  	if (debug_pagealloc_enabled_static())

include/linux/mm.h extern void __kernel_map_pages(struct page *page, int numpages, int enable); · line 4149

Quoted 10 lines
include/linux/mm.h +2 -0
  		__kernel_map_pages(page, numpages, 1);
  }
 
  static inline void debug_pagealloc_unmap_pages(struct page *page, int numpages)
  {
+	iommu_debug_check_unmapped(page, numpages);
+
  	if (debug_pagealloc_enabled_static())
  		__kernel_map_pages(page, numpages, 0);
  }

Other changes look good to me,

#13 lore raw

Re: [PATCH v4 4/4] iommu: debug-pagealloc: Check mapped/unmapped kernel memory

Quoted 17 lines · Baolu Lu
Quoted 13 lines · Mostafa Saleh

Now, as the page_ext holds count of IOMMU mappings, we can use it to assert that any page allocated/freed is indeed not in the IOMMU.

The sanitizer doesn’t protect against mapping/unmapping during this period. However, that’s less harmful as the page is not used by the kernel.

---
  drivers/iommu/iommu-debug-pagealloc.c | 23 +++++++++++++++++++++++
  include/linux/iommu-debug-pagealloc.h | 14 ++++++++++++++
  include/linux/mm.h                    |  5 +++++
  3 files changed, 42 insertions(+)

[...]

Quoted 15 lines

include/linux/iommu-debug-pagealloc.h DECLARE_STATIC_KEY_FALSE(iommu_debug_initialized); · line 20

Quoted 13 lines
include/linux/iommu-debug-pagealloc.h +7 -0
index a439d6815ca1..46c3c1f70150 100644
@@ -13,6 +13,20 @@ DECLARE_STATIC_KEY_FALSE(iommu_debug_initialized);
  extern struct page_ext_operations page_iommu_debug_ops;
+void __iommu_debug_check_unmapped(const struct page *page, int numpages);
+
+static inline void iommu_debug_check_unmapped(const struct page *page, int numpages)
+{
+	if (static_branch_unlikely(&iommu_debug_initialized))
+		__iommu_debug_check_unmapped(page, numpages);
+}

Same here, possible to make iommu_debug_initialized static?

This also is similar to the iommu ones to reduce overhead, the main feature have the same pattern with "_debug_pagealloc_enabled" static key.

Thanks,
Mostafa

Quoted 40 lines
Quoted 26 lines
include/linux/iommu-debug-pagealloc.h +6 -0
+#else
+static inline void iommu_debug_check_unmapped(const struct page *page,
+					      int numpages)
+{
+}
+
  #endif /* CONFIG_IOMMU_DEBUG_PAGEALLOC */
  #endif /* __LINUX_IOMMU_DEBUG_PAGEALLOC_H */
include/linux/mm.h +3 -0
index 7a1819c20643..3763b71a7d3e 100644
@@ -36,6 +36,7 @@
  #include <linux/rcuwait.h>
  #include <linux/bitmap.h>
  #include <linux/bitops.h>
+#include <linux/iommu-debug-pagealloc.h>
  struct mempolicy;
  struct anon_vma;
@@ -4133,12 +4134,16 @@ extern void __kernel_map_pages(struct page *page, int numpages, int enable);
  #ifdef CONFIG_DEBUG_PAGEALLOC
  static inline void debug_pagealloc_map_pages(struct page *page, int numpages)
  {
+	iommu_debug_check_unmapped(page, numpages);
+

include/linux/mm.h extern void __kernel_map_pages(struct page *page, int numpages, int enable); · line 4148

Quoted 10 lines
include/linux/mm.h +2 -0
  	if (debug_pagealloc_enabled_static())
  		__kernel_map_pages(page, numpages, 1);
  }
  static inline void debug_pagealloc_unmap_pages(struct page *page, int numpages)
  {
+	iommu_debug_check_unmapped(page, numpages);
+
  	if (debug_pagealloc_enabled_static())
  		__kernel_map_pages(page, numpages, 0);
  }

Other changes look good to me,