iommu: Add IOMMU_DEBUG_PAGEALLOC sanitizer
Subsystems (5)
v3 cover iommu: Add IOMMU_DEBUG_PAGEALLOC sanitizer
Overview
--------
This patch series introduces a new debugging feature, IOMMU_DEBUG_PAGEALLOC, designed to catch DMA use-after-free bugs and IOMMU mapping leaks from buggy drivers.
The kernel has powerful sanitizers like KASAN and DEBUG_PAGEALLOC for catching CPU-side memory corruption. However, there is limited runtime sanitization for DMA mappings managed by the IOMMU. A buggy driver can free a page while it is still mapped for DMA, leading to memory corruption or use-after-free vulnerabilities when that page is reallocated and used for a different purpose.
Inspired by DEBUG_PAGEALLOC, this sanitizer tracks IOMMU mappings on a per-page basis, as it’s not possible to unmap the pages, because it requires to lock and walk all domains on every kernel free, instead we rely on page_ext to add an IOMMU-specific mapping reference count for each page.
And on each page allocated/freed from the kernel we simply check the count and WARN if it is not zero, and dumping page owner information if enabled.
Concurrency
-----------
By design this check is racy where one caller can map pages just after the check, which can lead to false negatives.
In my opinion this is acceptable for sanitizers (for ex KCSAN have that property).
Otherwise we have to implement locks in iommu_map/unmap for all domains which is not favourable even for a debug feature.
The sanitizer only guarantees that the refcount itself doesn’t get corrupted using atomics. And there are no false positives.
CPU vs IOMMU Page Size
----------------------
IOMMUs can use different page sizes and which can be non-homogeneous; not even all of them have the same page size.
To solve this, the refcount is always incremented and decremented in units of the smallest page size supported by the IOMMU domain. This ensures the accounting remains consistent regardless of the size of the map or unmap operation, otherwise double counting can happen.
Testing & Performance
---------------------
This was tested on Morello with Arm64 + SMMUv3
Did some tests on Qemu including different SMMUv3/CPU page size (arm64).
I also ran dma_map_benchmark on Morello:
echo dma_map_benchmark > /sys/bus/pci/devices/0000\:06\:00.0/driver_override echo 0000:06:00.0 > /sys/bus/pci/devices/0000\:06\:00.0/driver/unbind echo 0000:06:00.0 > /sys/bus/pci/drivers/dma_map_benchmark/bind ./dma_map_benchmark -t $threads -g $nr_pages
CONFIG refers to "CONFIG_IOMMU_DEBUG_PAGEALLOC" cmdline refers to "iommu.debug_pagealloc"
Numbers are (map latency)/(unmap latency), lower is better.
CONFIG=n CONFIG=y CONFIG=y cmdline=0 cmdline=1 4K - 1 thread 0.1/0.6 0.1/0.6 0.1/0.7
4K - 4 threads 0.1/1.1 0.1/1.0 0.2/1.1
1M - 1 thread 0.8/21.2 0.7/21.2 5.4/42.3
1M - 4 threads 1.1/45.9 1.1/46.0 5.9/45.1
Main changes in v3: (Most of them addressing Will comments)
v2: https://lore.kernel.org/linux-iommu/[email protected]/
- Reword the Kconfig help
- Use unmap_begin/end instead of unmap/remap
- Use relaxed accessors when refcounting
- Fix a bug with checking the returned address from iova_to_phys
- Add more hardening checks (overflow)
- Add more debug info on assertions (dump_page_owner())
- Handle cases where unmap returns larger size as the core code seems to tolerate that.
- Drop Tested-by tags from Qinxin as the code logic changed
Main changes in v2:
v1: https://lore.kernel.org/linux-iommu/[email protected]/
- Address Jörg comments about #ifdefs and static keys
- Reword the Kconfig help
- Drop RFC
- Collect t-b from Qinxin
- Minor cleanups
Mostafa Saleh (4):
drivers/iommu: Add page_ext for IOMMU_DEBUG_PAGEALLOC drivers/iommu: Add calls for IOMMU_DEBUG_PAGEALLOC drivers/iommu-debug-pagealloc: Track IOMMU pages drivers/iommu-debug-pagealloc: Check mapped/unmapped kernel memory
.../admin-guide/kernel-parameters.txt | 6 + drivers/iommu/Kconfig | 19 ++ drivers/iommu/Makefile | 1 + drivers/iommu/iommu-debug-pagealloc.c | 172 ++++++++++++++++++ drivers/iommu/iommu.c | 12 +- include/linux/iommu-debug-pagealloc.h | 85 +++++++++ include/linux/mm.h | 5 + mm/page_ext.c | 4 + 8 files changed, 302 insertions(+), 2 deletions(-) create mode 100644 drivers/iommu/iommu-debug-pagealloc.c create mode 100644 include/linux/iommu-debug-pagealloc.h
base-commit: ac3fd01e4c1efce8f2c054cdeb2ddd2fc0fb150d
Signature · 1 line
2.52.0.460.gd25c4c69ec-goog
PATCH v3 1/4 drivers/iommu: Add page_ext for IOMMU_DEBUG_PAGEALLOC
Add a new config IOMMU_DEBUG_PAGEALLOC, which registers new data to page_ext.
This config will be used by the IOMMU API to track pages mapped in the IOMMU to catch drivers trying to free kernel memory that they still map in their domains, causing all types of memory corruption.
This behaviour is disabled by default and can be enabled using kernel cmdline iommu.debug_pagealloc.
--- .../admin-guide/kernel-parameters.txt | 6 ++++ drivers/iommu/Kconfig | 19 +++++++++++ drivers/iommu/Makefile | 1 + drivers/iommu/iommu-debug-pagealloc.c | 32 +++++++++++++++++++ include/linux/iommu-debug-pagealloc.h | 17 ++++++++++ mm/page_ext.c | 4 +++ 6 files changed, 79 insertions(+) create mode 100644 drivers/iommu/iommu-debug-pagealloc.c create mode 100644 include/linux/iommu-debug-pagealloc.h
Documentation/admin-guide/kernel-parameters.txt +6 -0
@@ -2557,6 +2557,12 @@ 1 - Bypass the IOMMU for DMA. unset - Use value of CONFIG_IOMMU_DEFAULT_PASSTHROUGH. + iommu.debug_pagealloc= + [KNL,EARLY] When CONFIG_IOMMU_DEBUG_PAGEALLOC is set, this + parameter enables the feature at boot time. By default, it + is disabled and the system behave the same way as a kernel + built without CONFIG_IOMMU_DEBUG_PAGEALLOC. + io7= [HW] IO7 for Marvel-based Alpha systems See comment before marvel_specify_io7 in arch/alpha/kernel/core_marvel.c.
drivers/iommu/Kconfig +19 -0
@@ -383,4 +383,23 @@ config SPRD_IOMMU Say Y here if you want to use the multimedia devices listed above. +config IOMMU_DEBUG_PAGEALLOC + bool "Debug IOMMU mappings against page allocations" + depends on DEBUG_PAGEALLOC && IOMMU_API && PAGE_EXTENSION + help + This enables a consistency check between the kernel page allocator and + the IOMMU subsystem. It verifies that pages being allocated or freed + are not currently mapped in any IOMMU domain. + + This helps detect DMA use-after-free bugs where a driver frees a page + but forgets to unmap it from the IOMMU, potentially allowing a device + to overwrite memory that the kernel has repurposed. + + These checks are best-effort and may not detect all problems. + + Due to performance overhead, this feature is disabled by default. + You must enable "iommu.debug_pagealloc" from the kernel command + line to activate the runtime checks. + + If unsure, say N. endif # IOMMU_SUPPORT
drivers/iommu/Makefile +1 -0
@@ -34,3 +34,4 @@ obj-$(CONFIG_IOMMU_SVA) += iommu-sva.o obj-$(CONFIG_IOMMU_IOPF) += io-pgfault.o obj-$(CONFIG_SPRD_IOMMU) += sprd-iommu.o obj-$(CONFIG_APPLE_DART) += apple-dart.o +obj-$(CONFIG_IOMMU_DEBUG_PAGEALLOC) += iommu-debug-pagealloc.o
drivers/iommu/iommu-debug-pagealloc.c new file +32 -0
@@ -0,0 +1,32 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Copyright (C) 2025 - Google Inc +* Author: Mostafa Saleh <[email protected]> +* IOMMU API debug page alloc sanitizer +*/ +#include <linux/atomic.h> +#include <linux/iommu-debug-pagealloc.h> +#include <linux/kernel.h> +#include <linux/page_ext.h> + +static bool needed; + +struct iommu_debug_metadata { +atomic_t ref; +}; + +static __init bool need_iommu_debug (void ) +{ +return needed; +} + +struct page_ext_operations page_iommu_debug_ops = { + .size =sizeof (struct iommu_debug_metadata ), + .need = need_iommu_debug, +}; + +static int __init iommu_debug_pagealloc (char *str) +{ +return kstrtobool (str, &needed); +} +early_param ("iommu.debug_pagealloc" , iommu_debug_pagealloc);
include/linux/iommu-debug-pagealloc.h new file +17 -0
@@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Copyright (C) 2025 - Google Inc +* Author: Mostafa Saleh <[email protected]> +* IOMMU API debug page alloc sanitizer +*/ + +#ifndef __LINUX_IOMMU_DEBUG_PAGEALLOC_H +#define __LINUX_IOMMU_DEBUG_PAGEALLOC_H + +#ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC + +extern struct page_ext_operations page_iommu_debug_ops; + +#endif /* CONFIG_IOMMU_DEBUG_PAGEALLOC */ + +#endif /* __LINUX_IOMMU_DEBUG_PAGEALLOC_H */
mm/page_ext.c +4 -0
@@ -11,6 +11,7 @@#include <linux/page_table_check.h> #include <linux/rcupdate.h> #include <linux/pgalloc_tag.h> +#include <linux/iommu-debug-pagealloc.h> /* * struct page extension
@@ -89,6 +90,9 @@ static struct page_ext_operations *page_ext_ops[] __initdata = {#ifdef CONFIG_PAGE_TABLE_CHECK &page_table_check_ops,#endif +#ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC + &page_iommu_debug_ops, +#endif };unsigned long page_ext_size;
Signature · 1 line
2.52.0.460.gd25c4c69ec-goog
Randy Dunlap
reviewerRe: [PATCH v3 1/4] drivers/iommu: Add page_ext for IOMMU_DEBUG_PAGEALLOC
Quoted 27 lines · Mostafa Saleh
Add a new config
IOMMU_DEBUG_PAGEALLOC, which registers new data topage_ext.This config will be used by the IOMMU API to track pages mapped in the IOMMU to catch drivers trying to free kernel memory that they still map in their domains, causing all types of memory corruption.
This behaviour is disabled by default and can be enabled using kernel cmdline iommu.debug_pagealloc.
Signed-off-by Mostafa Saleh <[email protected]>--- .../admin-guide/kernel-parameters.txt | 6 ++++ drivers/iommu/Kconfig | 19 +++++++++++ drivers/iommu/Makefile | 1 + drivers/iommu/iommu-debug-pagealloc.c | 32 +++++++++++++++++++ include/linux/iommu-debug-pagealloc.h | 17 ++++++++++ mm/page_ext.c | 4 +++ 6 files changed, 79 insertions(+) create mode 100644 drivers/iommu/iommu-debug-pagealloc.c create mode 100644 include/linux/iommu-debug-pagealloc.hDocumentation/admin-guide/kernel-parameters.txt +0
-0
Documentation/admin-guide/kernel-parameters.txt line 2563
Quoted 7 lines
Documentation/admin-guide/kernel-parameters.txt +4
-0@@ -2557,6 +2557,12 @@ 1 - Bypass the IOMMU for DMA. unset - Use value of CONFIG_IOMMU_DEFAULT_PASSTHROUGH. + iommu.debug_pagealloc= + [KNL,EARLY] When CONFIG_IOMMU_DEBUG_PAGEALLOC is set, this + parameter enables the feature at boot time. By default, it + is disabled and the system behave the same way as a kernel
behaves
Quoted 5 lines
Documentation/admin-guide/kernel-parameters.txt +2
-0+ built without CONFIG_IOMMU_DEBUG_PAGEALLOC. + io7= [HW] IO7 for Marvel-based Alpha systems See comment before marvel_specify_io7 in arch/alpha/kernel/core_marvel.c.
Signature · 1 line
~Randy
Randy Dunlap
reviewerRe: [PATCH v3 1/4] drivers/iommu: Add page_ext for IOMMU_DEBUG_PAGEALLOC
I missed one other thing:
Quoted 33 lines · Randy Dunlap
Quoted 30 lines · Mostafa Saleh
Add a new config
IOMMU_DEBUG_PAGEALLOC, which registers new data topage_ext.This config will be used by the IOMMU API to track pages mapped in the IOMMU to catch drivers trying to free kernel memory that they still map in their domains, causing all types of memory corruption.
This behaviour is disabled by default and can be enabled using kernel cmdline iommu.debug_pagealloc.
Signed-off-by Mostafa Saleh <[email protected]>--- .../admin-guide/kernel-parameters.txt | 6 ++++ drivers/iommu/Kconfig | 19 +++++++++++ drivers/iommu/Makefile | 1 + drivers/iommu/iommu-debug-pagealloc.c | 32 +++++++++++++++++++ include/linux/iommu-debug-pagealloc.h | 17 ++++++++++ mm/page_ext.c | 4 +++ 6 files changed, 79 insertions(+) create mode 100644 drivers/iommu/iommu-debug-pagealloc.c create mode 100644 include/linux/iommu-debug-pagealloc.hDocumentation/admin-guide/kernel-parameters.txt +0
-0@@ -2557,6 +2557,12 @@ 1 - Bypass the IOMMU for DMA. unset - Use value of CONFIG_IOMMU_DEFAULT_PASSTHROUGH.
Quoted 1 line
Documentation/admin-guide/kernel-parameters.txt line 2560
Quoted 1 line
Documentation/admin-guide/kernel-parameters.txt +1
-0+ iommu.debug_pagealloc=
Please note what format the option parameter value takes and possible values, like iommu.passthrough above here in the kernel-parameters.txt file.
Quoted 13 lines
Documentation/admin-guide/kernel-parameters.txt line 2563
Quoted 3 lines
Documentation/admin-guide/kernel-parameters.txt +3
-0+ [KNL,EARLY] When CONFIG_IOMMU_DEBUG_PAGEALLOC is set, this + parameter enables the feature at boot time. By default, it + is disabled and the system behave the same way as a kernelbehaves
Quoted 5 lines
Documentation/admin-guide/kernel-parameters.txt +2
-0+ built without CONFIG_IOMMU_DEBUG_PAGEALLOC. + io7= [HW] IO7 for Marvel-based Alpha systems See comment before marvel_specify_io7 in arch/alpha/kernel/core_marvel.c.
Signature · 1 line
~Randy
Re: [PATCH v3 1/4] drivers/iommu: Add page_ext for IOMMU_DEBUG_PAGEALLOC
Quoted 36 lines · Randy Dunlap
I missed one other thing:
Quoted 33 lines · Randy Dunlap
Quoted 30 lines · Mostafa Saleh
Add a new config
IOMMU_DEBUG_PAGEALLOC, which registers new data topage_ext.This config will be used by the IOMMU API to track pages mapped in the IOMMU to catch drivers trying to free kernel memory that they still map in their domains, causing all types of memory corruption.
This behaviour is disabled by default and can be enabled using kernel cmdline iommu.debug_pagealloc.
Signed-off-by Mostafa Saleh <[email protected]>--- .../admin-guide/kernel-parameters.txt | 6 ++++ drivers/iommu/Kconfig | 19 +++++++++++ drivers/iommu/Makefile | 1 + drivers/iommu/iommu-debug-pagealloc.c | 32 +++++++++++++++++++ include/linux/iommu-debug-pagealloc.h | 17 ++++++++++ mm/page_ext.c | 4 +++ 6 files changed, 79 insertions(+) create mode 100644 drivers/iommu/iommu-debug-pagealloc.c create mode 100644 include/linux/iommu-debug-pagealloc.hDocumentation/admin-guide/kernel-parameters.txt +0
-0@@ -2557,6 +2557,12 @@ 1 - Bypass the IOMMU for DMA. unset - Use value of CONFIG_IOMMU_DEFAULT_PASSTHROUGH.
Quoted 4 lines
Quoted 1 line
Documentation/admin-guide/kernel-parameters.txt line 2560
Quoted 1 line
Documentation/admin-guide/kernel-parameters.txt +1
-0+ iommu.debug_pagealloc=Please note what format the option parameter value takes and possible values, like iommu.passthrough above here in the kernel-parameters.txt file.
I see, I was following the kernel's “debug_pagealloc”. Although, the implementation understands “={0,1}”, I can add something as:
Format: { "0" | "1" }
0 - Sanitizer disabled.
1 - Sanitizer enabled, expect run-time overhead.
To make it more clear.
Thanks,
Mostafa
Quoted 18 lines
Quoted 14 lines
Documentation/admin-guide/kernel-parameters.txt line 2563
Quoted 4 lines
Documentation/admin-guide/kernel-parameters.txt +3
-0+ [KNL,EARLY] When CONFIG_IOMMU_DEBUG_PAGEALLOC is set, this + parameter enables the feature at boot time. By default, it + is disabled and the system behave the same way as a kernelbehaves
Quoted 5 lines
Documentation/admin-guide/kernel-parameters.txt +2
-0+ built without CONFIG_IOMMU_DEBUG_PAGEALLOC. + io7= [HW] IO7 for Marvel-based Alpha systems See comment before marvel_specify_io7 in arch/alpha/kernel/core_marvel.c.Signature · 1 line
~Randy
Re: [PATCH v3 1/4] drivers/iommu: Add page_ext for IOMMU_DEBUG_PAGEALLOC
In reply to Mostafa SalehQuoted 22 lines · Mostafa Saleh
Add a new config
IOMMU_DEBUG_PAGEALLOC, which registers new data topage_ext.This config will be used by the IOMMU API to track pages mapped in the IOMMU to catch drivers trying to free kernel memory that they still map in their domains, causing all types of memory corruption.
This behaviour is disabled by default and can be enabled using kernel cmdline iommu.debug_pagealloc.
Signed-off-by Mostafa Saleh <[email protected]>--- .../admin-guide/kernel-parameters.txt | 6 ++++ drivers/iommu/Kconfig | 19 +++++++++++ drivers/iommu/Makefile | 1 + drivers/iommu/iommu-debug-pagealloc.c | 32 +++++++++++++++++++ include/linux/iommu-debug-pagealloc.h | 17 ++++++++++ mm/page_ext.c | 4 +++ 6 files changed, 79 insertions(+)create mode 100644
drivers/iommu/iommu-debug-pagealloc.ccreate mode 100644include/linux/iommu-debug-pagealloc.h
[..]
Quoted 9 lines
include/linux/iommu-debug-pagealloc.h new file +3
-0@@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Copyright (C) 2025 - Google Inc
include/linux/iommu-debug-pagealloc.h line 13
Quoted 10 lines
include/linux/iommu-debug-pagealloc.h new file +10
-0+* Author: Mostafa Saleh <[email protected]> +* IOMMU API debug page alloc sanitizer +*/ + +#ifndef __LINUX_IOMMU_DEBUG_PAGEALLOC_H +#define __LINUX_IOMMU_DEBUG_PAGEALLOC_H + +#ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC + +extern struct page_ext_operations page_iommu_debug_ops;
How about moving above to below mm/page_ext.c and remove iommu-debug- pagealloc.h header file? ...
mm/page_ext.c line 14
Quoted 13 lines
include/linux/iommu-debug-pagealloc.h new file +4
-0+ +#endif /* CONFIG_IOMMU_DEBUG_PAGEALLOC */ + +#endif /* __LINUX_IOMMU_DEBUG_PAGEALLOC_H */ mm/page_ext.c +1
-0@@ -11,6 +11,7 @@#include <linux/page_table_check.h> #include <linux/rcupdate.h> #include <linux/pgalloc_tag.h> +#include <linux/iommu-debug-pagealloc.h>
... remove this include line and put the "extern" line here,
extern struct page_ext_operations page_iommu_debug_ops;
mm/page_ext.c static struct page_ext_operations *page_ext_ops[] __initdata = { · line 98
Quoted 13 lines
mm/page_ext.c +3
-0/* * struct page extension @@ -89,6 +90,9 @@ static struct page_ext_operations *page_ext_ops[] __initdata = {#ifdef CONFIG_PAGE_TABLE_CHECK &page_table_check_ops,#endif +#ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC + &page_iommu_debug_ops, +#endif };unsigned long page_ext_size;
Or, put it directly in linux/iommu.h?
include/linux/iommu-debug-pagealloc.h +0 -2
@@ -14,8 +14,6 @@ struct iommu_domain;DECLARE_STATIC_KEY_FALSE (iommu_debug_initialized); -extern struct page_ext_operations page_iommu_debug_ops; -void __iommu_debug_map (struct iommu_domain *domain,phys_addr_t phys,size_t size); void __iommu_debug_unmap_begin(struct iommu_domain *domain,
include/linux/iommu.h +4 -0
@@ -1185,6 +1185,10 @@ void iommu_detach_device_pasid(struct iommu_domain *domain, struct device *dev, ioasid_t pasid);ioasid_t iommu_alloc_global_pasid (struct device *dev);void iommu_free_global_pasid (ioasid_t pasid); + +#ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC +extern struct page_ext_operations page_iommu_debug_ops; +#endif /* CONFIG_IOMMU_DEBUG_PAGEALLOC */ #else /* CONFIG_IOMMU_API */ struct iommu_ops {};
mm/page_ext.c +1 -1
@@ -11,7 +11,7 @@#include <linux/page_table_check.h> #include <linux/rcupdate.h> #include <linux/pgalloc_tag.h> -#include <linux/iommu-debug-pagealloc.h> +#include <linux/iommu.h> /* * struct page extension
Thanks,
baolu
Re: [PATCH v3 1/4] drivers/iommu: Add page_ext for IOMMU_DEBUG_PAGEALLOC
Quoted 65 lines · Baolu Lu
Quoted 22 lines · Mostafa Saleh
Add a new config
IOMMU_DEBUG_PAGEALLOC, which registers new data topage_ext.This config will be used by the IOMMU API to track pages mapped in the IOMMU to catch drivers trying to free kernel memory that they still map in their domains, causing all types of memory corruption.
This behaviour is disabled by default and can be enabled using kernel cmdline iommu.debug_pagealloc.
Signed-off-by Mostafa Saleh <[email protected]>--- .../admin-guide/kernel-parameters.txt | 6 ++++ drivers/iommu/Kconfig | 19 +++++++++++ drivers/iommu/Makefile | 1 + drivers/iommu/iommu-debug-pagealloc.c | 32 +++++++++++++++++++ include/linux/iommu-debug-pagealloc.h | 17 ++++++++++ mm/page_ext.c | 4 +++ 6 files changed, 79 insertions(+)create mode 100644
drivers/iommu/iommu-debug-pagealloc.ccreate mode 100644include/linux/iommu-debug-pagealloc.h[..]
Quoted 9 lines
include/linux/iommu-debug-pagealloc.h new file +3
-0@@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Copyright (C) 2025 - Google Inc include/linux/iommu-debug-pagealloc.h line 13
Quoted 10 lines
include/linux/iommu-debug-pagealloc.h new file +10
-0+* Author: Mostafa Saleh <[email protected]> +* IOMMU API debug page alloc sanitizer +*/ + +#ifndef __LINUX_IOMMU_DEBUG_PAGEALLOC_H +#define __LINUX_IOMMU_DEBUG_PAGEALLOC_H + +#ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC + +extern struct page_ext_operations page_iommu_debug_ops;How about moving above to below
mm/page_ext.cand remove iommu-debug-pagealloc.hheader file? ...mm/page_ext.c line 14
Quoted 13 lines
include/linux/iommu-debug-pagealloc.h new file +4
-0+ +#endif /* CONFIG_IOMMU_DEBUG_PAGEALLOC */ + +#endif /* __LINUX_IOMMU_DEBUG_PAGEALLOC_H */ mm/page_ext.c +1
-0@@ -11,6 +11,7 @@#include <linux/page_table_check.h> #include <linux/rcupdate.h> #include <linux/pgalloc_tag.h> +#include <linux/iommu-debug-pagealloc.h> ... remove this include line and put the "extern" line here,
Quoted 1 line
extern struct
page_ext_operationspage_iommu_debug_ops;
This file is needed for other functions added later, which is then included by iommu.c mm.h
Also, the pattern in page_ext looks that users have their own headers:
include/linux/page_owner.h:extern struct page_ext_operations page_owner_ops; include/linux/page_table_check.h:extern struct page_ext_operations page_table_check_ops; include/linux/pgalloc_tag.h:extern struct page_ext_operations page_alloc_tagging_ops;
Quoted 15 lines
mm/page_ext.c static struct page_ext_operations *page_ext_ops[] __initdata = { · line 97
Quoted 12 lines
mm/page_ext.c +3
-0/* * struct page extension @@ -89,6 +90,9 @@ static struct page_ext_operations *page_ext_ops[] __initdata = {#ifdef CONFIG_PAGE_TABLE_CHECK &page_table_check_ops,#endif +#ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC + &page_iommu_debug_ops, +#endif };unsigned long page_ext_size;Or, put it directly in
linux/iommu.h?
I tried that, but in the last patch we need to include that in linux/mm.h which didn't compile and required including other files which seemed unnecessary and that it would be better to isolate this feature.
Thanks,
Mostafa
Quoted 46 lines
include/linux/iommu-debug-pagealloc.h +0
-2@@ -14,8 +14,6 @@ struct iommu_domain;DECLARE_STATIC_KEY_FALSE (iommu_debug_initialized); -extern struct page_ext_operations page_iommu_debug_ops; -void __iommu_debug_map (struct iommu_domain *domain,phys_addr_t phys,size_t size); void __iommu_debug_unmap_begin(struct iommu_domain *domain,include/linux/iommu.h +4
-0@@ -1185,6 +1185,10 @@ void iommu_detach_device_pasid(struct iommu_domain *domain, struct device *dev, ioasid_t pasid);ioasid_t iommu_alloc_global_pasid (struct device *dev);void iommu_free_global_pasid (ioasid_t pasid); + +#ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC +extern struct page_ext_operations page_iommu_debug_ops; +#endif /* CONFIG_IOMMU_DEBUG_PAGEALLOC */ #else /* CONFIG_IOMMU_API */ struct
iommu_ops{};mm/page_ext.c +1
-1@@ -11,7 +11,7 @@#include <linux/page_table_check.h> #include <linux/rcupdate.h> #include <linux/pgalloc_tag.h> -#include <linux/iommu-debug-pagealloc.h> +#include <linux/iommu.h> /* * struct page extension Thanks,
baolu
PATCH v3 2/4 drivers/iommu: Add calls for IOMMU_DEBUG_PAGEALLOC
In reply to Mostafa SalehAdd calls for the new iommu debug config IOMMU_DEBUG_PAGEALLOC:
- iommu_debug_init: Enable the debug mode if configured by the user.
- iommu_debug_map: Track iommu pages mapped, using physical address.
- iommu_debug_unmap_begin: Track start of iommu unmap operation, with IOVA and size.
- iommu_debug_unmap_end: Track the end of unmap operation, passing the actual unmapped size versus the tracked one at unmap_begin.
We have to do the unmap_begin/end as once pages are unmapped we lose the information of the physical address.
This is racy, but the API is racy by construction as it uses refcounts and doesn't attempt to lock/synchronize with the IOMMU API as that will be costly, meaning that possibility of false negative exists.
--- drivers/iommu/iommu-debug-pagealloc.c | 26 +++++++++++++ drivers/iommu/iommu.c | 12 +++++- include/linux/iommu-debug-pagealloc.h | 56 +++++++++++++++++++++++++++ 3 files changed, 92 insertions(+), 2 deletions(-)
drivers/iommu/iommu-debug-pagealloc.c +26 -0
@@ -5,11 +5,13 @@* IOMMU API debug page alloc sanitizer */ #include <linux/atomic.h> +#include <linux/iommu.h> #include <linux/iommu-debug-pagealloc.h> #include <linux/kernel.h> #include <linux/page_ext.h> static bool needed; +DEFINE_STATIC_KEY_FALSE (iommu_debug_initialized);struct iommu_debug_metadata {atomic_t ref;
@@ -25,6 +27,30 @@ struct page_ext_operations page_iommu_debug_ops = { .need = need_iommu_debug, }; +void __iommu_debug_map (struct iommu_domain *domain,phys_addr_t phys,size_t size) +{ +} + +void __iommu_debug_unmap_begin (struct iommu_domain *domain, +unsigned long iova,size_t size) +{ +} + +void __iommu_debug_unmap_end (struct iommu_domain *domain, +unsigned long iova,size_t size, +size_t unmapped) +{ +} + +void iommu_debug_init (void ) +{ +if (!needed) +return ; + +pr_info ("iommu: Debugging page allocations, expect overhead or disable iommu.debug_pagealloc" ); +static_branch_enable (&iommu_debug_initialized); +} +static int __init iommu_debug_pagealloc (char *str) {return kstrtobool (str, &needed);
drivers/iommu/iommu.c +10 -2
@@ -18,6 +18,7 @@#include <linux/errno.h> #include <linux/host1x_context_bus.h> #include <linux/iommu.h> +#include <linux/iommu-debug-pagealloc.h> #include <linux/iommufd.h> #include <linux/idr.h> #include <linux/err.h>
@@ -231,6 +232,8 @@ static int __init iommu_subsys_init(void)if (!nb)return -ENOMEM ; +iommu_debug_init (); +for (int i =0 ; i <ARRAY_SIZE (iommu_buses); i++) { nb[i].notifier_call = iommu_bus_notifier;bus_register_notifier (iommu_buses[i], &nb[i]);
@@ -2544,10 +2547,12 @@ int iommu_map_nosync(struct iommu_domain *domain, unsigned long iova, }/* unroll mapping in case something went wrong */ -if (ret) +if (ret) {iommu_unmap (domain, orig_iova, orig_size - size); -else + }else {trace_map (orig_iova, orig_paddr, orig_size); +iommu_debug_map (domain, orig_paddr, orig_size); + }return ret; }
@@ -2609,6 +2614,8 @@ static size_t __iommu_unmap(struct iommu_domain *domain,pr_debug ("unmap this: iova 0x%lx size 0x%zx\n" , iova, size); +iommu_debug_unmap_begin (domain, iova, size); +/* * Keep iterating until we either unmap 'size' bytes (or more) * or we hit an area that isn't mapped.
@@ -2629,6 +2636,7 @@ static size_t __iommu_unmap(struct iommu_domain *domain, }trace_unmap (orig_iova, size, unmapped); +iommu_debug_unmap_end (domain, orig_iova, size, unmapped);return unmapped; }
include/linux/iommu-debug-pagealloc.h +56 -0
66 lines not on this page · raw message
Signature · 1 line
2.52.0.460.gd25c4c69ec-goog
Re: [PATCH v3 2/4] drivers/iommu: Add calls for IOMMU_DEBUG_PAGEALLOC
Quoted 14 lines · Mostafa Saleh
Add calls for the new iommu debug config
IOMMU_DEBUG_PAGEALLOC:
-iommu_debug_init: Enable the debug mode if configured by the user.
-iommu_debug_map: Track iommu pages mapped, using physical address.
-iommu_debug_unmap_begin: Track start of iommu unmap operation, with IOVA and size.
-iommu_debug_unmap_end: Track the end of unmap operation, passing the actual unmapped size versus the tracked one atunmap_begin.We have to do the unmap_begin/end as once pages are unmapped we lose the information of the physical address.
This is racy, but the API is racy by construction as it uses refcounts and doesn't attempt to lock/synchronize with the IOMMU API as that will be costly, meaning that possibility of false negative exists.
Quoted 7 lines
Signed-off-by Mostafa Saleh <[email protected]>--- drivers/iommu/iommu-debug-pagealloc.c | 26 +++++++++++++ drivers/iommu/iommu.c | 12 +++++- include/linux/iommu-debug-pagealloc.h | 56 +++++++++++++++++++++++++++ 3 files changed, 92 insertions(+), 2 deletions(-)
Remove "drivers/" from the commit title.
$ git log --oneline drivers/iommu/iommu.c
[...]
Quoted 57 lines
include/linux/iommu-debug-pagealloc.h +45
-0@@ -8,10 +8,66 @@#ifndef __LINUX_IOMMU_DEBUG_PAGEALLOC_H#define __LINUX_IOMMU_DEBUG_PAGEALLOC_H +struct iommu_domain ; +#ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC +DECLARE_STATIC_KEY_FALSE (iommu_debug_initialized); +extern struct page_ext_operations page_iommu_debug_ops; +void __iommu_debug_map (struct iommu_domain *domain,phys_addr_t phys, +size_t size); +void __iommu_debug_unmap_begin (struct iommu_domain *domain, +unsigned long iova,size_t size); +void __iommu_debug_unmap_end (struct iommu_domain *domain, +unsigned long iova,size_t size,size_t unmapped); + +static inline void iommu_debug_map (struct iommu_domain *domain, +phys_addr_t phys,size_t size) +{ +if (static_branch_unlikely (&iommu_debug_initialized)) +__iommu_debug_map (domain, phys, size); +} + +static inline void iommu_debug_unmap_begin (struct iommu_domain *domain, +unsigned long iova,size_t size) +{ +if (static_branch_unlikely (&iommu_debug_initialized)) +__iommu_debug_unmap_begin (domain, iova, size); +} + +static inline void iommu_debug_unmap_end (struct iommu_domain *domain, +unsigned long iova,size_t size, +size_t unmapped) +{ +if (static_branch_unlikely (&iommu_debug_initialized)) +__iommu_debug_unmap_end (domain, iova, size, unmapped); +} + +void iommu_debug_init (void ); + +#else +static inline void iommu_debug_map (struct iommu_domain *domain, +phys_addr_t phys,size_t size) +{ +} + +static inline void iommu_debug_unmap_begin (struct iommu_domain *domain, +unsigned long iova,size_t size) +{ +}
include/linux/iommu-debug-pagealloc.h line 69
Quoted 10 lines
include/linux/iommu-debug-pagealloc.h +10
-0+ +static inline void iommu_debug_unmap_end (struct iommu_domain *domain, +unsigned long iova,size_t size, +size_t unmapped) +{ +} + +static inline void iommu_debug_init (void ) +{ +}
I suppose that all these should go to drivers/iommu/iommu-priv.h, as they are for use in other files inside the IOMMU subsystem.
include/linux/iommu-debug-pagealloc.h line 73
Quoted 4 lines
include/linux/iommu-debug-pagealloc.h +1
-0+#endif /* CONFIG_IOMMU_DEBUG_PAGEALLOC */ #endif /* __LINUX_IOMMU_DEBUG_PAGEALLOC_H */
Thanks,
baolu
Re: [PATCH v3 2/4] drivers/iommu: Add calls for IOMMU_DEBUG_PAGEALLOC
Quoted 23 lines · Baolu Lu
Quoted 14 lines · Mostafa Saleh
Add calls for the new iommu debug config
IOMMU_DEBUG_PAGEALLOC:
-iommu_debug_init: Enable the debug mode if configured by the user.
-iommu_debug_map: Track iommu pages mapped, using physical address.
-iommu_debug_unmap_begin: Track start of iommu unmap operation, with IOVA and size.
-iommu_debug_unmap_end: Track the end of unmap operation, passing the actual unmapped size versus the tracked one atunmap_begin.We have to do the unmap_begin/end as once pages are unmapped we lose the information of the physical address.
This is racy, but the API is racy by construction as it uses refcounts and doesn't attempt to lock/synchronize with the IOMMU API as that will be costly, meaning that possibility of false negative exists.Quoted 7 lines
Signed-off-by Mostafa Saleh <[email protected]>--- drivers/iommu/iommu-debug-pagealloc.c | 26 +++++++++++++ drivers/iommu/iommu.c | 12 +++++- include/linux/iommu-debug-pagealloc.h | 56 +++++++++++++++++++++++++++ 3 files changed, 92 insertions(+), 2 deletions(-)
Quoted 3 lines
Remove "
drivers/" from the commit title.$ git log --oneline drivers/iommu/iommu.c
My bad, I will fix it.
Quoted 61 lines
[...]
Quoted 59 lines
include/linux/iommu-debug-pagealloc.h +50
-0@@ -8,10 +8,66 @@ #ifndef __LINUX_IOMMU_DEBUG_PAGEALLOC_H #define __LINUX_IOMMU_DEBUG_PAGEALLOC_H +struct iommu_domain; + #ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC +DECLARE_STATIC_KEY_FALSE(iommu_debug_initialized); + extern struct page_ext_operations page_iommu_debug_ops; +void __iommu_debug_map(struct iommu_domain *domain, phys_addr_t phys, + size_t size); +void __iommu_debug_unmap_begin(struct iommu_domain *domain, + unsigned long iova, size_t size); +void __iommu_debug_unmap_end(struct iommu_domain *domain, + unsigned long iova, size_t size, size_t unmapped); + +static inline void iommu_debug_map(struct iommu_domain *domain, + phys_addr_t phys, size_t size) +{ + if (static_branch_unlikely(&iommu_debug_initialized)) + __iommu_debug_map(domain, phys, size); +} + +static inline void iommu_debug_unmap_begin(struct iommu_domain *domain, + unsigned long iova, size_t size) +{ + if (static_branch_unlikely(&iommu_debug_initialized)) + __iommu_debug_unmap_begin(domain, iova, size); +} + +static inline void iommu_debug_unmap_end(struct iommu_domain *domain, + unsigned long iova, size_t size, + size_t unmapped) +{ + if (static_branch_unlikely(&iommu_debug_initialized)) + __iommu_debug_unmap_end(domain, iova, size, unmapped); +} + +void iommu_debug_init(void); + +#else +static inline void iommu_debug_map(struct iommu_domain *domain, + phys_addr_t phys, size_t size) +{ +} + +static inline void iommu_debug_unmap_begin(struct iommu_domain *domain, + unsigned long iova, size_t size) +{ +} + +static inline void iommu_debug_unmap_end(struct iommu_domain *domain, + unsigned long iova, size_t size, + size_t unmapped) +{
Quoted 8 lines
include/linux/iommu-debug-pagealloc.h line 69
Quoted 5 lines
include/linux/iommu-debug-pagealloc.h +5
-0+} + +static inline void iommu_debug_init (void ) +{ +}I suppose that all these should go to
drivers/iommu/iommu-priv.h, as they are for use in other files inside the IOMMU subsystem.
It seemed better to have all the feature functions/declarations in one isolated file, as it is included outside of the iommu susbsystem also. I have no strong opinion, I can keep them in drivers/iommu/iommu-priv.h if you think it's better. But then we will have to include also "iommu-debug-pagealloc.h" for the static key to avoid including extra files to linux/mm.h.
Thanks,
Mostafa
Quoted 7 lines
include/linux/iommu-debug-pagealloc.h line 73
Quoted 4 lines
include/linux/iommu-debug-pagealloc.h +1
-0+#endif /* CONFIG_IOMMU_DEBUG_PAGEALLOC */ #endif /* __LINUX_IOMMU_DEBUG_PAGEALLOC_H */ Thanks,
baolu
Re: [PATCH v3 2/4] drivers/iommu: Add calls for IOMMU_DEBUG_PAGEALLOC
Quoted 108 lines · Mostafa Saleh
Quoted 23 lines · Baolu Lu
Quoted 14 lines · Mostafa Saleh
Add calls for the new iommu debug config
IOMMU_DEBUG_PAGEALLOC:
-iommu_debug_init: Enable the debug mode if configured by the user.
-iommu_debug_map: Track iommu pages mapped, using physical address.
-iommu_debug_unmap_begin: Track start of iommu unmap operation, with IOVA and size.
-iommu_debug_unmap_end: Track the end of unmap operation, passing the actual unmapped size versus the tracked one atunmap_begin.We have to do the unmap_begin/end as once pages are unmapped we lose the information of the physical address.
This is racy, but the API is racy by construction as it uses refcounts and doesn't attempt to lock/synchronize with the IOMMU API as that will be costly, meaning that possibility of false negative exists.Quoted 7 lines
Signed-off-by Mostafa Saleh <[email protected]>--- drivers/iommu/iommu-debug-pagealloc.c | 26 +++++++++++++ drivers/iommu/iommu.c | 12 +++++- include/linux/iommu-debug-pagealloc.h | 56 +++++++++++++++++++++++++++ 3 files changed, 92 insertions(+), 2 deletions(-)Quoted 3 lines
Remove "
drivers/" from the commit title.$ git log --oneline drivers/iommu/iommu.cMy bad, I will fix it.
Quoted 61 lines
[...]
Quoted 59 lines
include/linux/iommu-debug-pagealloc.h +50
-0@@ -8,10 +8,66 @@ #ifndef __LINUX_IOMMU_DEBUG_PAGEALLOC_H #define __LINUX_IOMMU_DEBUG_PAGEALLOC_H +struct iommu_domain; + #ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC +DECLARE_STATIC_KEY_FALSE(iommu_debug_initialized); + extern struct page_ext_operations page_iommu_debug_ops; +void __iommu_debug_map(struct iommu_domain *domain, phys_addr_t phys, + size_t size); +void __iommu_debug_unmap_begin(struct iommu_domain *domain, + unsigned long iova, size_t size); +void __iommu_debug_unmap_end(struct iommu_domain *domain, + unsigned long iova, size_t size, size_t unmapped); + +static inline void iommu_debug_map(struct iommu_domain *domain, + phys_addr_t phys, size_t size) +{ + if (static_branch_unlikely(&iommu_debug_initialized)) + __iommu_debug_map(domain, phys, size); +} + +static inline void iommu_debug_unmap_begin(struct iommu_domain *domain, + unsigned long iova, size_t size) +{ + if (static_branch_unlikely(&iommu_debug_initialized)) + __iommu_debug_unmap_begin(domain, iova, size); +} + +static inline void iommu_debug_unmap_end(struct iommu_domain *domain, + unsigned long iova, size_t size, + size_t unmapped) +{ + if (static_branch_unlikely(&iommu_debug_initialized)) + __iommu_debug_unmap_end(domain, iova, size, unmapped); +} + +void iommu_debug_init(void); + +#else +static inline void iommu_debug_map(struct iommu_domain *domain, + phys_addr_t phys, size_t size) +{ +} + +static inline void iommu_debug_unmap_begin(struct iommu_domain *domain, + unsigned long iova, size_t size) +{ +} + +static inline void iommu_debug_unmap_end(struct iommu_domain *domain, + unsigned long iova, size_t size, + size_t unmapped) +{Quoted 8 lines
include/linux/iommu-debug-pagealloc.h line 69
Quoted 5 lines
include/linux/iommu-debug-pagealloc.h +5
-0+} + +static inline void iommu_debug_init (void ) +{ +}I suppose that all these should go to
drivers/iommu/iommu-priv.h, as they are for use in other files inside the IOMMU subsystem.It seemed better to have all the feature functions/declarations in one isolated file, as it is included outside of the iommu susbsystem also. I have no strong opinion, I can keep them in
drivers/iommu/iommu-priv.hif you think it's better. But then we will have to include also "iommu-debug-pagealloc.h" for the static key to avoid including extra files tolinux/mm.h.
Quoted 3 lines
Thanks,
Mostafa
Hi Baolu,
I plan to post a new version soon. So far, I only have the commit subject rewrites. Do you have any opinion on the header file split or the unmap size part?
Thanks,
Mostafa
Quoted 7 lines
Quoted 7 lines
include/linux/iommu-debug-pagealloc.h line 73
Quoted 4 lines
include/linux/iommu-debug-pagealloc.h +1
-0+#endif /* CONFIG_IOMMU_DEBUG_PAGEALLOC */ #endif /* __LINUX_IOMMU_DEBUG_PAGEALLOC_H */ Thanks,
baolu
Re: [PATCH v3 2/4] drivers/iommu: Add calls for IOMMU_DEBUG_PAGEALLOC
Quoted 103 lines · Mostafa Saleh
Quoted 102 lines · Mostafa Saleh
Quoted 22 lines · Baolu Lu
Quoted 14 lines · Mostafa Saleh
Add calls for the new iommu debug config
IOMMU_DEBUG_PAGEALLOC:
-iommu_debug_init: Enable the debug mode if configured by the user.
-iommu_debug_map: Track iommu pages mapped, using physical address.
-iommu_debug_unmap_begin: Track start of iommu unmap operation, with IOVA and size.
-iommu_debug_unmap_end: Track the end of unmap operation, passing the actual unmapped size versus the tracked one atunmap_begin.We have to do the unmap_begin/end as once pages are unmapped we lose the information of the physical address.
This is racy, but the API is racy by construction as it uses refcounts and doesn't attempt to lock/synchronize with the IOMMU API as that will be costly, meaning that possibility of false negative exists.Quoted 7 lines
Signed-off-by Mostafa Saleh<[email protected]>--- drivers/iommu/iommu-debug-pagealloc.c | 26 +++++++++++++ drivers/iommu/iommu.c | 12 +++++- include/linux/iommu-debug-pagealloc.h | 56 +++++++++++++++++++++++++++ 3 files changed, 92 insertions(+), 2 deletions(-)Quoted 3 lines
Remove "
drivers/" from the commit title.$ git log --oneline drivers/iommu/iommu.cMy bad, I will fix it.
Quoted 59 lines
[...]
Quoted 58 lines
include/linux/iommu-debug-pagealloc.h +49
-0@@ -8,10 +8,66 @@#ifndef __LINUX_IOMMU_DEBUG_PAGEALLOC_H#define __LINUX_IOMMU_DEBUG_PAGEALLOC_H +struct iommu_domain ; +#ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC +DECLARE_STATIC_KEY_FALSE (iommu_debug_initialized); +extern struct page_ext_operations page_iommu_debug_ops; +void __iommu_debug_map (struct iommu_domain *domain,phys_addr_t phys, +size_t size); +void __iommu_debug_unmap_begin (struct iommu_domain *domain, +unsigned long iova,size_t size); +void __iommu_debug_unmap_end (struct iommu_domain *domain, +unsigned long iova,size_t size,size_t unmapped); + +static inline void iommu_debug_map (struct iommu_domain *domain, +phys_addr_t phys,size_t size) +{ +if (static_branch_unlikely (&iommu_debug_initialized)) +__iommu_debug_map (domain, phys, size); +} + +static inline void iommu_debug_unmap_begin (struct iommu_domain *domain, +unsigned long iova,size_t size) +{ +if (static_branch_unlikely (&iommu_debug_initialized)) +__iommu_debug_unmap_begin (domain, iova, size); +} + +static inline void iommu_debug_unmap_end (struct iommu_domain *domain, +unsigned long iova,size_t size, +size_t unmapped) +{ +if (static_branch_unlikely (&iommu_debug_initialized)) +__iommu_debug_unmap_end (domain, iova, size, unmapped); +} + +void iommu_debug_init (void ); + +#else +static inline void iommu_debug_map (struct iommu_domain *domain, +phys_addr_t phys,size_t size) +{ +} + +static inline void iommu_debug_unmap_begin (struct iommu_domain *domain, +unsigned long iova,size_t size) +{ +} + +static inline void iommu_debug_unmap_end (struct iommu_domain *domain, +unsigned long iova,size_t size, +size_t unmapped)Quoted 8 lines
include/linux/iommu-debug-pagealloc.h line 69
Quoted 6 lines
include/linux/iommu-debug-pagealloc.h +6
-0+{ +} + +static inline void iommu_debug_init (void ) +{ +}I suppose that all these should go to
drivers/iommu/iommu-priv.h, as they are for use in other files inside the IOMMU subsystem.It seemed better to have all the feature functions/declarations in one isolated file, as it is included outside of the iommu susbsystem also. I have no strong opinion, I can keep them in
drivers/iommu/iommu-priv.hif you think it's better. But then we will have to include also "iommu-debug-pagealloc.h" for the static key to avoid including extra files tolinux/mm.h.
Quoted 8 lines
Quoted 3 lines
Thanks,
MostafaHi Baolu,
I plan to post a new version soon. So far, I only have the commit subject rewrites. Do you have any opinion on the header file split or the unmap size part?
I have no option regarding the unmap size part, as the comments already declare the possible cases. For the header file split, I feel it's better to put staff that is only used in the IOMMU subsystem in drivers/ iommu in iommu-pri.h instead of the global ones under include/linux.
Thanks,
baolu
PATCH v3 3/4 drivers/iommu-debug-pagealloc: Track IOMMU pages
In reply to Mostafa SalehUsing the new calls, use an atomic refcount to track how many times a page is mapped in any of the IOMMUs.
For unmap we need to use iova_to_phys() to get the physical address of the pages.
We use the smallest supported page size as the granularity of tracking per domain.
This is important as it is possible to map pages and unmap them with larger sizes (as in map_sg()) cases.
--- drivers/iommu/iommu-debug-pagealloc.c | 91 +++++++++++++++++++++++++++ 1 file changed, 91 insertions(+)
drivers/iommu/iommu-debug-pagealloc.c +91 -0
@@ -27,19 +27,110 @@ struct page_ext_operations page_iommu_debug_ops = { .need = need_iommu_debug, }; +static struct page_ext *get_iommu_page_ext (phys_addr_t phys) +{ +struct page *page =phys_to_page (phys); +struct page_ext *page_ext =page_ext_get (page); + +return page_ext; +} + +static struct iommu_debug_metadata *get_iommu_data (struct page_ext *page_ext) +{ +return page_ext_data (page_ext, &page_iommu_debug_ops); +} + +static void iommu_debug_inc_page (phys_addr_t phys) +{ +struct page_ext *page_ext =get_iommu_page_ext (phys); +struct iommu_debug_metadata *d =get_iommu_data (page_ext); + +WARN_ON (atomic_inc_return_relaxed (&d->ref) <=0 ); +page_ext_put (page_ext); +} + +static void iommu_debug_dec_page (phys_addr_t phys) +{ +struct page_ext *page_ext =get_iommu_page_ext (phys); +struct iommu_debug_metadata *d =get_iommu_data (page_ext); + +WARN_ON (atomic_dec_return_relaxed (&d->ref) <0 ); +page_ext_put (page_ext); +} + +/* +* IOMMU page size doesn't have tomatch the CPU page size. So, we use +* the smallest IOMMU page size to refcount the pages in the vmemmap. +* That is important as both map and unmap has to use the same page size +* to update the refcount to avoid double counting the same page. +* And as we can't know from iommu_unmap() what was the original page size +* used for map, we just use the minimum supported one for both. +*/ +static size_t iommu_debug_page_size (struct iommu_domain *domain) +{ +return 1UL <<__ffs (domain->pgsize_bitmap); +} +void __iommu_debug_map (struct iommu_domain *domain,phys_addr_t phys,size_t size) { +size_t off, end; +size_t page_size =iommu_debug_page_size (domain); + +if (WARN_ON (!phys ||check_add_overflow (phys, size, &end))) +return ; + +for (off =0 ; off < size ; off += page_size) { +if (!pfn_valid (__phys_to_pfn (phys + off))) +continue ; +iommu_debug_inc_page (phys + off); + } +} + +static void __iommu_debug_update_iova (struct iommu_domain *domain, +unsigned long iova,size_t size,bool inc) +{ +size_t off, end; +size_t page_size =iommu_debug_page_size (domain); + +if (WARN_ON (check_add_overflow (iova, size, &end))) +return ; + +for (off =0 ; off < size ; off += page_size) { +phys_addr_t phys =iommu_iova_to_phys (domain, iova + off); + +if (!phys || !pfn_valid (__phys_to_pfn (phys))) +continue ; + +if (inc) +iommu_debug_inc_page (phys); +else +iommu_debug_dec_page (phys); + } }void __iommu_debug_unmap_begin (struct iommu_domain *domain,unsigned long iova,size_t size) { +__iommu_debug_update_iova (domain, iova, size, false); }void __iommu_debug_unmap_end (struct iommu_domain *domain,unsigned long iova,size_t size,size_t unmapped) { +if (unmapped == size) +return ; + +/* +* If unmap failed, re-increment the refcount, but if it unmapped +* larger size, decrement the extra part. +*/ +if (unmapped < size) +__iommu_debug_update_iova (domain, iova + unmapped, + size - unmapped, true); +else +__iommu_debug_update_iova (domain, iova + size, + unmapped - size, false); }void iommu_debug_init (void )
Signature · 1 line
2.52.0.460.gd25c4c69ec-goog
Re: [PATCH v3 3/4] drivers/iommu-debug-pagealloc: Track IOMMU pages
Quoted 7 lines · Mostafa Saleh
void __iommu_debug_unmap_end(struct
iommu_domain*domain,
unsigned long iova,size_tsize,size_tunmapped)
{drivers/iommu/iommu-debug-pagealloc.c +3
-0+if (unmapped == size) +return ; +
drivers/iommu/iommu-debug-pagealloc.c struct page_ext_operations page_iommu_debug_ops = { · line 133
Quoted 10 lines
drivers/iommu/iommu-debug-pagealloc.c +10
-0+/* +* If unmap failed, re-increment the refcount, but if it unmapped +* larger size, decrement the extra part. +*/ +if (unmapped < size) +__iommu_debug_update_iova (domain, iova + unmapped, + size - unmapped, true); +else +__iommu_debug_update_iova (domain, iova + size, + unmapped - size, false);
In any case, could the 'else' branch become a real operation?
In the __iommu_unmap():
/*
* Keep iterating until we either unmap 'size' bytes (or more)
* or we hit an area that isn't mapped.
*/
while (unmapped < size) {
size_t pgsize, count;
pgsize = iommu_pgsize(domain, iova, iova, size -
unmapped, &count);
unmapped_page = ops->unmap_pages(domain, iova, pgsize,
count, iotlb_gather);
if (!unmapped_page)
break;
pr_debug("unmapped: iova 0x%lx size 0x%zx\n",
iova, unmapped_page);
iova += unmapped_page;
unmapped += unmapped_page;
}The comments say that it is possible to unmap more bytes than 'size', but isn't it a bug if this helper unmaps more than the caller desired?
Thanks,
baolu
Re: [PATCH v3 3/4] drivers/iommu-debug-pagealloc: Track IOMMU pages
Quoted 44 lines · Baolu Lu
Quoted 7 lines · Mostafa Saleh
void __iommu_debug_unmap_end(struct
iommu_domain*domain,
unsigned long iova,size_tsize,size_tunmapped)
{drivers/iommu/iommu-debug-pagealloc.c +3
-0+if (unmapped == size) +return ; +drivers/iommu/iommu-debug-pagealloc.c struct page_ext_operations page_iommu_debug_ops = { · line 133
Quoted 10 lines
drivers/iommu/iommu-debug-pagealloc.c +10
-0+/* +* If unmap failed, re-increment the refcount, but if it unmapped +* larger size, decrement the extra part. +*/ +if (unmapped < size) +__iommu_debug_update_iova (domain, iova + unmapped, + size - unmapped, true); +else +__iommu_debug_update_iova (domain, iova + size, + unmapped - size, false);In any case, could the 'else' branch become a real operation?
In the
__iommu_unmap():/* * Keep iterating until we either unmap 'size' bytes (or more) * or we hit an area that isn't mapped. */ while (unmapped < size) { size_t pgsize, count; pgsize = iommu_pgsize(domain, iova, iova, size - unmapped, &count); unmapped_page = ops->unmap_pages(domain, iova, pgsize, count, iotlb_gather); if (!unmapped_page) break; pr_debug("unmapped: iova 0x%lx size 0x%zx\n", iova, unmapped_page); iova += unmapped_page; unmapped += unmapped_page; }
Quoted 2 lines
The comments say that it is possible to unmap more bytes than 'size', but isn't it a bug if this helper unmaps more than the caller desired?
I was wondering also why the core API allows that, I couldn't find useful information from "git blame". But I vaguely remember that some IOMMUs can't split blocks so they unmap the whole block when a smaller size is requested.
Thanks,
Mostafa
Quoted 3 lines
Thanks,
baolu
PATCH v3 4/4 drivers/iommu-debug-pagealloc: Check mapped/unmapped kernel memory
In reply to Mostafa SalehNow, as the page_ext holds count of IOMMU mappings, we can use it to assert that any page allocated/freed is indeed not in the IOMMU.
The sanitizer doesn’t protect against mapping/unmapping during this period. However, that’s less harmful as the page is not used by the kernel.
--- drivers/iommu/iommu-debug-pagealloc.c | 23 +++++++++++++++++++++++ include/linux/iommu-debug-pagealloc.h | 12 ++++++++++++ include/linux/mm.h | 5 +++++ 3 files changed, 40 insertions(+)
drivers/iommu/iommu-debug-pagealloc.c +23 -0
@@ -9,6 +9,7 @@#include <linux/iommu-debug-pagealloc.h> #include <linux/kernel.h> #include <linux/page_ext.h> +#include <linux/page_owner.h> static bool needed;DEFINE_STATIC_KEY_FALSE (iommu_debug_initialized);
@@ -71,6 +72,28 @@ static size_t iommu_debug_page_size(struct iommu_domain *domain)return 1UL <<__ffs (domain->pgsize_bitmap); } +static bool iommu_debug_page_count (const struct page *page) +{ +unsigned int ref; +struct page_ext *page_ext =page_ext_get (page); +struct iommu_debug_metadata *d =get_iommu_data (page_ext); + + ref =atomic_read (&d->ref); +page_ext_put (page_ext); +return ref !=0 ; +} + +void __iommu_debug_check_unmapped (const struct page *page,int numpages) +{ +while (numpages--) { +if (WARN_ON (iommu_debug_page_count (page))) { +pr_warn ("iommu: Detected page leak!\n" ); +dump_page_owner (page); + } + page++; + } +} +void __iommu_debug_map (struct iommu_domain *domain,phys_addr_t phys,size_t size) {size_t off, end;
include/linux/iommu-debug-pagealloc.h +12 -0
@@ -22,6 +22,7 @@ void __iommu_debug_unmap_begin(struct iommu_domain *domain, unsigned long iova, size_t size);void __iommu_debug_unmap_end (struct iommu_domain *domain,unsigned long iova,size_t size,size_t unmapped); +void __iommu_debug_check_unmapped (const struct page *page,int numpages);static inline void iommu_debug_map (struct iommu_domain *domain,phys_addr_t phys,size_t size)
@@ -45,6 +46,12 @@ static inline void iommu_debug_unmap_end(struct iommu_domain *domain, __iommu_debug_unmap_end(domain, iova, size, unmapped); } +static inline void iommu_debug_check_unmapped (const struct page *page,int numpages) +{ +if (static_branch_unlikely (&iommu_debug_initialized)) +__iommu_debug_check_unmapped (page, numpages); +} +void iommu_debug_init (void );#else
@@ -68,6 +75,11 @@ static inline void iommu_debug_init(void) { } +static inline void iommu_debug_check_unmapped (const struct page *page, +int numpages) +{ +} +#endif /* CONFIG_IOMMU_DEBUG_PAGEALLOC */ #endif /* __LINUX_IOMMU_DEBUG_PAGEALLOC_H */
include/linux/mm.h +5 -0
@@ -36,6 +36,7 @@#include <linux/rcuwait.h> #include <linux/bitmap.h> #include <linux/bitops.h> +#include <linux/iommu-debug-pagealloc.h> struct mempolicy ;struct anon_vma ;
@@ -3818,12 +3819,16 @@ extern void __kernel_map_pages(struct page *page, int numpages, int enable);#ifdef CONFIG_DEBUG_PAGEALLOC static inline void debug_pagealloc_map_pages (struct page *page,int numpages) { +iommu_debug_check_unmapped (page, numpages); +if (debug_pagealloc_enabled_static ())__kernel_map_pages (page, numpages,1 ); }static inline void debug_pagealloc_unmap_pages (struct page *page,int numpages) { +iommu_debug_check_unmapped (page, numpages); +if (debug_pagealloc_enabled_static ())__kernel_map_pages (page, numpages,0 ); }
Signature · 1 line
2.52.0.460.gd25c4c69ec-goog