iommu: Add page_ext for IOMMU_DEBUG_PAGEALLOC
Subsystems (5)
v6 cover iommu: Add page_ext for IOMMU_DEBUG_PAGEALLOC
Overview
--------
This patch series introduces a new debugging feature, IOMMU_DEBUG_PAGEALLOC, designed to catch DMA use-after-free bugs and IOMMU mapping leaks from buggy drivers.
The kernel has powerful sanitizers like KASAN and DEBUG_PAGEALLOC for catching CPU-side memory corruption. However, there is limited runtime sanitization for DMA mappings managed by the IOMMU. A buggy driver can free a page while it is still mapped for DMA, leading to memory corruption or use-after-free vulnerabilities when that page is reallocated and used for a different purpose.
Inspired by DEBUG_PAGEALLOC, this sanitizer tracks IOMMU mappings on a per-page basis, as it’s not possible to unmap the pages, because it requires to lock and walk all domains on every kernel free, instead we rely on page_ext to add an IOMMU-specific mapping reference count for each page.
And on each page allocated/freed from the kernel we simply check the count and WARN if it is not zero, and dumping page owner information if enabled.
Concurrency
-----------
By design this check is racy where one caller can map pages just after the check, which can lead to false negatives.
In my opinion this is acceptable for sanitizers (for ex KCSAN have that property).
Otherwise we have to implement locks in iommu_map/unmap for all domains which is not favourable even for a debug feature.
The sanitizer only guarantees that the refcount itself doesn’t get corrupted using atomics. And there are no false positives.
CPU vs IOMMU Page Size
----------------------
IOMMUs can use different page sizes and which can be non-homogeneous; not even all of them have the same page size.
To solve this, the refcount is always incremented and decremented in units of the smallest page size supported by the IOMMU domain. This ensures the accounting remains consistent regardless of the size of the map or unmap operation, otherwise double counting can happen.
Testing & Performance
---------------------
This was tested on Morello with Arm64 + SMMUv3
Did some testing Lenovo IdeaCentre X Gen 10 Snapdragon
Did some testing on Qemu including different SMMUv3/CPU page size (arm64).
I also ran dma_map_benchmark on Morello:
echo dma_map_benchmark > /sys/bus/pci/devices/0000\:06\:00.0/driver_override echo 0000:06:00.0 > /sys/bus/pci/devices/0000\:06\:00.0/driver/unbind echo 0000:06:00.0 > /sys/bus/pci/drivers/dma_map_benchmark/bind ./dma_map_benchmark -t $threads -g $nr_pages
CONFIG refers to "CONFIG_IOMMU_DEBUG_PAGEALLOC" cmdline refers to "iommu.debug_pagealloc"
Numbers are (map latency)/(unmap latency), lower is better.
CONFIG=n CONFIG=y CONFIG=y cmdline=0 cmdline=1 4K - 1 thread 0.1/0.6 0.1/0.6 0.1/0.7
4K - 4 threads 0.1/1.1 0.1/1.0 0.2/1.1
1M - 1 thread 0.8/21.2 0.7/21.2 5.4/42.3
1M - 4 threads 1.1/45.9 1.1/46.0 5.9/45.1
Changes in v6:
v5: https://lore.kernel.org/linux-iommu/[email protected]/
- Remove wrong logic for handling unmap with bigger size than requested and warn instead.
- Collect R-bs and Acked-by
Changes in v5:
v4: https://lore.kernel.org/all/[email protected]/
- Fix typo in comment
- Collect Baolu R-bs
Main changes in v4:
v3: https://lore.kernel.org/all/[email protected]/
- Update the kernel parameter format in docs based on Randy feedback
- Update commit subjects
- Add IOMMU only functions in iommu-priv.h based on Baolu feedback
Main changes in v3: (Most of them addressing Will comments)
v2: https://lore.kernel.org/linux-iommu/[email protected]/
- Reword the Kconfig help
- Use unmap_begin/end instead of unmap/remap
- Use relaxed accessors when refcounting
- Fix a bug with checking the returned address from iova_to_phys
- Add more hardening checks (overflow)
- Add more debug info on assertions (dump_page_owner())
- Handle cases where unmap returns larger size as the core code seems to tolerate that.
- Drop Tested-by tags from Qinxin as the code logic changed
Main changes in v2:
v1: https://lore.kernel.org/linux-iommu/[email protected]/
- Address Jörg comments about #ifdefs and static keys
- Reword the Kconfig help
- Drop RFC
- Collect t-b from Qinxin
- Minor cleanups
Mostafa Saleh (4):
iommu: Add page_ext for IOMMU_DEBUG_PAGEALLOC
iommu: Add calls for IOMMU_DEBUG_PAGEALLOC
iommu: debug-pagealloc: Track IOMMU pages
iommu: debug-pagealloc: Check mapped/unmapped kernel memory
.../admin-guide/kernel-parameters.txt | 9 + drivers/iommu/Kconfig | 19 ++ drivers/iommu/Makefile | 1 + drivers/iommu/iommu-debug-pagealloc.c | 167 ++++++++++++++++++ drivers/iommu/iommu-priv.h | 58 ++++++ drivers/iommu/iommu.c | 11 +- include/linux/iommu-debug-pagealloc.h | 32 ++++ include/linux/mm.h | 5 + mm/page_ext.c | 4 + 9 files changed, 304 insertions(+), 2 deletions(-) create mode 100644 drivers/iommu/iommu-debug-pagealloc.c create mode 100644 include/linux/iommu-debug-pagealloc.h
Signature · 1 line
2.52.0.457.g6b5491de43-goog
PATCH v6 1/4 iommu: Add page_ext for IOMMU_DEBUG_PAGEALLOC
Add a new config IOMMU_DEBUG_PAGEALLOC, which registers new data to page_ext.
This config will be used by the IOMMU API to track pages mapped in the IOMMU to catch drivers trying to free kernel memory that they still map in their domains, causing all types of memory corruption.
This behaviour is disabled by default and can be enabled using kernel cmdline iommu.debug_pagealloc.
--- .../admin-guide/kernel-parameters.txt | 9 ++++++ drivers/iommu/Kconfig | 19 +++++++++++ drivers/iommu/Makefile | 1 + drivers/iommu/iommu-debug-pagealloc.c | 32 +++++++++++++++++++ include/linux/iommu-debug-pagealloc.h | 17 ++++++++++ mm/page_ext.c | 4 +++ 6 files changed, 82 insertions(+) create mode 100644 drivers/iommu/iommu-debug-pagealloc.c create mode 100644 include/linux/iommu-debug-pagealloc.h
Documentation/admin-guide/kernel-parameters.txt +9 -0
@@ -2675,6 +2675,15 @@ Kernel parameters 1 - Bypass the IOMMU for DMA. unset - Use value of CONFIG_IOMMU_DEFAULT_PASSTHROUGH. + iommu.debug_pagealloc= + [KNL,EARLY] When CONFIG_IOMMU_DEBUG_PAGEALLOC is set, this + parameter enables the feature at boot time. By default, it + is disabled and the system behaves the same way as a kernel + built without CONFIG_IOMMU_DEBUG_PAGEALLOC. + Format: { "0" | "1" } + 0 - Sanitizer disabled. + 1 - Sanitizer enabled, expect runtime overhead. + io7= [HW] IO7 for Marvel-based Alpha systems See comment before marvel_specify_io7 in arch/alpha/kernel/core_marvel.c.
drivers/iommu/Kconfig +19 -0
@@ -384,6 +384,25 @@ config SPRD_IOMMU Say Y here if you want to use the multimedia devices listed above. +config IOMMU_DEBUG_PAGEALLOC + bool "Debug IOMMU mappings against page allocations" + depends on DEBUG_PAGEALLOC && IOMMU_API && PAGE_EXTENSION + help + This enables a consistency check between the kernel page allocator and + the IOMMU subsystem. It verifies that pages being allocated or freed + are not currently mapped in any IOMMU domain. + + This helps detect DMA use-after-free bugs where a driver frees a page + but forgets to unmap it from the IOMMU, potentially allowing a device + to overwrite memory that the kernel has repurposed. + + These checks are best-effort and may not detect all problems. + + Due to performance overhead, this feature is disabled by default. + You must enable "iommu.debug_pagealloc" from the kernel command + line to activate the runtime checks. + + If unsure, say N. endif # IOMMU_SUPPORT source "drivers/iommu/generic_pt/Kconfig"
drivers/iommu/Makefile +1 -0
@@ -36,3 +36,4 @@ obj-$(CONFIG_IOMMU_SVA) += iommu-sva.o obj-$(CONFIG_IOMMU_IOPF) += io-pgfault.o obj-$(CONFIG_SPRD_IOMMU) += sprd-iommu.o obj-$(CONFIG_APPLE_DART) += apple-dart.o +obj-$(CONFIG_IOMMU_DEBUG_PAGEALLOC) += iommu-debug-pagealloc.o
drivers/iommu/iommu-debug-pagealloc.c new file +32 -0
@@ -0,0 +1,32 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Copyright (C) 2025 - Google Inc +* Author: Mostafa Saleh <[email protected]> +* IOMMU API debug page alloc sanitizer +*/ +#include <linux/atomic.h> +#include <linux/iommu-debug-pagealloc.h> +#include <linux/kernel.h> +#include <linux/page_ext.h> + +static bool needed; + +struct iommu_debug_metadata { +atomic_t ref; +}; + +static __init bool need_iommu_debug (void ) +{ +return needed; +} + +struct page_ext_operations page_iommu_debug_ops = { + .size =sizeof (struct iommu_debug_metadata ), + .need = need_iommu_debug, +}; + +static int __init iommu_debug_pagealloc (char *str) +{ +return kstrtobool (str, &needed); +} +early_param ("iommu.debug_pagealloc" , iommu_debug_pagealloc);
include/linux/iommu-debug-pagealloc.h new file +17 -0
@@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Copyright (C) 2025 - Google Inc +* Author: Mostafa Saleh <[email protected]> +* IOMMU API debug page alloc sanitizer +*/ + +#ifndef __LINUX_IOMMU_DEBUG_PAGEALLOC_H +#define __LINUX_IOMMU_DEBUG_PAGEALLOC_H + +#ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC + +extern struct page_ext_operations page_iommu_debug_ops; + +#endif /* CONFIG_IOMMU_DEBUG_PAGEALLOC */ + +#endif /* __LINUX_IOMMU_DEBUG_PAGEALLOC_H */
mm/page_ext.c +4 -0
@@ -11,6 +11,7 @@#include <linux/page_table_check.h> #include <linux/rcupdate.h> #include <linux/pgalloc_tag.h> +#include <linux/iommu-debug-pagealloc.h> /* * struct page extension
@@ -89,6 +90,9 @@ static struct page_ext_operations *page_ext_ops[] __initdata = {#ifdef CONFIG_PAGE_TABLE_CHECK &page_table_check_ops,#endif +#ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC + &page_iommu_debug_ops, +#endif };unsigned long page_ext_size;
Signature · 1 line
2.52.0.457.g6b5491de43-goog
PATCH v6 2/4 iommu: Add calls for IOMMU_DEBUG_PAGEALLOC
In reply to Mostafa SalehAdd calls for the new iommu debug config IOMMU_DEBUG_PAGEALLOC:
- iommu_debug_init: Enable the debug mode if configured by the user.
- iommu_debug_map: Track iommu pages mapped, using physical address.
- iommu_debug_unmap_begin: Track start of iommu unmap operation, with IOVA and size.
- iommu_debug_unmap_end: Track the end of unmap operation, passing the actual unmapped size versus the tracked one at unmap_begin.
We have to do the unmap_begin/end as once pages are unmapped we lose the information of the physical address.
This is racy, but the API is racy by construction as it uses refcounts and doesn't attempt to lock/synchronize with the IOMMU API as that will be costly, meaning that possibility of false negative exists.
--- drivers/iommu/iommu-debug-pagealloc.c | 28 +++++++++++++ drivers/iommu/iommu-priv.h | 58 +++++++++++++++++++++++++++ drivers/iommu/iommu.c | 11 ++++- include/linux/iommu-debug-pagealloc.h | 1 + 4 files changed, 96 insertions(+), 2 deletions(-)
drivers/iommu/iommu-debug-pagealloc.c +28 -0
@@ -5,11 +5,15 @@* IOMMU API debug page alloc sanitizer */ #include <linux/atomic.h> +#include <linux/iommu.h> #include <linux/iommu-debug-pagealloc.h> #include <linux/kernel.h> #include <linux/page_ext.h> +#include "iommu-priv.h" +static bool needed; +DEFINE_STATIC_KEY_FALSE (iommu_debug_initialized);struct iommu_debug_metadata {atomic_t ref;
@@ -25,6 +29,30 @@ struct page_ext_operations page_iommu_debug_ops = { .need = need_iommu_debug, }; +void __iommu_debug_map (struct iommu_domain *domain,phys_addr_t phys,size_t size) +{ +} + +void __iommu_debug_unmap_begin (struct iommu_domain *domain, +unsigned long iova,size_t size) +{ +} + +void __iommu_debug_unmap_end (struct iommu_domain *domain, +unsigned long iova,size_t size, +size_t unmapped) +{ +} + +void iommu_debug_init (void ) +{ +if (!needed) +return ; + +pr_info ("iommu: Debugging page allocations, expect overhead or disable iommu.debug_pagealloc" ); +static_branch_enable (&iommu_debug_initialized); +} +static int __init iommu_debug_pagealloc (char *str) {return kstrtobool (str, &needed);
drivers/iommu/iommu-priv.h +58 -0
68 lines not on this page · raw message
drivers/iommu/iommu.c +9 -2
@@ -232,6 +232,8 @@ static int __init iommu_subsys_init(void)if (!nb)return -ENOMEM ; +iommu_debug_init (); +for (int i =0 ; i <ARRAY_SIZE (iommu_buses); i++) { nb[i].notifier_call = iommu_bus_notifier;bus_register_notifier (iommu_buses[i], &nb[i]);
@@ -2562,10 +2564,12 @@ int iommu_map_nosync(struct iommu_domain *domain, unsigned long iova, }/* unroll mapping in case something went wrong */ -if (ret) +if (ret) {iommu_unmap (domain, orig_iova, orig_size - size); -else + }else {trace_map (orig_iova, orig_paddr, orig_size); +iommu_debug_map (domain, orig_paddr, orig_size); + }return ret; }
@@ -2627,6 +2631,8 @@ static size_t __iommu_unmap(struct iommu_domain *domain,pr_debug ("unmap this: iova 0x%lx size 0x%zx\n" , iova, size); +iommu_debug_unmap_begin (domain, iova, size); +/* * Keep iterating until we either unmap 'size' bytes (or more) * or we hit an area that isn't mapped.
@@ -2647,6 +2653,7 @@ static size_t __iommu_unmap(struct iommu_domain *domain, }trace_unmap (orig_iova, size, unmapped); +iommu_debug_unmap_end (domain, orig_iova, size, unmapped);return unmapped; }
include/linux/iommu-debug-pagealloc.h +1 -0
@@ -9,6 +9,7 @@#define __LINUX_IOMMU_DEBUG_PAGEALLOC_H#ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC +DECLARE_STATIC_KEY_FALSE (iommu_debug_initialized);extern struct page_ext_operations page_iommu_debug_ops;
Signature · 1 line
2.52.0.457.g6b5491de43-goog
PATCH v6 3/4 iommu: debug-pagealloc: Track IOMMU pages
In reply to Mostafa SalehUsing the new calls, use an atomic refcount to track how many times a page is mapped in any of the IOMMUs.
For unmap we need to use iova_to_phys() to get the physical address of the pages.
We use the smallest supported page size as the granularity of tracking per domain.
This is important as it is possible to map pages and unmap them with larger sizes (as in map_sg()) cases.
--- drivers/iommu/iommu-debug-pagealloc.c | 84 +++++++++++++++++++++++++++ 1 file changed, 84 insertions(+)
drivers/iommu/iommu-debug-pagealloc.c +84 -0
@@ -29,19 +29,103 @@ struct page_ext_operations page_iommu_debug_ops = { .need = need_iommu_debug, }; +static struct page_ext *get_iommu_page_ext (phys_addr_t phys) +{ +struct page *page =phys_to_page (phys); +struct page_ext *page_ext =page_ext_get (page); + +return page_ext; +} + +static struct iommu_debug_metadata *get_iommu_data (struct page_ext *page_ext) +{ +return page_ext_data (page_ext, &page_iommu_debug_ops); +} + +static void iommu_debug_inc_page (phys_addr_t phys) +{ +struct page_ext *page_ext =get_iommu_page_ext (phys); +struct iommu_debug_metadata *d =get_iommu_data (page_ext); + +WARN_ON (atomic_inc_return_relaxed (&d->ref) <=0 ); +page_ext_put (page_ext); +} + +static void iommu_debug_dec_page (phys_addr_t phys) +{ +struct page_ext *page_ext =get_iommu_page_ext (phys); +struct iommu_debug_metadata *d =get_iommu_data (page_ext); + +WARN_ON (atomic_dec_return_relaxed (&d->ref) <0 ); +page_ext_put (page_ext); +} + +/* +* IOMMU page size doesn't have to match the CPU page size. So, we use +* the smallest IOMMU page size to refcount the pages in the vmemmap. +* That is important as both map and unmap has to use the same page size +* to update the refcount to avoid double counting the same page. +* And as we can't know from iommu_unmap() what was the original page size +* used for map, we just use the minimum supported one for both. +*/ +static size_t iommu_debug_page_size (struct iommu_domain *domain) +{ +return 1UL <<__ffs (domain->pgsize_bitmap); +} +void __iommu_debug_map (struct iommu_domain *domain,phys_addr_t phys,size_t size) { +size_t off, end; +size_t page_size =iommu_debug_page_size (domain); + +if (WARN_ON (!phys ||check_add_overflow (phys, size, &end))) +return ; + +for (off =0 ; off < size ; off += page_size) { +if (!pfn_valid (__phys_to_pfn (phys + off))) +continue ; +iommu_debug_inc_page (phys + off); + } +} + +static void __iommu_debug_update_iova (struct iommu_domain *domain, +unsigned long iova,size_t size,bool inc) +{ +size_t off, end; +size_t page_size =iommu_debug_page_size (domain); + +if (WARN_ON (check_add_overflow (iova, size, &end))) +return ; + +for (off =0 ; off < size ; off += page_size) { +phys_addr_t phys =iommu_iova_to_phys (domain, iova + off); + +if (!phys || !pfn_valid (__phys_to_pfn (phys))) +continue ; + +if (inc) +iommu_debug_inc_page (phys); +else +iommu_debug_dec_page (phys); + } }void __iommu_debug_unmap_begin (struct iommu_domain *domain,unsigned long iova,size_t size) { +__iommu_debug_update_iova (domain, iova, size, false); }void __iommu_debug_unmap_end (struct iommu_domain *domain,unsigned long iova,size_t size,size_t unmapped) { +if ((unmapped == size) ||WARN_ON_ONCE (unmapped > size)) +return ; + +/* If unmap failed, re-increment the refcount. */ +__iommu_debug_update_iova (domain, iova + unmapped, + size - unmapped, true); }void iommu_debug_init (void )
Signature · 1 line
2.52.0.457.g6b5491de43-goog
Re: [PATCH v6 3/4] iommu: debug-pagealloc: Track IOMMU pages
Quoted 110 lines · Mostafa Saleh
Using the new calls, use an atomic refcount to track how many times a page is mapped in any of the IOMMUs.
For unmap we need to use
iova_to_phys()to get the physical address of the pages.We use the smallest supported page size as the granularity of tracking per domain.
This is important as it is possible to map pages and unmap them with larger sizes (as inmap_sg()) cases.Reviewed-by Samiullah Khawaja <[email protected]>Reviewed-by Lu Baolu <[email protected]>Signed-off-by Mostafa Saleh <[email protected]>--- drivers/iommu/iommu-debug-pagealloc.c | 84 +++++++++++++++++++++++++++ 1 file changed, 84 insertions(+)drivers/iommu/iommu-debug-pagealloc.c +77
-0@@ -29,19 +29,103 @@ struct page_ext_operations page_iommu_debug_ops = { .need = need_iommu_debug, }; +static struct page_ext *get_iommu_page_ext (phys_addr_t phys) +{ +struct page *page =phys_to_page (phys); +struct page_ext *page_ext =page_ext_get (page); + +return page_ext; +} + +static struct iommu_debug_metadata *get_iommu_data (struct page_ext *page_ext) +{ +return page_ext_data (page_ext, &page_iommu_debug_ops); +} + +static void iommu_debug_inc_page (phys_addr_t phys) +{ +struct page_ext *page_ext =get_iommu_page_ext (phys); +struct iommu_debug_metadata *d =get_iommu_data (page_ext); + +WARN_ON (atomic_inc_return_relaxed (&d->ref) <=0 ); +page_ext_put (page_ext); +} + +static void iommu_debug_dec_page (phys_addr_t phys) +{ +struct page_ext *page_ext =get_iommu_page_ext (phys); +struct iommu_debug_metadata *d =get_iommu_data (page_ext); + +WARN_ON (atomic_dec_return_relaxed (&d->ref) <0 ); +page_ext_put (page_ext); +} + +/* +* IOMMU page size doesn't have to match the CPU page size. So, we use +* the smallest IOMMU page size to refcount the pages in the vmemmap. +* That is important as both map and unmap has to use the same page size +* to update the refcount to avoid double counting the same page. +* And as we can't know from iommu_unmap() what was the original page size +* used for map, we just use the minimum supported one for both. +*/ +static size_t iommu_debug_page_size (struct iommu_domain *domain) +{ +return 1UL <<__ffs (domain->pgsize_bitmap); +} +void __iommu_debug_map (struct iommu_domain *domain,phys_addr_t phys,size_t size) { +size_t off, end; +size_t page_size =iommu_debug_page_size (domain); + +if (WARN_ON (!phys ||check_add_overflow (phys, size, &end))) +return ; + +for (off =0 ; off < size ; off += page_size) { +if (!pfn_valid (__phys_to_pfn (phys + off))) +continue ; +iommu_debug_inc_page (phys + off); + } +} + +static void __iommu_debug_update_iova (struct iommu_domain *domain, +unsigned long iova,size_t size,bool inc) +{ +size_t off, end; +size_t page_size =iommu_debug_page_size (domain); + +if (WARN_ON (check_add_overflow (iova, size, &end))) +return ; + +for (off =0 ; off < size ; off += page_size) { +phys_addr_t phys =iommu_iova_to_phys (domain, iova + off); + +if (!phys || !pfn_valid (__phys_to_pfn (phys))) +continue ; + +if (inc) +iommu_debug_inc_page (phys); +else +iommu_debug_dec_page (phys); + } }void __iommu_debug_unmap_begin (struct iommu_domain *domain,unsigned long iova,size_t size) {
drivers/iommu/iommu-debug-pagealloc.c struct page_ext_operations page_iommu_debug_ops = { · line 125
Quoted 10 lines
drivers/iommu/iommu-debug-pagealloc.c +4
-0+__iommu_debug_update_iova (domain, iova, size, false); }void __iommu_debug_unmap_end (struct iommu_domain *domain,unsigned long iova,size_t size,size_t unmapped) { +if ((unmapped == size) ||WARN_ON_ONCE (unmapped > size)) +return ; +
Looks good!
drivers/iommu/iommu-debug-pagealloc.c struct page_ext_operations page_iommu_debug_ops = { · line 130
Quoted 5 lines
drivers/iommu/iommu-debug-pagealloc.c +3
-0+/* If unmap failed, re-increment the refcount. */ +__iommu_debug_update_iova (domain, iova + unmapped, + size - unmapped, true); }
Thanks!
Quoted 5 lines
drivers/iommu/iommu-debug-pagealloc.c +0
-0void iommu_debug_init (void )Signature · 1 line
2.52.0.457.g6b5491de43-goog
Re: [PATCH v6 3/4] iommu: debug-pagealloc: Track IOMMU pages
In reply to Mostafa SalehQuoted 7 lines · Mostafa Saleh
drivers/iommu/iommu-debug-pagealloc.c +7
-0+static struct page_ext *get_iommu_page_ext (phys_addr_t phys) +{ +struct page *page =phys_to_page (phys); +struct page_ext *page_ext =page_ext_get (page); + +return page_ext; +}
drivers/iommu/iommu-debug-pagealloc.c struct page_ext_operations page_iommu_debug_ops = { · line 48
Quoted 10 lines
drivers/iommu/iommu-debug-pagealloc.c +10
-0+ +static struct iommu_debug_metadata *get_iommu_data (struct page_ext *page_ext) +{ +return page_ext_data (page_ext, &page_iommu_debug_ops); +} + +static void iommu_debug_inc_page (phys_addr_t phys) +{ +struct page_ext *page_ext =get_iommu_page_ext (phys); +struct iommu_debug_metadata *d =get_iommu_data (page_ext);
You cannot do this - phys_to_page() can only be called if we already know that phys is a struct page backed item and by the time you get here that information is lost.
Probably the only way to resolve this is to somehow pass in an iommu prot flag that can tell the difference between struct page and non-struct page addresses.
But I have to NAK this approach of blindly calling phys_to_page().
Jason
Re: [PATCH v6 3/4] iommu: debug-pagealloc: Track IOMMU pages
Quoted 24 lines · Jason Gunthorpe
Quoted 7 lines · Mostafa Saleh
drivers/iommu/iommu-debug-pagealloc.c +7
-0+static struct page_ext *get_iommu_page_ext (phys_addr_t phys) +{ +struct page *page =phys_to_page (phys); +struct page_ext *page_ext =page_ext_get (page); + +return page_ext; +}drivers/iommu/iommu-debug-pagealloc.c struct page_ext_operations page_iommu_debug_ops = { · line 48
Quoted 10 lines
drivers/iommu/iommu-debug-pagealloc.c +10
-0+ +static struct iommu_debug_metadata *get_iommu_data (struct page_ext *page_ext) +{ +return page_ext_data (page_ext, &page_iommu_debug_ops); +} + +static void iommu_debug_inc_page (phys_addr_t phys) +{ +struct page_ext *page_ext =get_iommu_page_ext (phys); +struct iommu_debug_metadata *d =get_iommu_data (page_ext);You cannot do this -
phys_to_page()can only be called if we already know that phys is a struct page backed item and by the time you get here that information is lost.
Quoted 6 lines
Probably the only way to resolve this is to somehow pass in an iommu prot flag that can tell the difference between struct page and non-struct page addresses.
But I have to NAK this approach of blindly calling
phys_to_page().
The callers to this, first will check "pfn_valid", which is the right check AFAICT (looking at similar patterns in page_owner for example).
Thanks,
Mostafa
Quoted 2 lines
Jason
Re: [PATCH v6 3/4] iommu: debug-pagealloc: Track IOMMU pages
Quoted 25 lines · Mostafa Saleh
Quoted 24 lines · Jason Gunthorpe
Quoted 7 lines · Mostafa Saleh
drivers/iommu/iommu-debug-pagealloc.c +7
-0+static struct page_ext *get_iommu_page_ext (phys_addr_t phys) +{ +struct page *page =phys_to_page (phys); +struct page_ext *page_ext =page_ext_get (page); + +return page_ext; +}drivers/iommu/iommu-debug-pagealloc.c struct page_ext_operations page_iommu_debug_ops = { · line 48
Quoted 10 lines
drivers/iommu/iommu-debug-pagealloc.c +10
-0+ +static struct iommu_debug_metadata *get_iommu_data (struct page_ext *page_ext) +{ +return page_ext_data (page_ext, &page_iommu_debug_ops); +} + +static void iommu_debug_inc_page (phys_addr_t phys) +{ +struct page_ext *page_ext =get_iommu_page_ext (phys); +struct iommu_debug_metadata *d =get_iommu_data (page_ext);You cannot do this -
phys_to_page()can only be called if we already know that phys is a struct page backed item and by the time you get here that information is lost.
Quoted 8 lines
Quoted 5 lines
Probably the only way to resolve this is to somehow pass in an iommu prot flag that can tell the difference between struct page and non-struct page addresses.
But I have to NAK this approach of blindly calling
phys_to_page().The callers to this, first will check "
pfn_valid", which is the right check AFAICT (looking at similar patterns inpage_ownerfor example).
I'm not sure pfn_valid really works in all cases, it has a number of exclusions that can be relevant when phys_addr_t can be a MMIO address..
So far we haven't been using it in the DMA paths at all, I'm not so keen to see that start..
Jason
Re: [PATCH v6 3/4] iommu: debug-pagealloc: Track IOMMU pages
Quoted 36 lines · Jason Gunthorpe
Quoted 25 lines · Mostafa Saleh
Quoted 24 lines · Jason Gunthorpe
Quoted 7 lines · Mostafa Saleh
drivers/iommu/iommu-debug-pagealloc.c +7
-0+static struct page_ext *get_iommu_page_ext (phys_addr_t phys) +{ +struct page *page =phys_to_page (phys); +struct page_ext *page_ext =page_ext_get (page); + +return page_ext; +}drivers/iommu/iommu-debug-pagealloc.c struct page_ext_operations page_iommu_debug_ops = { · line 48
Quoted 10 lines
drivers/iommu/iommu-debug-pagealloc.c +10
-0+ +static struct iommu_debug_metadata *get_iommu_data (struct page_ext *page_ext) +{ +return page_ext_data (page_ext, &page_iommu_debug_ops); +} + +static void iommu_debug_inc_page (phys_addr_t phys) +{ +struct page_ext *page_ext =get_iommu_page_ext (phys); +struct iommu_debug_metadata *d =get_iommu_data (page_ext);You cannot do this -
phys_to_page()can only be called if we already know that phys is a struct page backed item and by the time you get here that information is lost.Quoted 8 lines
Quoted 5 lines
Probably the only way to resolve this is to somehow pass in an iommu prot flag that can tell the difference between struct page and non-struct page addresses.
But I have to NAK this approach of blindly calling
phys_to_page().The callers to this, first will check "
pfn_valid", which is the right check AFAICT (looking at similar patterns inpage_ownerfor example).
Quoted 7 lines
I'm not sure
pfn_validreally works in all cases, it has a number of exclusions that can be relevant whenphys_addr_tcan be a MMIO address..So far we haven't been using it in the DMA paths at all, I'm not so keen to see that start..
But I don’t see why not. from the documentation:
/**
* pfn_valid - check if there is a valid memory map entry for a PFN
* @pfn: the page frame number to check
*
* Check if there is a valid memory map entry aka struct page for the @pfn.
* Note, that availability of the memory map entry does not imply that
* there is actual usable memory at that @pfn. The struct page may
* represent a hole or an unusable page frame.
…
That means that struct page exists, which is all what we need here.
I can see many places have the same pattern in the kernel already, for example:
- vfio_iommu_type1.c, is_invalid_reserved_pfn() which does the same check which can include MMIO and then get the page struct.
- kvm_main.c: in __kvm_vcpu_map(), it distinguishes MMIO from memory and then accesses the page struct.
Thanks,
Mostafa
Quoted 1 line
Jason
Re: [PATCH v6 3/4] iommu: debug-pagealloc: Track IOMMU pages
Quoted 11 lines · Mostafa Saleh
But I don’t see why not. from the documentation:
/**
*pfn_valid- check if there is a valid memory map entry for a PFN
* @pfn: the page frame number to check
*
* Check if there is a valid memory map entry aka struct page for the @pfn.
* Note, that availability of the memory map entry does not imply that
* there is actual usable memory at that @pfn. The struct page may
* represent a hole or an unusable page frame.
…
Quoted 1 line
That means that struct page exists, which is all what we need here.
A struct page that has never been initialize shouldn't ever be read. I don't know how that relates to page_ext, but are you really sure that is all you need?
Quoted 3 lines
I can see many places have the same pattern in the kernel already, for example:
-vfio_iommu_type1.c,is_invalid_reserved_pfn()which does the same check which can include MMIO and then get the page struct.
This whole flow is nonsensical and wrong though, I wouldn't point to it as something reliable.
Quoted 2 lines
-
kvm_main.c: in__kvm_vcpu_map(), it distinguishes MMIO from memory and then accesses the page struct.
That's sure looks sketchy to me.. Eg if CONFIG_WANT_PAGE_VIRTUAL is set and you try to feed a MMIO through through that kmap() it will explode.
KVM can argue that it doesn't work with CONFIG_WANT_PAGE_VIRTUAL but iommu cannot.
So, again, IDK, we are trying not to use pfn_valid() in the DMA code.
Jason
Re: [PATCH v6 3/4] iommu: debug-pagealloc: Track IOMMU pages
Quoted 13 lines · Jason Gunthorpe
Quoted 11 lines · Mostafa Saleh
But I don’t see why not. from the documentation:
/**
*pfn_valid- check if there is a valid memory map entry for a PFN
* @pfn: the page frame number to check
*
* Check if there is a valid memory map entry aka struct page for the @pfn.
* Note, that availability of the memory map entry does not imply that
* there is actual usable memory at that @pfn. The struct page may
* represent a hole or an unusable page frame.
…
Quoted 6 lines
Quoted 1 line
That means that struct page exists, which is all what we need here.
A struct page that has never been initialize shouldn't ever be read. I don't know how that relates to
page_ext, but are you really sure that is all you need?
AFAIU, if pfn_valid() returns true, it means the struct page is valid, and lookup_page_ext() will check that a valid page_ext exists for this entry.
So, what is missing is the NULL check for the page_ext returned, as it can be NULL even if pfn_valid() was true.
But I can't see why we shouldn't use pfn_valid() at all in that path. I don't like the approach of using the prot to check that, as the driver can be buggy which is what the santizer is defending against. If we find some CONFIGs conflicting with it, we can just express that in Kconfig and disable the santaizer in that case.
Quoted 10 lines
Quoted 3 lines
I can see many places have the same pattern in the kernel already, for example:
-vfio_iommu_type1.c,is_invalid_reserved_pfn()which does the same check which can include MMIO and then get the page struct.This whole flow is nonsensical and wrong though, I wouldn't point to it as something reliable.
Quoted 2 lines
-
kvm_main.c: in__kvm_vcpu_map(), it distinguishes MMIO from memory and then accesses the page struct.
Quoted 7 lines
That's sure looks sketchy to me.. Eg if
CONFIG_WANT_PAGE_VIRTUALis set and you try to feed a MMIO through through thatkmap()it will explode.KVM can argue that it doesn't work with
CONFIG_WANT_PAGE_VIRTUALbut iommu cannot.
WANT_PAGE_VIRTUAL seems possible in loongarch which supports KVM.
Thanks,
Mostafa
Quoted 3 lines
So, again, IDK, we are trying not to use
pfn_valid()in the DMA code.Jason
Re: [PATCH v6 3/4] iommu: debug-pagealloc: Track IOMMU pages
Quoted 27 lines · Mostafa Saleh
Quoted 13 lines · Jason Gunthorpe
Quoted 11 lines · Mostafa Saleh
But I don’t see why not. from the documentation:
/**
*pfn_valid- check if there is a valid memory map entry for a PFN
* @pfn: the page frame number to check
*
* Check if there is a valid memory map entry aka struct page for the @pfn.
* Note, that availability of the memory map entry does not imply that
* there is actual usable memory at that @pfn. The struct page may
* represent a hole or an unusable page frame.
…Quoted 6 lines
Quoted 1 line
That means that struct page exists, which is all what we need here.
A struct page that has never been initialize shouldn't ever be read. I don't know how that relates to
page_ext, but are you really sure that is all you need?AFAIU, if
pfn_valid()returns true, it means the struct page is valid, andlookup_page_ext()will check that a validpage_extexists for this entry.
So, what is missing is the NULL check for thepage_extreturned, as it can be NULL even ifpfn_valid()was true.
Quoted 5 lines
But I can't see why we shouldn't use
pfn_valid()at all in that path. I don't like the approach of using the prot to check that, as the driver can be buggy which is what the santizer is defending against. If we find some CONFIGs conflicting with it, we can just express that inKconfigand disable the santaizer in that case.
That's a fair point.
How about adding a page_ext_from_phys() kind of function that could use pfn_valid internally and has documented semantics for what it even does for "holes" in the page map?
I'd be happier to see such a well defined API than randomly adding pfn_valid() and phys_to_page() when we are trying hard to not have those things in these paths.
Quoted 8 lines
Quoted 6 lines
That's sure looks sketchy to me.. Eg if
CONFIG_WANT_PAGE_VIRTUALis set and you try to feed a MMIO through through thatkmap()it will explode.KVM can argue that it doesn't work with
CONFIG_WANT_PAGE_VIRTUALbut iommu cannot.
WANT_PAGE_VIRTUALseems possible in loongarch which supports KVM.
Yikes, maybe that configuration doesn't run KVM?
Jason
David Hildenbrand (Red Hat)
maintainerRe: [PATCH v6 3/4] iommu: debug-pagealloc: Track IOMMU pages
In reply to Mostafa SalehQuoted 16 lines · Mostafa Saleh
Quoted 15 lines · Jason Gunthorpe
Quoted 12 lines · Mostafa Saleh
But I don’t see why not. from the documentation:
/**
*pfn_valid- check if there is a valid memory map entry for a PFN
* @pfn: the page frame number to check
*
* Check if there is a valid memory map entry aka struct page for the @pfn.
* Note, that availability of the memory map entry does not imply that
* there is actual usable memory at that @pfn. The struct page may
* represent a hole or an unusable page frame.
…That means that struct page exists, which is all what we need here.
Quoted 8 lines
Quoted 4 lines
A struct page that has never been initialize shouldn't ever be read. I don't know how that relates to
page_ext, but are you really sure that is all you need?AFAIU, if
pfn_valid()returns true, it means the struct page is valid, andlookup_page_ext()will check that a validpage_extexists for this entry.
Not always. Offline memory blocks have a memory map but no page ext. We allocate the page ext at memory onlining time.
Also, I'm not sure about ZONE_DEVICE memory, very likely we never allocate a page_ext for them?
I'd assume both cases are not relevant for your use case, though.
Signature · 3 lines
Cheers David
Re: [PATCH v6 3/4] iommu: debug-pagealloc: Track IOMMU pages
Quoted 26 lines · David Hildenbrand (Red Hat)
Quoted 16 lines · Mostafa Saleh
Quoted 15 lines · Jason Gunthorpe
Quoted 12 lines · Mostafa Saleh
But I don’t see why not. from the documentation:
/**
*pfn_valid- check if there is a valid memory map entry for a PFN
* @pfn: the page frame number to check
*
* Check if there is a valid memory map entry aka struct page for the @pfn.
* Note, that availability of the memory map entry does not imply that
* there is actual usable memory at that @pfn. The struct page may
* represent a hole or an unusable page frame.
…That means that struct page exists, which is all what we need here.
Quoted 8 lines
Quoted 4 lines
A struct page that has never been initialize shouldn't ever be read. I don't know how that relates to
page_ext, but are you really sure that is all you need?AFAIU, if
pfn_valid()returns true, it means the struct page is valid, andlookup_page_ext()will check that a validpage_extexists for this entry.
Quoted 7 lines
Not always. Offline memory blocks have a memory map but no page ext. We allocate the page ext at memory onlining time.
Also, I'm not sure about
ZONE_DEVICEmemory, very likely we never allocate apage_extfor them?I'd assume both cases are not relevant for your use case, though.
They are in the sense that those PFNs can get into these routines and still need to be handled in some appropriate way..
Jason
Re: [PATCH v6 3/4] iommu: debug-pagealloc: Track IOMMU pages
In reply to David Hildenbrand (Red Hat)Quoted 27 lines · David Hildenbrand (Red Hat)
Quoted 16 lines · Mostafa Saleh
Quoted 15 lines · Jason Gunthorpe
Quoted 12 lines · Mostafa Saleh
But I don’t see why not. from the documentation:
/**
*pfn_valid- check if there is a valid memory map entry for a PFN
* @pfn: the page frame number to check
*
* Check if there is a valid memory map entry aka struct page for the @pfn.
* Note, that availability of the memory map entry does not imply that
* there is actual usable memory at that @pfn. The struct page may
* represent a hole or an unusable page frame.
…That means that struct page exists, which is all what we need here.
Quoted 8 lines
Quoted 4 lines
A struct page that has never been initialize shouldn't ever be read. I don't know how that relates to
page_ext, but are you really sure that is all you need?AFAIU, if
pfn_valid()returns true, it means the struct page is valid, andlookup_page_ext()will check that a validpage_extexists for this entry.
Quoted 8 lines
Not always. Offline memory blocks have a memory map but no page ext. We allocate the page ext at memory onlining time.
Also, I'm not sure about
ZONE_DEVICEmemory, very likely we never allocate apage_extfor them?I'd assume both cases are not relevant for your use case, though.
From my understanding, in that case, page_ext_get() will return NULL.
So, as long as struct page exists, page_ext_get won't misbehave.
I am not sure about offline memory, but MMIO can be used. We use pfn_valid() before getting the struct page that we pass to page_ext. Would that be OK?
Thanks,
Mostafa
Quoted 4 lines
--
CheersDavid
David Hildenbrand (Red Hat)
maintainerRe: [PATCH v6 3/4] iommu: debug-pagealloc: Track IOMMU pages
Quoted 38 lines · Mostafa Saleh
Quoted 27 lines · David Hildenbrand (Red Hat)
Quoted 16 lines · Mostafa Saleh
Quoted 15 lines · Jason Gunthorpe
Quoted 12 lines · Mostafa Saleh
But I don’t see why not. from the documentation:
/**
*pfn_valid- check if there is a valid memory map entry for a PFN
* @pfn: the page frame number to check
*
* Check if there is a valid memory map entry aka struct page for the @pfn.
* Note, that availability of the memory map entry does not imply that
* there is actual usable memory at that @pfn. The struct page may
* represent a hole or an unusable page frame.
…That means that struct page exists, which is all what we need here.
Quoted 8 lines
Quoted 4 lines
A struct page that has never been initialize shouldn't ever be read. I don't know how that relates to
page_ext, but are you really sure that is all you need?AFAIU, if
pfn_valid()returns true, it means the struct page is valid, andlookup_page_ext()will check that a validpage_extexists for this entry.Quoted 8 lines
Not always. Offline memory blocks have a memory map but no page ext. We allocate the page ext at memory onlining time.
Also, I'm not sure about
ZONE_DEVICEmemory, very likely we never allocate apage_extfor them?I'd assume both cases are not relevant for your use case, though.
Quoted 7 lines
From my understanding, in that case,
page_ext_get()will return NULL.So, as long as struct page exists,
page_ext_getwon't misbehave.I am not sure about offline memory, but MMIO can be used. We use
pfn_valid()before getting the struct page that we pass topage_ext. Would that be OK?
It's tricky. If you look at lookup_page_ext(), it relies on extracting the pfn+nid from the "struct page".
If the "struct page" is uninitialized (e.g., offline memory) that cannot possibly work, as it could just give you random garbage.
(note that there are two implementations of lookup_page_ext(), both extracting the PFN but only one extracting the NID).
Signature · 3 lines
Cheers David
Re: [PATCH v6 3/4] iommu: debug-pagealloc: Track IOMMU pages
Quoted 47 lines · David Hildenbrand (Red Hat)
Quoted 38 lines · Mostafa Saleh
Quoted 27 lines · David Hildenbrand (Red Hat)
Quoted 16 lines · Mostafa Saleh
Quoted 15 lines · Jason Gunthorpe
Quoted 12 lines · Mostafa Saleh
But I don’t see why not. from the documentation:
/**
*pfn_valid- check if there is a valid memory map entry for a PFN
* @pfn: the page frame number to check
*
* Check if there is a valid memory map entry aka struct page for the @pfn.
* Note, that availability of the memory map entry does not imply that
* there is actual usable memory at that @pfn. The struct page may
* represent a hole or an unusable page frame.
…That means that struct page exists, which is all what we need here.
Quoted 8 lines
Quoted 4 lines
A struct page that has never been initialize shouldn't ever be read. I don't know how that relates to
page_ext, but are you really sure that is all you need?AFAIU, if
pfn_valid()returns true, it means the struct page is valid, andlookup_page_ext()will check that a validpage_extexists for this entry.Quoted 8 lines
Not always. Offline memory blocks have a memory map but no page ext. We allocate the page ext at memory onlining time.
Also, I'm not sure about
ZONE_DEVICEmemory, very likely we never allocate apage_extfor them?I'd assume both cases are not relevant for your use case, though.
Quoted 7 lines
From my understanding, in that case,
page_ext_get()will return NULL.So, as long as struct page exists,
page_ext_getwon't misbehave.I am not sure about offline memory, but MMIO can be used. We use
pfn_valid()before getting the struct page that we pass topage_ext. Would that be OK?
Quoted 9 lines
It's tricky. If you look at
lookup_page_ext(), it relies on extracting the pfn+nid from the "struct page".If the "struct page" is uninitialized (e.g., offline memory) that cannot possibly work, as it could just give you random garbage.
(note that there are two implementations of
lookup_page_ext(), both extracting the PFN but only one extracting the NID).
I see, thanks for the clarification.
I see that pfn_to_online_page() exits, which seems to handle online memory and ZONE_DEVICE. Would that be a suitable alternative?
Would you have a problem if we added a new function in page_ext "page_ext_from_phys()" as Jason suggested?
Thanks,
Mostafa
Thanks,
Mostafa
Quoted 5 lines
--
CheersDavid
Re: [PATCH v6 3/4] iommu: debug-pagealloc: Track IOMMU pages
Quoted 2 lines · Mostafa Saleh
Would you have a problem if we added a new function in
page_ext"page_ext_from_phys()" as Jason suggested?
Given the hidden complexity that David just pointed out I think this is essential to encapsulate it in a function.
The function must be able to accept any phys_addr_t and safely returns NULL if page_ext cannot be used for some reason.
Jason
David Hildenbrand (Red Hat)
maintainerRe: [PATCH v6 3/4] iommu: debug-pagealloc: Track IOMMU pages
Quoted 9 lines · Jason Gunthorpe
Quoted 2 lines · Mostafa Saleh
Would you have a problem if we added a new function in
page_ext"page_ext_from_phys()" as Jason suggested?Given the hidden complexity that David just pointed out I think this is essential to encapsulate it in a function.
The function must be able to accept any
phys_addr_tand safely returns NULL ifpage_extcannot be used for some reason.
Right. I think pfn_to_online_page() is the appropriate check.
@Mostafa, I saw you already sent a v7, is the pfn_to_online_page() stuff handled in there?
Signature · 3 lines
Cheers David
Re: [PATCH v6 3/4] iommu: debug-pagealloc: Track IOMMU pages
Quoted 9 lines · David Hildenbrand (Red Hat)
Quoted 7 lines · Jason Gunthorpe
Quoted 2 lines · Mostafa Saleh
Would you have a problem if we added a new function in
page_ext"page_ext_from_phys()" as Jason suggested?Given the hidden complexity that David just pointed out I think this is essential to encapsulate it in a function.
Quoted 8 lines
Quoted 2 lines
The function must be able to accept any
phys_addr_tand safely returns NULL ifpage_extcannot be used for some reason.Right. I think
pfn_to_online_page()is the appropriate check.@Mostafa, I saw you already sent a v7, is the
pfn_to_online_page()stuff handled in there?
Yes, it's in the first patch in function page_ext_get_phys(). After spending some time looking into it, that was my conclusion also.
Thanks,
Mostafa
Quoted 4 lines
--
CheersDavid
PATCH v6 4/4 iommu: debug-pagealloc: Check mapped/unmapped kernel memory
In reply to Mostafa SalehNow, as the page_ext holds count of IOMMU mappings, we can use it to assert that any page allocated/freed is indeed not in the IOMMU.
The sanitizer doesn’t protect against mapping/unmapping during this period. However, that’s less harmful as the page is not used by the kernel.
--- drivers/iommu/iommu-debug-pagealloc.c | 23 +++++++++++++++++++++++ include/linux/iommu-debug-pagealloc.h | 14 ++++++++++++++ include/linux/mm.h | 5 +++++ 3 files changed, 42 insertions(+)
drivers/iommu/iommu-debug-pagealloc.c +23 -0
@@ -9,6 +9,7 @@#include <linux/iommu-debug-pagealloc.h> #include <linux/kernel.h> #include <linux/page_ext.h> +#include <linux/page_owner.h> #include "iommu-priv.h"
@@ -73,6 +74,28 @@ static size_t iommu_debug_page_size(struct iommu_domain *domain)return 1UL <<__ffs (domain->pgsize_bitmap); } +static bool iommu_debug_page_count (const struct page *page) +{ +unsigned int ref; +struct page_ext *page_ext =page_ext_get (page); +struct iommu_debug_metadata *d =get_iommu_data (page_ext); + + ref =atomic_read (&d->ref); +page_ext_put (page_ext); +return ref !=0 ; +} + +void __iommu_debug_check_unmapped (const struct page *page,int numpages) +{ +while (numpages--) { +if (WARN_ON (iommu_debug_page_count (page))) { +pr_warn ("iommu: Detected page leak!\n" ); +dump_page_owner (page); + } + page++; + } +} +void __iommu_debug_map (struct iommu_domain *domain,phys_addr_t phys,size_t size) {size_t off, end;
include/linux/iommu-debug-pagealloc.h +14 -0
@@ -13,6 +13,20 @@ DECLARE_STATIC_KEY_FALSE(iommu_debug_initialized);extern struct page_ext_operations page_iommu_debug_ops; +void __iommu_debug_check_unmapped (const struct page *page,int numpages); + +static inline void iommu_debug_check_unmapped (const struct page *page,int numpages) +{ +if (static_branch_unlikely (&iommu_debug_initialized)) +__iommu_debug_check_unmapped (page, numpages); +} + +#else +static inline void iommu_debug_check_unmapped (const struct page *page, +int numpages) +{ +} +#endif /* CONFIG_IOMMU_DEBUG_PAGEALLOC */ #endif /* __LINUX_IOMMU_DEBUG_PAGEALLOC_H */
include/linux/mm.h +5 -0
@@ -36,6 +36,7 @@#include <linux/rcuwait.h> #include <linux/bitmap.h> #include <linux/bitops.h> +#include <linux/iommu-debug-pagealloc.h> struct mempolicy ;struct anon_vma ;
@@ -4133,12 +4134,16 @@ extern void __kernel_map_pages(struct page *page, int numpages, int enable);#ifdef CONFIG_DEBUG_PAGEALLOC static inline void debug_pagealloc_map_pages (struct page *page,int numpages) { +iommu_debug_check_unmapped (page, numpages); +if (debug_pagealloc_enabled_static ())__kernel_map_pages (page, numpages,1 ); }static inline void debug_pagealloc_unmap_pages (struct page *page,int numpages) { +iommu_debug_check_unmapped (page, numpages); +if (debug_pagealloc_enabled_static ())__kernel_map_pages (page, numpages,0 ); }
Signature · 1 line
2.52.0.457.g6b5491de43-goog
Re: [PATCH v6 4/4] iommu: debug-pagealloc: Check mapped/unmapped kernel memory
Quoted 7 lines · Mostafa Saleh
Now, as the
page_extholds count of IOMMU mappings, we can use it to assert that any page allocated/freed is indeed not in the IOMMU.The sanitizer doesn’t protect against mapping/unmapping during this period. However, that’s less harmful as the page is not used by the kernel.
Quoted 9 lines
Reviewed-by Samiullah Khawaja <[email protected]>Reviewed-by Lu Baolu <[email protected]>Signed-off-by Mostafa Saleh <[email protected]>--- drivers/iommu/iommu-debug-pagealloc.c | 23 +++++++++++++++++++++++ include/linux/iommu-debug-pagealloc.h | 14 ++++++++++++++ include/linux/mm.h | 5 +++++ 3 files changed, 42 insertions(+)
Thanks,
Praan
Quoted 99 lines
drivers/iommu/iommu-debug-pagealloc.c +23
-0@@ -9,6 +9,7 @@#include <linux/iommu-debug-pagealloc.h> #include <linux/kernel.h> #include <linux/page_ext.h> +#include <linux/page_owner.h> #include "iommu-priv.h" @@ -73,6 +74,28 @@ static size_t iommu_debug_page_size(struct iommu_domain *domain)return 1UL <<__ffs (domain->pgsize_bitmap); } +static bool iommu_debug_page_count (const struct page *page) +{ +unsigned int ref; +struct page_ext *page_ext =page_ext_get (page); +struct iommu_debug_metadata *d =get_iommu_data (page_ext); + + ref =atomic_read (&d->ref); +page_ext_put (page_ext); +return ref !=0 ; +} + +void __iommu_debug_check_unmapped (const struct page *page,int numpages) +{ +while (numpages--) { +if (WARN_ON (iommu_debug_page_count (page))) { +pr_warn ("iommu: Detected page leak!\n" ); +dump_page_owner (page); + } + page++; + } +} +void __iommu_debug_map (struct iommu_domain *domain,phys_addr_t phys,size_t size) {size_t off, end;include/linux/iommu-debug-pagealloc.h +14
-0@@ -13,6 +13,20 @@ DECLARE_STATIC_KEY_FALSE(iommu_debug_initialized);extern struct page_ext_operations page_iommu_debug_ops; +void __iommu_debug_check_unmapped (const struct page *page,int numpages); + +static inline void iommu_debug_check_unmapped (const struct page *page,int numpages) +{ +if (static_branch_unlikely (&iommu_debug_initialized)) +__iommu_debug_check_unmapped (page, numpages); +} + +#else +static inline void iommu_debug_check_unmapped (const struct page *page, +int numpages) +{ +} +#endif /* CONFIG_IOMMU_DEBUG_PAGEALLOC */ #endif /* __LINUX_IOMMU_DEBUG_PAGEALLOC_H */ include/linux/mm.h +5
-0@@ -36,6 +36,7 @@#include <linux/rcuwait.h> #include <linux/bitmap.h> #include <linux/bitops.h> +#include <linux/iommu-debug-pagealloc.h> struct mempolicy ;struct anon_vma ;@@ -4133,12 +4134,16 @@ extern void __kernel_map_pages(struct page *page, int numpages, int enable);#ifdef CONFIG_DEBUG_PAGEALLOC static inline void debug_pagealloc_map_pages (struct page *page,int numpages) { +iommu_debug_check_unmapped (page, numpages); +if (debug_pagealloc_enabled_static ())__kernel_map_pages (page, numpages,1 ); }static inline void debug_pagealloc_unmap_pages (struct page *page,int numpages) { +iommu_debug_check_unmapped (page, numpages); +if (debug_pagealloc_enabled_static ())__kernel_map_pages (page, numpages,0 ); }Signature · 1 line
2.52.0.457.g6b5491de43-goog
Quoted 6 lines · Mostafa Saleh
Mostafa Saleh (4):
iommu: Addpage_extforIOMMU_DEBUG_PAGEALLOC
iommu: Add calls forIOMMU_DEBUG_PAGEALLOC
iommu: debug-pagealloc: Track IOMMU pages
iommu: debug-pagealloc: Check mapped/unmapped kernel memory
Quoted 12 lines
.../admin-guide/kernel-parameters.txt | 9 + drivers/iommu/Kconfig | 19 ++ drivers/iommu/Makefile | 1 + drivers/iommu/iommu-debug-pagealloc.c | 167 ++++++++++++++++++ drivers/iommu/iommu-priv.h | 58 ++++++ drivers/iommu/iommu.c | 11 +- include/linux/iommu-debug-pagealloc.h | 32 ++++ include/linux/mm.h | 5 + mm/page_ext.c | 4 + 9 files changed, 304 insertions(+), 2 deletions(-) create mode 100644 drivers/iommu/iommu-debug-pagealloc.c create mode 100644 include/linux/iommu-debug-pagealloc.h
Applied, thanks.
Quoted 21 lines · Jörg Rödel
Quoted 6 lines · Mostafa Saleh
Mostafa Saleh (4):
iommu: Addpage_extforIOMMU_DEBUG_PAGEALLOC
iommu: Add calls forIOMMU_DEBUG_PAGEALLOC
iommu: debug-pagealloc: Track IOMMU pages
iommu: debug-pagealloc: Check mapped/unmapped kernel memoryQuoted 12 lines
.../admin-guide/kernel-parameters.txt | 9 + drivers/iommu/Kconfig | 19 ++ drivers/iommu/Makefile | 1 + drivers/iommu/iommu-debug-pagealloc.c | 167 ++++++++++++++++++ drivers/iommu/iommu-priv.h | 58 ++++++ drivers/iommu/iommu.c | 11 +- include/linux/iommu-debug-pagealloc.h | 32 ++++ include/linux/mm.h | 5 + mm/page_ext.c | 4 + 9 files changed, 304 insertions(+), 2 deletions(-) create mode 100644 drivers/iommu/iommu-debug-pagealloc.c create mode 100644 include/linux/iommu-debug-pagealloc.h
Quoted 1 line
Applied, thanks.
Thanks a lot! Based on the discussion on patch 3, I might need to respin a new version.
Thanks,
Mostafa