Fix stale IOTLB entries for kernel address space

From: Dave Hansen <[email protected]>

The page tables used to map the kernel and userspace often have very different handling rules. There are frequently *_kernel() variants of functions just for kernel page tables. That's not great and has lead to code duplication.

Instead of having completely separate call paths, allow a 'ptdesc' to be marked as being for kernel mappings. Introduce helpers to set and clear this status.

Note: this uses the PG_referenced bit. Page flags are a great fit for this since it is truly a single bit of information. Use PG_referenced itself because it's a fairly benign flag (as opposed to things like PG_lock). It's also (according to Willy) unlikely to go away any time soon.

PG_referenced is not in PAGE_FLAGS_CHECK_AT_FREE. It does not need to be cleared before freeing the page, and pages coming out of the allocator should have it cleared. Regardless, introduce an API to clear it anyway. Having symmetry in the API makes it easier to change the underlying implementation later, like if there was a need to move to a PAGE_FLAGS_CHECK_AT_FREE bit.

Signed-off-by Dave Hansen <[email protected]>

Signed-off-by Lu Baolu <[email protected]>

Reviewed-by Jason Gunthorpe <[email protected]>

Reviewed-by Kevin Tian <[email protected]>

https://lore.kernel.org/all/[email protected]/

include/linux/mm.h pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, · line 3206

Quoted 3 lines · Lu Baolu

include/linux/mm.h +1 -2

-#if defined(CONFIG_SPLIT_PMD_PTLOCKS)
-
+#if defined(CONFIG_SPLIT_PMD_PTLOCKS) || defined(CONFIG_X86_64)

What's with the #ifdef munging? It's not mentioned in the changelog.

I went looking at this because pmd_free_pte_page() is right in the middle of the action on this reported use after free:

Quoted 1 line

https://lore.kernel.org/all/[email protected]/

so something fishy is going on. It would be great to narrow that report down to a _specific_ patch in the series.

From: Dave Hansen <[email protected]>

There are a billion ways to refer to a physical memory address. One of the x86 PMD freeing code location chooses to use a 'pte_t *' to point to a PMD page and then call a PTE-specific freeing function for it. That's a bit wonky.

Just use a 'struct ptdesc *' instead. Its entire purpose is to refer to page table pages. It also means being able to remove an explicit cast.

Right now, pte_free_kernel() is a one-liner that calls pagetable_dtor_free(). Effectively, all this patch does is remove one superfluous __pa(__va(paddr)) conversion and then call pagetable_dtor_free() directly instead of through a helper.

Signed-off-by Dave Hansen <[email protected]>

Signed-off-by Lu Baolu <[email protected]>

Reviewed-by Jason Gunthorpe <[email protected]>

Reviewed-by Kevin Tian <[email protected]>

---
  include/linux/mm.h    |  6 ++++--
  arch/x86/mm/pgtable.c | 12 ++++++------
  2 files changed, 10 insertions(+), 8 deletions(-)

include/linux/mm.h +0 -0

index 15ce0c415d36..94e2ec6c5685 100644

include/linux/mm.h +1 -0

  static inline struct page *pmd_pgtable_page(pmd_t *pmd)
  {
  	unsigned long mask = ~(PTRS_PER_PMD * sizeof(pmd_t) - 1);

@@ -3215,6 +3214,9 @@ static inline struct ptdesc *pmd_ptdesc(pmd_t *pmd)
  {
  	return page_ptdesc(pmd_pgtable_page(pmd));
  }
+#endif

From: Dave Hansen <[email protected]>

This introduces a conditional asynchronous mechanism, enabled by CONFIG_ASYNC_KERNEL_PGTABLE_FREE. When enabled, this mechanism defers the freeing of pages that are used as page tables for kernel address mappings. These pages are now queued to a work struct instead of being freed immediately.

This deferred freeing allows for batch-freeing of page tables, providing a safe context for performing a single expensive operation (TLB flush) for a batch of kernel page tables instead of performing that expensive operation for each page table.

On x86, CONFIG_ASYNC_KERNEL_PGTABLE_FREE is selected if CONFIG_IOMMU_SVA is enabled, because both Intel and AMD IOMMU architectures could potentially cache kernel page table entries in their paging structure cache, regardless of the permission.

Signed-off-by Dave Hansen <[email protected]>

Signed-off-by Lu Baolu <[email protected]>

Reviewed-by Jason Gunthorpe <[email protected]>

Reviewed-by Kevin Tian <[email protected]>

---
  arch/x86/Kconfig     |  1 +
  mm/Kconfig           |  3 +++
  include/linux/mm.h   | 16 +++++++++++++---
  mm/pgtable-generic.c | 37 +++++++++++++++++++++++++++++++++++++
  4 files changed, 54 insertions(+), 3 deletions(-)

arch/x86/Kconfig +0 -0

index fa3b616af03a..ded29ee848fd 100644

Quoted 13 lines · Lu Baolu

From: Dave Hansen <[email protected]>

This introduces a conditional asynchronous mechanism, enabled by CONFIG_ASYNC_KERNEL_PGTABLE_FREE. When enabled, this mechanism defers the freeing of pages that are used as page tables for kernel address mappings. These pages are now queued to a work struct instead of being freed immediately.

This deferred freeing allows for batch-freeing of page tables, providing a safe context for performing a single expensive operation (TLB flush) for a batch of kernel page tables instead of performing that expensive operation for each page table.

Quoted 4 lines

On x86, CONFIG_ASYNC_KERNEL_PGTABLE_FREE is selected if CONFIG_IOMMU_SVA is enabled, because both Intel and AMD IOMMU architectures could potentially cache kernel page table entries in their paging structure cache, regardless of the permission.

See below, I assume this is patch #7 material.

Quoted 12 lines

Signed-off-by Dave Hansen <[email protected]>

Signed-off-by Lu Baolu <[email protected]>

Reviewed-by Jason Gunthorpe <[email protected]>

Reviewed-by Kevin Tian <[email protected]>

---
  arch/x86/Kconfig     |  1 +
  mm/Kconfig           |  3 +++
  include/linux/mm.h   | 16 +++++++++++++--- mm/pgtable-generic.c | 37 +++++++++++++++++++++++++++++++++++++

  4 files changed, 54 insertions(+), 3 deletions(-)

Quoted 9 lines

arch/x86/Kconfig +0 -0

index fa3b616af03a..ded29ee848fd 100644
@@ -279,6 +279,7 @@ config X86

      select HAVE_PCI
      select HAVE_PERF_REGS
      select HAVE_PERF_USER_STACK_DUMP
+    select ASYNC_KERNEL_PGTABLE_FREE     if IOMMU_SVA

That should belong into patch #7, no?

page last free pid 5965 tgid 5964 stack trace:
 reset_page_owner include/linux/page_owner.h:25 [inline]
 free_pages_prepare mm/page_alloc.c:1394 [inline]
 __free_frozen_pages+0xbc4/0xd30 mm/page_alloc.c:2906
 pmd_free_pte_page+0xa1/0xc0 arch/x86/mm/pgtable.c:783
 vmap_try_huge_pmd mm/vmalloc.c:158 [inline]

Here's the part that confuses me:

Quoted 6 lines · syzbot ci

page last free pid 5965 tgid 5964 stack trace:
  reset_page_owner include/linux/page_owner.h:25 [inline]
  free_pages_prepare mm/page_alloc.c:1394 [inline]
  __free_frozen_pages+0xbc4/0xd30 mm/page_alloc.c:2906
  pmd_free_pte_page+0xa1/0xc0 arch/x86/mm/pgtable.c:783
  vmap_try_huge_pmd mm/vmalloc.c:158 [inline]

...

So, vmap_try_huge_pmd() did a pmd_free_pte_page(). Yet, somehow, the PMD stuck around so that it *could* be used after being freed. It _looks_ like pmd_free_pte_page() freed the page, returned 0, and made vmap_try_huge_pmd() return early, skipping the pmd pmd_set_huge().

But I don't know how that could possibly happen.

while, in this patch, we need the page descriptor that a pmd entry points to. Perhaps we should roll back to the previous approach used in v5?

Quoted 9 lines · Dave Hansen

Here's the part that confuses me:

Quoted 6 lines · syzbot ci

page last free pid 5965 tgid 5964 stack trace:
   reset_page_owner include/linux/page_owner.h:25 [inline]
   free_pages_prepare mm/page_alloc.c:1394 [inline]
   __free_frozen_pages+0xbc4/0xd30 mm/page_alloc.c:2906
   pmd_free_pte_page+0xa1/0xc0 arch/x86/mm/pgtable.c:783
   vmap_try_huge_pmd mm/vmalloc.c:158 [inline]

Quoted 8 lines

...

So, vmap_try_huge_pmd() did a pmd_free_pte_page(). Yet, somehow, the PMD stuck around so that it *could* be used after being freed. It _looks_ like pmd_free_pte_page() freed the page, returned 0, and made vmap_try_huge_pmd() return early, skipping the pmd pmd_set_huge().

But I don't know how that could possibly happen.

The reported issue is only related to this patch:

- [PATCH v6 3/7] x86/mm: Use 'ptdesc' when freeing PMD pages

It appears that the pmd_ptdesc() helper can't be used directly here in this patch. pmd_ptdesc() retrieves the page table page that the PMD entry resides in:

static inline struct page *pmd_pgtable_page(pmd_t *pmd)

{
          unsigned long mask = ~(PTRS_PER_PMD * sizeof(pmd_t) - 1);
          return virt_to_page((void *)((unsigned long) pmd & mask));
}

static inline struct ptdesc *pmd_ptdesc(pmd_t *pmd)

{
          return page_ptdesc(pmd_pgtable_page(pmd));
}

while, in this patch, we need the page descriptor that a pmd entry points to.

Quoted 38 lines · Baolu Lu

Quoted 9 lines · Dave Hansen

Here's the part that confuses me:

Quoted 6 lines · syzbot ci

page last free pid 5965 tgid 5964 stack trace:
   reset_page_owner include/linux/page_owner.h:25 [inline]
   free_pages_prepare mm/page_alloc.c:1394 [inline]
   __free_frozen_pages+0xbc4/0xd30 mm/page_alloc.c:2906
   pmd_free_pte_page+0xa1/0xc0 arch/x86/mm/pgtable.c:783
   vmap_try_huge_pmd mm/vmalloc.c:158 [inline]

Quoted 8 lines

...

So, vmap_try_huge_pmd() did a pmd_free_pte_page(). Yet, somehow, the PMD stuck around so that it *could* be used after being freed. It _looks_ like pmd_free_pte_page() freed the page, returned 0, and made vmap_try_huge_pmd() return early, skipping the pmd pmd_set_huge().

But I don't know how that could possibly happen.

The reported issue is only related to this patch:

- [PATCH v6 3/7] x86/mm: Use 'ptdesc' when freeing PMD pages

It appears that the pmd_ptdesc() helper can't be used directly here in this patch. pmd_ptdesc() retrieves the page table page that the PMD entry resides in:

static inline struct page *pmd_pgtable_page(pmd_t *pmd)
{
          unsigned long mask = ~(PTRS_PER_PMD * sizeof(pmd_t) - 1); return virt_to_page((void *)((unsigned long) pmd & mask)); }

static inline struct ptdesc *pmd_ptdesc(pmd_t *pmd) {
          return page_ptdesc(pmd_pgtable_page(pmd)); }

Quoted 2 lines

while, in this patch, we need the page descriptor that a pmd entry points to.

Ah. But that's just pointing at a leaf page table, right?

Quoted 39 lines · David Hildenbrand

Quoted 38 lines · Baolu Lu

Quoted 9 lines · Dave Hansen

Here's the part that confuses me:

Quoted 6 lines · syzbot ci

page last free pid 5965 tgid 5964 stack trace:
    reset_page_owner include/linux/page_owner.h:25 [inline]
    free_pages_prepare mm/page_alloc.c:1394 [inline]
    __free_frozen_pages+0xbc4/0xd30 mm/page_alloc.c:2906
    pmd_free_pte_page+0xa1/0xc0 arch/x86/mm/pgtable.c:783
    vmap_try_huge_pmd mm/vmalloc.c:158 [inline]

Quoted 8 lines

...

So, vmap_try_huge_pmd() did a pmd_free_pte_page(). Yet, somehow, the PMD stuck around so that it *could* be used after being freed. It _looks_ like pmd_free_pte_page() freed the page, returned 0, and made vmap_try_huge_pmd() return early, skipping the pmd pmd_set_huge().

But I don't know how that could possibly happen.

The reported issue is only related to this patch:

- [PATCH v6 3/7] x86/mm: Use 'ptdesc' when freeing PMD pages

It appears that the pmd_ptdesc() helper can't be used directly here in this patch. pmd_ptdesc() retrieves the page table page that the PMD entry resides in:

static inline struct page *pmd_pgtable_page(pmd_t *pmd)
{
          unsigned long mask = ~(PTRS_PER_PMD * sizeof(pmd_t) - 1); return virt_to_page((void *)((unsigned long) pmd & mask));
}

static inline struct ptdesc *pmd_ptdesc(pmd_t *pmd) {
          return page_ptdesc(pmd_pgtable_page(pmd));
}

Quoted 4 lines

Quoted 2 lines

while, in this patch, we need the page descriptor that a pmd entry points to.

Ah. But that's just pointing at a leaf page table, right?

Yes, that points to a leaf page table.

This proposes a fix for a security vulnerability related to IOMMU Shared Virtual Addressing (SVA). In an SVA context, an IOMMU can cache kernel page table entries. When a kernel page table page is freed and reallocated for another purpose, the IOMMU might still hold stale, incorrect entries. This can be exploited to cause a use-after-free or write-after-free condition, potentially leading to privilege escalation or data corruption.

This solution introduces a deferred freeing mechanism for kernel page table pages, which provides a safe window to notify the IOMMU to invalidate its caches before the page is reused.

 arch/x86/Kconfig              |  1 +
 mm/Kconfig                    |  3 ++
 include/asm-generic/pgalloc.h | 18 +++++++++
 include/linux/iommu.h         |  4 ++
 include/linux/mm.h            | 71 ++++++++++++++++++++++++++++++++---
 arch/x86/mm/init_64.c         |  2 +-
 arch/x86/mm/pat/set_memory.c  |  2 +-
 arch/x86/mm/pgtable.c         | 12 +++---
 drivers/iommu/iommu-sva.c     | 29 +++++++++++++-
 mm/pgtable-generic.c          | 39 +++++++++++++++++++
 10 files changed, 167 insertions(+), 14 deletions(-)

Quoted 7 lines · On Tue, 14 Oct 2025 21:04:30 +0800 Lu Baolu <[email protected]> wrote:

This proposes a fix for a security vulnerability related to IOMMU Shared Virtual Addressing (SVA). In an SVA context, an IOMMU can cache kernel page table entries. When a kernel page table page is freed and reallocated for another purpose, the IOMMU might still hold stale, incorrect entries. This can be exploited to cause a use-after-free or write-after-free condition, potentially leading to privilege escalation or data corruption.

Is only x86 affected?

Quoted 4 lines

This solution introduces a deferred freeing mechanism for kernel page table pages, which provides a safe window to notify the IOMMU to invalidate its caches before the page is reused.

Thanks for working on this.

Can we expect any performance impact from this? Have any measurements been performed?

Only [7/7] has a cc:stable, even though that patch is not at all backportable. Please give some thought and suggestions regarding whether you think we should backport this into earlier kernels.

If "yes" then the size and scope of the series looks problematic. Is it possible to put together something simple and expedient just to plug the hole in older kernels?

Quoted 12 lines

  arch/x86/Kconfig              |  1 +
  mm/Kconfig                    |  3 ++
  include/asm-generic/pgalloc.h | 18 +++++++++
  include/linux/iommu.h         |  4 ++
  include/linux/mm.h            | 71 ++++++++++++++++++++++++++++++++---
  arch/x86/mm/init_64.c         |  2 +-
  arch/x86/mm/pat/set_memory.c  |  2 +-
  arch/x86/mm/pgtable.c         | 12 +++---
  drivers/iommu/iommu-sva.c     | 29 +++++++++++++-
  mm/pgtable-generic.c          | 39 +++++++++++++++++++
  10 files changed, 167 insertions(+), 14 deletions(-)

It isn't obvious which tree should carry this. Were you thinking the x86 tree?

Quoted 6 lines · Andrew Morton

Quoted 3 lines

This solution introduces a deferred freeing mechanism for kernel page table pages, which provides a safe window to notify the IOMMU to invalidate its caches before the page is reused.

Thanks for working on this.

Quoted 2 lines

Can we expect any performance impact from this?  Have any measurements been performed?

This change only defers page table page freeing, allows for batch- freeing of page table pages, and notifies the IOMMU driver to invalidate the related caches. It doesn't impose any overhead in any critical path; therefore, I don't see any potential performance impact.

Quoted 3 lines

Only [7/7] has a cc:stable, even though that patch is not at all backportable.  Please give some thought and suggestions regarding whether you think we should backport this into earlier kernels.

Yes. We should backport this series to stable kernels.

Quoted 3 lines

If "yes" then the size and scope of the series looks problematic.  Is it possible to put together something simple and expedient just to plug the hole in older kernels?

Squashing some patches is one way. But would it be workable to backport this series manually? Say, could we send a pull request to the stable mailing list after this series has landed?

Quoted 11 lines

Quoted 3 lines

Only [7/7] has acc:stable, even though that patch is not at all backportable.  Please give some thought and suggestions regarding whether you think we should backport this into earlier kernels.

Yes. We should backport this series to stable kernels.

Quoted 3 lines

If "yes" then the size and scope of the series looks problematic.  Is it possible to put together something simple and expedient just to plug the hole in older kernels?

Squashing some patches is one way. But would it be workable to backport this series manually? Say, could we send a pull request to the stable mailing list after this series has landed?

I honestly think we should just disable SVA in old kernels at compile time, or at least default it to be disabled at runtime. That's the simplest thing.

The other alternative is to have arch_vmap_pmd_supported() return false when SVA is active, or maybe when it's supported on the platform.

Either of those are 10-ish lines of code and easy to backport.

Quoted 15 lines · Dave Hansen

Quoted 11 lines

Quoted 3 lines

Only [7/7] has acc:stable, even though that patch is not at all backportable.  Please give some thought and suggestions regarding whether you think we should backport this into earlier kernels.

Yes. We should backport this series to stable kernels.

Quoted 3 lines

If "yes" then the size and scope of the series looks problematic.  Is it possible to put together something simple and expedient just to plug the hole in older kernels?

Squashing some patches is one way. But would it be workable to backport this series manually? Say, could we send a pull request to the stable mailing list after this series has landed?

I honestly think we should just disable SVA in old kernels at compile time, or at least default it to be disabled at runtime. That's the simplest thing.

Quoted 4 lines

The other alternative is to have arch_vmap_pmd_supported() return false when SVA is active, or maybe when it's supported on the platform.

Either of those are 10-ish lines of code and easy to backport.

Hi iommu folks, any insights on this?

Quoted 5 lines

Quoted 4 lines

The other alternative is to have arch_vmap_pmd_supported() return false when SVA is active, or maybe when it's supported on the platform.

Either of those are 10-ish lines of code and easy to backport.

Hi iommu folks, any insights on this?

IDK, the only SVA user on x86 I know is IDXD, so if you do the above plan you break IDXD in all stable kernels. Doesn't sound OK?

I'm thinking that even messing with arch_vmap_pmd_supported() would be suboptimal. The easiest thing is to just stick the attached patch in stable kernels and disable SVA at compile time.

Quoted 3 lines · Dave Hansen

I'm thinking that even messing with arch_vmap_pmd_supported() would be suboptimal. The easiest thing is to just stick the attached patch in stable kernels and disable SVA at compile time.

Gah, please just ignore the hunks in that patch other than the drivers/iommu/intel/Kconfig one.

Quoted 7 lines · Jason Gunthorpe

Quoted 5 lines

Quoted 4 lines

The other alternative is to have arch_vmap_pmd_supported() return false when SVA is active, or maybe when it's supported on the platform.

Either of those are 10-ish lines of code and easy to backport.

Hi iommu folks, any insights on this?

IDK, the only SVA user on x86 I know is IDXD, so if you do the above plan you break IDXD in all stable kernels. Doesn't sound OK?

Vinicius, any thoughts on this?

I'm thinking that even messing with arch_vmap_pmd_supported() would be suboptimal. The easiest thing is to just stick the attached patch in stable kernels and disable SVA at compile time.

There just aren't enough SVA users out in the wild to justify more complexity than this.

arch/x86/entry/vsyscall/vsyscall_64.c +7 -3

index c9103a6fa06e..0b0e0283994f 100644
@@ -124,7 +124,8 @@ bool emulate_vsyscall(unsigned long error_code,
 	if ((error_code & (X86_PF_WRITE | X86_PF_USER)) != X86_PF_USER)
 		return false;
 
-	if (!(error_code & X86_PF_INSTR)) {
+	/* Avoid emulation unless userspace was executing from vsyscall page: */
+	if (address != regs->ip) {
 		/* Failed vsyscall read */
 		if (vsyscall_mode == EMULATE)
 			return false;

@@ -136,13 +137,16 @@ bool emulate_vsyscall(unsigned long error_code,
 		return false;
 	}
 
+
+	/* X86_PF_INSTR is only set when NX is supported: */
+	if (cpu_feature_enabled(X86_FEATURE_NX))
+		WARN_ON_ONCE(!(error_code & X86_PF_INSTR));
+
 	/*
 	 * No point in checking CS -- the only way to get here is a user mode
 	 * trap to a high address, which means that we're in 64-bit user code.
 	 */
 
-	WARN_ON_ONCE(address != regs->ip);
-
 	if (vsyscall_mode == NONE) {
 		warn_bad_vsyscall(KERN_INFO, regs,
 				  "vsyscall attempted with vsyscall=none");

arch/x86/mm/tlb.c +5 -2

index 39f80111e6f1..e3ce9b0b2447 100644
@@ -665,6 +665,7 @@ static unsigned long mm_mangle_tif_spec_bits(struct task_struct *next)
 static void cond_mitigation(struct task_struct *next)
 {
 	unsigned long prev_mm, next_mm;
+	bool userspace_needs_ibpb = false;
 
 	if (!next || !next->mm)
 		return;

@@ -722,7 +723,7 @@ static void cond_mitigation(struct task_struct *next)
 		 */
 		if (next_mm != prev_mm &&
 		    (next_mm | prev_mm) & LAST_USER_MM_IBPB)
-			indirect_branch_prediction_barrier();
+			userspace_needs_ibpb = true;
 	}
 
 	if (static_branch_unlikely(&switch_mm_always_ibpb)) {

@@ -732,9 +733,11 @@ static void cond_mitigation(struct task_struct *next)
 		 * last on this CPU.
 		 */
 		if ((prev_mm & ~LAST_USER_MM_SPEC_MASK) != (unsigned long)next->mm)
-			indirect_branch_prediction_barrier();
+			userspace_needs_ibpb = true;
 	}
 
+	this_cpu_write(x86_ibpb_exit_to_user, userspace_needs_ibpb);
+
 	if (static_branch_unlikely(&switch_mm_cond_l1d_flush)) {
 		/*
 		 * Flush L1D when the outgoing task requested it and/or

drivers/iommu/intel/Kconfig +4 -1

index f2f538c70650..a5d66bfd9e50 100644
@@ -48,7 +48,10 @@ config INTEL_IOMMU_DEBUGFS
 
 config INTEL_IOMMU_SVM
 	bool "Support for Shared Virtual Memory with Intel IOMMU"
-	depends on X86_64
+	# The kernel does not invalidate IOTLB entries when freeing
+	# kernel page tables. This can lead to IOMMUs walking (and
+	# writing to) CPU page tables after they are freed.
+	depends on BROKEN
 	select MMU_NOTIFIER
 	select IOMMU_SVA
 	help

Quoted 10 lines · Dave Hansen

Quoted 7 lines · Jason Gunthorpe

Quoted 5 lines

Quoted 4 lines

The other alternative is to have arch_vmap_pmd_supported() return false when SVA is active, or maybe when it's supported on the platform.

Either of those are 10-ish lines of code and easy to backport.

Hi iommu folks, any insights on this?

IDK, the only SVA user on x86 I know is IDXD, so if you do the above plan you break IDXD in all stable kernels. Doesn't sound OK?

Vinicius, any thoughts on this?

This won't break IDXD exactly/totally, it would cause it to be impossible for users to create shared DSA/IAA workqueues (which are the nicer ones to use), and it will cause the driver to print some not happy messages in the kernel logs. The in-kernel users of IDXD (iaa_crypto for zswap, for example) will continue to work.

In short, I am not happy, but I think it's workable, even better if there are alternatives in case people complain.